A user recommendation system and method based on a multi-head attention mechanism

Abstract

The application discloses a user recommendation system and method based on a multi-head attention mechanism, the system comprising a user behavior data processing module, which is used for performing feature mapping on feature data of users and items to obtain a first feature matrix; a feature processing and fusion module, which is used for calculating the weight of features according to the first feature matrix to obtain a feature weight vector, and sequentially multiplying the feature weight vector with the first feature matrix to perform weight assignment to obtain a second feature matrix; calculating the correlation degree between features according to the second feature matrix and performing feature weighting combination to obtain a third feature matrix; and a deep recommendation module, which is used for connecting the second feature matrix and the third feature matrix to obtain a fourth feature matrix, calculating the predicted score of users for items, and recommending items to users according to the score; the application improves the convergence speed and robustness of the model, reduces the adverse effects of unimportant features on the prediction result, excavates the hidden relationship between features, and realizes personalized recommendation for users.

Description

Technical Field

[0001] This invention relates to a user recommendation system and method, and more particularly to a user recommendation system and method based on a multi-head attention mechanism. Background Technology

[0002] The rapid development of internet technology has propelled society into the information age, with people receiving massive amounts of information daily from internet terminals. User recommendation systems effectively filter this information, ensuring users only receive content that interests them. Furthermore, they can uncover potential interests based on user browsing history, thus providing better content recommendations. Currently, various online platforms, such as Taobao and QQ Music, employ recommendation algorithms to push content in real-time based on users' historical browsing information, not only shortening search time but also increasing user dependence on the platform.

[0003] Traditional recommendation algorithms suffer from slow processing speed and low recommendation accuracy when handling large-scale data, making it difficult to meet users' normal needs. Deep learning technology, with its parallel computing, autonomous learning, and strong fitting capabilities, is particularly suitable for analyzing and processing large-scale data. However, current deep learning-based recommendation algorithms neglect the importance of features themselves and the impact of the correlation between features on the effectiveness of feature combinations. This leads to a decrease in recommendation accuracy when unimportant features are introduced. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a user recommendation system and method based on a multi-head attention mechanism that focuses on the correlation between features to improve the accuracy of recommendation results.

[0005] Technical solution: The user recommendation system based on multi-head attention mechanism of the present invention is characterized by comprising:

[0006] The user behavior data processing module is used to perform feature mapping between user and project feature data to obtain the first feature matrix.

[0007] The feature processing and fusion module is used to calculate the weights of features based on the first feature matrix to obtain a feature weight vector, and then multiply it sequentially with the first feature matrix to assign weights, thereby obtaining a second feature matrix; and to calculate the correlation between pairs of features based on the second feature matrix and perform a weighted combination of features to obtain a third feature matrix.

[0008] The deep recommendation module is used to connect the second feature matrix and the third feature matrix to obtain the fourth feature matrix, calculate the user's predicted rating for the item, and recommend items to the user based on the rating.

[0009] Furthermore, the user behavior data processing module includes an input layer, which is used to vectorize the text data features of users and items to obtain a discrete vector representation of the feature data.

[0010] Furthermore, the user behavior data processing module includes a feature mapping layer, which encodes the original feature data and maps it to the latent space to obtain a first feature matrix; the feature mapping layer includes a three-layer neural network based on a feedback neural network.

[0011] The formula for mapping is e i =V i x i ;where V i It is the mapping matrix of feature i, x i It is the feature vector after feature i is one-hot encoded, e i It is the mapped feature vector.

[0012] Furthermore, the feature processing fusion module includes a weighting layer, which uses a squeeze-scaling network to calculate the first weight of each feature in the first feature matrix to obtain a feature weight vector, and then multiplies it sequentially with the first feature matrix to assign weights, thereby obtaining a second feature matrix.

[0013] Furthermore, the feature processing fusion module includes a feature cross layer, which uses a multi-head attention mechanism to calculate the correlation between pairs of features in the second feature matrix.

[0014] Furthermore, in the feature processing fusion module, calculating the correlation between pairs of features based on the second feature matrix and performing a weighted combination of features includes:

[0015] The formula for calculating the degree of correlation is:

[0016]

[0017] Where <·> represents the inner product operation, querying the matrix. Bond matrix These are the mapping matrices for the h-th attention head, where h = 1, 2, ..., H, and H is the number of attention heads. f e o Let f and o be the feature vectors of features f and o, respectively.

[0018] Furthermore, in the feature processing fusion module, calculating the correlation between pairs of features based on the second feature matrix and performing a weighted combination of features includes:

[0019] A second weight is assigned to the feature based on the degree of relevance. The formula for calculating the second weight is as follows:

[0020]

[0021] in, As the second weight, F h Let h be the feature set under the h-th attention head.

[0022] Furthermore, the deep recommendation module includes a hidden layer, which is used to perform transformation and fusion calculation on the fourth feature matrix to obtain the implicit expression matrix of user and item features;

[0023] Furthermore, the deep recommendation module includes an output layer, which is used to decode the implicit representation matrix using a deep residual network to obtain the user's predicted rating for the item.

[0024]

[0025] in, W represents the predicted score for each item. out y and b represent the weight matrix and bias vector of the output layer, respectively. DRN This represents the output vector of a deep residual network;

[0026] The user recommendation method based on multi-head attention mechanism described in this invention includes the following steps:

[0027] Step 1: Perform feature mapping between user and project feature data to obtain the first feature matrix;

[0028] Step 2: Calculate the weights of the features based on the first feature matrix to obtain a feature weight vector, and then multiply it sequentially with the first feature matrix to assign weights, thus obtaining a second feature matrix; calculate the correlation between each pair of features based on the second feature matrix and perform a weighted combination of features to obtain a third feature matrix;

[0029] Step 3: Connect the second feature matrix and the third feature matrix to obtain the fourth feature matrix, calculate the user's predicted rating for the project, and recommend projects to the user based on the rating.

[0030] Beneficial effects: Compared with the prior art, the advantages of the present invention are: (1) By processing user and project features, the importance of features is calculated and weighted, reducing the adverse effects of unimportant features on the final prediction results; (2) The correlation between features is calculated through a multi-head attention mechanism, and the weighted features are cross-referenced. Then, the combined features after cross-reference are assigned weights according to the degree of correlation, which increases the scale of the original data and alleviates the data sparsity problem; (3) The final recommendation results are predicted by using a deep residual network. Through the powerful learning ability of the deep network, the hidden connections between features are further explored, and personalized recommendation results for users are realized. Attached Figure Description

[0031] Figure 1 This is an architecture diagram of the user recommendation system based on the multi-head attention mechanism of the present invention.

[0032] Figure 2 This is a flowchart of the user recommendation method based on multi-head attention mechanism of the present invention. Detailed Implementation

[0033] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0034] like Figure 1 As shown, the user recommendation system based on the multi-head attention mechanism includes a user behavior data processing module, a feature processing and fusion module, and a deep recommendation module. These three modules collaborate and connect in series to calculate and analyze the importance of items and user features themselves, assign weights to the first feature matrix to obtain the second feature matrix, use the multi-head attention mechanism to calculate the correlation between features, analyze the hidden relationships between features, perform feature combination, i.e., feature cross, assign weights to the combined features according to the degree of correlation between features to obtain the third feature matrix, and connect it with the second feature matrix to obtain the fourth feature matrix. Finally, a deep network is used to predict the final recommendation result to achieve intelligent personalized recommendations for users.

[0035] The user behavior data processing module performs vector transformation and feature mapping on the original feature data of users and projects to obtain a first feature matrix. Specifically, the module preprocesses the original text data, fully explores and utilizes the hidden relationships between user and project features, calculates the weights of each feature, and inputs the weighted second feature matrix into the feature processing fusion module. The user behavior data processing module includes an input layer and a feature mapping layer, wherein:

[0036] The input layer is used to vectorize the output text information, resulting in discrete vector data. In this embodiment, the vectorized text data features of the input layer include user occupation, user age, rating, and project type. These text data are then encoded to obtain discrete vector representations of the features.

[0037] The feature mapping layer is used to map vector data to obtain behavioral information in the deep latent space. In this embodiment, the feature mapping layer establishes a three-layer deep learning neural network model based on a Back Propagation Neural Network (BPNN). Vector data with ratings is used as the labeled dataset, and the ratings are used as labels for deep learning model training. The trained model weights are obtained and mapped to the deep latent space to mine the implicit semantic information in the vector data. This information is then fused to obtain the behavioral information in the deep latent space. After encoding the user's age, gender, and item feature information, it is mapped to the latent space using the following formula:

[0038] e i =V i x i ;

[0039] Among them, V i It is the mapping matrix of feature i, x i It is the feature vector after feature i is one-hot encoded, e i It is the mapped feature vector.

[0040] The feature processing and fusion module, connected to the user behavior data processing module, calculates the first weight of each feature and the correlation between features based on the first feature matrix, resulting in a weighted second feature matrix. Then, it performs feature crossing and assigns second weights to the combined features based on the correlation, resulting in a third feature matrix. The third and second feature matrices are then combined to obtain a fourth feature matrix. Specifically, the feature processing and fusion module uses a Squeeze-and-Excitation Network (SEnet) to process the first feature matrix output by the user behavior data processing module, calculating the first weight of each feature vector and then weighting the first feature matrix. It also uses a multi-head attention mechanism to calculate the correlation between features, assigning second weights to the combined features based on the correlation, and concatenates the combined feature matrix (the third feature matrix) with the second feature matrix to form a fourth matrix, which is then input into the subsequent deep recommendation module. The feature processing and fusion module includes a weighting layer and a feature crossing layer, wherein:

[0041] The weighting layer processes the obtained first feature matrix, calculating a weight vector composed of the weights of each feature. This weight vector is then used to weight the feature matrix to obtain the second feature matrix. In this embodiment, the first feature matrix undergoes first weight calculation using a squeeze-scaling network. Each feature vector is treated as a transmission channel, and the squeeze-scaling network adaptively adjusts the weights of the channels, reducing the impact of unimportant channel information on the final recommendation result, thus achieving weighting of the feature matrix.

[0042] The feature cross-layer is used to calculate the correlation between features based on the obtained second feature matrix, and then performs weighted combinations based on the correlation level to obtain the third feature matrix. In this embodiment, the correlation between the weighted features is calculated, and feature cross-combinations are performed. Then, a second weight is assigned to the combined features based on the correlation level.

[0043] The degree of correlation is expressed as:

[0044]

[0045] Where, ψ h (e f ,e o ) represents the degree of correlation between feature f and feature o, e f e o Let f and o represent the eigenvectors of features f and o, respectively. The symbols <·> represent the inner product operation. The inner product operation is faster and yields simpler results than the Hadamard product, making it more suitable for large-scale data computation. (Query matrix) Bond matrix These represent the mapping matrices under the h-th attention head (h = 1, 2, ..., H), which map the features calculated under different attention heads (located in different latent spaces) to the same latent space, so that the output results under different attention heads can be better integrated later. Weights are assigned to the combined features according to their relevance, and the weight calculation formula is:

[0046]

[0047] Among them, F h Let e ​​be the feature set under the h-th attention head. f e o Let f and o represent the eigenvectors of features f and o, respectively.

[0048] The deep recommendation module, connected to the feature processing and fusion module, performs secondary modeling based on the processed fourth feature matrix of users and items. This further extracts the relationships between features and generates recommendation results. Specifically, the deep recommendation module uses a Deep Residual Network (DRN) to process the third and fourth feature matrices, further exploring the correlations between features, learning more layers of feature representation, and predicting recommendation results. The deep recommendation module includes an input layer, hidden layers, and an output layer, where:

[0049] The input layer is used to concatenate the second and third feature matrices to obtain the fourth feature matrix.

[0050] The hidden layer is used to perform transformation and fusion calculations on the fourth feature matrix to obtain the implicit expression matrix of user and project features.

[0051] The output layer decodes the implicit representation matrix of user and item features to obtain the user's predicted rating for the item, and uses the user's predicted rating for the item as the recommendation result. The result is expressed as follows:

[0052]

[0053] Among them, y DRN W represents the final output vector of the deep residual network. out b and b represent the weight matrix and bias vector of the output layer, respectively. This represents the model's predicted click values ​​for each item.

[0054] This embodiment experimentally analyzes the performance of the user recommendation system based on the multi-head attention mechanism, using two public datasets: Criteo and Amazon. The Criteo dataset, provided by the US advertising company Criteo, is used to predict ad click-through rates (CTR). This dataset contains data on user clicks on ads over a week, including 13 continuous features and 26 categorical features, covering 40 million training samples and 6 million test samples. The Amazon dataset is used to predict the CTR of products on the Amazon website. User information includes ID, age, occupation, etc., and product information includes name, price, brand, etc. Both datasets are divided into training and test sets in an 8:2 ratio. The training set is used to train the model parameters, and the test set is used to test the model's performance. The error and accuracy of the proposed system are analyzed against algorithms such as LR, FM, MLP, and DCN. The following table shows a performance comparison between the proposed system and various algorithms:

[0055]

[0056] like Figure 2As shown, based on the same inventive concept, the user recommendation method based on multi-head attention mechanism described in this invention includes the following steps:

[0057] Step 1: Perform feature mapping between user and project feature data to obtain the first feature matrix;

[0058] Step 2: Calculate the weights of the features based on the first feature matrix to obtain a feature weight vector, and then multiply it sequentially with the first feature matrix to assign weights, thus obtaining a second feature matrix; calculate the correlation between each pair of features based on the second feature matrix and perform a weighted combination of features to obtain a third feature matrix;

[0059] Step 3: Connect the second feature matrix and the third feature matrix to obtain the fourth feature matrix, calculate the user's predicted rating for the project, and recommend projects to the user based on the rating.

Claims

1. A user recommendation system based on a multi-head attention mechanism, characterized in that, include: The user behavior data processing module is used to perform feature mapping between user and project feature data to obtain the first feature matrix. The feature processing and fusion module is used to calculate the weights of features based on the first feature matrix to obtain a feature weight vector, and then multiply it sequentially with the first feature matrix to assign weights, thereby obtaining a second feature matrix; and to calculate the correlation between pairs of features based on the second feature matrix and perform a weighted combination of features to obtain a third feature matrix. The deep recommendation module is used to connect the second feature matrix and the third feature matrix to obtain the fourth feature matrix, calculate the user's predicted rating for the item, and recommend items to the user based on the rating. The user behavior data processing module includes an input layer, which is used to vectorize the text data features of users and items to obtain a discrete vector representation of the feature data. The vectorized text data features of the input layer include user occupation, user age, rating, and item type. The user behavior data processing module includes a feature mapping layer, which is used to encode the original feature data and map it to the latent space to obtain a first feature matrix. The feature mapping layer comprises a three-layer neural network based on a feedback neural network; The formula for mapping is ;where V i It is the mapping matrix of feature i, x i It is the feature vector after feature i is one-hot encoded, e i It is the mapped feature vector; The feature processing fusion module includes a weighting layer, which uses a squeeze-and-stretch network to calculate the first weight of each feature in the first feature matrix to obtain a feature weight vector, and then multiplies it sequentially with the first feature matrix to assign weights, thereby obtaining a second feature matrix. The feature processing fusion module includes a feature cross layer, which uses a multi-head attention mechanism to calculate the correlation between pairs of features in the second feature matrix.

2. The user recommendation system based on multi-head attention mechanism according to claim 1, characterized in that, In the feature processing fusion module, calculating the correlation between pairs of features based on the second feature matrix and performing a weighted combination of features includes: The formula for calculating the degree of correlation is: ； in, For inner product operations, query the matrix. Bond matrix Let H be the mapping matrix for the h-th attention head, where h = 1, 2, ..., H, and H is the number of attention heads. Let f and o be the feature vectors of features f and o, respectively.

3. The user recommendation system based on multi-head attention mechanism according to claim 2, characterized in that, In the feature processing fusion module, calculating the correlation between pairs of features based on the second feature matrix and performing a weighted combination of features includes: A second weight is assigned to the feature based on the degree of relevance. The formula for calculating the second weight is as follows: ； in, As the second weight, Let h be the feature set under the attention head. For relevance, Let f and o be the feature vectors of features f and o, respectively.

4. The user recommendation system based on multi-head attention mechanism according to claim 1, characterized in that, The deep recommendation module includes a hidden layer, which is used to transform and fuse the fourth feature matrix to obtain the implicit expression matrix of user and item features.

5. The user recommendation system based on multi-head attention mechanism according to claim 4, characterized in that, The deep recommendation module includes an output layer, which is used to decode the implicit representation matrix using a deep residual network to obtain the user's predicted rating for the item. ； in, This indicates the predicted score for each item. b and b represent the weight matrix and bias vector of the output layer, respectively. This represents the output vector of the deep residual network.

6. A user recommendation method based on a multi-head attention mechanism, applied to the system described in claim 1, characterized in that, The steps include the following: Step 1: Perform feature mapping between user and project feature data to obtain the first feature matrix; Step 2: Calculate the weights of the features based on the first feature matrix to obtain a feature weight vector, and then multiply it sequentially with the first feature matrix to assign weights, thus obtaining a second feature matrix; calculate the correlation between each pair of features based on the second feature matrix and perform a weighted combination of features to obtain a third feature matrix; Step 3: Connect the second feature matrix and the third feature matrix to obtain the fourth feature matrix, calculate the user's predicted rating for the project, and recommend projects to the user based on the rating.

Images