A residual neural collaborative filtering recommendation system fusing attention mechanism
The recommendation system, which combines multilayer perceptron and residual neural network with attention mechanism, solves the problems of insufficient feature fusion and gradient vanishing in existing technologies, and achieves more efficient recommendation results.
Patent Information
- Application Number
- CN202211568216.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-06
- Publication Date
- 2026-02-06
- Estimated Expiration
- 2042-12-06
AI Technical Summary
Existing recommendation algorithms ignore the linear relationship between users and items, and cannot effectively integrate low-order and high-order features, leading to gradient vanishing and information loss, which inhibits the performance improvement of recommendation systems.
By employing a multilayer perceptron and a residual neural network, combined with an attention mechanism, high-order features of users and items are extracted through an embedding layer, a preprocessing layer, a multilayer perceptron, and residual channels. Low-order features are then passed through the residual structure to prevent gradient vanishing. At the same time, different feature weights are fused through the attention mechanism to improve recommendation quality.
It effectively alleviates the problems of gradient vanishing and information loss, improves the performance of the recommendation system, and enhances the accuracy and personalization of recommendations.
Smart Images
Figure CN115858944B_ABST
Abstract
Description
[0001] The application belongs to the technical field of recommendation, in particular to a residual neural collaborative filtering recommendation system fusing an attention mechanism. BACKGROUND
[0002] Under the wave of digitization, the global industrial internet is stepping into the fast lane of development. The surge of indiscriminate information makes online users need to spend a lot of time and effort to filter these information to obtain content closer to their interest points. As the most simple and efficient information filtering technology at present, the recommendation system (RSs) can fully analyze the historical behavior information of users and construct a corresponding recommendation model to predict user preferences and push the most likely interesting content to them, which greatly improves the user experience and effectively alleviates the information overload problem. The collaborative filtering method based on deep neural network can well discover the complex relationship between users and items due to its strong nonlinear transformation ability and learning ability. However, this kind of method ignores the linear relationship between users and items, thereby inhibiting the further improvement of the model performance. Therefore, how to design a neural collaborative filtering algorithm to effectively fuse the low-order and high-order features between data to improve the performance of the recommendation system is particularly important.
[0003] At present, the recommendation algorithm is mostly realized by simple splicing of features to realize feature fusion, ignoring the influence of different feature combinations on the recommendation result. At the same time, when extracting high-order features, the gradient disappearance and information loss problems existing in the neural network cannot be effectively alleviated. Therefore, how to alleviate the gradient disappearance and information loss problems existing in the neural network when extracting high-order features, and efficiently fuse low-order and high-order features to improve the ability of the recommendation system has become a technical problem. SUMMARY
[0004] In view of the above problems, the application based on a multilayer perceptron proposes a residual neural collaborative filtering recommendation system fusing an attention mechanism, comprising:
[0005] The server is used for setting a data acquisition and preparation module, a preference information extraction module, a model training module, a model prediction and recommendation module. At the same time, the server is connected with a database to provide data information for the above-mentioned modules;
[0006] The database is connected with an intelligent terminal and is used for saving the behavior data information of users left on the intelligent terminal;
[0007] The intelligent terminal, i.e. the mobile phone or computer used by the user, can retain the behavior data information of the user browsing, clicking or purchasing items on the terminal and transmit it into the above-mentioned database for saving;
[0008] A data collection and preparation module is configured to extract user behavior data provided by the database, and clean and standardize the user behavior data to obtain a format required by the preference extraction module;
[0009] A preference information extraction module is configured to extract user preference information for items from the user behavior data provided by the data collection and preparation module, and model the user preference information.
[0010] A model training module is configured to solve a model provided by the preference information extraction module to obtain optimal parameters of the model.
[0011] A model prediction and recommendation module is configured to provide personalized item recommendations for each user based on the model provided by the model training module, so as to improve the satisfaction of the user.
[0012] Further, the overall steps of the system are as follows:
[0013] S1: The intelligent terminal sends interaction information between a user and an item in the recommendation system to a database.
[0014] S2: The data collection and preparation module obtains all behavior data from the database, including user id, item id, and rating information.
[0015] S2: The data collection and preparation module converts the user id into a vector E u using one-hot encoding.
[0016] S3: The data collection and preparation module converts the item id into a vector E i using one-hot encoding.
[0017] S4: The data collection and preparation module obtains the rating information by recording the items rated by the user as 1 and the items not rated by the user as 0.
[0018] S5: The data collection and preparation module stores the rating information in the form of a matrix, i.e., an implicit feedback matrix, wherein the row of the matrix represents a user u, the column of the matrix represents an item i, and the value r u,i in the matrix represents the rating of the user u for the item i.
[0019] S6: The preference information extraction module randomly initializes the embedding layer parameters (W Eu and W Ei ), the preprocessing layer parameters (W p0 and W p1 ), the multi-layer perception parameters (W1, W2, …, W n ), and the residual channel parameters (W 0,2 , W 0,3 , …, Wn-2,n );
[0020] S7: User vector and item vector are converted into dense embedding vectors by embedding layer and and the feature vector of user u to item i is generated by splicing ui ;
[0021] S8: The output vector after feature reconstruction is obtained according to the pre-processing layer parameters W p0 and W p1 The calculation method is as follows:
[0022]
[0023] S9: The output vector will obtain two types of feature outputs after passing through the first full connection layer and the residual channel, which are h1 and h 0,2 , and the calculation method is as follows:
[0024]
[0025] S10: The two types of output vectors will be calculated by the attention mechanism layer to obtain the final output h2 of the second hidden layer of the multi-layer perception by weighted summation, and the calculation method is as follows:
[0026]
[0027] S11: The subsequent calculation is similar to S5 and S6, and the calculation method of the final output of the model is as follows:
[0028]
[0029] S12: The model training module uses gradient descent method to iteratively optimize the target loss function until the model converges, and outputs the model parameters, including embedding layer parameters (W Eu and W Ei ), pre-processing layer parameters (W p0 and W p1 ), multi-layer perception parameters (W1, W2, …, W n ), and residual channel parameters (W 0,2 , W 0,3 , …, W n-2,n ). As shown in the following formula:
[0030]
[0031] Loss = Loss dense + αLoss pre
[0032]
[0033] wherein, Loss pre represents the loss of the preprocessing layer, Loss dense represents the loss of the model prediction, W dense represents the parameters of the multilayer perceptron and the parameters of the residual channel, W pre represents the parameters of the preprocessing layer, and alpha is a hyperparameter for controlling the preprocessing loss.
[0034] S13: The model prediction and recommendation module obtains the model parameters output by the model training module, including the embedding layer parameters (W Eu and W Ei ), the preprocessing layer parameters (W p0 and W p1 ), the multilayer perceptron parameters (W1, W2, …, W n ), and the residual channel parameters (W 0,2 , W 0,3 , …, W n-2,n ).
[0035] S14: The model prediction and recommendation module calculates the score of each user to each item by the following formula
[0036]
[0037] …
[0038]
[0039] S15: The model prediction and recommendation module fills the matrix according to the user and item id.
[0040] S16: The model prediction and recommendation module, for different users, fills the corresponding according to the score from large to small to recommend the items that the user has not interacted with.
[0041] The residual neural collaborative filtering recommendation system with attention mechanism proposed in the application uses a multilayer neural network to extract high-order features between users and items, and then uses a residual structure to pass low-order features to prevent the problem of gradient disappearance and solve the problem of information loss in the transmission of information by the neural network. At the same time, the system uses the attention mechanism to fuse high-order features and low-order features, assigns different feature weights to different users, so as to more fully reflect the personalized preferences of the users and improve the recommendation quality. BRIEF DESCRIPTION OF DRAWINGS
[0042] Figure 1 is a system framework diagram of the application.
[0043] Figure 2 This is a schematic diagram of the model process of the present invention;
[0044] Figure 3 This invention demonstrates the variation of NDCG values for different recommendation list lengths across three datasets.
[0045] Figure 4 This invention demonstrates the variation of F1 scores for different recommendation list lengths across three datasets. Detailed Implementation
[0046] To make the specific embodiments and technical solutions of the present invention clearer, further explanation is provided below in conjunction with the accompanying drawings.
[0047] Figure 1 This is a schematic diagram of the residual neural collaborative filtering recommendation system that incorporates the attention mechanism of the present invention. Figure 1 As can be seen, the recommendation system includes: a smart terminal, i.e., the user's mobile phone or computer, capable of retaining user behavior data information such as browsing, clicking, or purchasing items on the terminal and storing it in the aforementioned database; a database connected to the smart terminal, used to store user behavior data information left on the smart terminal; a server used to set up a data acquisition and preparation module, a preference information extraction module, a model training module, and a model prediction and recommendation module, and simultaneously connected to the database to provide data information for the above modules; a data acquisition and preparation module, used to extract user behavior data provided by the aforementioned database, and clean and standardize it to obtain the format required by the preference extraction module; a preference information extraction module, used to extract user preference information for items from the user behavior data provided by the data acquisition and preparation module, and use this to build a model; a model training module, used to solve the model provided by the preference information extraction module to obtain the optimal parameters of the model; and a model prediction and recommendation module, used to provide personalized item recommendations for each user based on the model provided by the model training module, in order to improve user satisfaction.
[0048] Furthermore, the following example illustrates this further:
[0049] Suppose there are M users U = {u1, u2, ..., u...} m} and n items I = {i1,i2,…,i n The interaction between users and the project is analyzed using an implicit feedback matrix. Representation; the implicit feedback matrix R is given by r u,i Fill in, r u,i This indicates whether user u has interacted with item i; if user u has rated item i, then assign them a rating r. u,i= 1, if no rating has been made, then let it be rated r u,i = 0; represents the vector representation of the system prediction of whether the user u will interact with all items.
[0050] First, the system data is obtained and cleaned through the data acquisition and preparation module to obtain each variable in the hypothesis. Figure 2 The entire process of the model processing data is shown, and the specific implementation steps are as follows:
[0051] S1: The data acquisition and preparation module obtains all the behavior data from the database, including user id, item id, and rating information.
[0052] S2: The data acquisition and preparation module converts the user id into a vector E u using one-hot encoding.
[0053] S3: The data acquisition and preparation module converts the item id into a vector E i using one-hot encoding.
[0054] S4: The data acquisition and preparation module obtains the rating information by marking the items rated by the user as 1 and marking the items not rated as 0.
[0055] S5: The data acquisition and preparation module stores the rating information in the form of a matrix, which is the implicit feedback matrix. The row of the matrix represents a certain user u, the column of the matrix represents a certain item i, and the value r u,i in the matrix represents the rating of the user u to the item i.
[0056] S6: The preference information extraction module randomly initializes the embedding layer parameters (W Eu and W Ei ), the preprocessing layer parameters (W p0 and W p1 ), the multi-layer perception parameters (W1, W2, …, W n ), and the residual channel parameters (W 0,2 , W 0,3 , …, W n-2,n ) according to the standard normal distribution.
[0057] S7: The user vector and the item vector are converted into dense embedding vectors and respectively through the embedding layer, and the feature vector V ui of the user u to the item i is generated by splicing.
[0058] S8: The preprocessing layer parameters W p0 and W p1, get the output vector of feature reconstruction The calculation is as follows:
[0059]
[0060] S9: Output vector Two types of feature outputs will be obtained through the first full connection layer and the residual channel, which are h1 and h 0,2 The calculation is as follows:
[0061]
[0062] S10: Two types of output vectors will be calculated by the attention mechanism layer to calculate the weight, and the final output h2 of the second hidden layer of the multi-layer perception will be generated by weighted summation. The calculation is as follows:
[0063]
[0064] S11: The subsequent calculation is similar to S5 and S6. The final output of the model is calculated as follows:
[0065]
[0066] S12: The model training module uses gradient descent method to iteratively optimize the target loss function until the model converges, and outputs the model parameters, including embedding layer parameters (W Eu and W Ei ), preprocessing layer parameters (W p0 and W p1 ), multi-layer perception parameters (W1, W2, …, W n ), and residual channel parameters (W 0,2 , W 0,3 , …, W n-2,n ). As shown in the following formula:
[0067]
[0068]
[0069] Loss = Loss dense + αLoss pre
[0070]
[0071] Wherein, Loss pre represents the loss of the preprocessing layer, Loss dense represents the loss of the model prediction, W dense represents the parameters of the multi-layer perception and the parameters of the residual channel, and W preparameters of the pre-processing layer, and a is a hyper-parameter controlling the pre-processing loss;
[0072] S13: The model prediction and recommendation module acquires the model parameters output by the model training module, including embedding layer parameters (W Eu and W Ei ), pre-processing layer parameters (W p0 and W p1 ), multi-layer perception parameters (W1, W2, …, W n ), and residual channel parameters (W 0,2 , W 0,3 , …, W n-2,n );
[0073] S14: The model prediction and recommendation module calculates the score of each user to each item by the following formula
[0074]
[0075] …
[0076]
[0077] S15: The model prediction and recommendation module fills the matrix according to the user and item id.
[0078] S16: The model prediction and recommendation module, for different users, recommends the items that the user has not interacted with according to the corresponding in descending order of score.
[0079] Figure 3 and Figure 4 measure the performance of the model in terms of F1 value and normalized discounted cumulative gain (NDCG), respectively. Their calculation methods and measurement contents are as follows:
[0080] F1 value: is an index combined by precision and recall, used to evaluate the comprehensive performance of the recommendation system accuracy. Among them, the precision in the present application measures the proportion of items that the user really likes in the item list recommended to the user; while the recall in the present application measures the proportion of the number of items that the user really likes in the item list recommended to the user in the total number of items that the user likes. Their calculation formulas are as follows:
[0081]
[0082] wherein represents the precision of the recommendation list, i.e. the proportion of the items that the user really likes in the item list I p (u ithe proportion of the set of items that the user actually purchased in the recommended list; represents the recall rate of the recommended list, that is, the proportion of the set of items that the user actually purchased in the recommended list to the set of items that the user actually purchased a (u i ).
[0083] Normalized Discounted Cumulative Gain: Considering the order of the items that the user actually likes in the recommended list, the earlier the position of the item that the user actually likes in the recommended list, the better the recommendation performance. The calculation formula is as follows:
[0084]
[0085] wherein, represents the discounted cumulative gain of the K-length recommended list for the user, id represents the ranking of the item in the recommended list, rel id represents whether the item ranked as id is in the set of items that the user actually purchased, if in rel id = 1, and 0 if not. IDCG u represents the maximum value that DCG u can reach.
Claims
1. A residual neural collaborative filtering recommendation system incorporating attention mechanisms, characterized in that, include: The server is used to set up the data acquisition and preparation module, the preference information extraction module, the model training module, and the model prediction and recommendation module; At the same time, the server connects to the database to provide data information to the above modules; A database, connected to a smart terminal, is used to store behavioral data information left by users on the smart terminal; Smart terminals, namely mobile phones or computers used by users, can retain behavioral data information of users browsing, clicking or purchasing items on the terminal and save it in the aforementioned database; The data acquisition and preparation module is used to extract user behavior data provided by the above database, and clean and standardize it to obtain the format required by the preference information extraction module. The preference information extraction module is used to extract user preference information for items from the user behavior data provided by the data collection and preparation module. The main steps are as follows: S1: Randomly initialize the embedding layer parameters W of user u and item i according to the standard normal distribution. Eu and W Ei Preprocessing layer parameters W p0 and W p1 Multilayer perceptron parameters W1, W2, ..., W n and residual channel parameter W 0,2 W 0,3 ... W n-2,n ; S2: Use one-hot encoding to convert user ID and item ID into user vector E. u and project vector E i They are then transformed into dense user embedding vectors through an embedding layer. and project embedding vector And by concatenating the data, a feature vector V for user u and item i is generated. ui ; S3: Based on the preprocessing layer parameter W p0 and W p1 The output vector after feature reconstruction is obtained. The calculation method is as follows: S4: Output Vector After passing through the first fully connected layer and the residual channel, two types of feature outputs are obtained, namely h1 and h2. 0,2 The calculation method is as follows: S5: The two types of output vectors are weighted by the attention mechanism layer, and then weighted and summed to generate the final output h2 of the second hidden layer of the multilayer perceptron. The calculation method is as follows: S6: Subsequent calculations follow the same pattern as S5 and S6, ultimately yielding the predicted score for user u in item i. The calculation formula is as follows: The model training module is used to solve the model provided by the preference information extraction module to obtain the optimal parameters of the model. The model prediction and recommendation module is used to provide personalized item recommendations for each user based on the model provided by the model training module, in order to improve user satisfaction.
2. The residual neural collaborative filtering recommendation system incorporating attention mechanisms according to claim 1, characterized in that, The data acquisition and preparation module set up on the server obtains all user behavior data from the recommendation system by connecting to the database, and cleans and standardizes the data according to the system requirements to create an implicit feedback matrix, specifically including: S1: Retrieve all behavioral data from the database, specifically including: user ID, project ID, and rating information; S2: Obtain the user vector E using one-hot encoding. u and project vector E i ; S3: For rating information, items that have been rated by the user are recorded as 1, and items that have not been rated are recorded as 0; S4: Store the rating information in a matrix format, which is the implicit feedback matrix, where each row of the matrix represents a user u, each column represents an item i, and the value r in the matrix... u,i This represents the rating that user u gives to item i.
3. The residual neural collaborative filtering recommendation system incorporating attention mechanisms according to claim 1, characterized in that, The model training module, located on the server, obtains the implicit feedback matrix provided by the data acquisition and preparation module, iteratively optimizes the objective loss function using gradient descent until the model converges, and finally outputs the optimal parameters of the model. Loss=Loss dense +αLoss pre , Among them, Loss pre This represents the loss of the preprocessing layer. dense W represents the loss predicted by the model. pre W represents the parameters of the preprocessing layer. dense α represents the parameters of the multilayer perceptron and the residual channel; α is the hyperparameter controlling the preprocessing loss.
4. The residual neural collaborative filtering recommendation system incorporating attention mechanisms according to claim 1, characterized in that, The model prediction and recommendation module set up on the server extracts the trained model parameters, predicts each user's preferences for all items, and makes recommendations according to preference scores from high to low. The specific steps are as follows: S1: Obtain the optimal parameters W output by the model training module. pre W dense and W att ; S2: Calculate the predicted rating for each user u for each item i. S3: Will Fill the matrix with user ID and project ID. S4: For different users u, assign their corresponding... The system recommends interactive items to users based on their scores, from highest to lowest.