Online knowledge sharing recommendation method based on interaction between attention mechanism and high-order features

Abstract

This invention relates to an online knowledge sharing recommendation method based on attention mechanisms and high-order feature interactions. A high-order feature interaction neural network is incorporated into the model, combining high-order feature interactions with the attention mechanism, and simultaneously introducing a deep neural network in conjunction with the interaction neural network. This invention improves the model's expressive power and enhances its generalization ability. It addresses the shortcomings of existing solutions in automatically exploring high-order feature interactions and distinguishing the importance of different features, as well as their inability to automatically extract features and uncover potential important features from the raw input.

Description

Technical Field

[0001] This invention relates to an online knowledge sharing and recommendation method based on attention mechanisms and high-order feature interactions. Background Technology

[0002] Online users post and answer questions from various disciplines on knowledge-sharing platforms (such as Quora and Stack Overflow overseas, and Zhihu in China). These platforms recommend new questions to users by analyzing their profiles and historical interactions. However, due to the wide range of users' interests and backgrounds, and the vast amount of online resources available on these platforms, users often have to choose from numerous questions, which may prevent them from directly accessing online resources that interest them. If the question selection process is too cumbersome, users may become frustrated and ultimately leave, which contradicts the platform's original intention of promoting user engagement and knowledge sharing.

[0003] Traditional recommendation system methods, such as content / use-based collaborative filtering, are gradually failing to meet the needs of today's large, complex, and dynamically changing data due to scalability challenges when dealing with large-scale datasets and difficulties in effectively modeling high-order features.

[0004] Previous research [2] proposed a hybrid model that combines a learning management system, a set of web 2.0 tools, and an e-learning recommendation system to enhance personalized online learning. However, this work did not provide technological innovation for online learning service recommendation systems; it simply combined existing technologies and could not specifically address the problems brought about by the characteristics of data in the education field. In a previous review [1], several recommendation methods for e-learning services were listed and analyzed. However, they were all too simple and could not be applied to online learning services in the context of big data in educational resources. Another study [3] proposed a hybrid recommendation algorithm that combines collaborative filtering and sequence pattern mining for peer-to-peer learning environments, but did not consider how to improve personalized learning services. [4] and [5] studied learning path recommendation; however, the proposed models mainly relied on demographic information for construction and did not provide sufficient space for in-depth exploration of individual preferences.

[0005] The existing solution has the following drawbacks:

[0006] 1. Existing solutions cannot fully explore and model the complex feature interactions at the underlying level of data in personalized knowledge recommendation services.

[0007] 2. Existing solutions cannot automatically explore high-order feature interactions or distinguish the importance of different features.

[0008] 3. Existing solutions are unable to automatically extract features or mine potential important features from raw input.

[0009] References:

[0010] [1] R.Sikka, A.Dhankhar and C.Rana, A survey paper on e-learningrecommender system. International Journal of Computer Applications.47(9).27-30(2012), DOI: http: / / dx.doi.org / 10.5120 / 7218-0024.

[0011] [2]N.Hoic-Bozic,MHDlab andV.Mornar,Recommender system andweb2.0tools to enhance ablended learning model.IEEE Transactions on education.59(1).39-44(2015).

[0012] [3]W.Chen,Z.Niu,X.Zhao andY.Li,A hybrid recommendation algorithm adapted in e-learning environments.WorldWide Web.17(2).271-284(2014).

[0013] [4]Z.Rusak.Exploitation of micro-learning for generating personalized learning paths.in DS 87-9Proceedings of the 21st International Conference onEngineering Design(ICED 17)Vol 9:Design Education,Vancouver,Canada,21-25.08.2017.2017,pp.129-138.

[0014] [5] Q. Zhao, Y. Zhang and J. Chen. An improved ant colony optimization algorithm for recommendation of micro-learning path. in 2016 IEEE International Conference on Computer and Information Technology (CIT). 2016, pp. 190-196, IEEE. Summary of the Invention

[0015] The purpose of this invention is to overcome the shortcomings of existing solutions and provide an online knowledge sharing and recommendation method based on attention mechanism and high-order feature interaction. This method improves the expressive power of the model and enhances its generalization ability.

[0016] To achieve the above objectives, the technical solution of the present invention is: an online knowledge sharing recommendation method based on attention mechanism and high-order feature interaction, which adds a high-order feature interaction neural network to the model, combines high-order feature interaction and attention mechanism, and introduces a deep neural network to combine with the interaction neural network.

[0017] In one embodiment of the present invention, the method includes:

[0018] Input steps:

[0019] Input a high-dimensional vector containing information about users and items;

[0020] Embedding steps:

[0021] High-dimensional vectors of user and item information are used to reduce the dimensionality and sparsity of the original data through an embedding layer;

[0022] Interactive neural network steps:

[0023] High-order feature interactions are automatically generated using interactive neural networks;

[0024] Deep neural network steps:

[0025] Deep neural networks implicitly capture latent information and feature combinations in a way that distinguishes them from interactive neural networks;

[0026] Residual connection steps:

[0027] Before feeding the information transformed by the deep neural network to the attention network, the residual is used to continuously add the original input information to the output of the deep neural network and the interaction neural network;

[0028] Attention network steps:

[0029] Attention networks are used to explain the differences in importance between various features; the calculated attention scores are projected back to the output of the residual connection step;

[0030] Output steps:

[0031] Output a decimal number in the range of 0-1, representing the user's level of interest in the given question.

[0032] In one embodiment of the present invention, the high-dimensional vector of user and item-related information includes the historical records of interactions between users and items, as well as additional information about users and items.

[0033] In one embodiment of the present invention, the embedding operation of the embedding layer is described as follows:

[0034] X embed，i =W embed，i X i

[0035] X embed，i For the embedding result of the i-th classification feature, W embed，i X is the embedding matrix that maps the i-th original classification feature to a low-dimensional space. i Let i be the i-th feature.

[0036] In one embodiment of the present invention, the interactive neural network consists of several layers, each layer being a feature interaction operation. The feature interaction operation for each interaction layer is described as follows:

[0037] X l+1 =X0X l W l +b l +X l

[0038] Where X l For the output of layer l, W l and b l Weights and bias parameters for each interaction layer.

[0039] In one embodiment of the present invention, each layer of the deep neural network is described as follows:

[0040] h l+1 =f(W l h l +b l )

[0041] Where h l The output of the l-th layer of the deep component is given by f(·), where f(·) is the activation function and W is the output of the l-th layer. l and b l These are the parameters of the l-th layer of the deep neural network.

[0042] In one embodiment of the present invention, the activation function is the ReLU activation function.

[0043] In one embodiment of the present invention, the attention mechanism of the attention network is described as follows:

[0044] a′ i =ReLU(WX i +b)

[0045]

[0046] W and b are model parameters, X i For the i-th feature, the attention score is calculated using the ReLU activation function.

[0047] In one embodiment of the present invention, the calculated attention score is projected back to the output of the residual connection step, as described below:

[0048] X s =a i WX

[0049] a i For the calculated attention score, W represents the weights used, and X represents the output of the residual connection step. s This is the output after applying the attention mechanism.

[0050] The present invention also provides an online knowledge sharing recommendation system based on attention mechanism and high-order feature interaction, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the steps of the method described above.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. This invention addresses the shortcomings of existing solutions in personalized knowledge recommendation services by incorporating high-order feature interaction neural networks into the model, which fail to fully explore and model the complex feature interactions at the underlying level of the data. This improves the model's expressive power and enhances its generalization ability.

[0053] 2. This invention combines high-order feature interaction and attention mechanism, which solves the shortcomings of existing solutions that cannot automatically explore high-order feature interaction and distinguish the importance of different features.

[0054] 3. By introducing a deep neural network and combining it with an interactive neural network, this invention solves the problem that existing solutions cannot automatically extract features and mine potential important features from the original input.

[0055] The advantages and uses of this invention are as follows:

[0056] Personalized Recommendations: Higher-order interactions help create more personalized recommendations. For example, in recommendation systems, considering higher-order feature interactions can help understand a user's subtle preferences based on different combinations of items.

[0057] Capturing complex patterns: Higher-order interactions can capture intricate, non-linear relationships between features that are often overlooked by models that only consider lower-order interactions. This helps in better understanding and modeling user behavior and item characteristics.

[0058] Improved prediction accuracy: By incorporating higher-order interactions, the model can make more accurate predictions. This is because it considers the synergistic effects of multiple feature interactions and can better interpret patterns in the underlying data.

[0059] Better generalization ability: Models that include higher-order interactions tend to perform better when faced with new, unseen data because they are able to learn more detailed and robust feature representations.

[0060] Powerful representational capabilities: Because deep neural networks have a multi-layered structure, they are able to learn rich and complex representations of data, thereby capturing complex patterns and relationships in the data.

[0061] Capturing key information: Attention mechanisms can dynamically assign different weights to different parts of the input sequence, allowing the model to focus on the most important information. This is particularly effective when processing long sequence data (such as text or time series). Attached Figure Description

[0062] Figure 1 This is a schematic diagram of the model structure of the present invention.

[0063] Figure 2 This is a flowchart of the steps of the method of the present invention. Detailed Implementation

[0064] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0065] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0066] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0067] This invention provides an online knowledge sharing recommendation method based on attention mechanism and high-order feature interaction. It adds a high-order feature interaction neural network to the model and combines high-order feature interaction with attention mechanism. At the same time, it introduces deep neural network and combines it with interaction neural network.

[0068] The following are specific implementation examples of the present invention.

[0069] like Figure 1 As shown, the model proposed in this invention includes three important networks:

[0070] An interactive neural network for exploring feature interactions, a deep neural network for mining potential information, and an attention network for distinguishing the importance of different features.

[0071] Figure 2 The flowchart below shows the steps of an online knowledge sharing and recommendation method based on attention mechanism and high-order feature interaction according to the present invention. The specific steps are as follows:

[0072] Input steps:

[0073] The model's input is a high-dimensional vector containing information about users and items. This includes a history of interactions between users and items, as well as additional information about users and items, such as user age and product category.

[0074] Embedding steps:

[0075] High-dimensional vectors containing user and item information are embedded to reduce the dimensionality and sparsity of the original data. Embedding not only reduces computation but also improves the model's recommendation performance.

[0076] X embed，i =W embed，i X i

[0077] X embed，i For the embedding result of the i-th classification feature, W embed，i X is the embedding matrix that maps the i-th original classification feature to a low-dimensional space. i Let i be the i-th feature.

[0078] Interactive neural network steps:

[0079] An interactive neural network is used to automatically generate high-order feature interactions. This network consists of several layers, each representing a feature interaction operation. The feature interaction operation for each interaction layer can be simply described as follows:

[0080] X l+1 =X0X l W l +b l +X l

[0081] Where X l For the output of layer l, W l and b l The weights and bias parameters for each interaction layer. This special network structure allows for increased interaction as the network deepens.

[0082] Deep neural network steps:

[0083] Deep networks implicitly capture latent information and feature combinations in a way that distinguishes them from interactive neural networks. Each layer of a deep neural network can be described as follows:

[0084] h l+1 =f(W l h l +b l )

[0085] Where h l This represents the output of the l-th layer of the deep component, where f(·) is the activation function; in this example, ReLU activation is used. l and b l These are the parameters of the l-th layer of the deep network.

[0086] Residual connection steps:

[0087] Before feeding the transformed information to the attention network, residuals are used to continuously add the original input information to the outputs of the deep network and the interactive neural network. The purpose of this is to preserve the original input information, which might be lost after passing through multiple layers of neural networks.

[0088] Attention network steps:

[0089] Attention networks are used to explain the differences in importance among various features. The attention mechanism can be described as follows:

[0090] a′ i =ReLU(WX i +b)

[0091]

[0092] W and b are model parameters. Note that the score is calculated using the ReLU activation function.

[0093] The calculated attention score is projected back into the output of the residual connection step. This process can be described as follows:

[0094] X s =a i WX

[0095] a i To calculate the attention score, W represents the weights used in the network, and X represents the output of the residual connection step. s This is the output after applying the attention mechanism.

[0096] Output steps:

[0097] The model outputs a decimal number in the range of 0-1, representing the user's level of interest in a given question.

[0098] The application process and method of the product using the method of this invention are as follows:

[0099] The system based on the method of this invention can analyze users' historical behavior and preferences to provide personalized recommendations for products or content.

[0100] 1. Data Input

[0101] The input data for the recommendation system includes:

[0102] User historical behavior data (such as browsing history, purchase history, etc.)

[0103] Product or content information (such as product description, category, price, etc.)

[0104] User personal information (such as age, gender, interests, etc.)

[0105] 2. Data Preprocessing

[0106] The input data undergoes the following preprocessing steps:

[0107] Data cleaning: removing missing values ​​and outliers

[0108] Formatting: Converting data into a uniform format

[0109] Feature extraction: Extracting useful features from raw data, such as user browsing frequency and product popularity.

[0110] 3. Model Training

[0111] The model training process using this product includes:

[0112] Adjust model parameters such as learning rate, batch size, and number of training epochs.

[0113] Model performance was evaluated using cross-validation and test sets.

[0114] 4. Real-time recommendations

[0115] The trained model is invoked to generate recommendation results based on the user's historical and current behavior, and these recommendations are then displayed to the user.

[0116] The present invention also provides an online knowledge sharing recommendation system based on attention mechanism and high-order feature interaction, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the steps of the method described above.

[0117] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0119] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0120] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0121] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An online knowledge sharing and recommendation method based on attention mechanism and high-order feature interaction, characterized in that, The model incorporates a high-order feature interaction neural network, combining high-order feature interaction with attention mechanisms, and also introduces deep neural networks, integrating them with the interaction neural network; including: Input steps: Input a high-dimensional vector containing information about users and items; Embedding steps: High-dimensional vectors of user and item information are used to reduce the dimensionality and sparsity of the original data through an embedding layer; Interactive neural network steps: High-order feature interactions are automatically generated using interactive neural networks; Deep neural network steps: Deep neural networks implicitly capture latent information and feature combinations in a way that distinguishes them from interactive neural networks; Residual connection steps: Before feeding the information transformed by the deep neural network to the attention network, the residual is used to continuously add the original input information to the output of the deep neural network and the interaction neural network; Attention network steps: Attention networks are used to explain the differences in importance between various features; the calculated attention scores are projected back to the output of the residual connection step; Output steps: Output a decimal number in the range of 0-1, representing the user's level of interest in the given question; An interactive neural network consists of several layers, each of which represents a feature interaction operation. The feature interaction operation for each interaction layer can be described as follows: X l+1 =X0X l W l +b l +X l Where X l For the output of layer l, W l and b l Weights and bias parameters for each interaction layer.

2. The online knowledge sharing and recommendation method based on attention mechanism and high-order feature interaction according to claim 1, characterized in that, A high-dimensional vector of information related to users and items, including a history of interactions between users and items, as well as additional information about users and items.

3. The online knowledge sharing and recommendation method based on attention mechanism and high-order feature interaction according to claim 1, characterized in that, The embedding operation of the embedding layer is described as follows: X embed,i =W embed,i X i X embed,i For the embedding result of the i-th classification feature, W embed,i X is the embedding matrix that maps the i-th original classification feature to a low-dimensional space. i Let i be the i-th feature.

4. The online knowledge sharing and recommendation method based on attention mechanism and high-order feature interaction according to claim 1, characterized in that, Each layer of a deep neural network is described as follows: h l+1 =f(W l h l +b l ) Where h l The output of the l-th layer of the deep component is given by f(·), where f(·) is the activation function and W is the output of the l-th layer. l and b l These are the parameters of the l-th layer of the deep neural network.

5. The online knowledge sharing and recommendation method based on attention mechanism and high-order feature interaction according to claim 4, characterized in that, The activation function uses the ReLU activation function.

6. The online knowledge sharing and recommendation method based on attention mechanism and high-order feature interaction according to claim 1, characterized in that, The attention mechanism of attention networks is described as follows: and i =ReLU(WX i +b) W and b are model parameters, X i For the i-th feature, the attention score is calculated using the ReLU activation function.

7. The online knowledge sharing and recommendation method based on attention mechanism and high-order feature interaction according to claim 1, characterized in that, The calculated attention score is projected back into the output of the residual connection step, as described below: X s =a i WX a i For the calculated attention score, W represents the weights used, and X represents the output of the residual connection step. s This is the output after applying the attention mechanism.

8. An online knowledge sharing and recommendation system based on attention mechanisms and high-order feature interactions, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, which, when executed by the processor, enable the implementation of the steps of the method as described in any one of claims 1-7.

Images