A news recommendation system based on attention and hierarchical candidate user interests

Abstract

The application relates to a news recommendation system based on attention and hierarchical candidate user interests, belonging to the field of natural language processing, and comprising a news encoder, a user encoder and an interest matching module; the news encoder adopts a trnasformer combined with attention to learn news text representation, carries out multi-granularity feature extraction on the news text, obtains semantic information, simultaneously combines with an interest tree to carry out multi-granularity interest capture, and matches with candidate news; the user encoder utilizes news category labels to construct hierarchical user interest representation, and captures diversified and multi-level user interests by learning a hierarchical interest tree; the interest matching module adopts candidate news and hierarchical user interest representation as input to calculate a user-level interest score.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of natural language processing, and relates to a news recommendation system based on attention and hierarchical candidate user interest BACKGROUND

[0002] With the rapid development and wide application of Internet information technology, many users have gradually changed their news reading habits from traditional paper media to digital news. However, in the face of such a large amount of information, Internet users often cannot quickly and effectively obtain the truly valuable news they need. The main goal of news recommendation is to help users quickly and efficiently obtain the most suitable news from the vast amount of news information on the Internet.

[0003] Most of the current mainstream research methods are based on deep learning and natural language processing. Wang et al. proposed a news recommendation method combined with knowledge graph information; An et al. used a GRU network to aggregate the clicked news sequence as an interest vector and combined it with the user ID embedding; Wu et al. proposed a sentiment-aware news encoder, which is jointly trained with an auxiliary sentiment prediction task to learn a sentiment-aware news representation; Zhang et al. concatenated the news text as the input of the BERT model; Qi et al. proposed to use news category labels to construct hierarchical user interest representation. These technologies have the following problems:

[0004] (1) When reading online news, the user's interest in news shows a diversified and multi-granularity characteristic. However, the current mainstream news encoder usually extracts features through a single indicator such as the title of the clicked news, which lacks semantic information and leads to poor matching of candidate news due to insufficient feature extraction.

[0005] (2) For two news of the same category clicked by the user, there may be a long time span, and the model easily ignores the association between the two news, thereby leading to weak ability of the model to capture the long-term interest of the user. SUMMARY

[0006] Therefore, the purpose of the present application is to solve the problems of insufficient feature extraction of the news encoder and weak ability of the user encoder to capture the long-term interest of the user, and to provide a news recommendation system based on attention and hierarchical candidate user interest. The news encoder based on transformer fusion attention is used to extract features of the title, theme, sub-theme and abstract of the news, and the convolutional network fusion self-attention is used to extract features of different click spans of the user, thereby improving the ability of the model to combine the long-term and short-term interests of the user.

[0007] To achieve the above purpose, the present application provides the following technical solutions:

[0008] A news recommendation system based on attention and hierarchical candidate user interest, comprising a news encoder, a user encoder and an interest matching module;

[0009] The news encoder adopts the trnasformer combined with attention to learn the news text representation, extracts multi-granularity features of the news text, obtains semantic information, and combines the interest tree to capture multi-granularity interest and match with the candidate news;

[0010] The user encoder utilizes news category labels to construct a hierarchical user interest representation, and captures diverse and multi-level user interests by learning a hierarchical interest tree;

[0011] The interest matching module takes the candidate news and the hierarchical user interest representation as input to calculate the user-level interest score.

[0012] Further, for the type of the tth news text, the news encoder learns the specific type of news embedding t i from is the jth text embedding sequence of the ith clicked news;

[0013] The bottom layer of the news encoder is a word embedding, which converts the words in the news title, the theme and sub-theme of the news, and the summary of the news into a low-dimensional embedding vector sequence; the next layer is a transformer, which is used to form a contextual word representation; finally, a word-level attention network is used to simulate the relative importance of different words and aggregate them together;

[0014] The attention weight β i w is calculated as:

[0015]

[0016] where q w , U w , u w are trainable parameters in the word attention network, and h i represents the multi-head representation of the ith word.

[0017] Further, the hierarchical interest tree in the user encoder is divided into three layers, which are the sub-theme layer, the theme layer and the user layer;

[0018] ​The sub-topic layer adopts a candidate-aware CNN network and a candidate-aware attention network to learn the user interest representation; the candidate-aware attention network adjusts the attention weight of the behavior context through the candidate news to select important news; the candidate-aware CNN network captures the local context between adjacent clicks by using candidate news information;

[0019] The user interest representation in the topic layer is obtained by the representations of d sub-topic layers of the same topic;

[0020] The overall interest of the user in the user layer is aggregated by the representations of each topic layer.

[0021] Further, the candidate-aware attention network first applies multiple self-attention heads to the click relevance model:

[0022]

[0023]

[0024] wherein, U w , u w is a trainable parameter in the word attention network, h i represents the multi-head representation of the i-th word; Q u is a projection matrix, is the parameter of the k-th attention head, represents the user's last N clicks.

[0025] Further, the candidate-aware CNN network applies multiple filters to capture the potential patterns between adjacent clicks and the context of the candidate news:

[0026] S i =W c [c i-h ;...; c i ;...; c i+h ; n c ]

[0027] wherein S i represents the context representation of the i-th click, 2h+1 is the window size of the CNN network, W c represents the parameters of the filters in the CNN network.

[0028] Further, in the topic layer, for topic i, the user interest representation is as follows:

[0029]

[0030]

[0031] wherein Zi denotes the i-th topic group representation, is the representation of the corresponding sub-topic group s; is obtained by jointly embedding the representation of the sub-topic s and the number of clicks, and j is the attention weight of

[0032] Further, in the user layer, the weight depends on the topic group representation and the number of clicked news under the topic, as follows:

[0033]

[0034]

[0035] wherein i denotes the attention weight of the i-th topic layer representation.

[0036] Further, the interest matching module takes the candidate news and the hierarchical user interest representation as input; the candidate news includes its representation nc, topic tc and sub-topic sc; first, the candidate news is matched with the overall user interest, and the user-level interest score og is calculated according to the correlation between nc and ug: og = nc·ug.

[0037] The present application has the beneficial effects that the present application proposes a combination of transformer and attention to learn news text representation, and performs multi-granularity feature extraction on news text to obtain more rich semantic information. At the same time, multi-granularity interest capture is combined with the interest tree to better match the candidate news. Considering that the news recommendation model usually ignores news with large time span of user clicks, resulting in weak ability of the model to capture long-term user interest and low matching rate of candidate news, the present application uses a candidate-aware self-attention network on the sub-title level interest to learn the long-term interest representation of the user. At the same time, a CNN network is used to learn the short-term interest representation of the user. The two are weighted to learn the user interest representation, so as to better match the candidate news and improve the accuracy of news recommendation.

[0038] Other advantages, objects, and features of the present application will be apparent to those skilled in the art from the following specification, and it is intended to cover any alternatives, modifications, or equivalents included within the scope of the present application. The objects and other advantages of the present application can be achieved and obtained by the following specification. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to make the objects, technical solutions and advantages of the present application clearer, the preferred detailed description of the present application will be combined with the drawings as follows, wherein:

[0040] Figure 1 Structure diagram of a news recommendation system based on attention and hierarchical candidate user interests. DETAILED DESCRIPTION

[0041] The present application also can be embodied in a diverse of ways, some of which have been described herein, and it is to be understood that the present application is not limited in scope to the particular embodiments disclosed herein. Indeed, those skilled in the art will recognize that other variations and modifications of the embodiments described herein can be made without departing from the spirit and scope of the present application. It is also to be understood that the terminology employed herein is used for the purpose of describing particular embodiments only and is not intended to be limiting, as the scope of the present application will be defined by the appended claims and equivalents thereof.

[0042] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings:

[0043] The same or similar components in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it should be understood that if the terms "upper", "lower", "left", "right", "front", "back" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the terms describing the positional relationship in the drawings are only for illustrative purposes, and cannot be understood as a limitation of the present application, for those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0044] Referring to Figure 1 , the present application provides a news recommendation system based on attention and hierarchical candidate user interests, which comprises a news encoder, a user encoder and an interest matching module.

[0045] News encoder: in order to learn the text representation of news information, the model adopts trnasformer combined with attention as the text encoder. Transformer is a language representation model based on attention mechanism, which encodes and represents the input sequence and contains rich text information.

[0046] In this embodiment, an encoder is used for the type of the tth news text to learn the specific type of news embedding t i .​ is the j-th text embedding sequence of the i-th clicked news. The text encoder is implemented by a stack of transformers and attention networks. The bottom layer is word embedding, which converts words in the news title, topics and subtopics of the news, and the summary of the news into a sequence of low-dimensional embedding vectors. The next layer is a transformer, which uses a transformer to form a contextual word representation. Finally, a word-level attention network is used to simulate the relative importance of different words and aggregate them together. The attention weight of the i-th word is i w is calculated as:

[0047]

[0048] where q w , U w , u w are trainable parameters in the word attention network, h i denotes the multi-head representation of the i-th word.

[0049] User encoder: The overall framework is to construct a hierarchical user interest representation using news category labels, and capture diverse and multi-level user interests by learning a hierarchical interest tree. First, learn multiple subtopic-level interest representations to model fine-grained user interests in different news subtopics (e.g. football and golf). Learn the subtopic-level interest representation of subtopic s ij from the N ij news clicked by the user in subtopic s ij . Since the long-distance context of the clicked news is usually the information to infer the user's long-term interest. In addition, different long-term behavior contexts usually have different importance in capturing different user long-term interests, we use a candidate self-attention network to model the subtopic-level user long-term interest. At the same time, a candidate-aware CNN network is used to model the subtopic user short-term interest, which is combined to learn the user interest in subtopic s ij .

[0050] In this embodiment, a hierarchical interest tree is used as a user encoder, which is divided into three layers, namely subtopic layer, topic layer and user layer. The subtopic layer uses a candidate-aware CNN network and a candidate-aware attention network to learn the user interest representation. The core of the candidate-aware attention network is to adjust the attention weight of the behavior context through the candidate news to select important news. First, apply multiple self-attention heads to the click relevance model:

[0051]

[0052]

[0053] where, U w , u w are trainable parameters in the word attention network, h i denotes the multi-head representation of the i-th word. Q u is a projection matrix, is the parameter of the k-th attention head, denotes the user's last N clicks.

[0054] The candidate-aware CNN network can capture the local context between adjacent clicks with the candidate news information. The present application applies multiple filters to capture the potential patterns between the context of adjacent clicks and candidate news: S i = W c [c i-h ;... ; c i ;... ; c i+h ; n c ], where S i denotes the context representation of the i-th click, 2h+1 is the window size of the CNN network, W c denotes the parameters of the filters in the CNN network.

[0055] While the topic layer user interest representation of the user encoder is obtained from the representation of d sub-topic layers of the same topic. For topic i, the user interest representation is as follows:

[0056]

[0057]

[0058] where Z i denotes the i-th topic group, is the representation of the corresponding sub-topic group s. is obtained by jointly embedding the representation of the sub-topic s and the number of clicks, β j is the attention weight of .

[0059] Finally, the overall interest of the user is aggregated from the representations of each topic layer, where the weight depends on the topic group representation and the number of clicked news under the topic, as follows:

[0060]

[0061]

[0062] where a i denotes the attention weight of the i-th topic layer representation.

[0063] Interest matching module: The matching between candidate news and user interests at different granularities can provide various clues for user interest positioning. Corresponding to the hierarchical user encoder is a hierarchical user interest matching framework, which takes candidate news (including its representation nc, topic tc and subtopic sc) and hierarchical user interest representation as input. First, the candidate news is matched with the overall user interest, and the user-level interest score og is calculated according to the correlation between nc and ug: og = nc·ug.

[0064] Embodiment one:

[0065] The data set of this embodiment, Microsoft News Dataset (MIND), is a large-scale data set for information recommendation research. The source of information collection is the anonymous behavior log of Microsoft News website. MIND contains about 160,000 English news articles and more than 15 million mapping logs generated by 100 million users. The training data and validation data are a compressed folder containing four different files, as shown in Table 1.

[0066] Table 1

[0067]

[0068] MRR, nDCG@K, area under ROC curve AUG are used as evaluation indexes.

[0069] MRR: Mean reciprocal rank is defined as:

[0070]

[0071] Where the larger the MRR value is, the higher the correct recommendation item is located at the top of the ranking list.

[0072] nDCG@K: Normalized discounted cumulative gain is defined as:

[0073]

[0074]

[0075] Where rel i represents the relevance degree of the ith recommendation, and when rel i =1, it means that the item is actually selected, otherwise rel i =0. IDCG is the normalized DCG, that is, the maximum DCG value obtained according to the ideal order.

[0076] AUG: Area under ROC curve, which represents the probability that the predicted value of the positive sample is greater than that of the negative sample from the sample, is defined as:

[0077]

[0078] Wherein M, N respectively represent the number of positive samples and negative samples, and S is the ranking of all samples.

[0079] The experimental environment of the application is based on a tensorflow+keras framework, an NVIDIA TESLA P100 GPU is used to train the model, and an Adam optimizer is used to train the model.

[0080] Table 2

[0081]

[0082] The news text title size accepted by the news encoder is set to 30 words, the entity is set to 5 words, the abstract size is set to 50 words, the maximum click is set to 50, and the sub-topic layer and the topic layer interest proportion of the whole interest is set to 0.2 and 0.2.

[0083] Finally, it should be pointed out that the above examples are only used to illustrate the technical solutions of the present application and are not limiting, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should be covered in the scope of the claims of the present application.

Claims

1. A news recommendation system based on attention and hierarchical candidate user interests, characterized in that: The news encoder, the user encoder and the interest matching module are included. The news encoder adopts the trnasformer combined attention method to learn the news text representation, extracts multi-granularity features of the news text, obtains semantic information, and combines the hierarchical interest tree to capture multi-granularity interest and match the candidate news. The user encoder constructs a hierarchical user interest representation by using news category labels, and captures diverse and multi-level user interests by learning a hierarchical interest tree. The interest matching module takes the candidate news and the hierarchical user interest representation as input, and calculates the user-level interest score. The news encoder is used for the type of the tth news text to learn the specific type of news embedding t j i i j i is the jth text embedding sequence of the ith clicked news;​​ The bottom layer of the news encoder is a word embedding, which converts the words in the news title, the theme and sub-theme of the news, and the summary of the news into a low-dimensional embedding vector sequence; the next layer is a transformer, which is used to form a context word representation; finally, a word-level attention network is used to simulate the relative importance of different words and aggregate them together; Attention weight β of the ith word i w is calculated as: where q w , U w , u w are trainable parameters in the word attention network, h i denotes the multi-headed representation of the i-th word; The hierarchical interest tree in the user encoder is divided into three layers, which are the sub-theme layer, the theme layer and the user layer. The sub-theme layer uses a candidate-aware CNN network and a candidate-aware attention network to learn the user interest representation; the candidate-aware attention network adjusts the attention weight of the behavior context by the candidate news to select important news; the candidate-aware CNN network captures the local context between adjacent clicks by using candidate news information; The theme layer user interest representation is obtained from the representation of d sub-theme layers of the same theme; The overall interest of the user in the user layer is aggregated from the representation of each theme layer; The candidate-aware attention network first applies multiple self-attention heads to the click relevance model: wherein Q u is a projection matrix, is a parameter of the kth attention head, denotes the user's last N clicks; The candidate-aware CNN network applies multiple filters to capture the potential patterns between the context of adjacent clicks and candidate news: S i = W c [c i-h ;... ; c i ;... ; c i+h ; n c ] where S i represents the context representation of the ith click, 2h+1 is the window size of the CNN network, represents the user's last N clicks, W c represents the parameters of the filter in the CNN network; In the theme layer, for theme i, the user interest representation is as follows: where Z i represents the i-th topic group, is the representation of the corresponding subtopic group s; is obtained by jointly embedding the representation of the subtopic s and the number of clicks, β j is the attention weight of In the user layer, the weight depends on the theme group representation and the number of clicked news under the theme, as follows: wherein a i denotes the attention weight of the i-th topic layer representation; The interest matching module takes candidate news and hierarchical user interest representation as input; the candidate news includes its representation nc, theme tc and sub-theme sc; first match the candidate news with the overall user interest, and calculate the user-level interest score og according to the correlation between nc and u g og = nc·u g .

Images