News recommendation method and system based on heterogeneous graph neural network and fusing multi-behavior and multi-modal data

Abstract

The invention provides a news recommendation method and system based on heterogeneous graph neural network fusion multi-behavior and multi-modal data, and relates to the technical field of recommendation systems. The method comprises the steps of obtaining user multi-behavior historical data and a candidate news set, wherein the user multi-behavior historical data comprises historical behavior records of a plurality of users for each piece of news in the historical news set; extracting a multi-modal feature of the historical news set, and generating a corresponding news embedding vector representation; the method comprises the following steps of: constructing a user multi-behavior heterogeneous graph by taking a user and news as nodes of the graph and taking various behaviors of the user on a single piece of news as a plurality of edges between two nodes, and initializing the user multi-behavior heterogeneous graph; adjusting an edge weight in the heterogeneous graph, and aggregating node information by using a graph attention layer to obtain a user interest feature; and calculating the similarity between the user interest feature and the multi-modal feature of each piece of news in the candidate news set, and determining the probability of recommending each piece of candidate news to the corresponding user so as to generate a recommendation list.

Description

Technical Field

[0001] The present invention relates to the technical field of recommendation systems, and in particular to a news recommendation method and system based on a heterogeneous graph neural network fusing multi-behavior and multi-modal data. Background Art

[0002] With the improvement of media technology and network service level, more and more users are accustomed to browsing online news through multimedia platforms such as Weibo. However, with the rapid increase in the amount of information, users often face the problem of information overload when obtaining news information.

[0003] Traditional news recommendation methods mainly include content-based news recommendation methods, collaborative filtering-based news recommendation methods, and hybrid recommendation-based news recommendation methods. The advantage of content-based news recommendation methods is that they do not require users' browsing history or behavior data. Therefore, for new users or users who have not logged in, news can be recommended only by extracting news content features, and there is no cold start problem. However, since only the similarity of news content features is considered without considering other browsing behaviors of users, changes in user interests may be ignored, resulting in too single recommendation results and failure to ensure the timeliness and diversity of recommendations; the advantage of collaborative filtering-based news recommendation methods is that they can make full use of users' browsing history and behavior data by analyzing the similarity between users and other users, and mine users' potential interests and preferences. However, since they need to rely on a large amount of user behavior data and the similarity of behavior data, and rarely consider the characteristics of the news content itself, they may lead to cold start problems and lack of explainability, thereby reducing the trust and satisfaction of recommendations; the advantage of hybrid recommendation-based news recommendation methods is that they combine the advantages of multiple recommendation algorithms and can more comprehensively consider various characteristics of users and news, thereby providing more personalized and accurate recommendation services. However, the need to integrate and optimize multiple algorithms and data sources may make the implementation of recommendation methods relatively complex, with high development and maintenance costs, while also increasing the complexity and uncertainty of the system.

[0004] In recent years, deep learning models have been widely used in news recommendation systems, and many new news recommendation methods based on deep learning have been generated. These models can effectively process unstructured data, capture the complex relationship between users and news, and improve the accuracy and diversity of recommendations. The news recommendation system based on deep learning establishes a user interest model by learning the user's historical click data and news features, thereby accurately predicting the user's interests. Compared with traditional recommendation systems, news recommendation systems based on deep learning can more effectively handle complex user interests and behavior patterns, improve recommendation accuracy and user experience. For example, "Meng Lingkang, Shi Chongyang. A Context-aware Interest Drift Network for Session-based News Recommendations [C]. 2020 IEEE 6th International Conference on Computer and Communications (ICCC), Chengdu, China, 2020" proposed a context-aware interest drift network for session-based news recommendation. The network combines rich context information, user interest modeling, and news text representation. By using a bidirectional attention recurrent network, multiple aspects of interest changes are integrated to simulate user interest drift. In addition, it combines environmental, breaking news, and article content information to provide rich functionality for user interest modeling. At the same time, the network uses multi-height filtered CNNs to process text content and additional news information to obtain effective text representation of article content, thereby alleviating the cold start problem. However, existing news is no longer limited to a single text form, but exists between multiple modalities and mixed modalities. Compared with news recommendations in a single text form, multimodal news article recommendations often bring some additional challenges, including how to unify different feature representations and structures representing data of different modalities. At the same time, when using graph neural networks for modeling, their generalization ability is often limited, and it is difficult to avoid noise problems in scene semantics. Summary of the invention

[0005] In view of the problem that existing news recommendation methods cannot be well applied to multimodal news articles, the present invention provides a news recommendation method and system based on heterogeneous graph neural network fusing multi-behavior and multimodal data.

[0006] In a first aspect, the present invention provides a news recommendation method based on a heterogeneous graph neural network fusing multi-behavior and multi-modal data, wherein the multi-modal data includes images and texts of the news, and the texts include titles, contents, and entities, and is characterized in that it includes:

[0007] Step 1: Obtain user multi-behavior history data and candidate news sets, wherein the user multi-behavior history data includes historical behavior records of several users on each news in the historical news set; the historical behavior records include clicks, likes, and shares;

[0008] Step 2: Extract the multimodal features of each news item in the historical news set, and integrate the news ID embedding with the multimodal features to generate the corresponding news embedding vector representation; wherein the multimodal features refer to the features obtained by extracting and fusing the image and text of the news;

[0009] Step 3: Consider users and news as nodes of the graph, and treat the user's various behaviors on a single piece of news as multiple edges between two nodes, one behavior corresponds to one edge, and construct a user multi-behavior heterogeneous graph and initialize it;

[0010] Step 4: Adjust the edge weights in the user multi-behavior heterogeneous graph, and use the graph attention layer to aggregate the node information after adjusting the edge weights to obtain the user interest features;

[0011] Step 5: Calculate the similarity between the user's interest features and the multimodal features of each news in the candidate news set, and calculate the probability of recommending each candidate news to the corresponding user based on the similarity, sort the candidate news by probability value, and generate a recommendation list.

[0012] Furthermore, in step 2, the image feature extraction process includes: extracting the region of interest ROI of the news image, performing feature extraction on the ROI, and obtaining the image feature P.

[0013] Furthermore, in step 2, the text feature extraction process includes: segmenting the news title, content and entity into words respectively, and generating a title word sequence W using a word-level multi-head self-attention network. T , content word sequence W b and entity word sequence W e .

[0014] Furthermore, step 2 specifically includes:

[0015] The Transformer attention mechanism is used to learn the cross-modal interaction between the image features and text features of the news, generating new image feature sequences P′ and text feature sequences W′;

[0016] Encode the news ID and generate the news ID embedding n And, embed the news ID into e n , image feature sequence P′ and text feature sequence W′ are combined to obtain the news embedding vector representation E = {e n , e m,i|m∈S}; where S={p,w} is the modality set, p represents the image modality, w represents the text modality, and e m,i represents the i-th eigenvector in mode m, e m，i ∈P′∪W′.

[0017] Furthermore, in step 3, the user multi-behavior heterogeneous graph is initialized, specifically including: initializing user nodes using user ID embedding, initializing news nodes using news embedding vector representation, and initializing the weights of each edge between two nodes using parameters of various user behaviors on a single piece of news.

[0018] Furthermore, step 4 specifically includes:

[0019] Use the learnable transformation matrix to unify the feature dimensions of different nodes as follows:

[0020] h′ θ =σ(W θ ·h θ +b φ )

[0021] Among them, σ is a nonlinear activation function, W θ is the learnable transformation matrix corresponding to the node type θ, bφ is the learnable bias vector, and h θ is the node feature, h′ θ is the node feature after transforming the feature dimension;

[0022] Assume u represents the user node and n represents the news node. The attention weight coefficient of node u and its neighbor node n is calculated according to the following formula:

[0023]

[0024] Among them, LeakyReLU is the activation function, a is the learnable attention weight vector, || represents the concatenation operation of the vector, and h′ u and h′ n They represent the user node features and news node features after the feature dimensions are transformed, and t represents the tth attention head;

[0025] The attention weight coefficient is normalized using the softmax function as follows:

[0026]

[0027] in, is the set of neighbor nodes of the current user node u;

[0028] According to the normalized attention weight coefficient, the multi-head attention mechanism is used to perform Transform, and then connect its features to obtain the output feature vector of node u in the l+1th graph attention layer

[0029]

[0030] Among them, || represents the concatenation operation of vectors, is the set of neighbor nodes of the current news node n, σ is a nonlinear activation function, T represents the number of heads in the multi-head attention mechanism, W( l , t) is the linear transformation matrix of the t-th attention head in the l-th layer.

[0031] Calculate the edge weight w between user node u and news node n u，n :

[0032]

[0033] Among them, w type Indicates the basic weight of the corresponding behavior type, represents the number of times user node u performs the corresponding behavior type on news node n, decay(·) is the time decay function, T current Indicates the current timestamp, Indicates the timestamp of the last time user node u performed the corresponding behavior type on news node n;

[0034] According to w u，n The user interest feature H is obtained according to the following formula u :

[0035]

[0036] Furthermore, step 4 also includes: calculating the abnormality degree of edge (u, n) according to the following formula: According to the degree of abnormality The edge weight w is calculated as follows u，n The adjustment coefficient γ is used to adjust the edge weight w u，n Make adjustments;

[0037]

[0038]

[0039] w′ u，n =γ·w u，n

[0040] in, represents the abnormality of the behavior corresponding to edge (u, n), x represents the browsing time or number of clicks corresponding to the behavior of edge (u, n), μ and σ represent the mean and standard deviation of the corresponding behavior type, and w′u，n represents the adjusted edge weight.

[0041] Furthermore, step 4 also includes: using the following loss function Train the graph attention layer:

[0042]

[0043] Among them, y u，n represents the actual interaction value between user node u and news node n, represents the prediction similarity, Represents y u，n and The cross entropy loss between un represents the adaptive parameter for adjusting the weight of edge (u, n), s un is the abnormality score of edge (u, n), h u and h n are the feature vectors of user node u and news node n, respectively, G Represents user multi-behavior as the edge set in the heterogeneous graph G.

[0044] In a second aspect, an embodiment of the present invention provides a news recommendation system based on a heterogeneous graph neural network that integrates multi-behavior and multi-modal data, wherein the multi-modal data includes images and texts of news, and the texts include titles, contents, and entities, including:

[0045] A data acquisition module, used to acquire user multi-behavior history data and candidate news sets, wherein the user multi-behavior history data includes historical behavior records of several users for each news in the historical news set;

[0046] A feature extraction module is used to extract the multimodal features of each news in the historical news set, and integrate the news ID embedding with the multimodal features to generate a corresponding news embedding vector representation; wherein the multimodal features refer to the features obtained by extracting and fusing the images and texts of the news;

[0047] A heterogeneous graph construction module is used to treat users and news as nodes of the graph, and treat the user's various behaviors on a single news as multiple edges between two nodes, one behavior corresponds to one edge, to construct a user multi-behavior heterogeneous graph and initialize it;

[0048] The clustering module is used to adjust the edge weights in the user multi-behavior heterogeneous graph, and use the graph attention layer to aggregate the node information after adjusting the edge weights to obtain the user interest features;

[0049] The recommendation module is used to calculate the similarity between the user's interest features and the multimodal features of each news in the candidate news set, and based on the similarity, calculate the probability of recommending each candidate news to the corresponding user, sort the candidate news by probability value, and generate a recommendation list.

[0050] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0051] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0052] Beneficial effects of the present invention:

[0053] (1) The news recommendation method and system based on heterogeneous graph neural network fusing multi-behavior and multi-modal data provided by the present invention are different from the traditional news recommendation method that only considers the single-modal news features and user click behavior. The present invention also considers the attraction of multimodal news to users due to the rich image and text semantic information contained in it, and the important influence of different user behaviors (such as clicks, likes, and shares) on user interest modeling. It comprehensively models the user interest feature vector, which can capture user interests more accurately, thereby providing personalized news recommendations and improving user satisfaction.

[0054] (2) To address the label noise problem caused by poor data quality, the present invention introduces an abnormal edge detection mechanism on the one hand, and then adjusts the edge weights of abnormal edges to reduce the impact of abnormal edges on model performance; on the other hand, a loss function with adaptive parameters is designed to dynamically adjust the weights of abnormal edges or delete abnormal edges, thereby reducing the impact of label noise on model optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 One of the flow charts of the news recommendation method based on heterogeneous graph neural network fusing multi-behavior and multi-modal data provided by an embodiment of the present invention;

[0056] Figure 2 A second flow chart of a news recommendation method based on a heterogeneous graph neural network integrating multi-behavior and multi-modal data provided in an embodiment of the present invention; wherein a represents an attention mechanism;

[0057] Figure 3 The process of constructing a heterogeneous graph of user multi-behavior provided by an embodiment of the present invention;

[0058] Figure 4The process of extracting user interest features provided by the embodiment of the present invention;

[0059] Figure 5 A schematic diagram of the structure of a news recommendation system based on a heterogeneous graph neural network integrating multi-behavior and multi-modal data provided by an embodiment of the present invention;

[0060] Figure 6 A structural block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] Combination Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a news recommendation method based on a heterogeneous graph neural network fusing multi-behavior and multi-modal data, wherein the multi-modal data includes images and texts of news, and the text includes news titles, contents and entities; wherein entities include people, places, organization names, and specific event names or activity names, etc., and includes the following steps:

[0063] S101: Obtain user multi-behavior history data and a candidate news set, wherein the user multi-behavior history data includes a number of users' historical behavior records on each news in the historical news set; it can be understood that the user's behavior on the news includes but is not limited to clicking, liking, sharing and commenting.

[0064] S102: extracting multimodal features of each news item in the historical news collection, and integrating the news ID embedding with the multimodal features to generate a corresponding news embedding vector representation; wherein the multimodal features refer to features obtained by extracting and fusing features of the news image and text;

[0065] S103: Considering users and news as nodes of a graph, and considering multiple behaviors of users on a single piece of news as multiple edges between two nodes, one behavior corresponds to one edge, thereby constructing a heterogeneous graph of user multi-behaviors and initializing it;

[0066] Specifically, the user multi-behavior historical data includes user ID, news ID, behavior type, etc., and the user behavior is used as the edge to model the user-news interaction relationship graph. The user multi-behavior heterogeneous graph is constructed through the user-news interaction relationship graph, such as Figure 3At the same time, the initialization process of the user multi-behavior heterogeneous graph includes: initializing user nodes using user ID embedding, initializing news nodes using news embedding vector representation, and initializing the weights of each edge between two nodes using the parameters of the user's various behaviors on a single piece of news.

[0067] S104: adjusting the edge weights in the user multi-behavior heterogeneous graph, and using the graph attention layer to aggregate the node information after the edge weights are adjusted to obtain the user interest features;

[0068] S105: Calculate the similarity between the user's interest features and the multimodal features of each news in the candidate news set, and calculate the probability of recommending each candidate news to the corresponding user based on the similarity, sort the candidate news features by probability value, and generate a recommendation list.

[0069] The news recommendation method provided by the embodiment of the present invention enables the recommendation system to integrate multiple information sources and improve the diversity and richness of recommended content by fusing text and image features. Through multimodal feature fusion, the present invention can deeply mine news information and comprehensively describe news content, which can more accurately capture user interests, thereby providing personalized news recommendations and improving user satisfaction. At the same time, by constructing a user multi-behavior heterogeneous graph neural network to integrate users, news and multiple behavioral relationships, the complex relationship between users and news is more comprehensively represented, thereby improving the accuracy of recommendations.

[0070] On the basis of the above-mentioned embodiments, in an embodiment of the present invention, a pre-trained VilBERT model is used to extract image features and text features of news, and to learn the cross-modal interaction between the image features and text features of news.

[0071] Specifically, usually, users and news have their own unique ID information, which is used to uniquely identify a user or news. Therefore, in the user multi-behavior history data, the ID information of the user and the news are related to each other. The embodiment of the present invention uses the IDs of the user and the news to initialize the user and news representation vectors, which are respectively denoted as e u and e n .

[0072] Image encoder: Since the information content of different areas in news images may be different, some areas such as the background in the image may not have important information when modeling news. Therefore, when extracting news image features, the pre-trained VilBERT model is used to first extract the ROI of the news image, and then further extract features from the ROI to form an image feature sequence of the ROI. Where pn represents the number of ROIs.

[0073] Text encoder: First, the news title, content and entity are separated by a tokenizer, and the word-level multi-head self-attention network is used to capture the interaction of words to learn the contextual representation of words. The corresponding attention mask is generated to represent the actual valid token position in the text (1 represents the valid position and 0 represents the fill position), and token_type_ids are used to distinguish different sentences or paragraphs. Finally, three word sequences are formed. and Among them, tn, bn, and en represent the number of words in the title, the number of words in the content, and the number of entities contained in the entity, respectively.

[0074] Cross-modal fusion: There is often a certain relationship between the text content and the image of a piece of news, and this relationship can help the present invention better understand the content of the news. The present invention uses a pre-trained VilBERT model to capture the intrinsic correlation between news text features and image features. The model uses the Transformer attention mechanism to capture the cross-modal interaction between images and text, and finally outputs a deep ROI image feature sequence P′=[p′1, p′2, ..., p′ pn ] and word sequence W′=[w′1,w′2,...,w′ wn ], where wn is the number of words in the sequence formed by integrating the three word sequences.

[0075] Based on the above content, each piece of historical data in the user's multi-behavior historical data can be integrated into a news embedding vector represented as:

[0076] E={e n , e m,i |m∈S} (1)

[0077] Among them, e n is the ID feature of the news, e m，i is the i-th eigenvector in the m-th mode, e m，i ∈P′∪W′, S={p,w} is the modal set, when m=p, e m，i is the ROI image feature When m=w, e m，i Word Features

[0078] On the basis of the above embodiments, an embodiment of the present invention provides a specific process of adjusting edge weights in a heterogeneous graph, aggregating node information after adjusting edge weights using a graph attention layer, and obtaining user interest features, including the following steps:

[0079] When processing heterogeneous graph data, the feature dimensions of each type of node may be different, resulting in certain difficulties in aggregating node information of different categories. The Graph Attention Layer (GAT) provides an effective method to aggregate different types of node features, thereby capturing the semantic relationship of high-order neighbors. Therefore, based on the above embodiments, the embodiments of the present invention provide a method of adjusting the coefficients of edges in a heterogeneous graph using an abnormal edge detection function and a loss function, and aggregating the adjusted node information using a graph attention layer to obtain a specific process of user interest features, including the following steps:

[0080] First, a learnable transformation matrix is ​​used to unify the feature dimensions of different nodes:

[0081] h′ θ =σ(W θ ·h θ +b φ ) (2)

[0082] Among them, σ is a nonlinear activation function, W θ is the learnable transformation matrix corresponding to the node type θ, bφ is the learnable bias vector, and h θ is the node feature, h′ θ is the node feature after dimension transformation.

[0083] Next, the node features are aggregated through the graph attention layer: u is set to represent the user node, and n is set to represent the neighbor node of node u, that is, the news node. On the one hand, the attention weight coefficient between node u and its neighbor node n is calculated, and then the attention weight coefficient is normalized. The features of the neighbor nodes are aggregated according to the normalized attention weight coefficient to obtain the final node features; on the other hand, the edge weights of user node u and news node n are calculated; and then, the user interest features are obtained based on the final node features and edge weights. The relevant formulas in the above process are as follows:

[0084] Attention weight coefficient:

[0085]

[0086] Use the softmax function to normalize the attention weight coefficient:

[0087]

[0088] According to the normalized attention weight coefficient, referring to the previous method, a multi-head attention mechanism is used to perform T attention mechanisms. Transform, and then connect its features to obtain the output feature vector of node u in the l+1th graph attention layer

[0089]

[0090] Among them, a is a learnable attention weight vector, || represents the concatenation operation of the vector, and LeakyReLU is the activation function. is the output feature vector of node u at layer l+1, is the set of neighbor nodes of the current news node n, σ is a nonlinear activation function, T represents the number of heads in the multi-head attention mechanism, W( l , t) is the linear transformation matrix of the t-th attention head in the l-th layer.

[0091] Assuming that this embodiment only considers the user's click, like and share behaviors, the calculation formula of the edge weight between the user node u and the news node n is:

[0092]

[0093] Among them, w type Indicates the basic weight of the corresponding behavior type, represents the number of times user u performs the corresponding behavior type on news n, decay(·) is the time decay function, T current Indicates the current timestamp, The timestamp indicating the type of action that user u last performed on news n.

[0094] User interest characteristics H u The calculation formula is:

[0095]

[0096] On the basis of the above embodiment, considering the label noise problem in the user multi-behavior historical data, in order to reduce the label noise, the embodiment of the present invention also introduces an abnormal edge detection function, such as Figure 2 and Figure 4 Specifically, for a specific behavior type, with the help of information such as the timestamp of the user behavior, the number of clicks during browsing time, etc., the mean and standard deviation of the behavior type can be calculated. Then, based on the mean (μ) and standard deviation (σ) of this behavior type, the abnormality degree z of a behavior belonging to this behavior type can be calculated using a standardized method (as shown in formula (12)), and abnormal edges that may contain noise can be identified. Then, the influence of noise can be reduced by adjusting the coefficient of the abnormal edge.

[0097]

[0098] Among them, x is the browsing time or number of clicks of the current behavior (determined by the behavior type of x). The higher the degree of abnormal behavior, the greater the absolute value of z.

[0099] For example, taking click behavior as an example and browsing time as an auxiliary, by counting the browsing time of all click behaviors, the average value and standard deviation of the behavior type of click behavior can be calculated; then, by obtaining the browsing time of the current click behavior, the abnormality of the current click behavior can be calculated according to formula (12).

[0100] The formula for adjusting edge weights is:

[0101]

[0102] w′ u，n =γ·w u，n (14)

[0103] Among them, w u，n is the edge weight calculated by the above formula (6), 0<γ<1 is the coefficient used to reduce the weight. When the z value is small, γ is close to 1, that is, the impact is small; when the z value is large, γ is close to 0, that is, the impact is large.

[0104] The embodiment of the present invention can identify and process the noise in the data by introducing the abnormal edge detection mechanism, reduce the impact of label noise on model performance, and improve the robustness of the model. It can be understood that in the embodiment of the present invention, the abnormal edge detection mechanism has been adopted to solve the label noise problem. Therefore, when training the graph attention neural network, the cross entropy loss is used as the loss function.

[0105] In the above embodiments, by constructing a heterogeneous graph neural network based on the user's multi-behavior historical data, the user's interests and preferences can be fully represented, and the interference caused by label noise can be avoided as much as possible by adjusting the weights of abnormal edges. In the embodiment of the present invention, in order to further avoid the impact of label noise on the model and more effectively reduce label noise, the embodiment of the present invention provides a new loss function with adaptive parameters to dynamically adjust the model parameters, such as Figure 2 and Figure 4 The adaptive noise reduction module shown in the figure can adapt to the weight changes of different user behaviors, adjust the weights of abnormal edges or delete abnormal edges to optimize model performance. In this embodiment, the loss function calculation formula with adaptive parameters is:

[0106]

[0107]

[0108] where l(·) is the cross entropy loss, is the predicted similarity, yu，n is the actual interaction value between user node u and news node n, α un is an adaptive parameter for adjusting the weight of edge (u, n), E G Represents user multi-behavior as the edge set in the heterogeneous graph G.

[0109] Adaptive parameter α un The calculation formula is as follows:

[0110]

[0111]

[0112] Among them, s un is the abnormality score of edge (u, n), h u and h n are the feature vectors of node u and node n respectively.

[0113] Prediction Similarity The calculation formula is:

[0114]

[0115] Among them, H C represents the news features of candidate news, H u Indicates that the user interest feature vector is obtained using the above graph attention neural network.

[0116] Finally, the Sigmoid activation function is used to convert the similarity score into a click probability, mapping the score to between [0, 1]:

[0117]

[0118] Among them, P represents user u clicking on news H C The probability of , σ is the Sigmoid activation function.

[0119] The news recommendation method based on heterogeneous graph neural network fusing multi-behavior and multimodal data provided by the embodiment of the present invention reduces the loss value of hard interaction by adjusting the coefficient of the learnable parameter of the loss function, reduces the interference of noise points on model optimization, and thus enables the multimodal news recommendation model to have better generalization performance.

[0120] like Figure 5 As shown, an embodiment of the present invention also provides a news recommendation system based on a heterogeneous graph neural network that integrates multi-behavior and multi-modal data, wherein the multi-modal data includes news images and news texts. The recommendation system includes: a data acquisition module, a feature extraction module, a heterogeneous graph construction module, a clustering module and a recommendation module.

[0121] Among them, the data acquisition module is used to obtain user multi-behavior historical data and candidate news sets, and the user multi-behavior historical data includes the historical behavior records of several users on each news in the historical news set; the feature extraction module is used to extract the multimodal features of each news in the historical news set, and integrate the news ID embedding with the multimodal features to generate the corresponding news embedding vector representation; wherein the multimodal features refer to the features obtained after feature extraction and fusion of the image and text of the news; the heterogeneous graph construction module is used to regard both users and news as nodes of the graph, and the user's multiple behaviors on a single news as multiple edges between two nodes, one behavior corresponds to one edge, to construct a user multi-behavior heterogeneous graph and initialize it; the clustering module is used to adjust the edge weights in the user multi-behavior heterogeneous graph, and use the graph attention layer to aggregate the node information after adjusting the edge weights to obtain the user interest features; the recommendation module is used to calculate the similarity between the user interest features and the multimodal features of each news in the candidate news set, and calculate the probability of recommending each candidate news to the corresponding user based on the similarity, sort the candidate news by probability value, and generate a recommendation list.

[0122] It should be noted that the news recommendation system based on heterogeneous graph neural network fusing multi-behavior and multi-modal data provided in the embodiment of the present invention is to realize the above method. Its specific functions can be referred to the above method embodiments, which will not be repeated here.

[0123] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor (processor) 601, a communication interface (Communications Interface) 602, a memory (memory) 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604. The processor 601 can call the logic instructions in the memory 603 to execute the multimodal news recommendation method, which includes: obtaining user multi-behavior history data and a candidate news set, wherein the user multi-behavior history data includes a number of users' historical behavior records for each news in the historical news set; extracting the multimodal features of each news in the historical news set, and integrating the news ID embedding with the multimodal features to generate a corresponding news embedding vector representation; wherein the multimodal features refer to the features obtained after feature extraction and fusion of the image and text of the news; regarding the user and the news as nodes of the graph, and regarding the user's multiple behaviors for a single news as multiple edges between two nodes, one behavior corresponds to one edge, constructing a user multi-behavior heterogeneous graph and initializing it; adjusting the edge weights in the user multi-behavior heterogeneous graph, and using the graph attention layer to aggregate the node information after the edge weights are adjusted to obtain the user interest features; calculating the similarity between the user interest features and the multimodal features of each news in the candidate news set, and calculating the probability of recommending each candidate news to the corresponding user based on the similarity, sorting the candidate news by the probability value, and generating a recommendation list.

[0124] In addition, when the logic instructions in the above-mentioned memory 603 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0125] The embodiment of the present invention further provides a computer program product, the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program includes program instructions, when the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining user multi-behavior history data and candidate news sets, the user multi-behavior history data includes a number of users' historical behavior records for each news in the historical news set; extracting multimodal features of each news in the historical news set, and integrating the news ID embedding with the multimodal features to generate a corresponding news embedding vector representation; wherein the multimodal features Features refer to the features obtained by extracting and fusing the features of the image and text of the news; users and news are regarded as nodes of the graph, and the user's various behaviors on a single news are regarded as multiple edges between two nodes, one behavior corresponds to one edge, and a user multi-behavior heterogeneous graph is constructed and initialized; the edge weights in the user multi-behavior heterogeneous graph are adjusted, and the node information after the edge weights are adjusted is aggregated using the graph attention layer to obtain user interest features; the similarity between the user interest features and the multimodal features of each news in the candidate news set is calculated, and the probability of recommending each candidate news to the corresponding user is calculated based on the similarity, and the candidate news is sorted by probability value to generate a recommendation list.

[0126] The embodiment of the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method provided by the above-mentioned method embodiments is implemented, for example, including: obtaining user multi-behavior history data and a candidate news set, wherein the user multi-behavior history data includes a number of users' historical behavior records for each news in the historical news set; extracting multimodal features of each news in the historical news set, and integrating the news ID embedding with the multimodal features to generate a corresponding news embedding vector representation; wherein the multimodal feature refers to the feature obtained after feature extraction and fusion of the image and text of the news; regarding the user and the news as nodes of the graph, and regarding the user's multiple behaviors for a single news as multiple edges between two nodes, one behavior corresponds to one edge, constructing a user multi-behavior heterogeneous graph and initializing it; adjusting the edge weights in the user multi-behavior heterogeneous graph, and using the graph attention layer to aggregate the node information after the edge weights are adjusted to obtain the user interest feature; calculating the similarity between the user interest feature and the multimodal features of each news in the candidate news set, and calculating the probability of recommending each candidate news to the corresponding user based on the similarity, sorting the candidate news by probability value, and generating a recommendation list.

[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A news recommendation method based on heterogeneous graph neural network integrating multi-behavior and multi-modal data, wherein the multi-modal data includes images and texts of news, and the texts include titles, contents and entities, characterized in that: include: Step 1: Obtain user multi-behavior history data and candidate news sets, wherein the user multi-behavior history data includes historical behavior records of several users for each news in the historical news set; Step 2: Extract the multimodal features of each news item in the historical news set, and integrate the news ID embedding with the multimodal features to generate the corresponding news embedding vector representation; wherein the multimodal features refer to the features obtained by extracting and fusing the image and text of the news; Step 3: Consider users and news as nodes of the graph, and treat the user's various behaviors on a single piece of news as multiple edges between two nodes, one behavior corresponds to one edge, and construct a user multi-behavior heterogeneous graph and initialize it; Step 4: Adjust the edge weights in the user multi-behavior heterogeneous graph, and use the graph attention layer to aggregate the node information after adjusting the edge weights to obtain the user interest features; Step 5: Calculate the similarity between the user's interest features and the multimodal features of each news in the candidate news set, and calculate the probability of recommending each candidate news to the corresponding user based on the similarity, sort the candidate news by probability value, and generate a recommendation list.

2. The news recommendation method based on heterogeneous graph neural network integrating multi-behavior and multi-modal data according to claim 1 is characterized in that: In step 2, the image feature extraction process includes: extracting the region of interest ROI of the news image, performing feature extraction on the ROI, and obtaining the image feature P.

3. The news recommendation method based on heterogeneous graph neural network integrating multi-behavior and multi-modal data according to claim 1 is characterized in that: In step 2, the text feature extraction process includes: segmenting the news title, content and entity into words respectively, and generating the title word sequence W using the word-level multi-head self-attention network T , content word sequence W b and entity word sequence W e .

4. The news recommendation method based on heterogeneous graph neural network integrating multi-behavior and multi-modal data according to any one of claims 1 to 3, characterized in that: Step 2 specifically includes: The Transformer attention mechanism is used to learn the cross-modal interaction between the image features and text features of the news, generating new image feature sequences P′ and text feature sequences W′; Encode the news ID and generate the news ID embedding n And, embed the news ID into e n , image feature sequence P′ and text feature sequence W′ are combined to obtain the news embedding vector representation E = {e n , e m，i |m∈S}; where S={p,w} is the modality set, p represents the image modality, w represents the text modality, and e m，i represents the i-th eigenvector in mode m, e m，i ∈P′∪W′.

5. The news recommendation method based on heterogeneous graph neural network integrating multi-behavior and multi-modal data according to claim 4 is characterized in that: Step 4 specifically includes: Use the learnable transformation matrix to unify the feature dimensions of different nodes as follows: h′ θ =σ(W θ ·h θ +b φ ) Among them, σ is a nonlinear activation function, W θ is the learnable transformation matrix corresponding to node type θ, b φ is the learnable bias vector, h θ is the node feature, h′ θ is the node feature after transforming the feature dimension; Assume u represents the user node and n represents the news node. The attention weight coefficient of node u and its neighbor node n is calculated according to the following formula: Among them, LeakyReLU is the activation function, a is the learnable attention weight vector, || represents the concatenation operation of the vector, and h′ u and h′ n They represent the user node features and news node features after the feature dimensions are transformed, and t represents the tth attention head; The attention weight coefficient is normalized using the softmax function as follows: in, is the set of neighbor nodes of the current user node u; According to the normalized attention weight coefficient, the multi-head attention mechanism is used to perform Transform, and then connect its features to obtain the output feature vector of node u in the l+1th graph attention layer Among them, || represents the concatenation operation of vectors, is the set of neighbor nodes of the current news node n, σ is a nonlinear activation function, T represents the number of heads in the multi-head attention mechanism, and W (l，t) is the linear transformation matrix of the t-th attention head in the l-th layer; Calculate the edge weight w between user node u and news node n u，n : Among them, w type Indicates the basic weight of the corresponding behavior type, represents the number of times user node u performs the corresponding behavior type on news node n, decay(·) is the time decay function, T current Indicates the current timestamp, Indicates the timestamp of the last time user node u performed the corresponding behavior type on news node n; According to w u，n The user interest feature H is obtained according to the following formula u :

6. The news recommendation method based on heterogeneous graph neural network integrating multi-behavior and multi-modal data according to claim 5 is characterized in that: Step 4 also includes: calculating the abnormality degree of edge (u, n) according to the following formula According to the degree of abnormality The edge weight w is calculated as follows u，n The adjustment coefficient γ is used to adjust the edge weight w u，n Make adjustments; w′ u，n =γ·w u，n in, represents the abnormality of the behavior corresponding to edge (u, n), x represents the browsing time or number of clicks corresponding to the behavior of edge (u, n), μ and σ represent the mean and standard deviation of the corresponding behavior type, and w′ u，n represents the adjusted edge weight.

7. The news recommendation method based on heterogeneous graph neural network integrating multi-behavior and multi-modal data according to claim 5 or 6 is characterized in that: Step 4 also includes: using the following loss function Train the graph attention layer: Among them, y u，n represents the actual interaction value between user node u and news node n, represents the prediction similarity, Represents y u，n and The cross entropy loss between un represents the adaptive parameter for adjusting the weight of edge (u, n), s un is the abnormality score of edge (u, n), h u and h n are the feature vectors of user node u and news node n, respectively, G Represents user multi-behavior as the edge set in the heterogeneous graph G.

8. A news recommendation system based on heterogeneous graph neural networks that fuses multi-behavior and multi-modal data, wherein the multi-modal data includes images and texts of news, and the texts include titles, contents, and entities, characterized in that: include: A data acquisition module, used to acquire user multi-behavior history data and candidate news sets, wherein the user multi-behavior history data includes historical behavior records of several users for each news in the historical news set; A feature extraction module is used to extract the multimodal features of each news in the historical news set, and integrate the news ID embedding with the multimodal features to generate a corresponding news embedding vector representation; wherein the multimodal features refer to the features obtained by extracting and fusing the images and texts of the news; A heterogeneous graph construction module is used to treat users and news as nodes of the graph, and treat the user's various behaviors on a single news as multiple edges between two nodes, one behavior corresponds to one edge, to construct a user multi-behavior heterogeneous graph and initialize it; The clustering module is used to adjust the edge weights in the user multi-behavior heterogeneous graph, and use the graph attention layer to aggregate the node information after adjusting the edge weights to obtain the user interest features; The recommendation module is used to calculate the similarity between the user's interest features and the multimodal features of each news in the candidate news set, and based on the similarity, calculate the probability of recommending each candidate news to the corresponding user, sort the candidate news by probability value, and generate a recommendation list.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images