Wisdom subject forwarding behavior prediction system based on heterogeneous graph neural network

Abstract

The application provides a wisdom subject forwarding behavior prediction system based on a heterogeneous graph neural network, the prediction system comprising: an embedding layer for establishing a heterogeneous graph related to a subject according to user information, publishing information and forwarding information; a convolutional neural network (CNN) module for generating information text feature representation through embedding nodes of related information in the user and information text in the heterogeneous graph; a heterogeneous graph neural network (GNN) module for obtaining first interest encoding information h v P through embedding nodes of long-term forwarding attention of the user in the heterogeneous graph; an attention mechanism LSTM module for obtaining second interest encoding information U s through embedding nodes of short-term forwarding attention of the user in the heterogeneous graph; and an integrated prediction module for projecting node embedding in the first interest encoding information and the second interest encoding information to a vector space with a required output dimension through a binary classification model to predict whether a user who has not generated a forwarding behavior will generate a forwarding behavior in the future, so that the system can alleviate the problem of data sparsity.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of data mining, and relates to link prediction, graph neural network, and in particular to a wisdom subject forwarding behavior prediction system based on a heterogeneous graph neural network. BACKGROUND

[0002] The wisdom subject, here, refers to a subject that can forward, including people and automatic forwarding robots and intelligent software. In the Swarm Intelligence 1.0 era (based on biological group behavior), a large number of simple individuals express collective wisdom beyond the limitations of individual intelligence by following certain rules. With the deep driving of the Internet of Things, mobile Internet development, and the Internet of Everything, co-construction and sharing, and big data, Swarm Intelligence has entered a new era, namely the Swarm Intelligence 2.0 era (wisdom era). In the mobile Internet era, Swarm Intelligence is a new industry that goes beyond individual limitations and is generated by a large number of independent individuals under the organization structure of the Internet.

[0003] With the rapid update and iteration of the Internet and mobile devices, microblogging, Twitter, and other information platforms have become an important way for people to obtain information on a daily basis. These information platforms are changing people's way of life and communication at all times. At the same time, these information platforms generate a large amount of data every day, including a large number of user behaviors and interactive information such as forwarding, browsing, liking, and commenting. For example, Sina Weibo, as an open information platform, has a very convenient use, making it an important platform integrating information diffusion and social entertainment. Therefore, if researchers can reasonably use the massive data generated by these information platforms and establish a complete social network system, relevant researchers can analyze the information propagation law in microblogging and predict some forwarding behavior characteristics of users.

[0004] Due to the strong openness and sharing characteristics of microblogging, Twitter, and other information platforms, a large amount of information on these platforms is transmitted every day under the forwarding behavior of users. Hot information often appears in message transmission, which refers to information that is forwarded at a frequency and total amount much higher than other information. These hot information may have a greater impact on social public opinion and value guidance, and may also bring some social adverse consequences, such as the spread of network rumors. Specifically, the prediction of microblogging forwarding behavior has the following important significance.

[0005] (1) Can dig out potential hot information in advance, through the early prediction of the frequency and propagation trend of these information, can for those have the possibility to produce a big impact of the bad information release public opinion early warning. At the same time, researchers can also analyze the factors related to the health hazards that make the network negative information become the focus of public opinion, so as to inform the relevant personnel can take various scientific targeted rules and regulations to optimize the network environment, crack down on criminal fraud behavior, avoid various malignant events.

[0006] (2) Can dig out the user's forwarding preference, through the forwarding prediction technology to explore the user's online behavior characteristics, and some long short-term interest, but also can dig out the user's characteristic information, using these characteristic information, many micro-blog based advertising company can carry out people classification on these micro-blog users, and then accurately put the advertisement to the appropriate crowd, reduce unnecessary advertising costs.

[0007] (3) In addition, in the personalized content recommendation of micro-blog platform and the marketing of micro-blog platform and other applications, the prediction of user's forwarding behavior also plays an important role. In addition, the prediction technology can also guide the development of enterprises and the decision-making of government.

[0008] The existing method still has some limitations when predicting the forwarding behavior. First, when using homogeneous information network to construct the social network in the forwarding behavior, the heterogeneous information nodes in the social network cannot be well represented. A large number of potential information in some heterogeneous information network cannot be represented in the attribute of the homogeneous information network. In addition, the homogeneous information network cannot well solve the sparsity problem of data, so it cannot capture the high-dimensional collaborative information between the relationship nodes. Finally, in the existing research, most of the attention is paid to the feature information on the static heterogeneous graph, and less attention is paid to the dynamic heterogeneous graph. And the existing dynamic heterogeneous graph analysis method does not pay attention to the time sequence evolution feature information on the static heterogeneous graph. Therefore, in the forwarding prediction problem, the time sequence characteristics of user behavior are not considered, and the forwarding mode in the heterogeneous graph cannot be modeled more comprehensively. In order to solve the above problems, the present application mainly researches from the following aspects:

[0009] (1) Because of the complexity of user forwarding behavior, the heterogeneous information network is used to model it, and the theme buffer is introduced to solve the data sparsity problem in the homogeneous network;

[0010] (2) Considering the structure information and content information in the heterogeneous information network, and combining the related characteristics of user forwarding behavior, a prediction method combining user long short-term interest is proposed;

[0011] (3) In order to mine the dynamic evolution mode in the data set, the application proposes a static heterogeneous network time sequence information snapshot capturing method based on hierarchical attention, and then extracts the time sequence evolution rule on the static heterogeneous graph through RNN time sequence attention, so as to improve the prediction effect.

[0012] The prediction of user forwarding behavior can be modeled as a link prediction problem, mainly by learning the existing user forwarding behavior in the user information network to evaluate the possibility of two users generating forwarding behavior at a certain time in the future. The link prediction method based on homogeneous information network cannot reflect the rich information in the user information network because of the singularity of nodes and edges, while the heterogeneous information network can well solve this problem. Therefore, domestic and foreign researchers more model this problem through heterogeneous information network. Under the background of current academic big data, through the related methods of data mining and the related theories of heterogeneous information network analysis, a new idea can be provided for predicting user forwarding behavior. Compared with traditional neural networks, graph neural networks have better adaptability to graph structure tasks. Therefore, with the continuous development of graph neural networks in recent years, the quality of the solution to related tasks is also continuously improved. Based on this, the application proposes a user forwarding behavior prediction model based on heterogeneous information network and graph neural network method, solves some existing deficiencies, and improves the prediction effect. SUMMARY

[0013] To achieve the above object, the application provides a wisdom subject forwarding behavior prediction system based on a heterogeneous graph neural network. The system introduces topic information in the data node embedding layer, and combines user attribute features and published forwarding text features as content feature representations of embedded nodes. The attention relationship and forwarding behavior of the user are extracted as structural feature representations. In this way, the problem of data sparsity can be alleviated. In order to pay attention to the long and short interests of the user, two independent models are used to add corresponding attention mechanisms to pay attention to the different interests of the user in the long and short term. In the process of extracting the long-term interest of the user, the attention mechanism applies intra-path aggregation to each meta-path. In this process, each target node extracts and combines information from the meta-path neighbor nodes connected based on the meta-path instance. In this way, the long-term interest extraction model captures the structure and semantic information of the heterogeneous graph from two adjacent nodes and the meta-path context between them.

[0014] The application is implemented by adopting the following technical solutions:

[0015] A wisdom subject forwarding behavior prediction system based on a heterogeneous graph neural network, the prediction system comprising:

[0016] The embedding layer establishes a heterogeneous graph including related topics according to user information, published information and forwarding information.

[0017] The convolutional neural network CNN module generates information text feature representation through embedding nodes of related information in the user and information text in the heterogeneous graph; that is:

[0018] U=f c ([B1; B2; B3; S])

[0019] Wherein: u is the feature representation of the user node, f c is a full connection layer; S is a structural feature; B1 is the feature attribute of the user itself, B2 is the feature attribute of the user sending information, and B3 is the feature attribute of the user forwarding information;

[0020] The heterogeneous graph neural network GNN module obtains the first interest encoding information through embedding nodes of long-term forwarding and attention of the user in the heterogeneous graph

[0021] The attention mechanism LSTM module obtains the second interest encoding information U s through embedding nodes of short-term forwarding and attention of the user in the heterogeneous graph; the integration prediction module predicts whether the user will generate a forwarding behavior in the future through a binary classification model for projecting node embedding in the first interest encoding information and the second interest encoding information to a vector space with a required output dimension; the binary classification model is:

[0022]

[0023] Wherein: σ is an activation function, W o is a weight matrix. is the final embedding representation of the node, that is, the first interest encoding information. Further, the heterogeneous graph establishment process is:

[0024] A heterogeneous network G=(V, E) containing user, information itself feature, and user-user interaction, user-information interaction is constructed, the nodes V and the edges E all contain different types; the network mode is S=(R, L), wherein R is a set of node types, and L is a set of edge types; the node type set R=(R U , R M , R Z ), wherein R U is a user type, R M is an information type, and R z is a theme type; the edge type set L=(L UU , L UM , L ZM ), wherein L UU is a user-user relationship edge type, L UM is a user-information relationship edge type, and LZM is a topic-information relationship edge type; the G=(V, E) network is constructed as follows:

[0025] g) node , i.e., a user node set;

[0026] h) node , i.e., an information node set;

[0027] i) node , i.e., a topic node set;

[0028] j) edge , a user-user relationship set, and the user follow relationship network is selected as the set herein;

[0029] k) edge , i.e., a user-information relationship set, and the user historical forwarding behavior is selected to construct the connection edge as the set. If the user has forwarded the information, the user-information connection edge is added to the set;

[0030] l) edge , i.e., a topic-information relationship set.

[0031] Further, the heterogeneous graph neural network GNN module further includes an intra-path aggregation layer, a first graph attention layer, a second graph attention layer, and an inter-path aggregation layer;

[0032] The intra-path aggregation layer adopts a special path instance encoder to convert the embedded node features of the long-term forwarding and follow-up of the user in the heterogeneous graph into a single vector;

[0033] The graph attention layer performs weighted summation on the single vector and the path instance related to the target node by the following formula:

[0034]

[0035] Wherein: is the normalized importance of the path instance P(v, u) of the node v in the kth attention head; given the projection feature vector h v , and the path set The intra-path aggregation of the path set generates M vector representations of the target node v∈V for a specific path, denoted as

[0036] The inter-path aggregation layer converts the node vectors of all nodes under a specific path by the element average value of the point vector to reveal the semantic information of all path combinations; the element average value of the point vector is:

[0037]

[0038] wherein: W and b are learnable parameters;

[0039] The second graph attention layer fuses the node output of the specific meta-path to obtain the first interest encoding information by the following formula

[0040]

[0041]

[0042] wherein: q is a parameterized attention vector for the node; β P can be interpreted as the relative importance of the node v to the meta-path P i ; when the β of each meta-path P P is calculated, the attention coefficient can be used to weight sum all the vectors of the node v for the specific meta-path.

[0043] Further, the attention mechanism LSTM module further includes an attention mechanism model on the forwarding content, an attention mechanism model on the forwarding time sequence, and

[0044] The attention mechanism model on the forwarding content is used to obtain the different influence α j

[0045]

[0046] wherein: u j and u are the attention values of the user u and the user who has the forwarding message behavior with the user u respectively; v T represents an attention vector;

[0047] The attention mechanism model on the forwarding content is used to obtain the attention weight and the user's respective feature vector to obtain the user's current representation on the forwarding content u c ;

[0048]

[0049] wherein: u c is the user's current interest representation based on the content, α j is the influence size;

[0050] The attention mechanism model on the forwarding time sequence is used to obtain the user's current representation on the forwarding content u c and the sequence feature representation S are spliced into a fully connected layer to obtain the second interest encoding information U s of the user:

[0051] U s =f c ([u c ;S])。

[0052] Beneficial effects:

[0053] Through the comparison of experimental results, it can be obtained that the model proposed in the application increases the precision, recall rate and F1 by 1%, 5% and 3% respectively on the data set with dense decision behavior compared with the optimal model. On the sparse data set, the precision rate decreases, but the precision rate increases by about 3.5%.

[0054] In general, the advantages of the model proposed in the application are in the following three aspects. First, the theme information is introduced, which can alleviate the sparsity problem of the interaction data between users and better reflect the interest of the user, that is, the information published by the user is little, and the extracted theme information can also be used to aggregate the data of the user. Second, the long and short-term interests of the user are combined in the model, and even if there is a short-term hotspot, the long-term interest of the user can also be reflected. Third, the order structure information embedded in the meta path is retained by using a suitable encoding function. The prediction accuracy on the heterogeneous graph is improved. BRIEF DESCRIPTION OF DRAWINGS

[0055] Fig. 1 The application relates to a user-information-theme heterogeneous network graph.

[0056] Fig. 2 The application relates to a model architecture diagram.

[0057] Fig. 3 Experimental result comparison, A is the result comparison of the dense data set, and B is the result comparison of the sparse data set.

[0058] Fig. 4 Parameter analysis diagram. DETAILED DESCRIPTION

[0059] The application provides a distribution type power source credible capacity evaluation method based on a power distribution network safety domain. Figs. 1-4 The application provides a distribution type power source credible capacity evaluation method based on a power distribution network safety domain.

[0060] The application provides a wisdom subject forwarding behavior prediction system based on a heterogeneous graph neural network.

[0061] The CNN is used for text feature extraction to convert the microblog text into content representation. The attribute feature is extracted by one-hot coding. The structural feature is extracted by the deepwalk method, and the input is a graph structure, and the output is a node representation.

[0062] GNN is used in long-term interest part, the long-term interest expression of microblog user is extracted by using multi-layer heterogeneous graph gnn, the input is heterogeneous graph, and the output is long-term interest expression of user, which is a vector.

[0063] Short-term interest uses LSTM, the input is the microblog expression recently forwarded by user, and two attentions of content and time sequence are used; the operation based on content is that attention mechanism is used to model the different influences of the information recently forwarded by user on potential forwarding user.

[0064] The operation of time sequence is that the vector is input into LSTM first, then the full connection layer output of each LSTM is used to attention mechanism. Two combinations obtain short-term interest expression.

[0065] Long-term interest expression contains intra-path aggregation and inter-path aggregation. Intra-path aggregation adopts a special meta-path instance encoder to convert all node features along the meta-path instance into a single vector. After encoding the meta-path instance into the vector representation, a graph attention layer is used to weight and sum the meta-path instances related to the target node. After aggregating the node and edge data in each meta-path, an inter-path aggregation layer is needed to combine the semantic information revealed by all meta-paths. The application assigns different weights to different meta-paths by using attention mechanism. Finally, the model uses an additional linear transformation with a nonlinear function to project the node embedding into a vector space with the required output dimension.

[0066] I. Model architecture design

[0067] The application designs a retweet prediction model based on attention and long-short heterogeneous graph neural network, which utilizes high-order structural information between published information and users. The model first establishes a heterogeneous graph including users, published information and topics, and then uses embedded information propagated by graph neural network to extract high-order information representation. The model mainly includes: a convolutional neural network CNN extracts relevant information features in user and information text, a heterogeneous graph neural network GNN obtains long-term attention and retweet interest of retweet users, an LSTM model based on attention mechanism obtains short-term attention and retweet interest of retweet users in short term, and finally a prediction module for integrating and predicting user retweet behavior. The convolutional neural network module is used to obtain the initial embedding representation of the user and the content of the published information. The module extracts information features in the text of the published information through CNN. The heterogeneous graph neural network module constructs a complete user-information-topic heterogeneous graph of user historical retweet records, and then obtains high-order coding information through graph neural network GNN. The LSTM model based on attention mechanism module is used to encode the recent retweet history information of the user to obtain the short-term interest of the user. Finally, the integration and prediction module includes a binary classification model for predicting whether a user who has not produced a retweet behavior will produce a retweet behavior in the future.

[0068] II. Embedding layer

[0069] The embedding layer includes two parts of content features and structure features of node attributes. The content features represent the user's own feature attributes B1, the user's sent information feature attributes B2 and the user's retweet information feature attributes B3, and the structure features represent the potential topological features of the user node in the heterogeneous graph. Combining the two kinds of feature attributes as the initial encoding of the embedded node can retain more information.

[0070] The application uses random walk to extract the structure features of the user node in the heterogeneous graph. The network is regarded as homogeneous to obtain the pre-trained structure embedding of each node. Specifically, the application regards the previously constructed heterogeneous graph as a homogeneous graph, temporarily ignores the attributes of edges and nodes as input, and sets the context window size, the space size of each node embedding, the iteration number and the length of random sampling, and inputs them into the DeepWalk algorithm. A plurality of rounds are sampled by random walk, and the sequence generated by sampling is learned by skip-gram, so as to obtain the structure embedding of each node.

[0071] The application uses a CNN to extract content features of a user node. The content features include user-specific feature attributes, user-sent microblog feature attributes, and user-forwarded microblog feature attributes. Therefore, three CNN modules are used in this paper as extractors of three kinds of feature information, and the three kinds of feature information are spliced to obtain the final representation of the node content features. Specifically, since a user may send or forward a large number of information, this section randomly selects a fixed number of 15 of the user's sent and forwarded information, and if there are less than 15, the number is repeated to fill the number.

[0072] Finally, the structural feature S and the content feature B need to be spliced, and the final information text feature representation is as shown in the formula.

[0073] U = f c ([B1; B2; B3; S])

[0074] Where u is the feature representation of the user node, f c is a fully connected layer.

[0075] The initial embedding of the information node and the topic node is obtained in a similar manner as the user node, but the difference is that the content feature of the information node and the topic node only includes their own text content feature. The content feature B and the structure feature S are spliced to obtain the initial embedding representation of the information node and the topic node. Their formula representation is shown in the following formula:

[0076] M = f c ([B; S])

[0077] T = f c ([B; S])

[0078] III. Heterogeneous graph neural network GNN module

[0079] (1) Long-term interest modeling, i.e., the first interest encoding information process:

[0080] In order to model the user's long-term forwarding interest, the application constructs a heterogeneous graph containing the user's complete forwarding history. And introduce the topic information, unlike the general user-information interaction information, adding the topic information extracted from the information can more accurately represent the user's forwarding interest, and to some extent, alleviate the problem of data sparsity. Then use GNN to propagate node features on the graph to learn the embedding representation of the user, and extract the high-order information representation between users.

[0081] Heterogeneous graph construction

[0082] First, a heterogeneous network containing users, information itself features, and user-user interaction, user-information interaction needs to be constructed. Define the heterogeneous network G=(V, E), the nodes V and edges E contain different types. The network model is S=(R, L), where R is the set of node types, L is the set of edge types. The node type set R=(R U ,R M ,R Z ), where R U is the user type, R M is the information type, and R z is the theme type. The edge type set L=(L UU ,L UM ,L ZM ), where L UU is the user-user relationship edge type, L UM is the user-information relationship edge type, and L ZM is the theme-information relationship edge type. The network is constructed as follows:

[0083] m) Node , that is, the user node set.

[0084] n) Node , that is, the information node set.

[0085] o) Node , that is, the theme node set.

[0086] p) Edge , that is, the user-user relationship set, and the user attention relationship network is selected as the set in this paper.

[0087] q) Edge , that is, the user-information relationship set, and the user historical forwarding behavior is selected to construct the connection edge as the set. If the user has forwarded the information, the user-information connection edge is added to the set.

[0088] r) Edge , that is, the theme-information relationship set, and the information forwarded by the user is summarized, the text content is arranged, and the text content of the information is recorded. The same LDA method is used for the information forwarded by the user to generate the theme probability distribution vector as the theme information of the information node. Then, the information M and the maximum probability theme Z are connected.

[0089] The application infers the theme of information through the LDA model, after introducing the theme of information, new information nodes that do not exist in the heterogeneous graph can be connected with the existing graph through the theme information, and then the purpose of updating the embedding in the graph convolution is achieved. In this way, the problem of data sparsity caused by the fact that part of the user forwards less historical information can be alleviated.

[0090] Heterogeneous graph neural network

[0091] The target of heterogeneous graph embedding is to map the nodes in the heterogeneous graph to a low-dimensional vector space.

[0092] A new heterogeneous graph embedding model is proposed in the application. The model includes intra-path aggregation and inter-path aggregation.

[0093] Intra-path aggregation layer

[0094] Given a meta-path P, the intra-path aggregation layer learns the structural and semantic information embedded in the target node, the meta-path-based neighbor nodes and the context between them by encoding the meta-path instance of P. Let P(v,u) be the meta-path instance connecting the target node $v$ and its meta-path neighbor node Further define the intermediate node of $P(v,u)$ as:{m P(v,u)}.

[0095] The intra-path aggregation adopts a special meta-path instance encoder to convert all node features along the meta-path instance into a single vector, as shown in the following formula.

[0096]

[0097] h P(v,u) is the aggregation result of the target node under the meta-path P(v,u), h v is the representation of the starting node of the meta-path, h u is the representation of the end node of the meta-path, and $n$ is the embedding representation of the intermediate node. The f function is a special meta-path encoding indicator. For simplicity, although there can be multiple instances connecting two nodes, P(v,u) is still used to represent a single instance.

[0098] After encoding the meta-path instance into a vector representation, a graph attention layer is used to weight sum the meta-path instances of P related to the target node v. The key idea is that different meta-path instances have different influences on the representation of the target node. A normalized importance weight All instances are weighted and summed to model this:

[0099]

[0100]

[0101] Here a T is the parameterized attention vector for meta-path P, || denotes the vector concatenation operation. denotes the importance of the meta-path instance P(v,u) to node v, LeakyReLU is the activation function, is the set of neighbor nodes, and then the importance weight coefficients (or attention coefficients) of all are normalized using the softmax function. When all get normalized importance weight coefficients (or attention coefficients) They are used to calculate the weighted combination of the representation of the meta-path instance about node v. Finally, an activation function σ(·) is output.

[0102] This attention mechanism can also be extended to multi-head, which helps to stabilize the learning process and reduce the high variance introduced by the heterogeneity of the graph. That is, K independent attention mechanisms are executed, and their outputs are spliced to obtain the following formula:

[0103]

[0104] where is the normalized importance of the meta-path instance P(v,u) of node v on the kth attention head. In summary, given the projected feature vector h v , and the meta-path set The meta-path internal aggregation of the model generates M vector representations for a specific meta-path for the target node v∈V, denoted as Each (assuming K=1) can be interpreted as a summary of the meta-path instance of the meta-path P i about node v, which shows a kind of semantic information implied in node v.

[0105] Inter-meta-path aggregation layer

[0106] After aggregating the node and edge data in each meta-path, the inter-meta-path aggregation layer is needed to combine the semantic information revealed by all meta-paths. For v∈V, |V| sets of hidden layer vectors are generated:

[0107] where W is the number of meta-paths of the node. A direct inter-meta-path aggregation method is to use the element average of these node vectors. The invention extends this method by using attention mechanism to assign different weights to different meta-paths. This operation is reasonable because meta-paths are not equally important in heterogeneous graphs.

[0108] First, for each meta-path The node vectors of all nodes under a specific meta-path are transformed and then averaged:

[0109]

[0110] where W and b are learnable parameters. Then the attention mechanism is used to fuse the nodes under a specific meta-path

[0111]

[0112]

[0113] where q is the parameterized attention vector for node v. β P can be interpreted as the relative importance of meta-path P i to node v. When the β for each meta-path P P is calculated, this attention coefficient can be used to weight-sum all the vectors of node v for a specific meta-path.

[0114] Finally, the model uses an additional linear transformation with a nonlinear function to project the node embedding into a vector space with the required output dimension:

[0115]

[0116] where σ is the activation function, and W o is the weight matrix. is the final embedding representation of node v, which is also the long-term interest representation U l of user v for the subsequent user retweet behavior prediction task.

[0117] (2) Short-term interest modeling, i.e., the second interest encoding information process:

[0118] The model of user's short-term retweet interest mainly uses the user's recent retweet history information to model the user's short-term interest. The user's short-term interest model mainly includes two parts. One part is the modeling of the user's interest attention mechanism on retweeting content, and the other part is the modeling of the user's attention on the potential timing information of the recently retweeted content.

[0119] Attention mechanism model on retweeted content

[0120] Given a network user u, this user has recently retweeted l pieces of information {n1, n2, …, nl l , the final influence α j of the user's recent retweeted information on the potential retweeting user u is modeled by using the attention mechanism.

[0121]

[0122] where u j and u are the attention values of user u and the users who recently retweeted the messages of user u, respectively. where v T represents the attention vector, and is the parameter vector to be learned. a j is the attention weight of all the retweeting users compared to the root node, i.e., the importance. The final attention weight is calculated and summed with the respective feature vector of the user to obtain the current representation of the user based on the retweet content u c .

[0123]

[0124] where u c is the current interest representation of the user based on the content, a j is the influence size.

[0125] Attention mechanism model on retweet timing

[0126] In addition to using the attention mechanism based on the retweet content to model the current interest of the user based on the retweet information content, the model proposed in the present application also focuses on the timing information hidden in the sequence of the retweet information between the user and other users, and therefore uses an attention mechanism based on the timing of the retweet history behavior to capture the timing features existing therein.

[0127] The LSTM takes the user u's recently retweeted user embedding as input, and outputs the sequence feature representation of user u. Since the retweet interest of each user is often influenced by the interaction history in the recent period of time. For example, the user can expand his attention object through the recently retweeted users. Therefore, the attention mechanism introduced above is applied to each hidden state h j and the previous hidden state {h1, h2, …, h j} to obtain the sequence feature representation s j at different times. Then, the CNN is used to fuse these features {s1, s2, …, s j}, and finally the sequence feature representation S of the user about the recent l pieces of retweet history information can be obtained.

[0128] Finally, the content-based retweet interest representation u c obtained by the attention mechanism module based on the retweet content is spliced with the sequence feature representation S obtained in this section to input into a fully connected layer to obtain the short-term retweet interest embedding feature representation U s of the user:

[0129] U s= f c ([u c ; S])

[0130] IV. Integration prediction layer

[0131] The prediction layer splices the long-term forwarding attention interest of the user and the forwarding interest embedding vector in the short term, and then linearly changes the spliced result to obtain the final representation U of the user node.

[0132] U = f c ([U s ; U l ])

[0133] For users i and j, whether they will produce forwarding behavior can be obtained by the following formula:

[0134]

[0135] where U i , U j represent the final embedding representation of the two users respectively, W and b are the weight and bias of the linear layer respectively, and Sigmoid is a nonlinear activation function. is the probability of whether there is a forwarding edge relationship between the two. If the two users produce forwarding behavior, the corresponding value is 1, otherwise it is 0. Finally, the loss function adopts the cross-entropy function for optimizing the model, as shown in the following formula:

[0136]

[0137] where Z represents the existing set of forwarding relationship edges. The loss function needs positive and negative examples of users with forwarding relationships, and negative examples need to be sampled from other users who have not produced forwarding behavior. ω(θ) is a regularization term, and λ is its weight.

[0138] V. Evaluation index

[0139] There are only two possible outcomes for predicting user retweeting behavior: the two users may have retweeting behavior in the future, or they may not have retweeting behavior in the future. Therefore, it is actually a typical binary classification problem. Table 1 shows the confusion matrix of the binary classification evaluation system. The true label of each sample in the binary classification problem is 1 (positive) or 0 (negative). However, the predicted output is in the interval [0, 1], so the usual method is to set a fixed threshold, such as 0.5, and divide the prediction higher than the threshold into 1 (positive), and divide the prediction lower than the threshold into 0 (negative). Therefore, there are four relationships between the true label and the predicted value: true positive (TP), false positive (FP), false negative (FN), and true negative (TN). The evaluation index of the classification problem is generally the accuracy of the classification, which is defined as the ratio of the total number of samples predicted accurately by the model to the total number of samples. Its calculation method is shown in the following formula.

[0140]

[0141] In order to better measure the performance of the model, this paper uses F1 score as an important indicator for evaluating the prediction accuracy of the predicted user retweeting behavior. F1 score is a comprehensive measure based on accuracy and recall. Precision is defined as the ratio of samples with a true positive label among all samples predicted to be positive; recall is the ratio of samples predicted to be positive among all samples with a true positive label. The calculation methods of precision P, recall R and F1 are shown in the following formulas.

[0142]

[0143]

[0144]

[0145] Six. Baseline model comparison

[0146] Random: Retweet prediction is a binary classification task, and the decision of whether to retweet is randomly selected.

[0147] CNN: Convolutional neural network method.

[0148] Metapath2vec In this model, the heterogeneous neighbor set is formed by meta-path-based random walk, and node representation is learned using heterogeneous skip graph. However, this method only supports one meta-path, and the user-information-user meta-path is selected here.

[0149] Hawkes Process (HP): This method is a special linear auto-regressive method. The strength of user relationship is taken as a latent factor, and the similarity of users and historical interactions are taken as the induction and representation of the latent factor. HP can be used to study the relationship between users and network structure, and can be combined with logistic regression to predict user behavior.

[0150] PSO-RBF neural network: PSO-RBF neural network is a kind of feedforward network. The algorithm uses radial basis function as the activation function of hidden layer neurons, and improves it with particle swarm optimization. It can approximate any continuous function with arbitrary precision and get classification results.

[0151] Dynamic-Triad, a dynamic homogeneous network method, which uses a triadic closure process to model dynamic network embedding and generates node embeddings for downstream tasks.

[0152] Meta-DynaMix, a dynamic heterogeneous network method, which is based on the dynamic heterogeneous information network embedding technology of meta-path, integrates the topological features and latent representations based on meta-path to learn heterogeneity and temporal evolution.

[0153] Other classifiers: Forward prediction can be regarded as a binary classification. Each message is labeled as a positive or negative instance, indicating whether it will be forwarded. In this paper, some classic classifiers are used, including SVM and random forest as baseline methods.

[0154] The present application is not limited to the above-described embodiments. The above description of specific embodiments is intended to describe and illustrate the technical solutions of the present application, and the specific embodiments described above are merely illustrative and not restrictive. Without departing from the purpose of the present application and the scope protected by the claims, those skilled in the art can make many forms of specific changes under the inspiration of the present application, which are all within the protection scope of the present application.

Claims

1. A crowd-based agent forwarding behavior prediction system based on heterogeneous graph neural networks, characterized in that: The prediction system includes: The embedding layer builds a heterogeneous graph that includes related topics based on user information, published information, and forwarded information; The Convolutional Neural Network (CNN) module generates feature representations of the information text by embedding relevant nodes in the heterogeneous graph between the user and the information text; that is: U1=f c ([B1;B2;B3;S]); in: It is a fully connected layer; S represents structural features; B1 represents the user's own feature attributes; B2 represents the feature attributes of the information sent by the user; B3 represents the feature attributes of the information forwarded by the user. The Heterogeneous Graph Neural Network (GNN) module obtains the first interest encoding information by identifying embedded nodes that users have long followed and forwarded in the heterogeneous graph. ; The attention mechanism LSTM module obtains the second interest encoding information by forwarding embedded nodes that users follow in a short period of time in a heterogeneous graph. ; The integrated prediction module projects the node embeddings from the first and second interest-encoded information into a vector space with the required output dimension to predict whether a user will forward a message that has not yet been forwarded in the future. The integrated prediction module: The user node's final representation U2 is obtained by concatenating the embedding vectors of the user's long-term forwarding interest and short-term forwarding interest, and then performing a linear transformation on the concatenated result. ； Whether users i and j will engage in forwarding behavior can be determined by the following formula: ； in: , The final embedded representations representing the two users respectively, and These represent the weights and partial rank of the linear layer, respectively. It is a non-linear activation function; This represents the probability that there will be a forwarding edge relationship between the two.

2. The crowd intelligence agent forwarding behavior prediction system based on heterogeneous graph neural networks according to claim 1, characterized in that: The heterogeneous graph construction process: Construct a heterogeneous network that includes the characteristics of users and information themselves, as well as user-user interactions and user-information interactions. , node and edge They all contain different types; their network modes are as follows: ,in It is a collection of node types. It is a set of edge types; its set of node types This paper selects the user attention relationship network as the set. It is the user type. It is the information type. It is a theme type; Its edge type set ,in It is a user-user relationship edge type. It is a user-information relationship edge type. It is a topic-information relationship edge type; the stated The network is constructed as follows: a) Node That is, the set of user nodes; b) Nodes That is, a set of information nodes; c) Node That is, the collection of topic nodes; d) edge The user-user relationship set is selected by choosing the network of user-related relationships. e) Side That is, the set of user-information relationships; Select the user's historical forwarding behavior to construct the connection edge as the set. If the user has forwarded the message, add the user-message connection edge to the set. f) edge That is, the set of topic-information relationships.

3. The crowd intelligence agent forwarding behavior prediction system based on heterogeneous graph neural networks according to claim 1, characterized in that: The heterogeneous graph neural network (GNN) module further includes an internal aggregation layer of metapaths, a first graph attention layer, a second graph attention layer, and an inter-metapath aggregation layer; The meta-path aggregation layer uses a special meta-path instance encoder to convert the embedded node features that users have been forwarding and paying attention to in the heterogeneous graph into a single vector. The graph attention layer concatenates and weights the meta-path instances related to the target node from a single vector using the following formula: ； in: For nodes Metapath instance at the k-th attention head The importance of normalization; given the projected feature vector set of metapaths The aggregation within the metapath of a set is the target node. generate A vector representation for a specific meta-path, denoted as ; The meta-path aggregation layer transforms the node vectors of all nodes under a specific meta-path by averaging the element-wise vectors of the point vectors, revealing the semantic information of all meta-path combinations; the element-wise average of the point vectors is: ； in: and These are learnable parameters; The second graph attention layer fuses the first interest encoding information from nodes along a specific meta-path using the following formula. : ； ； in: This is a parameterized attention vector for a node; Metapath The relative importance of nodes; when each metapath is calculated. of Through this attention coefficient, the node All vectors for a specific metapath are summed using a weighted average.

4. The crowd intelligence agent forwarding behavior prediction system based on heterogeneous graph neural networks according to claim 1, characterized in that: The attention mechanism LSTM module also includes an attention mechanism model for forwarded content, an attention mechanism model for forwarded timing, and a fully connected layer; The attention mechanism model for forwarded content is established by using the following formula to analyze information about a user's most recently forwarded content and potential forwarding users. Different Influence : ; in: and They are users and recent interactions with users Attention score of users who forward messages; Represents the attention vector; The attention mechanism model on the forwarded content obtains the user's current representation based on the forwarded content by weighting and summing the attention weights and the user's respective feature vectors using the following formula. ; ； in It represents the user's current content-based interests. It's about the size of the influence; The attention mechanism model based on the forwarding sequence uses the user's current representation based on the forwarded content. With sequence feature representation The input is concatenated into a fully connected layer to obtain the user's secondary interest encoding information. : 。

Images