Information cascade prediction system and method based on transformer enhanced hawkes process

Abstract

The application discloses a kind of information cascade prediction system and method based on the enhanced hawkes process of Transformer, including user embedding module, global dependence module, local dependence module, intensity function acquisition module and popularity prediction module, first, obtain user position-wise embedding and time coding as user embedding, then in the perspective of topological structure, introduce path perception hypothesis, and design two layers of attention layer from two angles of global embedding and local mode, parameterize the intensity function of hawkes process, and combine hawkes process, global embedding and local mode, learn the time and topological random characteristics coupled in information cascade diffusion process, to carry out popularity prediction;The application expands the traditional hawkes process, and effectively obtains knowledge from continuous time domain, improves the accuracy of popularity prediction.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of information processing, and relates to information diffusion, information cascade and popularity prediction based on deep learning, in particular to a method and system (Hawkesformer) for introducing a Transformer into a Hawkes process to model the time characteristics and topological structure of a cascade and finally realize cascade prediction. BACKGROUND

[0002] Sharing content through social media platforms such as Twitter, Facebook and Weibo has become the main channel for individuals to express their opinions and reactions on various topics. This trend has generated a large amount of data describing the way information spreads and has brought unprecedented convenience for generating, delivering and disseminating information items. The initial published information (e.g., Tweet) and subsequent shared retweets (e.g., Retweet) form an information cascade, representing the so-called information diffusion process. Similar phenomena have also been found in other (non-social media) settings: paper citations, blog space and email forwarding. A large number of user sharing behaviors have facilitated the rapid and massive spread of information cascades. A typical task of information cascade is to predict the size of a cascade (tweets, Weibo, etc.) after a certain period of time, i.e., the popularity prediction (the size of the information cascade), which can bring great economic and social value in many downstream applications. For example, understanding which type of advertisement maximizes the impact can make advertising more accurate; predicting the potential impact of rumors on users can enable administrators to intervene early to avoid serious consequences, etc.

[0003] Existing methods can be mainly divided into three categories: (i) Feature-based models mainly explore feature sets extracted from structure, time series, user content and profile; however, they rely on a large number of hand-crafted feature engineering and cannot be generalized from one domain to another; (ii) Probabilistic generative models aim to model events through classic examples of temporal point processes (TPP), including Poisson process and Hawkes process; however, existing methods make strong assumptions on the diffusion mechanism of cascades and have limited learning ability on large-scale cascade data; (iii) Deep learning models pay more attention to static temporal cascade diffusion within a specific snapshot in the discrete time domain, and pay little attention to the whole evolution process and the full use of all cascade dynamics, mainly using simple event sequence models (such as RNN) to model cascade events, or supplemented by GNN to model time features and topological features separately; but the two are essentially coupled, and this way of separate modeling loses cross-domain information and reduces the expression and prediction ability of the model. SUMMARY

[0004] In view of the problems of limited learning ability and cross-domain information loss existing in the current information cascade prediction method, the purpose of the present application aims to provide a novel modeling method for the diffusion process of the cascade, summarize the diffusion process of the cascade into a directed acyclic graph (DAG) that continuously diffuses in the continuous time domain, and introduce the Transformer (attention mechanism) into the Hawkes process, so as to model the coupling of time features and topological features in the continuous time domain by introducing the attention mechanism into the Hawkes process, realize effective modeling of the diffusion process of the cascade, and improve the accuracy of the popularity prediction.

[0005] The general idea of the present application is to connect two levels of attention modules together to parameterize the intensity function of the Hawkes process. The intensity function ensures the coupling of time and topological dependence of the cascade in the continuous time domain. Specifically, on the first level, a global dependence module is designed to dynamically capture the long-term diffusion process of past events in the cascade, and a primary / non-primary path hypothesis is proposed to adaptively integrate the diffusion process into the underlying DAG. This allows each node participating in the cascade at any location to update its current hidden state. On the second level, a local module is designed to capture the short-term evolution rate of the information cascade by encoding the local patterns in a fixed time slice window. Through the design of the two levels, we get the representation of the node on the global and local, and finally send it to a fully connected layer for popularity prediction.

[0006] Before giving a specific process, some basic terms and definitions are given. Given an information cascade C k And its observed time t oForwarding history record where t j ∈(0,t o ], L is the number of forwarding in the observation time window (t0,t0+t o ), t0 represents the start time. Each triple (t j ,v j ,u j ) represents that user u j forwarded user v j at time t j . For example, the forwarding history record in the appendix is: Figure 1 { (t4,u1,u4),(t5,u3,u5)},for simplicity, the latter is omitted as { (t4,u1,u4),(t5,u3,u5)},for simplicity, the latter is omitted as In addition, we use to represent the set of timestamps before the current time.

[0007] Based on the above inventive idea, the information cascade prediction system based on the Transformer enhanced Hox process provided by the application comprises:

[0008] A user embedding module (Initial User Embeddings) is used to obtain user position-wise embedding and time coding, and the user position-wise embedding and time coding are used as user embedding.

[0009] A global dependency module (Global Dependency Module) is used to distinguish all predecessor nodes as main path nodes and non-main path nodes for each user node in the user embedding, and obtain the attention probability matrix of the main path nodes and the non-main path nodes, and then obtain the global dependency of each node, and then obtain the global embedding of all nodes through full connection.

[0010] A local dependency module (Local Dependency Module) is used to obtain the local mode of each node according to the set time slice window, then the local mode of each node is subjected to a pooling operation to obtain the pooling representation of each node, and finally the hidden vector of the local mode of each node is generated based on the mask attention mechanism.

[0011] A strength function obtaining module is used to parameterize the strength function of the Hox process according to the global embedding and the local mode hidden vector.

[0012] A popularity prediction module is used to combine the integral of the Hox process and the full connection processing result of the global embedding and the local mode hidden vector to generate a popularity prediction result.

[0013] The user embedding module described above, user embedding X is composed of position-wise embedding and temporal encoding, used as the input of the subsequent module (global dependency module).

[0014] The user embedding module includes a Position-wise embedding obtaining submodule and a temporal encoding obtaining submodule.

[0015] 1. The Position-wise embedding obtaining submodule includes a shared matrix U where D is the dimension of user embedding, and L is the maximum number of forwarding. The jth column of U represents the user of the jth forwarding, and each cascade has the same order of nodes in the time axis, and shares an initial embedding. The shared matrix U is randomly initialized first, and then the subsequent context embedding related to the underlying cascade is dynamically updated through the subsequent global dependency module. The shared matrix U greatly reduces the computational overhead, making the system more efficient. In a cascade, for the user u j , let v j represent the one-hot index vector of u j on the shared matrix U, then the position-wise embedding of user u j is obtained by Uv j , where represents the one-hot index set of all user embeddings. Therefore, based on the shared matrix U, the user Position-wise embedding obtained by the Position-wise embedding obtaining submodule is UV.

[0016] 2. The temporal encoding obtaining submodule (Temporal Encoding) is a key component of the present application, which is a two-layer attention module. Since the attention mechanism does not consider sequential dependencies, the temporal encoding obtaining submodule uses temporal encoding z to represent sequence positions, which is defined as follows:

[0017]

[0018] The temporal encoding is a variant of the original Transformer absolute position encoding. Where is the temporal encoding, D is the dimension of the temporal encoding, and [z(t j )] i represents the value of encoding the ith dimension, i∈[1,D]. is the set of temporal encodings.

[0019] Therefore, the initial user embedding of the forwarding sequence can be finally recorded as: X=UV+Z. Therefore, the initial user embedding X is composed of learnable shared position-wise embedding UV and fixed temporal encoding Z.

[0020] The aforementioned global dependency module, referred to as the global module; the global dependency of information cascading can be regarded as an indication that a user may promote or inhibit another user in the long-term diffusion process.

[0021] This global dependency module includes:

[0022] The path-aware attention layer is used to distinguish all predecessor nodes from main path nodes and non-main path nodes for each user node in the user embedding, and to obtain the long-term dependency attention scores of main path nodes and non-main path nodes, and to construct the attention probability matrix of main path and non-main path; then, based on the attention probability matrix of main path and non-main path of each node, its global dependency is obtained.

[0023] The first output network is used to perform fully connected processing on the global dependencies of each node, resulting in the global embedding of all nodes.

[0024] After obtaining the user embedding, a path-aware attention layer was designed in the global module to parameterize the Hawkesformer intensity function, which is used to capture long-term diffusion dependencies between input cascade nodes, and the user complete diffusion dependency extraction submodule is used to extract the temporal topological dependencies coupled between the main path nodes and non-main path nodes.

[0025] Hawkes processes demonstrate that the final popularity of a cascading information chain is related to users' past participation. This means that the current node not only influences its immediate forwarders but also, through transitivity, its non-immediate forwarders. To represent the impact of each node on the future popularity of the cascading chain from a topological perspective, a path-aware hypothesis, namely the main path / non-main path hypothesis, is proposed in the global module: it is assumed that nodes on the main path influence the current node u. j The contribution of nodes on the main path is relatively large, followed by nodes on non-main paths. j The arrival contribution of a node is relatively small, but it still affects the final popularity, which is consistent with the assumptions of the Hawkes process. For the current node u j All predecessor nodes They are divided into two categories: (1) Nodes on the main path, denoted as It includes the previous node u j (1) All previous nodes along the entire forwarding path back to the root node u0; (2) Nodes on non-main paths, i.e., all nodes other than the main path nodes, denoted as Appendix Figure 2 An example is given, where, for the current node u4, ① and ② represent nodes on the main path, and ③ represents nodes on the non-main path.

[0026] Based on the above explanation, the implementation steps of the global dependency module are as follows:

[0027] Given a current node u j , the long-term dependency attention score of u in the main path is obtained:

[0028]

[0029] Where a, b and represent the user embedding matrix X converted by the main path Q (Query) matrix , the non-main path Q (Query) matrix and the K (Key) matrix , respectively. These three conversion matrices are used to distinguish the role of users in long-term dependencies. Where <·,·> represents the inner product function. Similarly, the long-term dependency attention score of u j in the non-main path is:

[0030]

[0031] Then, the attention weighted sum is calculated by the following formula, and the complete diffusion dependency e j of user u j is obtained:

[0032]

[0033] Finally, according to the attention scores of the main path and the non-main path of each node, the context embedding of each node is obtained: E = {e0, e1, …, e j , …, e L}, where each row in E represents the global dependency of a node in the cascade. represents the user embedding vector converted by the V (Value) matrix .

[0034] The above calculations are performed for the dependency of the current single user u j . In actual operation, the above attention dependency is used in a matrix parallel manner to speed up the operation. The matrix calculation process of global dependency can be referred to in the attached Figure 3 .

[0035] First, based on the path-aware attention layer, the attention scores of the current node in the main path and the non-main path are calculated by the following formula:

[0036]

[0037] ​

[0038] Where M P , One-hot matrices are used to retain the attention scores of nodes on the main path and non-main path, respectively. Then element Correspondingly, if Finally, the attention probability matrices for the main path and non-main paths can be derived as follows:

[0039]

[0040] Where D is the dimension of the input embedding. Mask matrix Used to prevent the system from spying on future data, if i <j,M i,j =0, otherwise M i,j =-∞. Matrix M forces the softmax function only when u j Effective long-term dependent attention is allocated to previous forwarders, while subsequent users (i.e., those later than t) receive more attention. j The activated user is assigned 0. Each column vector in matrix A represents u. j For its pioneer node Attention, including the main path Non-main path The nodes on.

[0041] Then, the matrix form of the global dependencies of each node is obtained according to the following formula:

[0042] E = (W V X)A

[0043] To learn more semantic information, the path-aware attention layer can be designed as a module based on a multi-head attention mechanism, mapping the context embedding to different subspaces. These subspaces can be computed in parallel using the matrix multiplication method given earlier, thereby calculating multiple heads E1, E2, ..., E H Then connect them to form E=W O Concat(E1,E2,…,E H ),in It is an aggregation matrix. Finally, we feed E into the first output network (here we are using a fully connected neural network FCNN) to obtain all the cascaded global embeddings:

[0044]

[0045] h(t j )=H(:,j)

[0046] in, are parameters of the neural network; h(t j ) is the j-th column in H u j (t) represents the long-term dependence representation of user u

[0047] The above local dependence module can be simply referred to as a local module; the growth rate of the forwarding quantity in the information cascade is not stable, and the short-term burst in the propagation process has a great influence on the final popularity. In order to better embed the evolution rate of the information cascade, here, the pooling attention is used to capture the local pattern. Based on the user global embedding representation learned in the global dependence module, the pooling operation is performed on the local pattern of each current node.

[0048] The local dependence module comprises:

[0049] A local pattern acquisition submodule is configured to obtain, according to a set time slice window, a part located before each node from the global dependence to obtain a local pattern of each node.

[0050] A pooling submodule is configured to perform a pooling operation on the local pattern of each node to obtain a pooling representation of each node.

[0051] A mask attention layer is configured to dynamically aggregate the local pattern pooling representation of each node and the node before it to generate a hidden vector of the local pattern of each node.

[0052] Each current node has a local pattern, and a fixed time slice window t s is set, and the local pattern acquisition submodule is used to obtain, from the global dependence, a part located before each node to obtain a local pattern of each node. j Therefore, the local pattern of the current node is composed of the global dependence of the current node u represents a set of all local patterns of an information cascade.

[0053] For each current node u j , the pooling submodule is used to perform a pooling operation on the local pattern of the current node to aggregate the features in the time slice t s to generate a new representation Therefore, the total pooling representation of all local patterns can be formalized as: Each column in H u (t) corresponds to the pooling representation of the local pattern.

[0054] In order to capture the evolution rate related to the short-term burst / crash of the cascade, the mask attention layer designed by the application is used to combine the influence of the local pattern. Based on the mask attention mechanism, the mask attention layer is used to dynamically aggregate the current time t jAll previous local pattern embeddings (i.e., pooling representations) generate a hidden vector to summarize the evolution of all previous patterns:

[0055]

[0056] g(·) is a linear transformation function, γ = {v(t0), v(t1), …, v(t j ), …, v(t L )} is the set of local dependence of evolution rate. The similarity function f(·,·) is specified as:

[0057]

[0058] The intensity function acquisition module is used to parameterize the intensity function of the Hawkes process through the global module and the local module to obtain the intensity function λ(t) of the Hawkesformer. Given a message cascade C k and its retweet history within the observation time t o represents the sequence before the time stamp t j but does not include the time t j . Here, the intensity function acquisition module uses the following conditional intensity function to model the continuous dynamics of cascade diffusion:

[0059]

[0060] Where Θ represents the parameters of the model, the retweet time t is defined in the interval [t j , t j+1 ), where f(x) = βlg(1+exp(x / β)) is the softplus function, and the parameter β is used to constrain the intensity function to be positive; The first term current represents the evolution process in the continuous time domain, and α controls the importance of the interpolation node within the two time stamps; w1, w2 are the learning weights of the global dependence and the local pattern at t, respectively, and the intensity function λ(t) represents the tweet arrival rate at t. The global and local dependence of Hawkesformer have expressiveness and flexibility, and the design of the intensity function enables Hawkesformer to capture the coupled spatiotemporal dynamics from the continuous time domain.

[0061] The popularity prediction module is mainly used to predict the popularity of information. According to the Hawkes theory, the survival function at t is: This function represents the conditional probability that no event occurs between [t n , t]. Subsequently, we can derive the probability of observing an event at any time t' as:

[0062] ​

[0063] Finally, the final popularity is predicted by the popularity prediction module, combining the time point process and the two-level attention mechanism module (global module and local module) output, according to the following formula

[0064]

[0065] The first term Lambda represents the integral of the Hawkes process, indicating the expected popularity size; t o represents the observation time; t p represents the prediction time; the second term represents the output result after the splicing result of the global embedding and the local mode hidden vector is input into the second output network (here, a fully connected neural network FCNN is used).

[0066] The information cascade prediction system based on the Transformer enhanced Hawkes process provided by the application further comprises an optimization module for parameter optimization in the system training process.

[0067] A cascade C k forwarding time The joint likelihood can be expressed as:

[0068]

[0069] Where represents the set of times up to time t j .

[0070] Suppose there are N information cascade label samples for training, {C1, C2, …, C N}, the optimization module optimizes by minimizing the mean square log error (MSLE) loss of the popularity and maximizing the log likelihood of all information cascades:

[0071]

[0072] y k represents the true popularity of the kth information cascade label sample, represents the predicted popularity corresponding to the kth information cascade label sample.

[0073] Then the Adam gradient descent algorithm is used to optimize the system network parameters.

[0074] The application further provides an information cascade prediction method based on the Transformer enhanced Hawkes process, which comprises the following steps:

[0075] S1 obtains user position-wise embedding and time encoding, and takes the user position-wise embedding and the time encoding as user embedding;

[0076] S2 distinguishes all predecessor nodes as main path nodes and non-main path nodes for each user node in the user embedding, and obtains attention probability matrices of the main path nodes and the non-main path nodes respectively, and then obtains global dependence of each node, and then obtains global embedding of all nodes through full connection processing;

[0077] S3 obtains local mode of each node according to a set time slice window, then performs a pooling operation on the local mode of each node to obtain a pooling representation of each node, and then generates a local mode hidden vector of each node based on a mask attention mechanism;

[0078] S4 is used to parameterize an intensity function of the Hawkes process according to the global embedding and the local mode hidden vector;

[0079] S5 is used to combine the integral of the Hawkes process and the full connection processing result of the global embedding and the local mode hidden vector to generate a popularity prediction result.

[0080] At this point, the system and method for coupling modeling and prediction of time characteristics and topological characteristics in a continuous time domain by introducing two layers of attention mechanisms into the Hawkes process are realized. In the first layer, the DAG is distinguished according to the main path and the non-main path, and this layer is used to capture the global dependence relationship between nodes, that is, the coupled time and topological dependence; in the second layer, a local pool attention module is proposed to embed the cascading evolution rate to simulate the burst and crash of the short-term forwarding quantity.

[0081] Compared with the prior art, the present application has the following beneficial effects:

[0082] 1、The present application firstly obtains user position-wise embedding and time encoding as user embedding, then introduces path perception hypothesis from the perspective of topological structure, and designs two layers of attention layers from the two angles of global embedding and local mode to parameterize the intensity function of the Hawkes process, and combines the Hawkes process, the global embedding and the local mode to learn the coupled time and topological random characteristics of the information cascade diffusion process, so as to perform popularity prediction; the present application extends the traditional Hawkes process, and effectively obtains knowledge from the continuous time domain, and improves the popularity prediction accuracy.

[0083] 2. The application proposes a two-level attention architecture, which parameterizes the intensity function of the Hox process; first, a global module attention layer based on path perception is designed to capture the long-term diffusion dependence between nodes in the dynamic diffusion topology, and then a local module is proposed to capture the short-term evolution rate in information cascade.

[0084] 3. The application customizes a learnable position-wise user embedding method for information cascade directed acyclic graph, improves parameter efficiency and reduces computational burden.

[0085] 4. The application is extremely important for understanding the evolution process of social network platform and explaining the popular reason of cascade; for example, through the application, the number of reposts of a microblog in a future period of time is predicted, the popular factors are explained, and the downstream tasks such as marketing design, rumor prediction and influence maximization can be used. BRIEF DESCRIPTION OF DRAWINGS

[0086] Figure 1 To present the information cascade diffusion process of directed acyclic graph (DAG). The modeling method of existing models models the time characteristics of node appearance and the topological characteristics of appearance in the graph separately, for example, the time characteristics are sent into the RNN, and the topological characteristics are sent into the GNN. But in essence, the two are coupled to appear, when the forwarding time of a certain node is determined, the topological structure in the DAG is also determined; for example, the forwarding behavior of u2 depends on u0 instead of its previous node u1 in time; therefore, the DAG forms a coupled time-topological dependence, and separate modeling cannot capture the coupled dependence.

[0087] Figure 2 The main path and non-main path assumption in the global dependence module, which is different from the chain structure based on RNN sequence model; the assumption can directly obtain the long-term dependence on the main path (non-main path).

[0088] Figure 3 The implementation details of the main path and non-main path assumption in the global dependence module, that is, the implementation details of Figure 2 .

[0089] Figure 4 The model diagram of the information cascade prediction system of the application based on the enhanced Hox process of the Transformer, including initial user embedding, global dependence module, local dependence module and prediction module.

[0090] Figure 5For visualization of the attention probability matrix A, we plot the heatmaps and cascade structure of three randomly selected cascades in the Weibo dataset. The diagonal line in the heatmap represents the attention of the current node to itself, and each row represents the attention of the current node to all predecessor nodes. Each entry ranges from [0, 1], and the darker the color, the higher the attention. The sum of each row is 1.

[0091] Terminology

[0092] Information Cascade: The term "information cascade" refers to the process of information spreading through social networks. In this process, users receive information from their friends and followers and then forward it to their own friends and followers. This creates a chain of information spreading through the network, hence the term "cascade." Figure 1 To illustrate this process, consider the following example: A root node posts a public content, and the root node's friends and followers see the public content and then forward it one after another. In this way, the public content spreads through the edges of the social network and creates an information cascade. A typical task of an information cascade is to predict the size of the potentially affected users after a certain observation period for a given cascade (tweets, microblogs, etc.). The theoretical basis can be found in the literature

J. Cheng, L. Adamic, P. A. Dow, J. M. Kleinberg, and J. Leskovec. Can cascades be predicted? In Proc. of WWW, 2014.

[0093] Hawkes Process (HP): Hawkes Process is a mainstream temporal point process (TPP). HP has three key factors: (1) user influence, influential users contribute more to the arrival rate of new forwarding, indicating that tweets forwarded by influential users are often forwarded more; (2) self-activation mechanism, each forwarding affects the arrival rate of future new forwarding; (3) time decay effect. The influence of forwarding decays as time increases. However, traditional HP simplifies the arrival rate of events, using, for example, exponential or power-law functions to model the arrival of events, and makes strong assumptions about the diffusion mechanism in cascade events, limiting their learning ability on large-scale cascade data.

[0094] Self-Attention Mechanism: Self-Attention Mechanism is a special case of attention mechanism recently proposed in Transformer

Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems 30 (2017).

[0095] DETAILED DESCRIPTION

[0096] The application will be further described in conjunction with the accompanying drawings.

[0097] Embodiment 1

[0098] For convenience, in the rest of the application, we will take the Twitter cascade as an example setting, but the application is also applicable to other types of information cascades, such as microblogging, academic citation, etc., and we will give examples in the experimental part.

[0099] The information cascade prediction system based on Transformer enhanced Hammersley process provided by the embodiment, as shown in Figure 4 , includes a user embedding module, a global dependence module, a local dependence module, a strength function acquisition module and a popularity prediction module.

[0100] (1) User embedding module

[0101] The user embedding module is used to obtain user position-wise embedding and time coding, and take the user position-wise embedding and time coding as user embedding.

[0102] The user embedding module includes a Position-wise embedding acquisition submodule and a time coding acquisition submodule.

[0103] The Position-wise embedding acquisition submodule includes a shared matrix In a cascade, for the jth retweet user u j , let v j represent its one-hot index vector on the shared matrix U, then the user u jThe position-wise embedding is obtained by Uv j The index set of one-hot is denoted by The one-hot index set representing all user embeddings is denoted by Thus, based on the shared matrix U, the user position-wise embedding obtained by the position-wise embedding obtaining submodule is UV.

[0104] The time encoding obtaining submodule utilizes time encoding z to represent the sequence position, which is defined as follows:

[0105]

[0106] The time encoding is a variant from the original Transformer absolute position encoding. Wherein is the time encoding, D is the dimension of the time encoding, [z(t j )] i represents the value of the i-th dimension, i∈[1, D]. is the set of time encodings.

[0107] Therefore, the user embedding can finally be denoted as: X = UV + Z. So the initial user embedding X is composed of the learnable shared position-wise embedding UV and the fixed time encoding Z.

[0108] The shared matrix U is initialized at the beginning, and its parameters are dynamically updated together with the back propagation of the designed optimization module to obtain the subsequent context embedding related to the underlying cascade.

[0109] (2) Global dependency module

[0110] The global dependency module is used to distinguish all predecessor nodes as main path nodes and non-main path nodes for each user node in the user embedding, and obtain the attention probability matrix of the main path nodes and the non-main path nodes, and then obtain the global dependency of each node, and then obtain the global embedding of all nodes through full connection processing.

[0111] The global dependency module includes a path-aware attention layer and a first output network.

[0112] The path-aware attention layer is used to distinguish all predecessor nodes as main path nodes and non-main path nodes for each user node in the user embedding, and obtain the long-term dependency attention score of the main path nodes and the non-main path nodes, and construct the attention probability matrix of the main path and the non-main path; and then obtain the global dependency of each node according to the attention probability matrix of the main path and the non-main path of each node.

[0113] As Figure 2 and Figure 3As shown, a path-aware assumption, i.e., the main path / non-main path assumption, is proposed in the global module. Given a current node u j , the long-term dependency attention score of u in the main path is obtained as follows:

[0114]

[0115] where a, b and represent the user embedding matrix X transformed by the main path Q (Query) matrix , the non-main path Q (Query) matrix and the K (Key) matrix , respectively. The three transformation matrices are used to distinguish the roles of users in long-term dependencies. Here, <·,·> represents the inner product function. Similarly, the long-term dependency attention score of u j in the non-main path is as follows:

[0116]

[0117] Then, the attention weighted sum is calculated by the following formula, and the complete diffusion dependency e j of user u j is obtained:

[0118]

[0119] Finally, the global dependency of all nodes is constructed according to the complete diffusion dependency of each node. The context embedding of each node is obtained according to the attention score of the main path and the non-main path of each node: E = {e0, e1, …, e j , …, e L}, where each row in E represents the global dependency of a node in the cascade. represents the user embedding matrix X transformed by the V (Value) matrix .

[0120] The above dependency of the current single user u j is calculated. In actual operation, the attention dependency above is used in a matrix parallel manner to speed up the operation.

[0121] Based on the path-aware attention layer, the attention scores of the current node in the main path and the non-main path are calculated by the following formula:

[0122]

[0123] ​

[0124] Where M P , One-hot matrices are used to retain the attention scores of nodes on the main path and non-main path, respectively. Then element Correspondingly, if Finally, the attention probability matrices for the main path and non-main paths can be derived as follows:

[0125]

[0126] Where D is the dimension of the input embedding. Mask matrix Used to prevent the system from spying on future data, if i <j,M i,j =0, otherwise M i,j =-∞. Matrix M forces the softmax function only when u j Effective long-term dependent attention is allocated to previous forwarders, while subsequent users (i.e., those later than t) receive more attention. j The activated user is assigned 0. Each column vector in matrix A represents u. j For its pioneer node Attention, including the main path Nodes on non-main paths

[0127] Then, the matrix form of the global dependencies of each node was obtained according to the following formula:

[0128] E = (W V X)A

[0129] To learn more semantic information, the path-aware attention layer is designed as a module based on a multi-head attention mechanism. Context embeddings are mapped to different subspaces, which can be computed in parallel using the matrix multiplication method described earlier, thereby calculating multiple heads E1, E2, ..., E H Then connect them to form E=W O Concat(E1,E2,…,E H ),in It is an aggregation matrix. The first output network is used to perform fully connected processing on the global dependencies of all nodes to obtain the global embeddings of all nodes. In this embodiment, a fully connected neural network FCNN is used as the first output network. The global dependencies E of all nodes are fed into the first output network to obtain the global embeddings of all nodes:

[0130]

[0131] h(t j )=H(:,j)

[0132] wherein, are parameters of the neural network; h(t j is the j-th column of H(t j , representing the long-term dependency representation of user u

[0133] (3) Local dependency module

[0134] The local dependency module is configured to obtain a local pattern of each node according to a set time slice window, and then perform a pooling operation on the local pattern of each node to obtain a pooling representation of each node; and then generate a hidden vector of the local pattern of each node based on a mask attention mechanism.

[0135] The local dependency module comprises a local pattern obtaining submodule, a pooling submodule, and a mask attention layer.

[0136] The local pattern obtaining submodule is configured to obtain a local pattern of each node by intercepting a part before each node from the global dependency according to a set time slice window. A fixed time slice window t s is set, and the local pattern obtaining submodule is configured to obtain a local pattern of each node by intercepting a part before each node from the global dependency; therefore, the local pattern of the current node is composed of the global dependency of the current node u j and part of the predecessor nodes, and is expressed as represents a set of all local patterns of an information cascade.

[0137] The pooling submodule is configured to perform a pooling operation on the local pattern of each node to obtain a pooling representation of each node. For each current node u j , the pooling submodule is configured to perform a pooling operation on the local pattern of the current node to aggregate features in the time slice t s to generate a new representation form Therefore, the total pooling representation of all local patterns can be formalized as: Each column in H(t corresponds to the pooling representation of the local pattern.

[0138] The mask attention layer is configured to dynamically aggregate the local pattern pooling representation of each node and the node before it to generate a hidden vector of the local pattern of each node. Based on the mask attention mechanism, the mask attention layer is configured to dynamically aggregate all local pattern embeddings (i.e., the pooling representation) before the current time t j to generate a hidden vector to summarize the evolution process of all previous patterns:

[0139]

[0140] g(·) is a linear transformation function, Y = {v(t 01 ), v(t1), …, v(tj), …, v(t L )} is a set of local dependence of evolution rate. The similarity function f(·,·) is specified as:

[0141]

[0142] (4) Intensity function obtaining module

[0143] The intensity function obtaining module is used to parameterize the intensity function of the traditional Hox process according to the global embedding and the local pattern hidden vector. Here, the intensity function obtaining module is used to model the continuous dynamics of cascading diffusion using the following conditional intensity function:

[0144]

[0145] where Θ represents the parameters of the model, the forwarding time t is defined in the interval [t j , t j+1 ), where f(x) = βlg(1+exp(x / β)) is the softplus function, and the parameter β is used to constrain the intensity function to be positive; the first term current represents the evolution process in the continuous time domain, and α controls the importance of the interpolation nodes within the two time stamps; w1 and w2 are the learning weights of the global dependence and the local pattern at time t, respectively, and λ(t) represents the tweet arrival rate at time t.

[0146] (5) Popularity prediction module

[0147] The popularity prediction module is used to combine the integral of the Hox process and the full connection processing result of the global embedding and the local pattern hidden vector to generate the popularity prediction result. Here, the final popularity is predicted according to the following formula

[0148]

[0149] The first term Λ represents the integral of the Hox process, which represents the expected popularity size; t o represents the observation time; t p represents the prediction time; and the second term represents the output result of the splicing result of the global embedding and the local pattern hidden vector after the second output network (here, a fully connected neural network FCNN is used).

[0150] Embodiment 2

[0151] This embodiment is a further improvement based on embodiment 1.

[0152] The information cascade prediction system based on the Transformer enhanced Hawkes process provided in the embodiment further comprises an optimization module for parameter optimization in the system training process.

[0153] Suppose there are N information cascade label samples for training, {C1, C2, …, C N The optimization module optimizes by minimizing the mean square log error (MSLE) loss of popularity and maximizing the log likelihood of all information cascades:

[0154]

[0155] y k y represents the true popularity of the kth information cascade label sample, y represents the predicted popularity corresponding to the kth information cascade label sample.

[0156] Then, the Adam gradient descent algorithm is used to optimize the system network parameters.

[0157] Embodiment 3

[0158] The embodiment provides an information cascade prediction method based on a Transformer enhanced Hawkes process, which is performed according to the following steps:

[0159] S1 obtains user position-wise embedding and time encoding, and takes the user position-wise embedding and time encoding as user embedding.

[0160] In this step, the user embedding module is used to obtain shared position-wise embedding UV and fixed time encoding Z, respectively, to obtain the final user embedding X = UV + Z.

[0161] S2 distinguishes all predecessor nodes as main path nodes and non-main path nodes for each user node in the user embedding, respectively obtains the long-term dependency attention score of the main path nodes and the non-main path nodes, and then obtains the global dependency of each node, and then obtains the global embedding of all nodes through full connection processing.

[0162] Here, first, the path-aware attention layer is used to distinguish all predecessor nodes as main path nodes and non-main path nodes for each user node in the user embedding, and obtain the attention probability matrix of the main path nodes and the non-main path nodes; then, the global dependency of all nodes is constructed according to the attention probability matrix of the main path and the non-main path of each node; and then, the global embedding of all nodes is obtained through full connection processing of the first output network.

[0163] S3 obtains the local pattern of each node according to the set time slice window, and then performs a pooling operation on the local pattern of each node to obtain the pooling representation of each node; then, based on the mask attention mechanism, a local pattern hidden vector of each node is generated.

[0164] Here, first, the local pattern acquisition submodule obtains the local pattern of each node by intercepting the part before each node from the global dependence according to the set time slice window; then, the pooling submodule performs a pooling operation on the local pattern of each node to obtain the pooling representation of each node; then, the mask attention layer is used to dynamically aggregate the local pattern pooling representation of each node and the node before it to generate a hidden vector of the local pattern of each node.

[0165] S4 is used to parameterize the intensity function of the Hawkes process according to the global embedding and the local pattern hidden vector.

[0166] Here, the intensity function acquisition module is used to parameterize the intensity function of the traditional Hawkes process according to the global embedding and the local pattern hidden vector to obtain the intensity function of the Hawkesformer model.

[0167] S5 is used to combine the integral of the Hawkes process and the full connection processing result of the global embedding and the local pattern hidden vector to generate a popularity prediction result.

[0168] Here, the popularity prediction module combines the integral of the Hawkes process and the full connection processing result of the global embedding and the local pattern hidden vector to generate a popularity prediction result.

[0169] Application example

[0170] This application example uses three different real data sets (Twitter, Weibo, and APS, the first data set is from reference

Lilian Weng, Filippo Menczer, and Yong-Yeol Ahn. 2013. Virality prediction and community structure in social networks. Scientific Reports 3 (2013)

Qi Cao, Huawei Shen, Keting Cen, Wentao Ouyang, and Xueqi Cheng. 2017. Deep-Hawkes: Bridging the gap between prediction and understanding of information cascades. In CIKM. 1149-1158

https: / / journals.aps.org / datasets

[0171] The three data sets are divided into training set, validation set and test set according to the proportion of 70%, 15% and 15% respectively.

[0172] Firstly, the training set is used to train and optimize the information cascade prediction system based on the Transformer enhanced Hawkes process (Hawkesformer) provided in embodiment 2. After each training, the validation set is used to verify the system obtained by training until the loss function of the system tends to be stable.

[0173] Then, the trained Hawkesformer is tested on the test set according to the information cascade prediction method based on the Transformer enhanced Hawkes process provided in embodiment 3, and compared with 7 different baseline models (Feature-Deep, DeepHawkes, CasCN, FOREST, VaCas, TempCas, Hawkesformer), and the commonly used measurement method MSLE (the smaller the value, the better the prediction effect) is adopted.

[0174] Table 1: Performance comparison of Hawkesformer and baseline models on three data sets

[0175]

[0176] ​Note: Bold values indicate better performance than other settings. All experimental results were obtained best in 10 trials, statistical significance p < 0.05.

[0177] The 7 baseline methods in Table 1 are introduced as follows:

[0178] Feature-Deep: Extract structural features, temporal features, and user features, and input them into a two-layer MLP.

Xu, Xovee, et al. "CasFlow: Exploring hierarchical structures and propagation uncertainty for cascade prediction." IEEE Transactions on Knowledge and Data Engineering (2021).

[0179] DeepHawkes: Combines deep learning and Hawkes' self-exciting point process, bridging the gap between prediction performance and interpretability.

Qi Cao, Huawei Shen, Keting Cen, Wentao Ouyang, and Xueqi Cheng. 2017. Deep-Hawkes: Bridging the gap between prediction and understanding of information cascades. In CIKM. 1149-1158.

[0180] CasCN: A hybrid model that stacks GNNs to obtain node embeddings and uses RNN to simulate the evolution of diffusion.

Chen, Xueqin, et al. "Information diffusion prediction via recurrent cascades convolution." 2019 IEEE 35th International Conference on Data Engineering (ICDE). IEEE, 2019.

[0181] FOREST: Combines reinforcement learning and RNN to handle multi-scale cascade prediction problems.

Yang, Cheng, et al. "Multi-scale Information Diffusion Prediction with Reinforced Recurrent Networks." IJCAI. 2019.

[0182] VaCas: A hierarchical diffusion model that integrates information cascade and spatio-temporal structure features, while also capturing diffusion uncertainty.

Fan Zhou, Xovee Xu, Kunpeng Zhang, Goce Trajcevski, and Ting Zhong. 2020. Variational information diffusion for probabilistic cascades prediction. In INFOCOM. 1618-1627.

[0183] TempCas: A heuristic method is used to embed the cascade graph using RNN, and LSTM is used on a specially designed attention-based CNN to learn the historical short-term trend.

Tang, Xiangyun, et al. "Fully exploiting cascade graphs for real-time forwarding prediction." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 1. 2021.

[0184] From the experimental results in Table 1, it can be seen that the information cascade prediction system based on the Transformer enhanced Hawkes process proposed in the present application can greatly improve the accuracy of popularity prediction compared with other baseline models.

[0185] In addition, in order to study the effectiveness of the global dependency module provided by the present application in capturing the long-term diffusion dependency relationship between nodes in the dynamic diffusion topology, the application further obtains an attention probability matrix A, as shown in Figure 5 The following is analyzed step by step:

[0186] (1) For the smaller (13 users) information cascade C1, the information cascade prediction system provided by the present application pays more attention to the main path and the early users participating in diffusion. For each row of matrix A, we can see that the users on the main path usually have the most striking color, which verifies our main / non-main hypothesis in the information propagation process (similar phenomena are also shown in cascades C2 and C3). For node 9 in cascade C1, Hawkesformer detects that 0→2→9 is the main path, indicating that nodes 0 and 2 have a greater impact on node 9, while other nodes on the non-main path have a smaller impact on the forwarding behavior of node 9. For node 8, Hawkesformer detects that 0→4→8 is the main path, indicating that nodes 0 and 4 have a greater impact on node 8, while other nodes on the non-main path have a smaller impact on the forwarding behavior of node 8.

[0187] (2) For the medium (58 users) information cascade C2, it is observed that there is a long-term dependence between the root user and the subsequent nodes. This indicates the need to model the long-term dependence between nodes in the dynamic diffusion topology, because sequence models such as RNN are difficult to train on long sequences and face the problem of gradient vanishing / explosion when the sequence is too long. In addition, several short-term outbreaks are also observed, which again verifies the assumption of the present application that the information cascade learns local patterns.

[0188] (3) At the same time, a virus propagation cascade C3 is shown, which has 106 users and a maximum depth of 8. Most of the retweets (84%) in this cascade are from indirect users to the root. We found that even the "tail" users at the end of the diffusion tree can trigger an outbreak (for example, users 19 and 43). Some users act as pivots connecting other influential users, bringing them into the information propagation process.

[0189] Therefore, the information cascade prediction system based on the Transformer enhanced Hawkes process proposed by the present application not only improves the accuracy of the popularity prediction, but also has strong explainability. The experiments on three real data sets prove the superior performance of the present application relative to the most advanced baseline model. The performance improvement of the proposed scheme shows that combining the advantages of deep learning (Transformer) and probabilistic models (Hawkes process) is a new direction to effectively simulate the information cascade diffusion process and improve the prediction accuracy.

[0190] In summary, the modeling method of the information propagation process is crucial to understanding the information propagation mechanism and the popularity prediction task, and the evolution of information diffusion presents a Directed Acyclic Graph (DAG), which is essentially coupled with topology features and time. However, the existing research adopts a separate modeling method for the two most important features of the cascade (topology features and time features), thus losing cross-domain information and reducing the expression and prediction ability of the model. The present application links the multi-layer attention mechanism to the Hawkes self-excitation point process to achieve more effective and reasonable modeling of the information cascade, so as to achieve more accurate popularity prediction. Specifically, a two-layer attention structure is used to parameterize the intensity function of the Hawkes process, which has realized effective representation knowledge from the continuous time domain.

[0191] Those skilled in the art will appreciate that the embodiments described herein are presented for purposes of illustration and that the inventive principles are not limited to these particular embodiments. Other variations and modifications can be made to the embodiments without departing from the spirit and scope of the inventive principles.

Claims

1. An information cascade prediction system based on Transformer enhanced Hawkes process, characterized in that, Comprise: A user embedding module for obtaining user position-wise embedding and time encoding, and taking the user position-wise embedding and time encoding as user embedding; A global dependency module for distinguishing all predecessor nodes as main path nodes and non-main path nodes for each user node in the user embedding, and obtaining attention probability matrix of the main path nodes and the non-main path nodes, and then obtaining global dependency of each node, and then obtaining global embedding of all nodes through full connection; A local dependency module for obtaining local mode of each node according to a set time slice window, then performing pooling operation on the local mode of each node to obtain pooling representation of each node, and finally generating hidden vector of the local mode of each node based on a mask attention mechanism; An intensity function obtaining module for parameterizing an intensity function of a Hawkes process according to the global embedding and the hidden vector of the local mode; A popularity prediction module for generating a popularity prediction result by combining integral of the Hawkes process and full connection processing result of the global embedding and the hidden vector of the local mode.

2. The Transformer-enhanced Hawkes process-based information cascade prediction system according to claim 1, wherein, The user embedding module comprises a Position-wise embedding obtaining submodule and a time encoding obtaining submodule; The Position-wise embedding acquisition submodule includes a shared matrix U, and for the jth forwarded user u j , let v j represent its one-hot index vector on the shared matrix U, then the position-wise embedding of the user u j is obtained by Uv j , where v represents a one-hot index set of all user embeddings; the Position-wise embedding obtained by the Position-wise embedding acquisition submodule is UV; The time encoding obtaining submodule uses time encoding z to represent sequence position, and is defined as follows: wherein is a time encoding, D is the dimension of the time encoding, [z(t j )] i represents the value encoding the i-th dimension, is a set of time encodings; The user embedding is finally denoted as: X=UV+Z. 3.The information cascade prediction system based on Transformer enhanced Hawkes process of claim 1, wherein, The global dependency module comprises: A path-aware attention layer for distinguishing all predecessor nodes as main path nodes and non-main path nodes for each user node in the user embedding, and obtaining long-term dependency attention scores of the main path nodes and the non-main path nodes, and constructing attention probability matrix of the main path and the non-main path; and then obtaining global dependency of each node according to the attention probability matrix of the main path and the non-main path of each node; A first output network for performing full connection processing on the global dependency of each node to obtain global embedding of all nodes.

4. The Transformer-enhanced Hoax process-based information cascade prediction system according to claim 3, wherein, Firstly, the path-aware attention layer is used to calculate the attention scores of the current node on the main path and the non-main path respectively by the following formula: P score = (W P X) T (W K X)⊙M P N score = (W N X) T (W K X)⊙M N where M P , one-hot matrices used to preserve the attention scores of the nodes on the main path and non-main path, respectively, if denote the set of main path nodes, then the element correspondingly, if denote the set of non-main path nodes, Finally, the attention probability matrices of the main path and non-main path are derived as: where D is the dimension of the input embedding; mask matrix used to prevent the system from peeking into future data, if i < j, M i,j = 0, otherwise M i,j = -∞; each column vector in matrix A represents u j attention to its predecessor nodes including nodes on the main path and non-main path Then, the matrix form of global dependency of each node is obtained by the user global dependency extraction submodule: E = (W V X)A; The global embedding of the nodes in all cascades is obtained by performing full connection on E through the first output network: h(t j ) = H(:,j) where, are parameters of the neural network; h(t j ) is the j-th column of H j , representing the long-term dependency representation of user u 5. The Transformer-enhanced Hoax process-based information cascade prediction system according to any one of claims 1 to 4, characterized in that, The local dependency module comprises: A local mode obtaining submodule for obtaining local mode of each node by intercepting part before each node from the global dependency according to a set time slice window; A pooling submodule for performing pooling operation on the local mode of each node to obtain pooling representation of each node; A mask attention layer for dynamically aggregating the local mode pooling representation of each node and the nodes before it to generate hidden vector of the local mode of each node. 6.The information cascade prediction system based on Transformer enhanced Hawkes process of claim 5, wherein, Dynamically aggregate the current time t with the mask attention layer j All previous local pattern embeddings generate a hidden vector to summarize the evolution of all previous patterns: g(·) is a linear transformation function, γ = {v(t0), v(t1),..., v(t j ),..., v(t L )} is a set of local dependence of evolution rate; a similarity function f(·, ·) is specified as:

7. The Transformer-enhanced Hox process-based information cascade prediction system according to claim 6, wherein, The intensity function obtaining module uses the following conditional intensity function to model continuous dynamics of cascade diffusion: where Θ represents the parameters of the model, the retransmission time t is defined in the interval [t j ,t j+1 ), where f(x) = βlg(l + exp(x / β)) is the softplus function, and the parameter β is used to constrain the intensity function to be positive; the first term current represents the evolution process in continuous time domain, and α controls the importance of the interpolation nodes within the two timestamps; w1, w2 are the learning weights of the global dependence and local pattern at time t, respectively, and λ(t) represents the tweet arrival rate at time t.

8. The information cascade prediction system based on Transformer enhanced Hawkes process according to claim 7, characterized in that, The popularity prediction module predicts the final popularity according to the following formula The first term Λ represents the integral of the Hawkes process, representing the expected prevalence size; t o represents the observation time; t p denotes the prediction time; the second term denotes the output result of the concatenation result of the global embedding and the local mode hidden vector after the second output network. 9.The information cascade prediction system based on Transformer enhanced Hawkes process of claim 8, wherein, Further comprising an optimization module for parameter optimization in the system training process; the optimization module optimizes by minimizing mean square logarithmic error loss of popularity and maximizing logarithmic likelihood of all information cascades: y k represents the true prevalence of the kth information cascade label sample, represents the predicted prevalence of the kth information cascade label sample, l(C k ) represents the joint likelihood of a cascade C k , and N represents the number of information cascade label samples.

10. A method for predicting information cascade based on a Transformer enhanced Hawkes process, comprising the following steps: S1 obtains user position-wise embedding and time encoding, and takes the user position-wise embedding and time encoding as user embedding; S2 distinguishes all predecessor nodes as main path nodes and non-main path nodes for each user node in the user embedding, and respectively obtains the long-term dependent attention score of the main path nodes and the non-main path nodes, and then obtains the global dependence of each node, and then obtains the global embedding of all nodes through full connection processing; S3 obtains the local mode of each node according to the set time slice window, then performs pooling operation on the local mode of each node to obtain the pooling representation of each node, and then generates the local mode hidden vector of each node based on the mask attention mechanism; S4 is used to parameterize the intensity function of the Hawkes process according to the global embedding and the local mode hidden vector; S5 is used to combine the integral of the Hawkes process and the full connection processing result of the global embedding and the local mode hidden vector to generate the popularity prediction result.

Images