User prediction method and device based on discrete time dynamic graph

Abstract

The embodiment of the invention provides a user prediction method and device based on a discrete time dynamic graph. The discrete time dynamic graph comprises a plurality of graph snapshots arranged according to time before the target moment. The method comprises the following steps: sequentially obtaining a first neighbor node in which a target user node exists from a plurality of graph snapshots, and forming a first neighbor sequence according to the time sequence of the graph snapshots; forming a plurality of first feature sequences according to related features of each neighbor node in the first neighbor sequence; processing a first input sequence formed based on the plurality of first feature sequences by using a first neural network based on a self-attention mechanism to obtain a user representation of a target user corresponding to the target user node; and predicting the user behavior of the target user at the target moment at least according to the user representation.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of machine learning, and in particular, to a method and device for user prediction based on discrete-time dynamic graphs using machine learning. Background Art

[0002] Dynamic graphs have been widely used in many fields such as fraud detection, recommendation systems, and e-commerce because of their ability to display entities that change over time and the ever-changing interactions between entities.

[0003] Dynamic graphs are divided into discrete-time dynamic graphs (DTDG) and continuous-time dynamic graphs (CTDG). DTDG is superior to CTDG in terms of simplifying data processing and analysis. Therefore, the user behavior of target users in specific business scenarios can be predicted based on DTDG.

[0004] How to design an effective user prediction solution based on discrete-time dynamic graphs is a technical problem that needs to be solved urgently. Summary of the invention

[0005] One or more embodiments of the present specification describe a user prediction scheme based on a discrete-time dynamic graph, which can improve the learning effect of the user representation by utilizing a first neural network based on a self-attention mechanism to learn the user representation of the target user corresponding to the target user node, thereby improving the accuracy of downstream prediction tasks related to the target user.

[0006] According to a first aspect, a user prediction method based on a discrete time dynamic graph is provided, wherein the discrete time dynamic graph includes a plurality of graph snapshots arranged in time before a target moment, and the method includes:

[0007] Sequentially obtain the first neighbor node of the target user node from the multiple graph snapshots, and form a first neighbor sequence according to the time sequence of the graph snapshots;

[0008] According to relevant features of each neighbor node in the first neighbor sequence, forming a plurality of first feature sequences;

[0009] Processing a first input sequence formed based on the plurality of first feature sequences using a first neural network based on a self-attention mechanism to obtain a user representation of a target user corresponding to the target user node;

[0010] A user behavior of the target user at a target time is predicted based at least on the user representation.

[0011] In some embodiments, a plurality of first feature sequences are formed according to relevant features of each neighbor node in the first neighbor sequence, specifically including: forming a neighbor feature sequence according to node features of each neighbor node itself and classifying the plurality of first feature sequences.

[0012] In some embodiments, several first feature sequences are formed based on relevant features of each neighbor node in the first neighbor sequence, specifically including: forming an edge feature sequence based on edge features of the connecting edges between each neighbor node and the target user node, and incorporating the edge feature sequences into the several first feature sequences.

[0013] In some embodiments, several first feature sequences are formed based on the relevant features of each neighbor node in the first neighbor sequence, specifically including: encoding the sequential index of the graph snapshot where each neighbor node is located in the multiple graph snapshots to obtain the corresponding position code, and the position code of each neighbor node forms a position feature sequence, which is included in the several first feature sequences.

[0014] In some embodiments, several first feature sequences are formed based on relevant features of each neighbor node in the first neighbor sequence, specifically including: for any neighbor node, the appearance features of the neighbor node are determined according to the number of interactions of the neighbor node with the target user node in the multiple graph snapshots, and the appearance features of each neighbor node form an appearance feature sequence, which is included in the several first feature sequences.

[0015] In some embodiments, the method further includes: obtaining second neighbor nodes of the target object node from the multiple graph snapshots in sequence, and forming a second neighbor sequence in the time order of the graph snapshots; forming a number of second feature sequences based on relevant features of each neighbor node in the second neighbor sequence; using the first neural network to process a second input sequence formed based on the number of second feature sequences to obtain a corresponding object representation; predicting the user behavior of the target user at the target time includes: determining a prediction result of whether the target user and the target object interact at the target time based on the user representation and the object representation.

[0016] In a further embodiment, the target object is one of the following: a commodity, a content object, another user different from the target user.

[0017] In a further embodiment, a plurality of first feature sequences are formed according to relevant features of each neighbor node in the first neighbor sequence, specifically including: for any neighbor node in the first neighbor sequence, according to the number of interactions of the neighbor node with the target user node and the number of interactions with the target object in the multiple graph snapshots, determining the intersection matrix of the neighbor node, encoding the intersection matrix into an intersection feature; the intersection features of each neighbor node form an intersection feature sequence, which is included in the plurality of first feature sequences.

[0018] In a further embodiment, the method further includes: determining a behavior label between the target user and the target object based on a graph snapshot at a target moment; determining a prediction loss based on the behavior label and the prediction result; and training the first neural network based on the prediction loss.

[0019] In some embodiments, the method further includes: performing a recombination operation corresponding to M different sizes on the plurality of first feature sequences, wherein the recombination operation corresponding to the first size includes dividing the first feature sequence into blocks of a first size, using the blocks as sequence elements to form a recombined sequence corresponding to the first size; and forming a single first input sequence from a plurality of recombined sequences of the same size, thereby obtaining M first input sequences corresponding to M different sizes.

[0020] In a further embodiment, the method of using a first neural network based on a self-attention mechanism to process a first input sequence formed based on the plurality of first feature sequences includes: using M first neural networks to process the M first input sequences respectively to obtain M embedded representations; and using a multilayer perceptron to process a combination vector of the M embedded representations to obtain the user representation.

[0021] According to a second aspect, a user prediction device based on a discrete time dynamic graph is provided, wherein the discrete time dynamic graph includes a plurality of graph snapshots arranged in time before a target moment, and the device includes:

[0022] A first forming unit is configured to sequentially obtain a first neighbor node of a target user node from the plurality of graph snapshots, and form a first neighbor sequence according to a time sequence of the graph snapshots;

[0023] a second forming unit, configured to form a plurality of first feature sequences according to relevant features of each neighbor node in the first neighbor sequence;

[0024] A processing unit is configured to process a first input sequence formed based on the plurality of first feature sequences using a first neural network based on a self-attention mechanism to obtain a user representation of a target user corresponding to the target user node;

[0025] The prediction unit is configured to predict the user behavior of the target user at a target time at least based on the user representation.

[0026] According to a third aspect, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0027] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect.

[0028] According to a fifth aspect, a computing device is provided, comprising a memory and a processor, wherein an executable code is stored in the memory, and when the processor executes the executable code, the method of the first aspect is implemented.

[0029] In the embodiments of this specification, the GNN+RNN architecture is abandoned and the first neural network (such as Transformer encoder) based on the self-attention mechanism is used to learn the user representation of the target user corresponding to the target user node, thereby solving the defects of the GNN+RNN architecture such as over-smoothing problems, long-term dependency problems, and unsuitability for large-scale data. On the basis of this technical concept, in the embodiments of this specification, a first neighbor sequence of the target user node is formed according to the time sequence of the graph snapshot, and a number of first feature sequences are formed according to the relevant features of each neighbor node in the first neighbor sequence. Based on the first feature sequences, a first input sequence suitable for processing by the first neural network is formed, which can improve the node representation learning effect of the target user node, thereby improving the accuracy of the downstream prediction task. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0031] Figure 1 A schematic diagram of the technical concept of performing edge prediction tasks based on discrete-time dynamic graphs is shown.

[0032] Figure 2 A flowchart of a user prediction method based on a discrete-time dynamic graph according to one embodiment is shown.

[0033] Figure 3 A schematic diagram of constructing feature sequences for node i and node j respectively is shown.

[0034] Figure 4 A schematic diagram showing a multi-size recombination operation performed on a feature sequence of a node i and a feature sequence of a node j is shown.

[0035] Figure 5 A schematic structural diagram of a user prediction device according to an embodiment is shown. DETAILED DESCRIPTION

[0036] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0037] In the description of the embodiments of the present application, words such as "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a concrete way.

[0038] In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the term "plurality" means two or more.

[0039] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprises", "has" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0040] The solution provided in this specification is used to predict the user behavior of a target user at a target time based on a discrete-time dynamic graph.

[0041] The discrete time dynamic graph includes multiple graph snapshots arranged in time before the target time. The discrete time dynamic graph can be denoted by G. G={G1,G2,…,G T-1}, represents the graph snapshots arranged in chronological order before the target time T (such as up to time T-1). Each graph snapshot G in G t It can be regarded as a static graph structure data. t =(V t ,E t ), 1≤t≤T-1. V t Represents G tThe set of nodes in E t Represents G t The set of edges in .

[0042] The nodes in the graph snapshot may include target user nodes and other object nodes. Objects may be, but are not limited to, commodities, content objects (such as transaction records, texts, pictures, videos, etc.), or another user different from the target user. The target user node may be regarded as a special object node. If there is a connecting edge between two object nodes in a graph snapshot, it means that there is a certain association relationship (such as purchase, attention, and other interactive relationships) between the two object nodes at the specific moment corresponding to the graph snapshot. The connecting edges between different object nodes in the same graph snapshot may indicate the same or different association relationships. There may be zero, one, or more connecting edges between any two object nodes in a graph snapshot. Different connecting edges between two object nodes may represent different association relationships.

[0043] The process of predicting the user behavior of the target user at the target time based on the discrete-time dynamic graph can be roughly summarized as learning the node representation of the target user node based on the discrete-time dynamic graph, and predicting the user behavior of the target user at the target time based on at least the learned node representation (including learning the node representation of the target object node in the edge prediction scenario).

[0044] User behavior prediction includes node prediction and edge prediction. Node prediction refers to the node representation of the target user node learned based on the discrete-time dynamic graph, predicting the user behavior of the target user at the target time. For example, predicting the probability of the target user performing a specific behavior at the target time, predicting the user type to which the target user belongs at the target time, and so on. Edge prediction refers to the node representation of the target user node and the node representation of the target object node learned based on the discrete-time dynamic graph, predicting the probability of the target user interacting with the target object at the target time, that is, predicting the possibility of the target user node having a connection edge with the target object node at the target time.

[0045] The existing technology mainly uses the GNN (Graph Neural Network) + RNN (Recurrent Neural Network) architecture to learn the node representation of object nodes in discrete-time dynamic graphs.

[0046] The GNN+RNN architecture inherits the inherent limitations of both GNN and RNN. GNN has an over-smoothing problem as the depth of the model architecture increases, which reduces the effect of node representation learning. RNN has a long-term dependency problem and is difficult to capture long-distance dependencies in the sequence, resulting in performance degradation, which also affects the effect of node representation learning.

[0047] In addition, discrete-time dynamic graphs are usually large in size. When using the GNN+RNN architecture to learn node representations, storing large-sized discrete-time dynamic graphs in GPU memory may cause out-of-memory (OOM) problems. As the number of graph snapshots increases, RNN needs to over-process long sequences, which will further increase the demand for computing resources. Therefore, the GNN+RNN architecture is difficult to adapt to large-sized graph data and long sequences.

[0048] In addition, the existing technology usually calculates node representations independently when performing edge prediction tasks, and only focuses on the characteristics of a single node, while ignoring the intersection characteristics between the two object nodes involved in the edge prediction task (such as two nodes appearing in the same context or having the same neighbor nodes), which in turn affects the accuracy of the edge prediction task.

[0049] In view of this, the embodiment of this specification proposes a node representation learning method of the first neural network based on the self-attention mechanism (Self-Attention) to solve the many defects of the GNN+RNN architecture, improve the learning effect of node representation, and thereby improve the accuracy of downstream prediction tasks.

[0050] The first neural network refers to a neural network that can process sequence data using a self-attention mechanism. The first neural network may specifically be, but is not limited to, a Transformer encoder, BERT, and its variants (such as RoBERTa, ERNIE, etc.).

[0051] Figure 1 A schematic diagram of the technical concept of performing edge prediction tasks based on discrete-time dynamic graphs is shown.

[0052] Figure 1 The edge prediction task shown is to learn the node representations of node i and node j respectively based on the graph snapshots at time t1, t2, and t3 before time t4, and then predict the possibility of the existence of a connecting edge between node i and node j in the graph snapshot at time t4 based on the node representations of node i and node j. The graph snapshot can be an interaction graph containing user nodes. When both node i and node j are user nodes, the existence of a connecting edge between them means that the two users have a specific type of association or interaction at time t4. When one of node i and node j is a user node and the other is other objects (commodities, content, etc.), the existence of a connecting edge between them means that the user has a specific type of interaction with the object at time t4, such as purchase, collection, add to cart, etc.

[0053] like Figure 1As shown in the figure, first, the neighbor nodes of node i and node j can be obtained from the three graph snapshots in turn to obtain the neighbor sequence of node i and the neighbor sequence of node j. Then, for each neighbor node in the neighbor sequence of node i and node j, features can be designed and encoded to obtain the feature sequence of node i and the feature sequence of node j. The length of the feature sequence is equal to the length of the neighbor sequence.

[0054] Exemplarily, for each neighbor node in the neighbor sequence of node i or node j, one or more of the following features can be constructed: node features, edge features, position features that can reflect the time order of the graph snapshot where the neighbor node is located in multiple graph snapshots, appearance features that can reflect the number of interactions between the neighbor node and node i or node j in multiple graph snapshots, and intersection features that can reflect the number of interactions between the neighbor node and node i and node j in multiple graph snapshots. In this way, five different feature sequences can be constructed for node i and node j. These five feature sequences can be stacked into one feature sequence according to the nodes. The stacking operation can occur before the multi-size reorganization operation or after the multi-size reorganization operation.

[0055] After obtaining the feature sequence of node i and the feature sequence of node j, the feature sequence of node i and the feature sequence of node j can be reorganized in multiple sizes to obtain block sequences of node i and node j at different sizes. Next, the corresponding block sequences of different sizes of node i and node j can be input into the Transformer encoder to obtain the embedded representations of node i and node j at different granularities. Then, the embedded representations of node i and node j at different granularities can be processed (such as splicing and projection) to obtain the final embedded representations of node i and node j. Finally, based on the final embedded representations of node i and node j, a prediction can be made to obtain the prediction result of whether there is a connecting edge between node i and node j at time t4.

[0056] By abandoning the traditional GNN+RNN architecture and instead using the first neural network based on the self-attention mechanism to learn the node representation of object nodes (such as target user nodes, target object nodes), the defects of the GNN+RNN architecture can be avoided.

[0057] The self-attention mechanism enables the model to assign different attention weights to each element in the sequence when processing the sequence. This mechanism allows the model to consider not only the current element but also other elements in the sequence when making predictions, thereby capturing the complex relationships between elements, helping the model to efficiently focus on more relevant elements, helping to alleviate the problem of over-smoothing and improve the learning effect of node representation. In addition, compared with the GNN+RNN architecture, the first neural network based on the self-attention mechanism is more suitable for processing large-scale data including a large number of nodes, edges, and a large time span.

[0058] Furthermore, the introduction of multi-size reorganization operations can obtain a variety of reorganized sequences that can display the dependencies between sequence elements at different sizes, so that the model can effectively capture information at different granularities and manage the input sequence length, so that it can reduce the memory space occupied while efficiently processing large-size data, thereby avoiding the problem of insufficient memory.

[0059] In addition, by designing a variety of feature sequences, the node representation learning effect can be improved, thereby improving the accuracy of the prediction results based on the final learned node representation to predict the target user behavior.

[0060] The specific implementation process of the above technical concept is described below.

[0061] Figure 2 A flowchart of a user prediction method based on a discrete time dynamic graph according to an embodiment is shown. It is understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 2 The model architecture corresponding to the technical concept will be used in both the training and use stages. Figure 2 The forward inference process shown makes predictions.

[0062] like Figure 2 As shown, in step S21, the first neighbor node of the target user node is obtained from multiple graph snapshots in sequence, and a first neighbor sequence is formed according to the time sequence of the graph snapshots. The above multiple graph snapshots are graph snapshots before the target time t to be predicted in the discrete dynamic graph, and are arranged in time to form a graph snapshot sequence.

[0063] The first neighbor node may include a one-hop neighbor node and / or a multi-hop neighbor node of the target user node. A one-hop neighbor node refers to a node that is directly and indirectly connected to the target user node. A multi-hop neighbor node refers to a node that is not directly connected to the target user but is indirectly connected to the target user node through other neighbor nodes.

[0064] In some embodiments, a one-hop neighbor node of the target user node may be obtained to obtain a first neighbor sequence including the one-hop neighbor node. In some other embodiments, multi-hop neighbor nodes of the target user node may be obtained to obtain a first neighbor sequence including multi-hop neighbor nodes. Exemplarily, a random walk algorithm may be used to sample neighbor nodes of the target user node, and a first neighbor node sequence may be obtained based on the sampled neighbor nodes.

[0065] In step S22, a plurality of first feature sequences are formed according to the relevant features of each neighbor node in the first neighbor sequence.

[0066] The first feature sequences may include one or more feature sequences, including but not limited to one or more of a neighbor feature sequence, an edge feature sequence, a position feature sequence, and an appearance feature sequence. When the user behavior prediction refers to edge prediction, the first feature sequences may also include an intersection feature sequence.

[0067] The following is a further explanation of these various characteristic sequences.

[0068] In some embodiments, step S22 may include, according to the node features of each neighbor node, forming a neighbor feature sequence, and classifying it into a plurality of first feature sequences. The length of the neighbor feature sequence is equal to the length of the first neighbor sequence. The node feature can be regarded as an abstract vector obtained by encoding one or more attribute information included in the neighbor node.

[0069] In some embodiments, step S22 may include, according to the edge features of the connection edges between each neighbor node and the target user node, forming an edge feature sequence, and classifying it into a plurality of first feature sequences. The length of the edge feature sequence is equal to the length of the first neighbor sequence. The edge feature can be regarded as an abstract vector obtained by encoding one or more attribute information included in the connection edge between the neighbor node and the target user node. When there are multiple connection edges between the neighbor node and the target user node, the attribute information included in the multiple connection edges can be encoded into one edge feature.

[0070] Taking node i as the target user node as an example, the first neighbor sequence of node i can be recorded as N i The neighbor feature sequence of node i can be recorded as The edge feature sequence of node i can be recorded as

[0071]

[0072] node n Represents one of the neighbor nodes n, d of node i N Represents the dimension of node features. (i,n,t′)Represents the connection edge between node i and its neighbor node n in the graph snapshot at time t′. E Represents the dimension of edge features.

[0073] In some embodiments, step S22 may include encoding the sequential index of the graph snapshot where each neighbor node in the first neighbor sequence is located in multiple graph snapshots to obtain a corresponding position code, and the position code of each neighbor node forms a position feature sequence, which is included in a plurality of first feature sequences. The length of the position feature sequence is equal to the length of the first neighbor sequence. Each position code in the position feature sequence can be regarded as a position feature. Different neighbor nodes appearing in the same graph snapshot in the first neighbor sequence can have the same position code.

[0074] Exemplarily, a sequential index p that can characterize the temporal order can be assigned to each graph snapshot according to the temporal order of the graph snapshots in the multiple graph snapshots. For example, Arabic numerals can be used to sequentially number the multiple graph snapshots. The sequential index p can be encoded using a variety of encoding methods. Exemplarily, the sequential index can be encoded using a time encoding function. The time encoding function is a technology that can map a sequential index to a low-dimensional vector.

[0075] The encoding formula of the time encoding function can be exemplarily expressed as:

[0076]

[0077] Where P represents the position feature and p is the sequential index. The dimension of the position feature P is 2·d p . w1,…,w d All of them can be trainable parameters that can be updated during the training phase. For example, the position feature sequence of node i can be recorded as Among them, P n Represents the location features of node i’s neighbor node n.

[0078] Graph snapshots arranged in chronological order can naturally reflect time series information. By constructing a position feature sequence based on the position encoding obtained by encoding the sequential index of the graph snapshot where the neighbor nodes are located in multiple graph snapshots, time sequence attributes can be given to the neighbor nodes, effectively utilizing time-related information in discrete-time dynamic graphs, thereby improving the model effect.

[0079] In some embodiments, step S22 may include, for any neighbor node in the first neighbor sequence, determining the appearance characteristics of the neighbor node according to the number of interactions of the neighbor node with the target user node in multiple graph snapshots, and the appearance characteristics of each neighbor node form an appearance characteristic sequence, which can be classified into several first characteristic sequences.

[0080] The occurrence feature can reflect the interaction frequency or pattern reflected in the historical behavior of the neighbor node, and can enhance the node representation. For any neighbor node in the first neighbor sequence, an occurrence vector can be constructed according to the number of interactions between the neighbor node and the target user node in multiple graph snapshots. For example, for each neighbor node node i n The constructed occurrence vector can be recorded as O n . O n The length of is equal to the number of T-1 graph snapshots before the target time T. Represents node i and its neighbor node n The number of interactions in the graph snapshot corresponding to time t, t=1,2,Ω,T-1. n Each vector element in corresponds to a number of interactions. n Each vector element in is not encoded but still has an explicit meaning. n Encoding is processed into appearance features. For example, a multilayer perceptron (MLP) and an activation function (such as ReLU) can be used to convert O n The encoding is processed as an occurrence feature.

[0081] Based on the appearance features of all neighbor nodes, the appearance feature sequence can be constructed. The appearance feature sequence can be recorded as

[0082]

[0083] In some embodiments, when user behavior prediction refers to edge prediction, step S22 may include, for any neighbor node in the first neighbor sequence, determining the intersection matrix of the neighbor node according to the number of interactions of the neighbor node with the target user node in multiple graph snapshots, and the number of interactions of the neighbor node with the target object.

[0084] In a specific example, the intersection matrix may include two rows of elements, the number of elements in each row is equal to the number of graph snapshots, one row of elements is used to represent the number of interactions between the neighbor node and the target user node in multiple graph snapshots, and the other row of elements is used to represent the number of interactions between the neighbor node and the target object node in multiple graph snapshots.

[0085] For example, node i represents the target user node and node j represents the target object node. For each neighbor node of node i, n The constructed intersection matrix can be recorded as A n . A n It can be expressed as,

[0086]

[0087] in, Represents the neighbor node node n The number of interactions with node i in graph snapshot t, Represents the neighbor node node n The number of interactions with node j in graph snapshot t.

[0088] Next, the intersection matrix can be encoded and processed into an intersection feature. For example, a neural network (such as GRU, MLP, etc.) or a SUM operation (such as weighted summation) can be used to process the intersection matrix into an intersection feature with a specific dimension. n The intersection feature obtained by encoding can be recorded as The dimension of the intersection feature is d I .

[0089] Then, the intersection features of each neighbor node in the first neighbor sequence can form an intersection feature sequence, which is classified into several first feature sequences. The intersection feature sequence can be recorded as

[0090]

[0091] Existing methods mainly learn node representations based on the information of a single node, while ignoring the intersection information between node pairs. In discrete-time dynamic graphs, synchronous interactions or shared neighbor nodes play a key role in shaping node representations. Intuitively, if two nodes have common neighbor nodes at some historical moments, the possibility of these two nodes interacting in the future will increase with the increase in the frequency of interaction between the common neighbor nodes. For example, in a social network, if two independent individuals follow the same person or have the same hobbies at the same time, the association between the two independent individuals will be strengthened. In addition, this regularity in the data may show periodicity. For example, in an e-commerce network, individuals who are interested in camping often tend to buy related items mainly during weekends. This phenomenon shows that the periodic pattern in consumer behavior is extremely critical to predicting users' future purchasing trends. Therefore, an intersection feature sequence can be constructed for the target user node and the target object node respectively to reflect the regularity or pattern that the intersection information between node pairs in the discrete-time dynamic graph may show.

[0092] In step S23, a first neural network based on a self-attention mechanism is used to process a first input sequence formed based on a plurality of first feature sequences to obtain a user representation of a target user corresponding to a target user node.

[0093] In some embodiments, several first feature sequences may be formed into a single first input sequence, and the first input sequence may be processed using a first neural network based on a self-attention mechanism to obtain a user representation of a target user corresponding to a target user node. Among them, features corresponding to the same neighbor node in several first feature sequences and taken from different first feature sequences may be spliced ​​into one feature, so that several first feature sequences may be processed into one first input sequence. Alternatively, multiple first feature sequences may be spliced ​​or stacked in sequence to obtain a total feature sequence as the first input sequence.

[0094] Figure 3 A schematic diagram of constructing feature sequences for node i and node j respectively is shown.

[0095] like Figure 3 As shown, for each neighbor node in the neighbor sequence of node i and node j, node features, edge features, position features, occurrence features, and intersection features can be constructed respectively to form five corresponding feature sequences, and the dimensions of each element in the five feature sequences can be unified using MLP. Different feature sequences can be input into MLPs with different structures.

[0096] After obtaining the unified feature sequence of five dimensions, the unified feature sequence of five dimensions can be stacked according to the nodes. For example, the features corresponding to the same neighbor node in the unified feature sequence of five dimensions of node i and taken from different feature sequences can be spliced ​​into one feature. In this way, several feature sequences can be processed into one feature sequence, which can be input into the first neural network as the first input sequence.

[0097] In some other embodiments, a plurality of first feature sequences may be subjected to a recombination operation corresponding to M different sizes. The first size may be used to represent one of the M sizes. The recombination operation corresponding to the first size includes dividing the first feature sequence into blocks of the first size, using the blocks as sequence elements, and forming a recombination sequence corresponding to the first size.

[0098] For each block in the recombinant sequence, multiple features included in the block may be aggregated (eg, averaged), and the features obtained after the aggregation process are used as features of the sequence element represented by the block.

[0099] M is an integer greater than or equal to 1. Generally speaking, the larger the value of M, the better the final model performance, but the calculation time will also increase accordingly. Therefore, the value of M can be determined by comprehensively considering the model performance and calculation efficiency.

[0100] For example, S m represents the first size. The length of the recombinant sequence corresponding to the first size can be recorded as λ m ,λm =|N i |*S m , where |N i | represents the length of the first feature sequence of node i (i.e., the target user node). If |N i |Cannot be S m If the first block is divisible by the last block, a padding operation can be performed on the last block to fill the size of the block to the first size. The padding operation refers to filling one or more features after the feature sequence contained in the last block so that the size of the filled block is the first size. The size of the block, that is, the number of features contained in the block, is also the length of the block. The filled feature can be a pre-set feature that is independent of the block to be filled. For example, the filled feature can be a feature vector with the same dimension as the feature in the first feature sequence and all element values ​​are zero. The filled feature can also be related to the block to be filled. For example, the feature to be filled can be determined based on the features contained in the block.

[0101] Figure 4 A schematic diagram showing a multi-size recombination operation performed on a feature sequence of a node i and a feature sequence of a node j is shown.

[0102] like Figure 4 As shown, the M types of block sizes may include 2, 3, and 4. For the feature sequence of node i and the feature sequence of node j, they can be divided and filled according to the block sizes of 2, 3, and 4, respectively. Taking the example that the feature sequence of node i and the feature sequence of node j both include eight sequence elements, when the feature sequence of node i or node j is divided into blocks using a block size of 2, since the feature sequence is divisible by 2, a recombinant sequence consisting of four blocks of size 2 can be directly obtained, and there is no need to perform a filling operation. When the feature sequence of node i or node j is divided into blocks using a block size of 3, the feature sequence is not divisible by 3, so it is necessary to perform a filling operation on the last block, that is, fill the last block with a feature so that the size of the last block is 3.

[0103] Taking the first characteristic sequences of node i including the five characteristic sequences mentioned above as an example, each of the five characteristic sequences can be subjected to M different size recombination operations. m The five recombinant sequences obtained by recombination can be recorded as,

[0104] Next, several recombinant sequences of the same size can be formed into a single first input sequence, thereby obtaining M first input sequences corresponding to M different sizes. Exemplarily, MLP can be used to unify the feature dimensions of each sequence element in several recombinant sequences of the same size, and then several recombinant sequences of the same size can be spliced ​​into a single first input sequence through a splicing operation. m For example, the first input sequence after concatenation can be recorded as

[0105]

[0106] Here, the symbol “‖” represents the concatenation operation.

[0107]

[0108] Among them, FEAT* can be NODE, EDGE, POS or INT, d C Represents the unified feature dimension.

[0109] Therefore, step S23 may include: using M first neural networks to process the M first input sequences respectively to obtain M embedded representations; using a multilayer perceptron to process the combination vectors of the M embedded representations to obtain a user representation.

[0110] The parameters of the M first neural networks can be the same, such as sharing parameters, which can be understood as running the same first neural network M times to process M first input sequences of different sizes. The parameters of the M first neural networks can also be different, such as M first neural networks with different parameters. Each first neural network is used to receive a first input sequence of a size, so different first neural networks can capture information of different granularities. The output of each first neural network (which can also be a sequence data) can be regarded as a node representation of the target user node at the corresponding granularity.

[0111] Taking the first neural network as a Transformer encoder as an example, the output obtained by the Transformer encoder processing the first input sequence is a first output sequence with a length equal to the first input sequence. Each sequence element in the first output sequence can be aggregated, and the feature vector finally obtained by aggregation is used as the embedded representation of the target user node at the size (i.e., specific granularity) corresponding to the first input sequence. Aggregation processing may include but is not limited to averaging processing and weighted summation processing.

[0112] The combination vector of the M embedded representations may refer to a vector obtained by performing a concatenation operation on the M embedded representations. The dimension of the combination vector may be equal to the sum of the dimensions of the M embedded representations.

[0113] Multi-sized recombination operations can reduce the computational complexity of the first neural network, and can also help the first neural network capture local information in the original sequence data at different granularities. Typically, sequence elements (i.e., blocks) in small-sized recombinant sequences can help the first neural network focus on capturing fine-grained local features, while sequence elements in large-sized recombinant sequences contain more contextual information, which can help the first neural network capture coarser-grained features. The recombination operation makes the length of the sequence input to the first neural network controllable, thereby allowing the model complexity to be kept within a reasonable range. In addition, multi-sized recombination operations enable the model to capture historical information of multiple granularities at the same time, thereby improving the expressiveness of the model.

[0114] In step S24, the user behavior of the target user at the target time is predicted based at least on the user representation.

[0115] Taking user behavior prediction as an example, user representation can be input into the prediction module, and the output of the prediction module is the prediction result of the user behavior of the target user at the target time. The prediction result can represent the probability of the target user performing one or more user behaviors at the target time. The prediction module can be, but is not limited to, MLP.

[0116] In some embodiments, user behavior prediction may refer to edge prediction, that is, predicting whether a target user interacts with a target object at a target time. In this case, the user behavior prediction may also be performed according to the above combined method. Figure 1 The method for obtaining the user representation of the target user described above obtains the object representation of the target object corresponding to the target object node.

[0117] Specifically, first, the second neighbor nodes of the target object node can be obtained from multiple graph snapshots in turn, and a second neighbor sequence can be formed according to the time sequence of the graph snapshots. Then, several second feature sequences are formed according to the relevant features of each neighbor node in the second neighbor sequence. Then, the second input sequence formed based on the several second feature sequences is processed by the first neural network to obtain the corresponding object representation. The second feature sequence can include the five feature sequences mentioned above.

[0118] Thus, step S24 may include: determining a prediction result of whether the target user and the target object interact at the target time according to the user representation and the object representation. Exemplarily, the user representation and the object representation may be spliced, and the spliced ​​result may be used as an input of the prediction module to obtain a prediction result of whether the target user and the target object interact at the target time output by the prediction module.

[0119] Exemplarily, a plurality of second characteristic sequences may be subjected to recombination operations corresponding to M different sizes, wherein the recombination operation corresponding to the first size includes dividing the second characteristic sequence into blocks of the first size, using the blocks as sequence elements to form a recombined sequence corresponding to the first size; and forming a single second input sequence from a plurality of recombined sequences of the same size, thereby obtaining M second input sequences corresponding to M different sizes.

[0120] In order to capture the interactive dependency between the target user node and the target object node, the first input sequence and the second input sequence, for example, the first input sequence and the second input sequence corresponding to the same size obtained based on the multi-size recombination operation, can be stacked into one input sequence (the first input sequence can be in front and the second input sequence can be in the back; or the first input sequence can be in the back and the second input sequence can be in the front). Then the input sequence is input into the first neural network. Then, according to the positional relationship between the first input sequence and the second input sequence in the input sequence, the output sequence corresponding to the first input sequence and the output sequence corresponding to the second input sequence can be found from the sequence output by the first neural network. Then, the two sequences are processed separately to obtain the embedded representation of the target user node and the embedded representation of the target object node.

[0121] For example, the M first input sequences of node i and the input sequences of the same size in the M second input sequences of node j can be combined into one input sequence X according to the following formula: (0) ,

[0122]

[0123] The M first input sequences of node i and the M second input sequences of node j correspond to the same size S m The input sequence of . express Length, express The length of the first neural network can be a Transformer encoder with L layers, X (0) is the input of the Transformer encoder, and the output of the Transformer encoder in the last layer (i.e., the Lth layer) can be expressed as H = X (L) , the entire processing process of the Transformer encoder can be expressed as the following formula,

[0124]

[0125] Represents the Transformer encoder for the input sequence The output sequence obtained by processing. Output sequence The length of the front is The sequence is the node representation of node i at the current granularity, and the output sequence The length of the front is The sequence after the sequence is the node representation of node j at the current granularity. The embedding representation of node i and node j at the current granularity can be obtained by averaging according to the following formula.

[0126]

[0127] in, Indicates that node i is at the current granularity (i.e., with size S m The corresponding granularity) of the embedded representation, represents the embedding representation of node j at the current granularity, Represents the output sequence The length of the front is sequence, Represents the output sequence The length of the middle is MEAN represents the average processing operation on the elements in the sequence.

[0128] Finally, the embedded representations of the nodes at different granularities can be concatenated and then passed through an MLP to obtain the final node representation of the node.

[0129]

[0130] Among them, FinalEmb i Represents the final learned node representation of node i, FinalEmb j Represents the final learned node representation of node j.

[0131] In the training phase, after step S24, a training step may be included, wherein the behavior label between the target user and the target object may be determined based on the graph snapshot at the target moment; the prediction loss may be determined based on the behavior label and the prediction result, and the first neural network may be trained based on the prediction loss. Training the first neural network means updating the model parameters of the first neural network. In some embodiments, other model parameters involved in steps S21 to S24 may also participate in the training.

[0132] For example, the prediction result can be recorded as

[0133]

[0134] Cross-Entropy can be used to construct a loss function that characterizes the prediction loss.

[0135]

[0136] Among them, L represents the prediction loss, Y ij Indicates the behavior label, that is, the actual result.

[0137] In other embodiments, other conventional loss terms may be introduced into the training loss, which are not limited here and will not be expanded one by one.

[0138] After the training phase is completed, the trained first neural network (optionally, other trained model structures may also be included) may be used for user behavior prediction.

[0139] According to an embodiment of another aspect, a user prediction device based on a discrete-time dynamic graph is provided. Figure 5 A schematic diagram of the structure of a user prediction device according to an embodiment is shown. The user prediction device can be deployed in any device, platform or device cluster with data storage, computing and processing capabilities.

[0140] like Figure 5 As shown, the user prediction device 700 includes:

[0141] A first forming unit 71 is configured to sequentially obtain the first neighbor node of the target user node from the multiple graph snapshots, and form a first neighbor sequence according to the time sequence of the graph snapshots;

[0142] A second forming unit 72 is configured to form a plurality of first feature sequences according to relevant features of each neighbor node in the first neighbor sequence;

[0143] The processing unit 73 is configured to process the first input sequence formed based on the plurality of first feature sequences by using a first neural network based on a self-attention mechanism to obtain a user representation of the target user corresponding to the target user node;

[0144] The prediction unit 74 is configured to predict the user behavior of the target user at a target time at least based on the user representation.

[0145] In some embodiments, the second forming unit 72 is specifically configured to form a neighbor feature sequence according to the node features of each neighbor node itself, and classify the neighbor feature sequences into the plurality of first feature sequences.

[0146] In some embodiments, the second forming unit 72 is specifically configured to: form an edge feature sequence according to the edge features of the connection edges between each neighbor node and the target user node, and classify the edge feature sequences into the plurality of first feature sequences.

[0147] In some embodiments, the second forming unit 72 is specifically configured to: encode the sequential index of the graph snapshot where each neighbor node is located in the multiple graph snapshots to obtain the corresponding position code, and the position code of each neighbor node forms a position feature sequence, which is included in the plurality of first feature sequences.

[0148] In some embodiments, the second forming unit 72 is specifically configured as follows: for any neighbor node, the appearance characteristics of the neighbor node are determined according to the number of interactions of the neighbor node with the target user node in the multiple graph snapshots, and the appearance characteristics of each neighbor node form an appearance feature sequence, which is included in the plurality of first feature sequences.

[0149] In some embodiments, the first forming unit 71 may also be configured to sequentially obtain second neighbor nodes of the target object node from the multiple graph snapshots, and form a second neighbor sequence according to the time sequence of the graph snapshots;

[0150] The second forming unit 72 may also be configured to form a plurality of second feature sequences according to relevant features of each neighbor node in the second neighbor sequence;

[0151] The processing unit 73 may also be configured to use the first neural network to process a second input sequence formed based on the plurality of second feature sequences to obtain a corresponding object representation;

[0152] The prediction unit 74 is specifically configured to determine, based on the user representation and the object representation, a prediction result of whether the target user and the target object interact at the target moment.

[0153] In a further embodiment, the second forming unit 72 is specifically configured as follows: for any neighbor node in the first neighbor sequence, according to the number of interactions of the neighbor node with the target user node and the number of interactions with the target object in the multiple graph snapshots, determine the intersection matrix of the neighbor node, and encode the intersection matrix into an intersection feature; the intersection features of each neighbor node form an intersection feature sequence, which is included in the several first feature sequences.

[0154] In some embodiments, the user prediction device 700 may further include:

[0155] a recombination unit configured to perform recombination operations corresponding to M different sizes on the plurality of first characteristic sequences, wherein the recombination operation corresponding to the first size includes dividing the first characteristic sequence into blocks of the first size, using the blocks as sequence elements, and forming a recombined sequence corresponding to the first size;

[0156] The third forming unit is configured to form a single first input sequence from a plurality of recombinant sequences of the same size, thereby obtaining M first input sequences corresponding to M different sizes.

[0157] In a further embodiment, the processing unit 73 is specifically configured to: use M first neural networks to process the M first input sequences respectively to obtain M embedded representations; use a multilayer perceptron to process the combination vector of the M embedded representations to obtain the user representation.

[0158] In some embodiments, the user prediction device 700 may further include:

[0159] A first determining unit, configured to determine a behavior label between the target user and the target object according to a graph snapshot at a target moment;

[0160] A second determining unit, configured to determine a prediction loss according to the behavior label and the prediction result;

[0161] A training unit is configured to train the first neural network according to the prediction loss.

[0162] For the specific implementation examples of each unit in the above device, please refer to the previous Figure 2 Through the above device, the first neural network based on the self-attention mechanism can be used to improve the learning effect of node representation, thereby improving the accuracy of downstream prediction tasks.

[0163] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 3 The method described.

[0164] According to another embodiment, a computer program product is also provided, including a computer program / instruction, which is executed by a processor to implement the aforementioned combination Figure 3 The method steps described.

[0165] According to another embodiment of the present invention, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the Figure 3 The method described.

[0166] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0167] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A user prediction method based on a discrete time dynamic graph, wherein the discrete time dynamic graph includes a plurality of graph snapshots arranged in time before a target time, and the method includes: Sequentially obtain the first neighbor node of the target user node from the multiple graph snapshots, and form a first neighbor sequence according to the time sequence of the graph snapshots; According to relevant features of each neighbor node in the first neighbor sequence, forming a plurality of first feature sequences; Processing a first input sequence formed based on the plurality of first feature sequences using a first neural network based on a self-attention mechanism to obtain a user representation of a target user corresponding to the target user node; A user behavior of the target user at a target time is predicted based at least on the user representation.

2. The method according to claim 1, wherein: According to the relevant features of each neighbor node in the first neighbor sequence, a plurality of first feature sequences are formed, including: A neighbor feature sequence is formed according to the node features of each neighbor node itself and included in the plurality of first feature sequences.

3. The method according to claim 1, wherein: According to the relevant features of each neighbor node in the first neighbor sequence, a plurality of first feature sequences are formed, including: An edge feature sequence is formed according to the edge features of the connection edges between each neighbor node and the target user node, and is included in the plurality of first feature sequences.

4. The method according to claim 1, wherein: According to the relevant features of each neighbor node in the first neighbor sequence, a plurality of first feature sequences are formed, including: The sequential index of the graph snapshot where each neighbor node is located in the multiple graph snapshots is encoded to obtain a corresponding position code, and the position code of each neighbor node forms a position feature sequence, which is included in the plurality of first feature sequences.

5. The method according to claim 1, wherein: According to the relevant features of each neighbor node in the first neighbor sequence, a plurality of first feature sequences are formed, including: For any neighbor node, the appearance feature of the neighbor node is determined according to the number of interactions of the neighbor node with the target user node in the multiple graph snapshots, and the appearance features of each neighbor node form an appearance feature sequence, which is included in the multiple first feature sequences.

6. The method according to any one of claims 1 to 5, further comprising: Sequentially acquiring second neighbor nodes of the target object node from the multiple graph snapshots, and forming a second neighbor sequence according to the time sequence of the graph snapshots; According to the relevant features of each neighbor node in the second neighbor sequence, forming a plurality of second feature sequences; Processing a second input sequence formed based on the plurality of second feature sequences using the first neural network to obtain a corresponding object representation; The predicting the user behavior of the target user at the target time includes: A prediction result of whether the target user and the target object interact at the target time is determined according to the user representation and the object representation.

7. The method according to claim 6, wherein: The target object is one of the following: a commodity, a content object, or another user different from the target user.

8. The method according to claim 6, wherein: According to the relevant features of each neighbor node in the first neighbor sequence, a plurality of first feature sequences are formed, including: For any neighbor node in the first neighbor sequence, an intersection matrix of the neighbor node is determined according to the number of interactions of the neighbor node with the target user node and the number of interactions of the neighbor node with the target object in the multiple graph snapshots, and the intersection matrix is ​​encoded and processed into an intersection feature; the intersection features of each neighbor node form an intersection feature sequence, which is included in the multiple first feature sequences.

9. The method according to claim 1, further comprising: Performing recombination operations corresponding to M different sizes on the plurality of first characteristic sequences, wherein the recombination operation corresponding to the first size includes dividing the first characteristic sequence into blocks of the first size, using the blocks as sequence elements, and forming a recombined sequence corresponding to the first size; Several recombined sequences of the same size are formed into a single first input sequence, thereby obtaining M first input sequences corresponding to M different sizes.

10. The method according to claim 9, wherein the first input sequence formed based on the plurality of first feature sequences is processed using a first neural network based on a self-attention mechanism, comprising: Using M first neural networks, respectively process the M first input sequences to obtain M embedded representations; The combination vector of the M embedded representations is processed by a multi-layer perceptron to obtain the user representation.

11. The method according to claim 6, further comprising: Determine a behavior label between the target user and the target object according to the graph snapshot at the target moment; Determining a prediction loss according to the behavior label and the prediction result; The first neural network is trained according to the prediction loss.

12. A user prediction device based on a discrete time dynamic graph, wherein the discrete time dynamic graph comprises a plurality of graph snapshots arranged in time before a target time, the device comprising: A first forming unit is configured to sequentially obtain a first neighbor node of a target user node from the plurality of graph snapshots, and form a first neighbor sequence according to a time sequence of the graph snapshots; a second forming unit, configured to form a plurality of first feature sequences according to relevant features of each neighbor node in the first neighbor sequence; A processing unit is configured to process a first input sequence formed based on the plurality of first feature sequences using a first neural network based on a self-attention mechanism to obtain a user representation of a target user corresponding to the target user node; The prediction unit is configured to predict the user behavior of the target user at a target time at least based on the user representation.

13. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

14. A computing device comprising a memory and a processor, characterized in that: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 11 is implemented.

Images