Method, device, electronic device and storage medium for processing dynamic image data

Abstract

The present application provides a method, device, electronic device and storage medium for processing dynamic graph data. The method for processing dynamic graph data includes: obtaining M sub-graph data that are time-continuous in the dynamic graph data; processing the M sub-graph data through a neural network to obtain M first output results, the M first output results are used to characterize the structural characteristics and timing characteristics of the M sub-graph data; determining M second output results based on the M first output results, the M second output results are used to characterize the timing characteristics of the M first output results; determining the prediction result of the Mth sub-graph data in the M sub-graph data based on the M second output results. Not only the parameters of the graph convolutional neural network in the neural network are evolved, but also the output results of the convolutional neural network are evolved, so that the dynamic sequence information in the dynamic graph data can be accurately extracted, thereby improving the accuracy of dynamic graph data prediction.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for processing dynamic image data. Background Art

[0002] With the continuous deepening of artificial neural network research, Graph Neural Networks (GNN) have been widely used in the modeling, learning and representation of graph structure data. Given that there are a large number of dynamic scenes in real life, in many cases, it is the dynamic behaviors in these dynamic scenes that can express some important information. In dynamic scenes, there are dynamic graph data whose structure and properties change over time, and the network that processes this dynamic graph data is Dynamic Graph Neural Networks (DGNN).

[0003] Generally speaking, the dynamic graph neural network DGNN combines the graph neural network GNN with the recurrent neural network (RNN). The graph neural network GNN captures the structural features of each subgraph data in the dynamic graph data, and the recurrent neural network RNN ​​captures the temporal features of the dynamic graph data. In this way, the dynamic graph neural network DGNN processes the dynamic graph data. For example, the existing EvolveGCN uses the recurrent neural network RNN ​​to evolve the parameters in the graph convolutional neural network (GCN), thereby capturing the dynamic information of the graph sequence in the dynamic graph data.

[0004] However, the learning ability of EvolveGCN is limited. In terms of feature extraction, EvolveGCN cannot accurately extract dynamic sequence information in dynamic graph data, thereby reducing the accuracy of dynamic graph data prediction. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for processing dynamic graph data, so as to accurately extract dynamic sequence information in dynamic graph data, thereby improving the accuracy of dynamic graph data prediction.

[0006] In order to solve the above technical problems, the embodiments of the present application provide the following technical solutions:

[0007] In a first aspect, the present application provides a method for processing dynamic graph data, the method comprising: obtaining M sub-graph data that are time-continuous in the dynamic graph data, the prediction results of the nodes of the first M-1 sub-graph data in the M sub-graph data are known, and M is a positive integer greater than or equal to 2; processing the M sub-graph data through a neural network to obtain M first output results, and the M first output results are used to characterize the structural characteristics and timing characteristics of the M sub-graph data; determining M second output results based on the M first output results, and the M second output results are used to characterize the timing characteristics of the M first output results; and determining the prediction result of the Mth sub-graph data in the M sub-graph data based on the M second output results.

[0008] The second aspect of the present application provides a processing device for dynamic graph data, the device comprising: a receiving module, used to obtain M sub-graph data that are continuous in time in the dynamic graph data, the prediction results of the nodes of the first M-1 sub-graph data in the M sub-graph data are known, and M is a positive integer greater than or equal to 2; a first processing module, used to process the M sub-graph data through a neural network to obtain M first output results, and the M first output results are used to characterize the structural characteristics and timing characteristics of the M sub-graph data; a second processing module, used to determine M second output results based on the M first output results, and the M second output results are used to characterize the timing characteristics of the M first output results; a prediction module, used to determine the prediction result of the Mth sub-graph data in the M sub-graph data based on the M second output results.

[0009] The third aspect of the present application provides an electronic device, comprising: a processor, a memory, and a bus; wherein the processor and the memory communicate with each other through the bus; and the processor is used to call program instructions in the memory to execute the method in the first aspect.

[0010] A fourth aspect of the present application provides a computer-readable storage medium, comprising: a stored program; wherein, when the program is executed, the device where the storage medium is located is controlled to execute the method in the first aspect.

[0011] Compared with the prior art, the method for processing dynamic graph data provided in the first aspect of the present application first obtains M time-continuous sub-graph data in the dynamic graph data; then, the M sub-graph data are processed by a neural network to obtain M first output results, and the M first output results are used to characterize the structural characteristics and timing characteristics of the M sub-graph data; then, M second output results are determined based on the M first output results, and the M second output results are used to characterize the timing characteristics of the M first output results; finally, the prediction result of the Mth sub-graph data in the M sub-graph data is determined based on the M second output results. After processing multiple sub-graph data in the dynamic graph data by a neural network, the multiple output results obtained are further processed. In this way, not only the parameters of the graph convolutional neural network in the neural network are evolved, but also the output results of the convolutional neural network are evolved. Through dual-channel evolution, the dynamic sequence information in the dynamic graph data can be accurately extracted, thereby improving the accuracy of dynamic graph data prediction.

[0012] The dynamic image data processing device provided in the second aspect of the present application, the electronic device provided in the third aspect, and the computer-readable storage medium provided in the fourth aspect have the same or similar beneficial effects as the dynamic image data processing method provided in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easy to understand. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0014] Figure 1 The process diagram of the method for processing dynamic image data in the embodiment of the present application is as follows Figure 1 ;

[0015] Figure 2 This is a schematic diagram of dynamic graph data in an embodiment of the present application;

[0016] Figure 3 Schematic diagram of the network architecture for processing dynamic graph data in the embodiment of the present application Figure 1 ;

[0017] Figure 4 The process diagram of the method for processing dynamic image data in the embodiment of the present application is as follows Figure 2 ;

[0018] Figure 5 This is a schematic diagram of the architecture of the compression and excitation network SENet in an embodiment of the present application;

[0019] Figure 6 Schematic diagram of the network architecture for processing dynamic graph data in the embodiment of the present application Figure 2 ;

[0020] Figure 7 This is a schematic diagram of the architecture of dynamic graph data in an embodiment of the present application;

[0021] Figure 8 The structure of the dynamic image data processing device in the embodiment of the present application is shown in FIG. Figure 1 ;

[0022] Fig. 9 The structure of the dynamic image data processing device in the embodiment of the present application is shown in FIG. Figure 2 ;

[0023] Fig.10 Schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0025] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0026] In the prior art, dynamic graph neural networks DGNN, such as EvolveGCN, can use recurrent neural networks RNN to evolve parameters in graph convolutional neural networks GCN, thereby capturing the dynamic information of graph sequences in dynamic graph data. However, the learning ability of EvolveGCN is limited, and it is still unable to accurately extract the dynamic sequence information in dynamic graph data, thereby reducing the accuracy of dynamic graph data prediction.

[0027] In view of this, an embodiment of the present application provides a method for processing dynamic graph data. After the dynamic graph data is processed by a neural network, the multiple output results obtained are further processed. That is, not only the parameters of the graph convolutional neural network in the neural network are evolved, but also the output results of the convolutional neural network are evolved. Through dual-channel evolution, the dynamic sequence information in the dynamic graph data can be accurately extracted, thereby improving the accuracy of dynamic graph data prediction.

[0028] Next, the method for processing dynamic image data provided in the embodiment of the present application is described in detail.

[0029] Figure 1The process diagram of the method for processing dynamic image data in the embodiment of the present application is as follows Figure 1 , see Figure 1 As shown, the method may include:

[0030] S101: Obtain M time-continuous sub-image data in dynamic image data.

[0031] Among them, the prediction results of the nodes of the first M-1 sub-graph data among the M sub-graph data are known. M is a positive integer greater than or equal to 2.

[0032] The time continuity indicates the time continuity of the dynamic graph data change. Compared with the static graph, the nodes and edges in the dynamic graph will change over time. However, the nodes and edges corresponding to each time point in the dynamic graph are fixed. Therefore, after obtaining the dynamic graph data, the subgraph data corresponding to each time point in the dynamic graph data can be obtained through the dynamic graph data, and then the dynamic graph data can be processed based on the subgraph data with fixed nodes and edges.

[0033] Figure 2 This is a schematic diagram of dynamic graph data in the embodiment of the present application, see Figure 2 As shown, in this dynamic graph, there are M subgraph data. In each subgraph data, the nodes and edges are constant. However, between each subgraph data, the nodes and edges will change over time.

[0034] Generally speaking, the purpose of processing dynamic graph data is to achieve prediction. That is, the content represented by the nodes or edges in the Mth subgraph data is predicted (the content represented by the nodes or edges in the Mth subgraph data is currently unknown) through the first M-1 subgraph data in the dynamic graph data (the content represented by the nodes and edges in the first M-1 subgraph data is known). In order to ensure the accuracy of the prediction, M subgraph data that are continuous in time in the dynamic graph data are generally obtained.

[0035] It should be noted that M here can be the number of all sub-image data in the dynamic image data, or the number of some sub-image data in the dynamic image data. The specific value of M can be determined according to the dynamic image data and the actual situation of the prediction, and is not specifically limited here. However, the value of M must be a positive integer greater than or equal to 2, so that it can be called dynamic image data.

[0036] S102: Process M sub-graph data through a neural network to obtain M first output results.

[0037] The M first output results are used to characterize the structural features and timing features of the M sub-graph data.

[0038] In order to process the dynamic graph data and obtain the structural features and timing features of the dynamic graph data to achieve dynamic graph prediction, after obtaining M sub-graph data that are time-continuous in the dynamic graph data, the M sub-graph data can be input into the neural network, and the neural network can process the M sub-graph data and output M first output results after processing.

[0039] It should be noted here that, in fact, only the Mth output result among the M first output results can be output, that is, the prediction of the Mth subgraph data. The first M-1 first output results among the M first output results are hidden states in the neural network and are not directly output. However, the neural network is able to know that there are M first output results corresponding to the M subgraph data. Each output result is a prediction of the corresponding subgraph data.

[0040] In practical applications, the neural network may be a dynamic graph neural network DGNN, such as EvolveGCN, variants of EvolveGCN (EGCN-H, EGCN-O), etc. The specific type of the neural network is not limited here.

[0041] S103: Determine M second output results according to the M first output results.

[0042] The M second output results are used to characterize the timing characteristics of the M first output results.

[0043] That is to say, after the neural network processes the M sub-graph data and obtains M first output results, the M first output results obtained after the neural network processing are processed again to obtain M second output results. That is, not only the parameters of the neural network are evolved in step S102, but also the output of the neural network is evolved in step S103. The accuracy of extracting dynamic sequence information from dynamic graph data is improved through double evolution, and then the accuracy of dynamic graph prediction is improved.

[0044] In the specific implementation process, the M first output results can be input into a recurrent neural network for processing, and the output of the recurrent neural network is the M second output results. Since the recurrent neural network is used to extract time series features, the M second output results can represent the time series features of the M first output results. By extracting features from the output of the previous neural network again through the recurrent neural network, the accuracy of extracting dynamic sequence information in dynamic graph data can be improved, thereby improving the accuracy of dynamic graph prediction.

[0045] In practical applications, recurrent neural network is also called Recurrent Neural Network, or RNN.

[0046] The M second output results here are similar to the M first output results mentioned above. Only the Mth second output result among the M second output results can actually be output. The first M-1 second output results among the M second output results are output inside the recurrent neural network, and only the recurrent neural network can know them.

[0047] S104: Determine a prediction result of the Mth sub-image data among the M sub-image data according to the M second output results.

[0048] After obtaining M second output results, since the Mth output result among the M second output results is obtained based on the first M-1 second output results, the prediction result of the Mth sub-image data among the M sub-image data can be accurately obtained based on the M second output results.

[0049] In the specific implementation process, the M second output results can be substituted into the fully connected (FC) layer, and the prediction results of the M sub-graphs can be output through the fully connected FC layer. The prediction result of the Mth sub-graph among the prediction results of the M sub-graphs is the prediction of the dynamic graph required in the end.

[0050] The prediction results of the M sub-graphs here are similar to the M first output results mentioned above. Only the prediction result of the Mth sub-graph among the prediction results of the M sub-graphs can be actually output. The prediction results of the first M-1 sub-graphs among the prediction results of the M sub-graphs are output inside the fully connected FC layer, and only the fully connected FC layer can know them.

[0051] Next, in combination with the network architecture for processing dynamic data in the embodiment of the present application, Figure 1 The processing method of dynamic graph data in is explained.

[0052] Figure 3 Schematic diagram of the network architecture for processing dynamic graph data in the embodiment of the present application Figure 1 , see Figure 3 As shown, first, the M subgraph data (node ​​feature matrices X1, X2, ..., X M , adjacency matrix A1, A2, ..., A M ) is input into the neural network, and M first output results (H1, H2, ..., H M Then, (H1, H2, ..., H M ) is input into the recurrent neural network RNN ​​to obtain M second output results (not shown in the figure). Then, the M second output results are substituted into the fully connected FC layer to obtain the prediction results corresponding to the M sub-graph data (Z1, Z2, ..., Z M ). Among the M prediction results, the Mth prediction result (Z M) is the prediction result of the dynamic graph required in the end.

[0053] It can be seen from the above content that the method for processing dynamic graph data provided in the embodiment of the present application first obtains M sub-graph data that are time-continuous in the dynamic graph data; then, the M sub-graph data are processed by a neural network to obtain M first output results, and the M first output results are used to characterize the structural characteristics and timing characteristics of the M sub-graph data; then, M second output results are determined based on the M first output results, and the M second output results are used to characterize the timing characteristics of the M first output results; finally, the prediction result of the Mth sub-graph data in the M sub-graph data is determined based on the M second output results. After processing multiple sub-graph data in the dynamic graph data by a neural network, the multiple output results obtained are further processed. In this way, not only the parameters of the graph convolutional neural network in the neural network are evolved, but also the output results of the convolutional neural network are evolved. Through dual-channel evolution, the dynamic sequence information in the dynamic graph data can be accurately extracted, thereby improving the accuracy of dynamic graph data prediction.

[0054] Further, as Figure 1 As a refinement and extension of the method shown, an embodiment of the present application also provides a method for processing dynamic graph data. Figure 4 The process diagram of the method for processing dynamic image data in the embodiment of the present application is as follows Figure 2 , see Figure 4 As shown, the method may include:

[0055] S401: Obtain M time-continuous sub-image data in the dynamic image data.

[0056] S402: Process M sub-graph data through a first recurrent neural network and a graph convolutional neural network to obtain M first output results.

[0057] Since the subsequent processing of the M sub-graph data in the dynamic graph data uses a recurrent neural network, in order to facilitate the description of the processing process of the sub-graph data, a certain recurrent process is used as an example for description. However, this is not intended to limit the method provided in the embodiment of the present application to only one recurrent process.

[0058] Assume that the M sub-image data include the first sub-image data and the second sub-image data. The first sub-image data and the second sub-image data are any two sub-image data that are adjacent in time sequence among the M sub-image data. The time sequence of the first sub-image data is before the time sequence of the second sub-image data. The following is an example of processing the second sub-image data.

[0059] In addition, the neural network includes a first recurrent neural network and a graph convolutional neural network. The network composed of the first recurrent neural network and the graph convolutional neural network can be set to one layer or two layers. The following is an example of a two-layer first recurrent neural network and a graph convolutional neural network, that is, the neural network is: one layer of recurrent neural network, one layer of graph convolutional neural network, two layers of recurrent neural network, and two layers of graph convolutional neural network. The graph convolutional neural network contains a parameter matrix to be trained.

[0060] In practical applications, the first recurrent neural network is called Recurrent Neural Network, or RNN. In order to distinguish it from the second recurrent neural network, it can be called RNN1. The graph convolutional neural network is called Graph Convolutional Network, or GCN.

[0061] Specifically, step S402 may include:

[0062] S4021: Input the first parameter matrix into a layer of recurrent neural network to obtain a second parameter matrix.

[0063] The first parameter matrix is ​​a parameter matrix when a layer of graph convolutional neural network processes the first sub-graph data. When the first sub-graph data is the first sub-graph data among the M sub-graph data, the first parameter matrix is ​​a parameter matrix randomly generated in a layer of graph convolutional neural network.

[0064] The first parameter matrix is ​​input into a layer of recurrent neural network for optimization to obtain an optimized second parameter matrix. The optimized second parameter matrix is ​​used to replace the original parameter matrix (here is the first parameter matrix) in a layer of graph convolutional neural network. This enables the layer of graph convolutional neural network replaced by the second parameter matrix to more accurately extract the structural features and temporal features of the second sub-graph data, thereby improving the accuracy of the final prediction results.

[0065] S4022: Input the second parameter matrix and the second subgraph data into a layer of graph convolutional neural network to obtain a fourth latent variable.

[0066] The second parameter matrix is ​​the parameters evolved in a layer of graph convolutional neural network when a layer of graph convolutional neural network processes the second sub-graph data. At this time, the evolved second parameter matrix is ​​substituted into a layer of graph convolutional neural network, and then the second sub-graph data is substituted into a layer of graph convolutional neural network after parameter evolution. The output of a layer of graph convolutional neural network can be regarded as a prediction result (but this prediction result only has one layer of graph convolutional neural network, and it is not the prediction result finally output in the embodiment of the present application), that is, the fourth hidden variable.

[0067] Here, the output of the first subgraph data after passing through a layer of graph convolutional neural network is the third latent variable, and the output of the second subgraph data after passing through a layer of graph convolutional neural network is the fourth latent variable.

[0068] At this point, the processing of the first recurrent neural network and graph convolutional neural network in the first layer has been completed. Of course, at this time, the fourth latent variable output by the first layer of graph convolutional neural network can also be used as the first output result. In this way, the processing of the first recurrent neural network and graph convolutional neural network in the second layer can be omitted, which can improve the prediction speed.

[0069] In order to improve the prediction accuracy, the processing of the first recurrent neural network and the graph convolutional neural network of the second layer can also be continued. The processing of the first recurrent neural network and the graph convolutional neural network of the second layer is further described below.

[0070] S4023: Input the second parameter matrix into the two-layer recurrent neural network to obtain a fourth parameter matrix.

[0071] Among them, the second parameter matrix is ​​the parameter matrix when a layer of graph convolutional neural network processes the second sub-graph data. At this time, the second parameter matrix is ​​input into the two-layer recurrent neural network for optimization to obtain the fourth parameter matrix after quadratic optimization. The fourth parameter matrix after quadratic optimization is used to replace the original parameter matrix (here is the third parameter matrix) in the two-layer graph convolutional neural network. The two-layer graph convolutional neural network replaced with the fourth parameter matrix can accurately extract the structural features and time series features of the second sub-graph data (i.e., the fourth latent variable output in the first layer) again, thereby improving the accuracy of the final prediction result.

[0072] S4024: Input the fourth parameter matrix, the fourth latent variable, and the adjacency matrix in the second subgraph data into a two-layer graph convolutional neural network to obtain a second latent variable.

[0073] Among them, the second latent variable is one of the M first output results.

[0074] The processing procedures of step S4022 and step S4024 are roughly the same. The difference is that after the evolved fourth parameter matrix is ​​substituted into the two-layer graph convolutional neural network, what is substituted into the two-layer graph convolutional neural network after parameter evolution is no longer the second subgraph data, but the fourth latent variable and the adjacency matrix in the second subgraph data. The subgraph data includes a node matrix and an adjacency matrix. The node matrix is ​​used to characterize the information of each node in the subgraph data, while the adjacency matrix is ​​used to characterize the information between each node in the subgraph data. In step S4024, only the adjacency matrix in the second subgraph data is input into the two-layer graph convolutional neural network in order to further and better extract the time series features in the second subgraph data, thereby improving the accuracy of the final prediction result.

[0075] After being processed by the two-layer graph convolutional neural network, the prediction result corresponding to the second subgraph data, that is, the second latent variable, can be output. This prediction result is still invisible and only known by the graph convolutional neural network. The Mth subgraph data will output the corresponding latent variable, that is, the Mth first output result.

[0076] S403: Determine M weights corresponding to the M first output results.

[0077] After the M sub-graph data are processed by the two-layer first recurrent neural network and the graph convolutional neural network, M first output results are obtained. However, the output Mth first output result cannot take into account the global features, thereby reducing the accuracy of the result prediction. After a lot of research, the applicant found that the fundamental reason why the Mth first output result cannot take into account the global features lies in the inherent defects of the recurrent neural network, that is, it can only take into account the features of the sub-graph data at the closer time points, while forgetting the features of the sub-graph data at the farther time points. For example, assuming that the dynamic graph data contains sub-graph data at 10 time points, when the recurrent neural network processes the sub-graph data at the 10th time point, it can only take into account the features of the sub-graph data at the 7th, 8th, and 9th time points, but cannot take into account the features of the sub-graph data at the earlier time points of 1, 2, and 3.

[0078] In view of this, in order to take into account the global features and thus improve the accuracy of the result prediction. After the M sub-graph data are processed by the recurrent neural network and the graph convolutional neural network to obtain M first output results, different weights, that is, M weights, are assigned to the M first output results according to the effective information amount of the M sub-graph data corresponding to the M first output results. Among them, the values ​​of the M weights are positively correlated with the effective information amount of the M sub-graph data. In other words, the more important the sub-graph data is and the more effective information it has, the higher its corresponding weight value is.

[0079] In practical applications, when determining the M weights corresponding to the M first output results, a weight calculation network can be used. That is, the M first output results are input into the weight calculation network, and the weight calculation network will have M outputs, and these M outputs are the M weights corresponding to the M first output results.

[0080] In practical applications, the weight calculation network can be a Squeeze-and-Excitation Network (SENet). Figure 5 This is a schematic diagram of the architecture of the compression and excitation network SENet in the embodiment of the present application, see Figure 5As shown in Figure 2, the compression and excitation network SENet includes a global pooling layer and two fully connected FC layers (FC1, FC2). When it is necessary to obtain M first output results (H1, H2, ..., H M ) corresponding to the M weights (α1, α2, …, α M ), the M first output results (H1, H2, ..., H M ) Input the compression and excitation network SENet, and the compression and excitation network SENet can output M first output results (H1, H2, ..., H M ) corresponding to the M weights (α1, α2, …, α M ). The compression and excitation network SENet is an existing network for calculating the weights of each part of a picture, and the specific process of calculating the weights of the compression and excitation network SENet in the embodiment of the present application is similar to the existing process of calculating the weights of each part of a picture, so it will not be repeated here. Of course, the weight calculation network can also be other types of neural networks for calculating weights. The specific type of the weight calculation network is not limited here.

[0081] Of course, in addition to using the weight calculation network to calculate the M weights corresponding to the M first output results, other calculation methods can also be used for calculation, such as: Attention mechanism. The specific calculation method of the weights is not limited here.

[0082] In this way, the M first output results after being assigned weights are subsequently input into the recurrent neural network and the fully connected FC layer, and the final output result can take into account the global characteristics, thereby improving the accuracy of the prediction.

[0083] S404: Determine M second output results according to the M first output results and the M weights.

[0084] The first output result is the hidden variable. There are as many hidden variables as there are first output results. That is, if there are M first output results, there are M hidden variables. In order to facilitate the description of the subsequent loop process, two loop processes are used for description. However, it is not intended to be only two loop processes.

[0085] Assume that the M first output results include a first latent variable and a second latent variable. The first latent variable and the second latent variable are any two first output results that are adjacent in time sequence. The time sequence of the first latent variable is before the time sequence of the second latent variable. In this way, the first latent variable and the second latent variable are input into the second recurrent neural network, and the second recurrent neural network can process the first latent variable and the second latent variable, and realize the prediction of the result of the next subgraph data through the relationship between the first latent variable and the second latent variable.

[0086] Specifically, step S407 may include:

[0087] S4041: Input the first latent variable and its weight into the second recurrent neural network to obtain the fifth latent variable.

[0088] After the first recurrent neural network and the graph convolutional neural network process the first subgraph data and the second subgraph data, the first latent variable and the second latent variable are obtained. Then, the first latent variable and its weight are input into the second recurrent neural network to obtain the fifth latent variable.

[0089] In practical applications, the second recurrent neural network is also called Recurrent Neural Network, or RNN. In order to distinguish it from the first recurrent neural network, it can be called RNN2.

[0090] S4042: Input the fifth latent variable and the second latent variable into the second recurrent neural network to obtain the sixth latent variable.

[0091] Among them, the sixth hidden variable is one of the M second output results.

[0092] There are M sub-graph data, there are M cycles in the second recurrent neural network, and then there are M corresponding hidden variables, that is, M second output results.

[0093] S405: Determine a prediction result of the Mth sub-image data among the M sub-image data according to the M second output results.

[0094] In the specific implementation process, the M second output results can be substituted into the fully connected FC layer, and the fully connected FC layer can calculate the M prediction results corresponding to the M sub-graph data. The prediction results corresponding to the first M sub-graph data are invisible and exist in the fully connected FC layer. The actual output is only the prediction result of the Mth sub-graph data, that is, the prediction result of the final dynamic graph.

[0095] At this point, the prediction process of the dynamic graph is over.

[0096] Next, in combination with the network architecture for processing dynamic data in the embodiment of the present application, Figure 4 The processing method of dynamic graph data in is explained.

[0097] Figure 6 Schematic diagram of the network architecture for processing dynamic graph data in the embodiment of the present application Figure 2 , see Figure 6 As shown, processing dynamic graph data may include the following steps:

[0098] 1. Input the parameter matrix W in the graph convolutional neural network GCN into the first recurrent neural network RNN1 in sequence to obtain the parameter matrix (W1, W2, ..., WM ).

[0099] Right now,

[0100] Where t represents a time point. A time point corresponds to a sub-graph data in the dynamic graph data. l represents the number of layers of graph convolution.

[0101] 2. The M sub-graph data in the dynamic graph data (node ​​feature matrix X1, X2, ..., X M , adjacency matrix A1, A2, ..., A M ) and parameter matrices (W1, W2, ..., W M ) are input into the graph convolutional neural network GCN in sequence, and then pass through the first recurrent neural network RNN1 and the graph convolutional neural network GCN to obtain M first output results (H1, H2, ..., H M ), that is, the hidden state, and the first layer Initial parameter matrix is randomly generated.

[0102] Right now,

[0103] Since the feature extraction capability of a graph convolutional neural network GCN with more than two layers will decrease, the embodiment of the present application uses a combined network of the first recurrent neural network RNN1 and the graph convolutional neural network GCN twice.

[0104] 3. The M first output results (H1, H2, ..., H M ) Input the compression and excitation network SENet, and get M weights (α1, α2, ..., α M ), and then obtain the weighted M first output results (α1H1, α2H2, ..., α M H M ).

[0105] Right now,

[0106] 4. The weighted M first output results (α1H1, α2H2, ..., α M H M ) is input into the second recurrent neural network RNN2 to obtain M second output results, and the M second output results are represented by a hidden variable ", that is, the M second output results include hidden variable 1", hidden variable 2", ..., hidden variable M" (not shown in the figure).

[0107] 5. Input the M second output results (first latent variable ', second latent variable ', ..., latent variable M") into the fully connected FC layer to obtain M prediction results (Z1, Z2, ..., ZM ).

[0108] Right now,

[0109] Among them, Z1, Z2, ..., Z M-1 is the prediction result of node properties in the training set (before training, the corresponding nodes are labeled), Z M is the prediction result of the test sample (before training, the corresponding node has no label). M It is the output matrix at the Mth time point, which represents the prediction result of the node properties, that is, the final dynamic graph prediction result required.

[0110] In the above step 1, the first recurrent neural network RNN1 is used to evolve the parameters in the graph convolutional neural network GCN. In the above step 4, the second recurrent neural network RNN2 is used to evolve the output of the graph convolutional neural network GCN. Taking the Long-Short Term Memory (LSTM) network as an example, the specific calculation process is as follows:

[0111] F t =sigmoid(V F X t +U F Y t-1 +B F )

[0112] I t =sigmoid(V I X t +U I Y t-1 +B I )

[0113] O t =sigmoid(V O X t +U O Y t-1 +B O )

[0114]

[0115]

[0116] Y t =O t *tanh(C t )

[0117] Among them, W x , U x and B xetc. are parameters to be trained, x∈{F,I,O,C}, sigmoid and tanh are activation functions, and * indicates element-wise multiplication. t is the input at the (current) time t, Y t-1 is the hidden state at time t-1 (previous time), Y t is the hidden state of the output. t is the unit state at time t. C0 and Y0 come from random initialization. F t It is called the forget gate, t It is called the input gate, O t It is called the output gate.

[0118] The first recurrent neural network RNN1 and the second recurrent neural network RNN 2的 The difference is that the input of the first recurrent neural network RNN1 is X t =Y t-1 =W t-1 , the output is Y t =W t At this time, Y t-1 It is not a hidden state, but input data. The input of the second recurrent neural network RNN2 is X t =α t H t , the output is the hidden state Y at time t t .

[0119] In practical applications, the above-mentioned dynamic graph data processing method can be specifically processed by a dual-evolution dynamic graph neural network model. Figure 6 As shown, Figure 6 The structure of the dual-evolving dynamic graph neural network model is shown in FIG. In the dual-evolving dynamic graph neural network model, the following are built in sequence: a first recurrent neural network, a graph convolutional neural network, a first recurrent neural network, a graph convolutional neural network, a weight calculation network, a second recurrent neural network, and a fully connected layer. After the dual-evolving dynamic graph neural network model is built, the M subgraph data in the dynamic graph data are input into the built dual-evolving dynamic graph neural network model, and the output of the dual-evolving dynamic graph neural network model is the prediction result of the Mth subgraph data.

[0120] In other words, the dual-evolution dynamic graph neural network model is used while being trained. That is, among the M subgraph data in the dynamic graph data, the prediction results of the first M-1 subgraph data are known, and the prediction results of the Mth subgraph data are unknown. Then, the first M-1 subgraph data are divided into a training set and a validation set according to a certain ratio, and the Mth subgraph data is used as a test set. In order to obtain the prediction result of the Mth subgraph data in the test set, the training set, the validation set, and the test set are all input into the dual-evolution dynamic graph neural network model at the same time. Since the parameters in the dual-evolution dynamic graph neural network model are random at the initial moment, that is, random parameters are used to process the first subgraph data in the training set. When the dual-evolution dynamic graph neural network model processes the second subgraph data, the parameters will be adjusted based on the output results of the first subgraph data. When the dual-evolution dynamic graph neural network model processes the third subgraph data, the parameters will be adjusted based on the output results of the second subgraph data. That is to say, the parameters in the dual-evolution dynamic graph neural network model will be adjusted over time based on the output results of each subgraph data in the training set until the Mth subgraph data is processed. In this way, the prediction of the Mth subgraph data is achieved, and the prediction result is correlated with the characteristics of the previous subgraph data, thus achieving the prediction of the dynamic graph.

[0121] Next, taking two specific scenarios in the financial field as examples, the use scenarios of the dynamic graph data processing method provided in the embodiments of the present application are explained.

[0122] In practical applications, the input data is the sub-graph data in the dynamic graph data, which are discrete data. These sub-graph data are divided into training set, validation set and test set. Figure 7 This is a schematic diagram of the architecture of the dynamic graph data in the embodiment of the present application, see Figure 7 As shown, in the dynamic graph data, the first subgraph data, the second subgraph data, and the third subgraph data can be used as training sets and validation sets, and the t-1th subgraph data and the tth subgraph data can be used as test sets. Since the model for processing dynamic graph data in the embodiment of the present application is a semi-supervised learning model, it is necessary to input all the training sets, validation sets, and test sets into the model.

[0123] 1. Suspicious transaction identification (node ​​classification)

[0124] In each subgraph data of the dynamic graph data, nodes are used to represent transactions, and edges are used to represent the association between transactions. The output of the model is obtained by inputting all the training sets, validation sets, and test sets corresponding to the dynamic graph data into the model. That is, the characteristics of the central node itself are predicted (updated) by the characteristics of the adjacent nodes and the characteristics of the central node itself. Its purpose is to predict the properties of each transaction node in the test set, that is, to classify the transaction nodes according to their characteristics and judge whether the transactions corresponding to the nodes are legal / illegal, normal / abnormal, etc. For example: If the properties of the transaction node (central node) adjacent to the illegal transaction node are similar (transaction amount, transaction location, etc.), there is a certain probability that it is also an illegal transaction (property prediction).

[0125] 2. Link recommendation (link prediction)

[0126] In each subgraph of the dynamic graph data, nodes are used to represent users and products, and edges are used to represent user consumption behaviors. The output of the model is obtained by inputting all the training sets, validation sets, and test sets corresponding to the dynamic graph data into the model. That is, graph representation learning is performed at each time point, the structural information of user nodes and product nodes is collected, and it is determined whether the representation vectors of user nodes and product nodes are close. If they are close, a connection edge is established, that is, the product is recommended to the user. That is, in the test set, it is predicted whether there is an edge between the user node and the product node, that is, whether future transactions will be carried out.

[0127] For example, suppose that in the test set, the user node u and the product node v are not connected at time point t. The historical transaction situation is used to predict whether there is an edge between the user node u and the product node. Here, the fully connected FC layer needs to be adjusted: the Embeddings of the user node u and the product node v at time point t are concatenated and substituted into a fully connected FC layer to determine whether the two Embeddings are similar. The output is the evaluation score of whether there is a connection. 1 means that the two Embeddings are similar and a connection edge is established. 0 means that the two Embeddings are not similar and no connection edge is established.

[0128] Finally, a set of comparative experiments is used to illustrate the feature extraction effect of the dynamic image data processing method provided in the embodiment of the present application.

[0129] The comparative experiment is used for suspicious transaction identification. The Bitcoin dataset provided by Elliptic (a Bitcoin monitoring company in the UK) is used. The Bitcoin dataset is a discrete dynamic graph data with time attributes, including legal transactions (from legal exchanges, services, etc.) and illegal money laundering transactions (from fraud, malware, terrorist financing, ransomware, Ponzi schemes, etc.). Nodes represent transaction entities, and edges represent continuous transactions of Bitcoin in a short period of time. The experiment uses the first 94 regular features of the transaction, using the first 38 time point data, 30 for training, and 8 for testing.

[0130] In the experiment, the processing method of dynamic graph data provided by the embodiment of the present application is compared with two variants of EvolveGCN (EGCN-H and EGCN-O). The three algorithms have the same parameter settings, with a learning rate of 0.001 and a number of iterations of 1000. Due to data imbalance, the experiment only evaluates illegal transactions. The experimental results are shown in the following table:

[0131] serial number algorithm Precise Recall F1 1 EGCN-H 0.681 0.848 0.755 2 EGCN-O 0.667 0.691 0.681 3 This application 0.857 0.765 0.808

[0132] Among them, Precise represents accuracy, Recall represents memory, and F1 represents score.

[0133] By comparison, it can be seen that EGCN-H and EGCN-O focus on the evolution of the parameters of the graph convolutional network GCN. EGCN-H performs a single evolution of the parameters of the graph convolutional network GCN, and considers more the extraction of node (self) features in the subgraph data. EGCN-O uses the output information of the graph convolutional network GCN to perform a single evolution of the parameters of the graph convolutional network GCN, and considers more the extraction of the structural information of the nodes. The task learning capabilities of both are limited. In contrast, the present application performs a dual evolution of the graph convolutional network GCN parameters and the graph convolutional network GCN output, while paying attention to node features and node structure information, and is optimal under both Precise (accuracy) and F1 (score) indicators, proving that the method for processing dynamic graph data provided in the embodiment of the present application can accurately extract dynamic sequence information in dynamic graph data, thereby improving the accuracy of dynamic graph data prediction.

[0134] Based on the same inventive concept, as an implementation of the above method, an embodiment of the present application also provides a device for processing dynamic graph data. Figure 8 The structure of the dynamic image data processing device in the embodiment of the present application is shown in FIG. Figure 1 , see Figure 8 As shown, the device may include:

[0135] The receiving module 801 is used to obtain M sub-graph data that are continuous in time in the dynamic graph data, where the prediction results of the nodes of the first M-1 sub-graph data in the M sub-graph data are known, and M is a positive integer greater than or equal to 2.

[0136] The first processing module 802 is used to process the M sub-graph data through a neural network to obtain M first output results, and the M first output results are used to characterize the structural characteristics and time series characteristics of the M sub-graph data.

[0137] The second processing module 803 is used to determine M second output results according to the M first output results, and the M second output results are used to characterize the time series characteristics of the M first output results.

[0138] The prediction module 804 is configured to determine a prediction result of the Mth sub-image data among the M sub-image data according to the M second output results.

[0139] Further, as Figure 8 As a refinement and extension of the method shown, an embodiment of the present application also provides a device for processing dynamic graph data. Fig. 9 The structure of the dynamic image data processing device in the embodiment of the present application is shown in FIG. Figure 2 , see Fig. 9 As shown, the method may include:

[0140] The receiving module 901 is used to obtain M sub-graph data that are continuous in time in the dynamic graph data, where the prediction results of the nodes of the first M-1 sub-graph data in the M sub-graph data are known, and M is a positive integer greater than or equal to 2.

[0141] The neural network includes a first recurrent neural network and a graph convolutional neural network, the M sub-graph data include first sub-graph data and second sub-graph data, the first sub-graph data and the second sub-graph data are any two temporally adjacent sub-graph data, and the timing of the first sub-graph data is before the timing of the second sub-graph data.

[0142] The first processing module 902 is used to process the M sub-graph data through a neural network to obtain M first output results, and the M first output results are used to characterize the structural characteristics and time series characteristics of the M sub-graph data.

[0143] Specifically, the first processing module 902 includes:

[0144] The first evolution unit 9021 is used to input a first parameter matrix into the first recurrent neural network to obtain a second parameter matrix, where the first parameter matrix is ​​a parameter matrix when the graph convolutional neural network processes the first subgraph data.

[0145] The first extraction unit 9022 is used to input the second parameter matrix and the second subgraph data into the graph convolutional neural network to obtain a fourth latent variable.

[0146] The second evolution unit 9023 is used to input the second parameter matrix into the first recurrent neural network to obtain a fourth parameter matrix.

[0147] The second extraction unit 9024 is used to input the fourth parameter matrix, the fourth latent variable and the adjacency matrix of the second subgraph data into the graph convolutional neural network to obtain a second latent variable, where the second latent variable is one of the M first output results.

[0148] The second processing module 903 is used to determine M second output results according to the M first output results, and the M second output results are used to characterize the time series characteristics of the M first output results.

[0149] Specifically, the second processing module 903 includes:

[0150] The determination unit 9031 is used to determine M weights corresponding to the M first output results, and the values ​​of the M weights are positively correlated with the effective information volume of the M sub-graph data.

[0151] The determining unit 9031 is specifically configured to input the M first output results into a weight calculation network to obtain the M weights.

[0152] The processing unit 9032 is used to determine the M second output results according to the M first output results and the M weights.

[0153] The M first output results include a first latent variable and a second latent variable, the first latent variable and the second latent variable are any two first output results that are adjacent in time sequence, and the time sequence of the first latent variable is located before the time sequence of the second latent variable.

[0154] The processing unit 9032 includes:

[0155] The first recurrent unit 3021a is used to input the first latent variable into the second recurrent neural network to obtain a fifth latent variable.

[0156] The second cyclic unit 9032b is used to input the fifth latent variable and the second latent variable into the second cyclic neural network to obtain a sixth latent variable, where the sixth latent variable is one of the M second output results.

[0157] The prediction module 904 is configured to determine a prediction result of the Mth sub-image data among the M sub-image data according to the M second output results.

[0158] It should be noted here that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0159] Based on the same inventive concept, an embodiment of the present application also provides an electronic device. Fig.10 This is a schematic diagram of the structure of the electronic device in the embodiment of the present application, see Fig.10 As shown, the electronic device may include: a processor 1001, a memory 1002, and a bus 1003; wherein the processor 1001 and the memory 1002 communicate with each other via the bus 1003; the processor 1001 is used to call program instructions in the memory 1002 to execute the methods in one or more of the above embodiments.

[0160] It should be noted that the description of the above electronic device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the electronic device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0161] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium, which may include: a stored program; wherein, when the program is running, the device where the storage medium is located is controlled to execute the method in one or more of the above embodiments.

[0162] It should be noted here that the description of the above storage medium embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the storage medium embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0163] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for processing dynamic graph data, characterized in that: The method comprises: Obtaining M subgraph data that are time-continuous in the dynamic graph data, wherein the prediction results of the nodes of the first M-1 subgraph data in the M subgraph data are known, and M is a positive integer greater than or equal to 2; the M subgraph data include nodes and edges, wherein the nodes are used to represent transactions, and the edges are used to represent the association between transactions; or, the nodes are used to represent users and commodities, and the edges are used to represent the consumption behavior of the users; Processing the M sub-graph data by a neural network to obtain M first output results, wherein the M first output results are used to characterize structural features and temporal features of the M sub-graph data; Determine M second output results according to the M first output results, wherein the M second output results are used to characterize the time series characteristics of the M first output results; Determine a prediction result of the Mth sub-image data among the M sub-image data according to the M second output results; The neural network includes a first recurrent neural network and a graph convolutional neural network, the M sub-graph data include first sub-graph data and second sub-graph data, the first sub-graph data and the second sub-graph data are any two sub-graph data that are adjacent in time sequence, and the time sequence of the first sub-graph data is located before the time sequence of the second sub-graph data; The processing of the M sub-graph data by a neural network to obtain M first output results includes: Processing a first parameter matrix through the first recurrent neural network to obtain a second parameter matrix, wherein the first parameter matrix is ​​a parameter matrix when the graph convolutional neural network processes the first subgraph data; A second latent variable is obtained based on the second parameter matrix, the second subgraph data and the graph convolutional neural network, where the second latent variable is one of the M first output results.

2. The method according to claim 1, characterized in that The determining M second output results according to the M first output results comprises: Determine M weights corresponding to the M first output results, where the values ​​of the M weights are positively correlated with the effective information amount of the M sub-graph data; The M second output results are determined according to the M first output results and the M weights.

3. The method according to claim 2, characterized in that The determining M weights corresponding to the M first output results includes: The M first output results are input into a weight calculation network to obtain the M weights.

4. The method according to any one of claims 1 to 3, characterized in that The first recurrent neural network includes a layer of recurrent neural network and a layer of recurrent neural network, and the graph convolutional neural network includes a layer of graph convolutional neural network and a layer of graph convolutional neural network; The step of processing the first parameter matrix by the first recurrent neural network to obtain a second parameter matrix includes: Inputting a first parameter matrix into the one-layer recurrent neural network to obtain a second parameter matrix, wherein the first parameter matrix is ​​a parameter matrix when the one-layer graph convolutional neural network processes the first subgraph data; Inputting the second parameter matrix into the two-layer recurrent neural network to obtain a fourth parameter matrix; The obtaining a second latent variable based on the second parameter matrix, the second subgraph data and the graph convolutional neural network includes: Inputting the second parameter matrix and the second subgraph data into the one-layer graph convolutional neural network to obtain a fourth latent variable; The adjacency matrix, the fourth parameter matrix and the fourth latent variable in the second subgraph data are input into the two-layer graph convolutional neural network to obtain a second latent variable, where the second latent variable is one of the M first output results.

5. The method according to any one of claims 1 to 3, characterized in that The M first output results include a first latent variable and a second latent variable, the first latent variable and the second latent variable are any two first output results adjacent in time sequence, and the time sequence of the first latent variable is located before the time sequence of the second latent variable; The determining M second output results according to the M first output results comprises: The first latent variable and the second latent variable are input into a second recurrent neural network to obtain a sixth latent variable, where the sixth latent variable is one of the M second output results.

6. The method according to claim 5, characterized in that The step of inputting the first latent variable and the second latent variable into a second recurrent neural network to obtain a sixth latent variable comprises: Inputting the first latent variable into the second recurrent neural network to obtain a fifth latent variable; The fifth latent variable and the second latent variable are input into the second recurrent neural network to obtain a sixth latent variable.

7. A device for processing dynamic graph data, characterized in that: The device comprises: A receiving module is used to obtain M sub-graph data that are continuous in time in the dynamic graph data, wherein the prediction results of the nodes of the first M-1 sub-graph data in the M sub-graph data are known, and M is a positive integer greater than or equal to 2; the M sub-graph data include nodes and edges, wherein the nodes are used to represent transactions, and the edges are used to represent the association between transactions; or, the nodes are used to represent users and commodities, and the edges are used to represent the consumption behavior of the users; A first processing module, used for processing the M sub-graph data by a neural network to obtain M first output results, wherein the M first output results are used for characterizing structural features and time series features of the M sub-graph data; A second processing module, used to determine M second output results according to the M first output results, wherein the M second output results are used to characterize the time series characteristics of the M first output results; A prediction module, used for determining a prediction result of the Mth sub-image data among the M sub-image data according to the M second output results; The neural network includes a first recurrent neural network and a graph convolutional neural network, the M sub-graph data include first sub-graph data and second sub-graph data, the first sub-graph data and the second sub-graph data are any two sub-graph data that are adjacent in time sequence, and the time sequence of the first sub-graph data is located before the time sequence of the second sub-graph data; The first processing module is specifically used for: Processing a first parameter matrix through the first recurrent neural network to obtain a second parameter matrix, wherein the first parameter matrix is ​​a parameter matrix when the graph convolutional neural network processes the first subgraph data; A second latent variable is obtained based on the second parameter matrix, the second subgraph data and the graph convolutional neural network, where the second latent variable is one of the M first output results.

8. An electronic device, characterized in that: include: Processor, memory, bus; Wherein, the processor and the memory communicate with each other through the bus; The processor is configured to call program instructions in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: include: A stored program; wherein, when the program is running, the device where the storage medium is located is controlled to execute the method as described in any one of claims 1 to 6.

Images