Classification method, device and storage medium for large-scale attributeless graphs based on graph transformer

Abstract

The present invention discloses a classification method, device and storage medium for a large-scale attributeless graph based on a graph transformer, comprising: obtaining an adjacency matrix and a node feature matrix of a large-scale attributeless graph to be classified; determining a connection mode between operations in a basic stacking unit; determining an operation mode and a unit layer number of each operation in the basic stacking unit; constructing a stacked graph neural network based on the connection mode, the operation mode and the unit layer number; determining a multi-layer perceptron module, an LLA module, a CLA module and a global mean pooling module, and constructing a local-global combined Transformer network by using the multi-layer perceptron module, the LLA module, the CLA module and the global mean pooling module; determining a classification network and a linear mapping network, and determining a graph transformer by using a stacked graph neural network based on random connections, a local-global combined Transformer network, a linear mapping network and a classification network; inputting the adjacency matrix and the node feature matrix into the graph transformer to obtain the category of the large-scale attributeless graph.

Description

Technical Field

[0001] The present invention relates to the technical field of attributeless graph processing, and in particular to a classification method, device and storage medium for large-scale attributeless graphs based on graph transformer. Background Art

[0002] Graph data is widely present in the real world and can naturally simulate real-world entities and complex relationships between entities. The topic of graph data processing, especially graph classification, involves building a model by learning from training data to predict the class label of a given object. However, a large number of graphs available in the real world lack sufficient node attributes or even do not contain node attributes (called attributeless graphs). For example, in the citation network classification task, the original attributes or detailed descriptions of some papers may not be accessible due to copyright protection; in the social network classification task, each graph represents a discussion thread, where the nodes correspond to users, and due to privacy protection, the user's attribute information may not be available.

[0003] Although attributeless graphs do not contain direct attribute information of nodes, their structure itself (such as node connection patterns, the existence of subgraphs, etc.) may contain important hidden patterns or rules. By classifying attributeless graphs, these hidden patterns can be revealed, helping people better explore the valuable information behind graph data.

[0004] The development of graph neural networks has led to great progress in graph classification tasks. Graph classification based on graph neural networks usually uses a stack of graph convolutional layers and graph pooling layers to learn and extract features from graphs. The superior performance is mainly based on the case where the graph has natural node features (attribute graphs) when applying graph neural networks. In order to apply graph neural networks to attributeless graphs, node features are usually initialized manually. However, existing attributeless graph classification methods only use graph neural networks to capture the neighborhood structure information of graph nodes, which makes it difficult to effectively mine more effective information in large-scale graphs. There are problems such as large errors and insufficient accuracy in classifying large-scale attributeless graphs. Summary of the invention

[0005] The present invention provides a classification method, device and storage medium for large-scale attributeless graphs based on graph transformer, which mainly aims to improve the classification accuracy of large-scale attributeless graphs.

[0006] According to a first aspect of the present invention, a large-scale attributeless graph classification method based on graph Transformer is provided, comprising:

[0007] Obtain the adjacency matrix and node feature matrix of the large-scale attribute-free graph to be classified;

[0008] A random connection generation algorithm is used to determine the connection pattern between operations in the basic stacking unit;

[0009] Calling a neural architecture search algorithm to determine an operation mode of each operation in the basic stacking unit and to determine the number of unit layers of the basic stacking unit;

[0010] Based on the connection mode, operation mode, and number of unit layers between the operations in the basic stacking unit, a stacking graph neural network based on random connection is constructed;

[0011] Determine a multi-layer perceptron module, an LLA module, a CLA module, and a global mean pooling module, and construct a local-global combined Transformer network by using the multi-layer perceptron module, the LLA module, the CLA module, and the global mean pooling module;

[0012] Determine a classification network and a linear mapping network, and determine a graph Transformer by using the randomly connected stacked graph neural network, the local-global combined Transformer network, the linear mapping network, and the classification network;

[0013] The adjacency matrix and the node feature matrix are input into the graph Transformer for classification, and features of the large-scale attributeless graph to be classified are extracted through the stacked graph neural network based on random connections to obtain a first attributeless graph feature output by the stacked graph neural network based on random connections, and features of the large-scale attributeless graph to be classified are extracted through the local-global combined Transformer network to obtain a second attributeless graph feature output by the local-global combined Transformer network, and the first attributeless graph feature is combined with the second attributeless graph feature to obtain a combined feature, and the combined feature is mapped through the linear mapping network to obtain a mapping graph representation feature, and the mapping graph representation feature is input into the classification network to obtain a category of the large-scale attributeless graph.

[0014] According to a second aspect of the present invention, a large-scale attributeless graph classification device based on graph transformer is provided, comprising:

[0015] An acquisition unit, used to acquire an adjacency matrix and a node feature matrix of a large-scale attribute-free graph to be classified;

[0016] A generation unit, used for determining a connection pattern between operations in a basic stacking unit by using a random connection generation algorithm;

[0017] A first determining unit, configured to call a neural architecture search algorithm to determine an operation mode of each operation in the basic stacking unit, and to determine the number of unit layers of the basic stacking unit;

[0018] A first construction unit is used to construct a stacked graph neural network based on random connections based on the connection mode, operation mode, and number of unit layers between the operations in the basic stacking unit;

[0019] The second construction unit is used to determine a multi-layer perceptron module, an LLA module, a CLA module, and a global mean pooling module, and construct a local-global combined Transformer network by using the multi-layer perceptron module, the LLA module, the CLA module, and the global mean pooling module;

[0020] A second determination unit is used to determine a classification network and a linear mapping network, and a graph Transformer is determined by the stacked graph neural network based on random connections, the local-global combined Transformer network, the linear mapping network, and the classification network;

[0021] A classification unit is used to input the adjacency matrix and the node feature matrix into the graph Transformer for classification, perform feature extraction on the large-scale attributeless graph to be classified through the stacked graph neural network based on random connections, and obtain a first attributeless graph feature output by the stacked graph neural network based on random connections, perform feature extraction on the large-scale attributeless graph to be classified through the local-global combined Transformer network, and obtain a second attributeless graph feature output by the local-global combined Transformer network, combine the first attributeless graph feature with the second attributeless graph feature to obtain a combined feature, map the combined feature through the linear mapping network to obtain a mapping graph representation feature, and input the mapping graph representation feature into the classification network to obtain a category of the large-scale attributeless graph.

[0022] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned large-scale attributeless graph classification method based on graph Transformer.

[0023] According to a fourth aspect of the present invention, there is provided a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the above-mentioned large-scale attributeless graph classification method based on graph Transformer is implemented.

[0024] According to a classification method, device and storage medium for large-scale attributeless graphs based on graph Transformer provided by the present invention, compared with the current attributeless graph classification method based on graph neural network, the present invention obtains the adjacency matrix and node feature matrix of the large-scale attributeless graph to be classified; and uses a random connection generation algorithm to determine the connection mode between each operation in the basic stacking unit; at the same time, calls a neural architecture search algorithm to determine the operation mode of each operation in the basic stacking unit, and determines the number of unit layers of the basic stacking unit; and based on the connection mode, operation mode and unit layer number between each operation in the basic stacking unit, constructs a stacked graph neural network based on random connection; at the same time, determines a multi-layer perceptron module, an LLA module, a CLA module and a global mean pooling module, and constructs a local-global combined Transformer network by the multi-layer perceptron module, the LLA module, the CLA module and the global mean pooling module; then determines a classification network and a linear mapping network, and the random-based A stacked graph neural network, a local-global combined Transformer network, a linear mapping network, and a classification network are connected to determine a graph Transformer; finally, the adjacency matrix and the node feature matrix are input into the graph Transformer for classification, and features of the large-scale attributeless graph to be classified are extracted by the stacked graph neural network based on random connections to obtain a first attributeless graph feature output by the stacked graph neural network based on random connections; features of the large-scale attributeless graph to be classified are extracted by the local-global combined Transformer network to obtain a second attributeless graph feature output by the local-global combined Transformer network; the first attributeless graph feature is combined with the second attributeless graph feature to obtain a combined feature; the combined feature is mapped by the linear mapping network to obtain a mapping graph representation feature; the mapping graph representation feature is input into the classification network to obtain a category of a large-scale attributeless graph.Therefore, by formulating a dedicated graph Transformer adapted to the large-scale attributeless graph, the large-scale attributeless graph is classified, so that the present invention can improve the classification accuracy of the large-scale attributeless graph. At the same time, the stacked graph neural network based on random connections in the graph Transformer can capture the first attributeless graph feature in the large-scale attributeless graph, and the LLA module (Local Linear Self-Attention, local grouped linear self-attention) in the local-global combined Transformer network is used to capture the fine-grained and short-distance information in the attributeless graph. Then, through the CLA module (Cross-Layer Attention, cross-group linear self-attention), the long-distance and global information in the attributeless graph is captured, and finally the second attributeless graph representation feature is obtained through global mean pooling. Finally, the large-scale attributeless graph is classified by performing a comprehensive analysis of the above feature information. During classification, not only the neighborhood node information of the attributeless graph can be mined, but also the information from the remote nodes in the large-scale attributeless graph can be effectively mined, avoiding the problem of only mining the shallow information of the attributeless graph, so that the present invention can improve the classification accuracy of the large-scale attributeless graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0026] Figure 1 A flow chart of a large-scale attributeless graph classification method based on graph transformer provided by an embodiment of the present invention is shown;

[0027] Figure 2 A schematic diagram of the structure of a graph Transformer model provided by an embodiment of the present invention is shown;

[0028] Figure 3 A flow chart of another large-scale attributeless graph classification method based on graph transformer provided by an embodiment of the present invention is shown;

[0029] Figure 4 A schematic diagram of a homogeneous / heterogeneous stacked graph neural network provided by an embodiment of the present invention is shown;

[0030] Figure 5 A schematic diagram of a random connection generation graph provided by an embodiment of the present invention is shown;

[0031] Figure 6 A schematic diagram of the structure of a large-scale attributeless graph classification device based on graph transformer provided by an embodiment of the present invention is shown;

[0032] Figure 7 A schematic diagram of the structure of another large-scale attributeless graph classification device based on graph transformer provided by an embodiment of the present invention is shown;

[0033] Figure 8 A schematic diagram of the physical structure of a computer device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0034] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0035] Existing attributeless graph classification methods only use graph neural networks to capture the domain structure information of graph nodes, which makes it difficult to effectively mine information from remote nodes in large-scale graphs. There are problems such as large errors and insufficient accuracy in classifying large-scale attributeless graphs.

[0036] In order to solve the above problems, the embodiment of the present invention provides a large-scale attributeless graph classification method based on graph transformer, such as Figure 1 As shown, the method includes:

[0037] 101. Obtain the adjacency matrix and node feature matrix of the large-scale attributeless graph to be classified.

[0038] For the embodiment of the present invention, a large-scale attributeless graph refers to a graph structure containing a large number of nodes and edges, but the nodes themselves do not have any additional attributes or features. This graph structure is widely used in many fields, such as the field of malware classification, using a graph data structure to explicitly express the execution logic of a program. A large-scale attributeless graph can be extracted from malware codes of different formats, such as binary executable files, vulnerability codes found in network traffic, attack codes linked together from gadgets in return-oriented programming attacks, etc. By classifying the large-scale attributeless graph corresponding to the execution logic of the program, the classification result of the malware can be obtained. Specifically, if the embodiment of the present invention is to perform malware identification, it is first necessary to construct a large-scale attributeless graph of the execution logic of the program. The adjacency matrix is ​​a form of expression of a large-scale attributeless graph, which is used to represent the connection relationship between the vertices in the graph. Then, the degree of the node is used as a feature, and the EBOH (Exponential Binned One-Hot Degree Encoding) encoding format is used to generate a node feature matrix. The EBOH encoding format uses a fixed number of buckets to represent the One-Hot degree, and the bucket range is designed as follows: from 0 to 2 αA set of one-to-one mappings of sequential buckets, where 1≤α≤3, α represents the number of buckets, then, from 2 α Start adding a set of buckets with many-to-one mappings, with exponential stepping of the bucket range, and an upper limit (inclusive) of 2 α+i , where i∈[1,k] and k∈N * , i represents the number of increased buckets, k represents the upper limit of the number of increased buckets, N * Represents a natural number. For example, the example of finding the EBOH encoding of the node with target degree value = 7792, where = 2, k = 3, the target degree value 7792 is mapped to its associated storage bucket (33 to ∞), then the bucket index (8) is represented by One-Hot encoding (the position where the value 8 exists is 1, otherwise it is 0), so that the encoding corresponding to each target degree value can be obtained, and finally the node feature matrix corresponding to the structure graph is generated according to the encoding corresponding to each target degree value Among them, H represents the node feature matrix, R represents the code corresponding to each node in the structure graph, v represents the number of nodes in the structure graph, and d represents the feature dimension.

[0039] 102. A random connection generation algorithm is used to determine the connection pattern between operations in a basic stacking unit.

[0040] 103. Call a neural architecture search algorithm to determine an operation mode of each operation in the basic stacking unit, and determine the number of unit layers of the basic stacking unit.

[0041] 104. Based on the connection mode, operation mode, and number of unit layers between the operations in the basic stacking unit, a stacking graph neural network based on random connections is constructed.

[0042] Among them, the basic stacking unit includes L GCN (Graph Convolutional Network) aggregation operations and a pre-processing operation 0 consisting of a multi-layer perceptron and a post-processing operation L+1, and the last layer is the readout layer.

[0043] Detailed design of the stacked graph neural network module based on random connections: First, execute the random connection generation algorithm (Algorithm 1) to determine the connections between the operations in the basic stacking unit, and then call the neural architecture search algorithm (Algorithm 2) to design the operation modes in the basic stacking unit. The network model that uses different units with different connection schemes and merging operations in the searched graph neural network is called a heterogeneous stacked graph neural network; in contrast, the network model that uses the same units in the searched graph neural network is called a homogeneous stacked graph neural network.

[0044] The implementation process of Algorithm 1 is as follows: Random connection generation: Generate a directed acyclic graph (DAG) based on the Erdos-Renyi random graph model, where the adjacency matrix of the DAG is a strict upper triangular matrix, and its entries are realizations of Bernoulli random variables with probability p. Then, the connection between each operation in the basic stacking unit is determined according to the adjacency matrix of the generated DAG, as follows:

[0045] (1) Input: number of nodes N, probability p (the edge between two nodes is connected with probability p);

[0046] (2) Generate Erdos-Renyi random graph (also referred to as ER random graph model, an important model at the intersection of graph theory and probability theory);

[0047] (3) Generate a directed acyclic graph by marking the nodes in the random graph. The adjacency matrix of the directed acyclic graph is a strict upper triangular matrix, and its entries are realizations of Bernoulli random variables with probability p;

[0048] (4) Determine the connections between the operations in the basic stacking unit based on the generated DAG (Directed Acyclic Graph), that is, the adjacency matrix of the large-scale attributeless graph to be classified.

[0049] The implementation process of Algorithm 2 is as follows: Neural architecture search algorithm: automatically search and determine the optimal stacked graph neural network structure, including the number of unit layers, connection mode, etc., as follows:

[0050] (1) Input: search space, where the search space consists of candidate operations:

[0051] ① For the merging operation of each layer feature: including concatenation (CONCAT), LSTM (Long Short-Term Memory, a special recurrent neural network architecture) unit, attention mechanism (ATT), summation (SUM), average (MEAN) and maximum (MAX) merging, expressed as follows:

[0052]

[0053] ②For the readout operation: global average pooling (GMEAN), global maximum pooling (GMAX), global sum pooling (GSUM), global sort pooling (GSORT), global attention pooling (GATT), set to sequence pooling (SET2SET), average maximum pooling (MEMA) are expressed as follows:

[0054]

[0055] (2) Use the Gumbel Softmax function (the Gumbel-Softmax function is a combination of the Gumbel distribution and the Softmax function. The Gumbel distribution is a continuous probability distribution, and the Softmax function is a function that maps multiple input values ​​to a probability distribution) to relax the search space, where G k =-log(-log(U k )) is a Gumbel random variable, U k is a uniform random variable, λ is the temperature of softmax, α= represents The neural architecture design that needs to be searched in . The relaxation process is shown as follows

[0056]

[0057] In the above formula, g(ο,α,k) represents the Gumbel Softmax function. represents the candidate operation, α represents the neural architecture design to be searched, and k is a subscript, indicating A value in represents the number of candidate operation sets, α k represents the kth neural architecture design to be searched, G j represents the jth Gumbel random variable, α j represents the jth neural architecture design that needs to be searched.

[0058] (3) Convert the problems involved in the graph neural network model architecture into a two-layer optimization problem:

[0059]

[0060] in, represents the search space, α=α m ,α r express The neural architecture design that needs to be searched in , W represents the model parameters in the search space, W * (α) represents the corresponding operation parameters after model training, and denote the training and validation loss functions respectively.

[0061] (4) The gradient descent method is used to optimize the parameters α and W to solve the optimization problem and return the design of each operation in the basic stacking unit.

[0062] Therefore, the connection mode, operation mode, and number of unit layers between the operations in the basic stacking unit can be determined in the above manner, and a stacked graph neural network based on random connections is constructed based on the above information. Furthermore, the fine-grained and short-distance information in the attributeless graph is captured by the LLA module in the local-global combined Transformer network, and the long-distance and global information in the attributeless graph is captured by the CLA module. This can capture the features in large-scale attributeless graphs more comprehensively, thereby improving the classification accuracy of large-scale attributeless graphs.

[0063] 105. Determine the multi-layer perceptron module, LLA module, CLA module, and global mean pooling module, and construct a local-global combined Transformer network by the multi-layer perceptron module, LLA module, CLA module, and global mean pooling module.

[0064] For the embodiment of the present invention, a multi-layer perceptron module, an LLA (Local Linear Self-Attention) module, a CLA (Cross-Layer Attention) module, and a global mean pooling module are trained and constructed, and then a local-global combined Transformer network is constructed by the multi-layer perceptron module, the LLA module, the CLA module, and the global mean pooling module.

[0065] In order to enable the graph Transformer classifier to better capture the implicit dependencies between graph nodes, such as long-range interactions or unobserved potential connections, the Transformer network is introduced. At the same time, in order to effectively reduce the complexity of self-attention calculation, a hierarchical self-attention mechanism is specially adopted. The detailed design of the Transformer network is as follows: First, the input features are mapped to f by a two-layer perceptron. I (·) to the latent space to obtain: H( 0 )∈R v×d

[0066] H (O) =f I (H)

[0067] In the above formula, H represents the node feature matrix H, f1(H) represents the linear mapping of the two-layer perceptron to H, R represents the real number set, v represents the graph node, and d represents the node feature dimension.

[0068] Furthermore, Algorithm 3 in the LLA module is called to calculate the pairwise influence between any pair of nodes in the group, and then Algorithm 4 in the CLA module is called to realize the communication between the node features of each group. Among them, the LLA module captures fine-grained and short-distance information, and the CLA module processes long-distance and global information. Finally, the graph node feature representation Z is globally mean pooled to obtain the graph representation GMEAN(Z).

[0069] The implementation process of Algorithm 3 is as follows: Locally-grouped linear attention: In order to reduce the computational complexity of the self-attention operation, the input features are spatially divided into non-overlapping groups so that self-attention communication occurs only within each group, as follows:

[0070] (1) Input: Graph node feature matrix mapped into latent space

[0071] (2) The feature matrix Divide into m groups to get

[0072] (3) In each local group, attention is calculated on the node features, that is, each node only interacts with other nodes in the same group. Each local group uses a linear attention function, which is defined as follows:

[0073]

[0074] where f Q 、f K 、f V They are all multilayer perceptrons, ||·|| represents the Frobenius norm, and 1 represents dimension is a column vector, the diag operation will The dimension column vector is transformed into diagonal matrix, β is the hyperparameter of the residual connection, Z i It combines all-pair attention propagation on all nodes in a local group and self-loop propagation. The former allows the model to capture the influence of other nodes, while the latter retains the information of the central node. i represents the i-th query vector, represents the feature matrix of the i-th node, represents the normalized value of the i-th query vector, K i represents the i-th key vector, represents the normalized value of the i-th key vector, V i represents the i-th value vector, D i represents the i-th scaling factor, and N represents the total number of vectors.

[0075] (4) Return the graph node feature representation Z of each local groupi .

[0076] The implementation process of Algorithm 4 is as follows: Cross-grouped linear attention: In order to expand the receptive field, a single representative is used to summarize the important information of each local group, and the representative is used to communicate with other local groups to realize cross-group information exchange.

[0077] (1) Input: Graph node feature representation Z of each local group i ;

[0078] (2) Perform subsampling operations on the graph node feature representations of each local group to obtain the representative features Φ(Z i );

[0079] (3) Combine the representative features of each group Φ(Z i ): i represents the i-th feature, H( 1 ) represents the concatenated features.

[0080] H (1) =[Φ(Z1);Φ(Z2);...;Φ(Z i )]

[0081] (4) In order to capture long-range and global information, H (1) Execute the linear attention function to achieve cross-group information exchange:

[0082]

[0083] In the above formula, Q represents the query vector of the features after cross-group exchange. The normalized vector of the query vector representing the features after cross-group exchange, H (1) represents the concatenated features, K represents the key vector of the features after cross-group exchange, represents the normalized vector of the key vector of the feature after cross-group exchange, V represents the value vector after cross-group exchange, and D represents the scaling factor.

[0084] (5) Return the graph node feature representation Z.

[0085] On the basis of capturing graph representation features based on randomly connected stacked graph neural networks, the fine-grained and short-distance information in the attributeless graph is captured through the LLA module in the newly designed local-global combined Transformer network, and the long-distance and global information in the attributeless graph is captured through the CLA module. This can capture the features of large-scale attributeless graphs more comprehensively, thereby improving the classification accuracy of large-scale attributeless graphs.

[0086] 106. Determine the classification network and linear mapping network, and determine the graph Transformer by stacking graph neural network based on random connection, local-global combined Transformer network, linear mapping network, and classification network.

[0087] For the embodiment of the present invention, based on the classification network and linear mapping network in the graph neural network (GNNs) architecture, a stacked graph neural network based on random connections and a Transformer network combining local and global are introduced, thereby determining a graph Transformer model, i.e., a classification model for large-scale attributeless graphs, by stacking graph neural networks based on random connections, a Transformer network combining local and global, a linear mapping network, and a classification network, with a structure as follows: Figure 2 As shown, Z0 represents the combined graph representation feature, Softmax is the function of the classification network, Represents the mapping graph obtained in the linear mapping network to represent the features.

[0088] 107. Input the adjacency matrix and the node feature matrix into the graph Transformer for classification, extract features of the large-scale attributeless graph to be classified through a stacked graph neural network based on random connections, and obtain the first attributeless graph feature, extract features of the large-scale attributeless graph to be classified through a local-global combined Transformer network, and obtain the second attributeless graph feature, combine the first attributeless graph feature with the second attributeless graph feature to obtain a combined feature, and map the combined feature through a linear mapping network to obtain a mapping graph representation feature, and input the mapping graph representation feature into the classification network to obtain the category of the large-scale attributeless graph.

[0089] For an embodiment of the present invention, after obtaining the adjacency matrix and node feature matrix of the large-scale attributeless graph to be classified, the adjacency matrix and the node feature matrix are input into the graph Transformer for classification. The graph Transformer model first learns the graph representation features based on the stacked graph neural network constructed based on the random connection architecture, that is, captures the first attributeless graph feature, and further captures the fine-grained and short-distance information in the attributeless graph through the LLA module in the local-global combined Transformer network, and then captures the long-distance and global information in the attributeless graph through the CLA module. Finally, the second attributeless graph feature is obtained through global mean pooling, the first attributeless graph feature is combined with the second attributeless graph feature to obtain a combined feature, and the combined feature is mapped through a linear mapping network to obtain a mapped graph representation feature, and the mapped graph representation feature is input into the classification network to obtain the category of the large-scale attributeless graph. Therefore, the embodiment of the present invention classifies the large-scale attributeless graph by formulating an exclusive graph Transformer adapted to the large-scale attributeless graph, so that the present invention can improve the classification accuracy of the large-scale attributeless graph. At the same time, the stacked graph neural network based on random connections in the graph Transformer can capture the first attributeless graph feature in the large-scale attributeless graph, and the LLA module in the local-global combined Transformer network can capture the fine-grained and short-distance information in the attributeless graph. The CLA module captures the long-distance and global information in the attributeless graph and finally obtains the second attributeless graph feature through global mean pooling. Finally, the above-mentioned feature information is comprehensively analyzed to classify the large-scale attributeless graph. When classifying, it can not only mine the neighborhood node information of the attributeless graph, but also effectively mine the information from the remote nodes in the large-scale attributeless graph, thereby improving the classification accuracy of the large-scale attributeless graph.

[0090] According to a classification method for large-scale attributeless graphs based on graph Transformer provided by the present invention, compared with the current graph classification method based on graph neural network, the present invention obtains the adjacency matrix and node feature matrix of the large-scale attributeless graph to be classified; and uses a random connection generation algorithm to determine the connection mode between each operation in the basic stacking unit; at the same time, calls a neural architecture search algorithm to determine the operation mode of each operation in the basic stacking unit, and determines the number of unit layers of the basic stacking unit; and based on the connection mode, operation mode, and unit layer number between each operation in the basic stacking unit, a stacked graph neural network based on random connection is constructed; at the same time, a multi-layer perceptron module, an LLA module, a CLA module, and a global mean pooling module are determined, and a local-global combined Transformer network is constructed by the multi-layer perceptron module, the LLA module, the CLA module, and the global mean pooling module; then a classification network and a linear mapping network are determined, and the stacked graph neural network based on random connection is constructed. A graph transformer is determined by using a graph neural network, a local-global combined Transformer network, a linear mapping network, and a classification network; finally, the adjacency matrix and the node feature matrix are input into the graph transformer for classification; the features of the large-scale attributeless graph to be classified are extracted by the stacked graph neural network based on random connections to obtain a first attributeless graph feature output by the stacked graph neural network based on random connections; the features of the large-scale attributeless graph to be classified are extracted by the local-global combined Transformer network to obtain a second attributeless graph feature output by the local-global combined Transformer network; the first attributeless graph feature is combined with the second attributeless graph feature to obtain a combined feature; the combined feature is mapped by the linear mapping network to obtain a mapping graph representation feature; the mapping graph representation feature is input into the classification network to obtain a category of a large-scale attributeless graph.Therefore, by formulating a dedicated graph Transformer that is compatible with the large-scale attributeless graph, the large-scale attributeless graph is classified, so that the present invention can improve the classification accuracy of the large-scale attributeless graph. At the same time, the stacked graph neural network based on random connections in the graph Transformer can capture the first attributeless graph feature in the large-scale attributeless graph, and the LLA module in the local-global combined Transformer network captures the fine-grained and short-distance information in the attributeless graph. The CLA module captures the long-distance and global information in the attributeless graph and finally obtains the second attributeless graph feature through global mean pooling. Finally, the large-scale attributeless graph is classified by performing a comprehensive analysis of the above feature information. During classification, not only the neighborhood node information of the attributeless graph can be mined, but also the information from the remote nodes in the large-scale attributeless graph can be effectively mined, avoiding the problem of only mining the shallow information of the attributeless graph. Therefore, the present invention can improve the classification accuracy of the large-scale attributeless graph.

[0091] Further, in order to better illustrate the above process of classifying a large-scale attributeless graph, as a refinement and extension of the above embodiment, the embodiment of the present invention provides another large-scale attributeless graph classification method based on graph transformer, such as Figure 3 As shown, the method includes:

[0092] 201. Obtain the adjacency matrix and node feature matrix of the large-scale attributeless graph to be classified.

[0093] For an embodiment of the present invention, for example, when malware is identified, an attributeless graph is generated based on the execution logic of the program, and then the attributeless graph is classified, so that the malware category can be identified, and the accuracy of malware identification can be improved. After determining a large-scale attributeless graph, the degree is used as the feature representation of the graph node, and the node feature matrix of the large-scale attributeless graph to be classified is generated using the EBOH encoding format (Exponential Binned One-Hot Degree Encoding), where the degree represents the number of adjacent vertices of the graph node.

[0094] Furthermore, the adjacency matrix is ​​a commonly used data structure in large-scale attributeless graphs to be classified. It is a matrix used to represent the adjacent relationship between vertices in the structure graph. For example, if G = (V, E) is an attributeless structure graph, where V is the vertex set in the structure graph, and E is the edge set in the structure graph, the adjacency matrix of G is a two-dimensional array, where each row and column represents a vertex. The values ​​of the elements in the adjacency matrix represent the relationship between the vertices in the structure graph. For undirected structure graphs, the adjacency matrix is ​​symmetric, and the elements on the main diagonal are all 0. For directed structure graphs, the adjacency matrix is ​​not necessarily symmetric, and the elements on the main diagonal may also be non-zero. The EBOH encoding format is further used to generate the node feature matrix of the large-scale attributeless graph.

[0095] 202. Determine the connection pattern between operations in the basic stacking unit using a random connection generation algorithm.

[0096] 203. Calling a neural architecture search algorithm to determine an operation mode of each operation in a basic stacking unit, and determining the number of unit layers of the basic stacking unit.

[0097] 204. Based on the connection mode, operation mode and number of unit layers between the operations in the basic stacking unit, a stacking graph neural network based on random connection is constructed.

[0098] For the embodiment of the present invention, the connection between the operations in the basic stacking unit can be determined by executing the random connection generation algorithm, and then the neural architecture search algorithm is called to design the operation modes in the basic stacking unit, such as Figure 4 As shown, the network model using different unit layers with different connection schemes and merging operations in the searched stacked graph neural network is called a heterogeneous stacked graph neural network; in contrast, the network model using the same unit in the searched stacked graph neural network is called a homogeneous stacked graph neural network, thereby being able to construct a stacked graph neural network based on random connections. Specifically, calling a neural architecture search algorithm to determine the operation mode of each operation in a basic stacked unit, and the method for determining the number of unit layers of a basic stacked unit includes: determining the search space of the basic stacked unit, and relaxing the search space using a preset probability distribution function; obtaining the required neural architecture of the model to be searched, the model parameters in the relaxed search space, the model operation parameters, the model training function, and the model verification loss function; optimizing the required neural architecture and the model operation parameters using a preset gradient descent method, and determining the operation mode of each operation in the basic stacked unit based on the optimized required neural architecture, the optimized model operation parameters, the model parameters, the model training function, and the model verification loss function, and determining the number of unit layers of the basic stacked unit.

[0099] Specifically, define the search space, such as the operation mode: First, it is necessary to define the types of operations that may be included in the basic stacking unit, such as convolution layers (different convolution kernel sizes, step sizes, padding methods, etc.), pooling layers (max pooling, average pooling, etc.), activation functions (ReLU, Tanh, etc.), normalization layers (BatchNorm, etc.), etc. These operation types constitute the search space of the operation mode, and use the preset probability distribution function to relax the search space (wherein the preset probability distribution function can be selected according to actual needs). Number of unit layers: At the same time, it is also necessary to define the search range of the number of unit layers. This usually depends on the specific application scenario and the limitations of computing resources. For example, the minimum and maximum values ​​of the number of unit layers can be set, or a certain strategy (such as gradually increasing the number of layers and evaluating the performance) can be adopted to determine the optimal number of layers. Design evaluation criteria: In order to evaluate the performance of different architectures (i.e., different operation modes and layer combinations), one or more evaluation criteria need to be designed. Common evaluation criteria include accuracy, loss function value, computational complexity (such as FLOPs), number of parameters, etc. These criteria will be used to evaluate the performance of candidate architectures during the search process and guide the search algorithm to find the best architecture. Selecting a search algorithm: There are many kinds of N algorithms, including random search, greedy search, gradient-based search (such as DARTS), reinforcement learning, evolutionary algorithms, etc. Each algorithm has its advantages and disadvantages and applicable scenarios. When selecting a search algorithm, it is necessary to consider the specific requirements of the problem, the limitations of computing resources, and the efficiency and effectiveness of the algorithm. Random search: Explore the search space by randomly selecting candidate architectures. Greedy search: Gradually select the current best architecture and build a new candidate architecture based on it. Gradient-based search (such as DARTS): Find the best architecture by optimizing the structural parameters of the neural network. Reinforcement learning: Use reinforcement learning algorithms (such as RNN controllers) to generate and evaluate candidate architectures. Evolutionary algorithm: Simulate the natural evolution process to search for the best architecture. Perform a search process: According to the selected search algorithm and the defined search space and evaluation criteria, a search process is performed to find the best basic stacking unit architecture and various model parameters. This usually involves multiple iterations, each of which generates a new candidate architecture, evaluates its performance, and updates the current best architecture, thereby determining the operation mode of each operation in the basic stacking unit and determining the number of unit layers of the basic stacking unit. Determine the final architecture: When the search process is over, the final basic stacked unit architecture is determined according to the evaluation criteria, including the operation mode of each operation and the number of unit layers. This architecture can then be used for subsequent training and deployment processes.

[0100] 205. Determine the multi-layer perceptron module, the LLA module, the CLA module, and the global mean pooling module, and construct a local-global combined Transformer network by using the multi-layer perceptron module, the LLA module, the CLA module, and the global mean pooling module.

[0101] Specifically, a hybrid architecture can be designed based on a multi-layer perceptron module, an LLA module, a CLA module, and a global mean pooling module to obtain a local-global combined Transformer network.

[0102] 206. Determine the classification network and linear mapping network, and determine the graph Transformer by stacking graph neural network based on random connection, local-global combined Transformer network, linear mapping network, and classification network.

[0103] The algorithm is based on the Transformer (a neural network model based on the attention mechanism) as the basic architecture, which includes a classification network and a linear mapping network. A stacked graph neural network based on random connections and a local-global combined Transformer network are introduced on the basic architecture to obtain a special graph Transformer.

[0104] Furthermore, after constructing the graph Transformer, it needs to be trained, such as Figure 5 As shown, Figure 5 The DAG (Directed Acyclic Graph, directed acyclic graph or large-scale attributeless graph) in the graph is obtained. Based on this, the method includes: obtaining a sample attributeless graph data set, wherein the sample attributeless graph data set includes sample attributeless graphs of multiple categories; using a preset five-fold cross-validation method to classify each sample attributeless graph in the sample attributeless graph data set to obtain the actual category corresponding to each sample attributeless graph; determining a sample adjacency matrix and a sample node feature matrix of each sample attributeless graph; inputting the sample adjacency matrix and the sample node feature matrix into the graph transformer to obtain the predicted category of the sample attributeless graph; constructing a loss function of the graph transformer based on the predicted category and the actual category of the same sample attributeless graph; based on the loss function, iteratively training the graph transformer to obtain a trained graph transformer; inputting the adjacency matrix and the node feature matrix into the trained graph transformer for classification to obtain the category of the large-scale attributeless graph.

[0105] Specifically, first, a sample attributeless graph dataset is generated, wherein the sample attributeless graph dataset includes sample large-scale attributeless graphs of multiple categories, so that a sample adjacency matrix of each sample large-scale attributeless graph can be generated. Then, the degree of the node is used as a feature, and the sample node feature matrix of each sample large-scale attributeless graph is generated in the EBOH encoding format. Then, the sample attributeless graph dataset is divided by a five-fold cross-validation method, that is, each sample attributeless graph in the sample attributeless graph dataset is classified to obtain the actual category corresponding to each sample attributeless graph. Then, the sample adjacency matrix and the sample node feature matrix are input into a graph transformer to obtain the predicted category of the sample attributeless graph. According to the predicted category and the actual category of the same sample attributeless graph, a loss function of the graph transformer is constructed. Finally, based on the loss function, the graph transformer is iteratively trained to obtain a trained graph transformer whose loss meets the preset requirements. Finally, the trained graph transformer is used to classify the large-scale attributeless graph to be classified.

[0106] 207. Input the adjacency matrix and the node feature matrix into a randomly connected stacked graph neural network for feature extraction, and obtain a first attributeless graph feature output by the randomly connected stacked graph neural network.

[0107] Specifically, the adjacency matrix and the node feature matrix are jointly input into a randomly connected stacked graph neural network to extract short-distance node dependency features, and an attributeless graph feature output by the randomly connected stacked graph neural network is obtained, namely, the first attributeless graph feature.

[0108] 208. Input the node feature matrix into the local-global combined Transformer network for feature extraction to obtain a second attributeless graph feature output by the local-global combined Transformer network.

[0109] For the embodiment of the present invention, after capturing the first attributeless graph feature of the large-scale attributeless graph, in order to make the captured information more comprehensive, it is also necessary to capture the second attributeless graph feature of the large-scale attributeless graph. Based on this, step 208 specifically includes: mapping the node feature matrix to a preset latent space through the multi-layer perceptron module to obtain a graph node feature matrix; dividing the graph node feature matrix into a preset number of groups through the LLA module to obtain an intra-group graph node feature matrix corresponding to each group; within each of the groups, using a preset linear attention function to interact each graph node of the graph node feature matrix with other nodes in the corresponding group to obtain a graph node feature matrix after the interaction nodes; capturing the pairwise influence information between any node pairs in the graph node feature matrix after the interaction nodes, and in each of the pairwise influence The method comprises the following steps: determining the influence information of the central node from the response information; determining the short-distance fine-grained features corresponding to each group based on the influence information of the central node corresponding to each group; performing a subsampling operation on the short-distance fine-grained features of each group through the CLA module to obtain the representative graph features of each group; splicing the representative graph features of each group respectively to obtain the spliced ​​graph features of each group; performing cross-group feature exchange on the spliced ​​graph features of each group using a preset linear attention function to obtain the exchanged spliced ​​graph features corresponding to each group; determining the long-distance global features based on the exchanged spliced ​​graph features corresponding to each group; extracting features of the long-distance global features through the global mean pooling module to obtain the attributeless graph features output by the Transformer network combining local and global of the large-scale attributeless graph to be classified.

[0110] Specifically, the input node feature matrix is ​​first mapped to a preset latent space (a space determined according to actual needs) through a multi-layer perceptron module to obtain a graph node feature matrix. The graph node feature matrix is ​​then input into the LLA module to obtain an intra-group graph node feature matrix corresponding to each group. Then, within each group, each graph node in the graph node feature matrix is ​​interacted with other nodes in the corresponding group using a preset linear attention function to obtain a graph node feature matrix after the interaction nodes. The pairwise influence information between any node pairs in the graph node feature matrix after the interaction nodes is captured, and the influence information of the central node is determined in each pairwise influence information. Based on the influence information of the central node corresponding to each group, the short-distance fine-grained features corresponding to each group are determined. Then, the corresponding graph nodes of each group are The short-distance fine-grained features are sub-sampled through the CLA module to obtain representative graph features of each group, and the representative graph features of each group are horizontally spliced ​​to obtain spliced ​​graph features of each group containing more invisible features, and the preset linear attention function is used to perform cross-group feature exchange on the spliced ​​graph features of each group to obtain the exchanged spliced ​​graph features corresponding to each group, and the long-distance global features are determined based on the exchanged spliced ​​graph features corresponding to each group, and the long-distance global features are extracted through the global mean pooling module, that is, the fine-grained and short-distance information in the attributeless graph is captured through the LLA module, and the long-distance and global information of the attributeless graph is captured through the CLA module, and the above information is subjected to global mean pooling to obtain the second attributeless graph feature. Therefore, the attributeless graph features of large-scale attributeless graphs can be extracted through the stacked graph neural network based on random connections, the fine-grained and short-distance information in the attributeless graph is captured through the LLA module in the Transformer network, and the long-distance and global information of the attributeless graph is captured through the CLA module. The above information is then pooled by the global mean to obtain the second attributeless graph feature. Through comprehensive analysis of the above feature information, all feature information of large-scale attributeless graphs can be captured, thereby improving the classification accuracy of large-scale attributeless graphs.

[0111] 209. Combine the first attributeless graph feature with the second attributeless graph feature to obtain a combined feature, map the combined feature through a linear mapping network to obtain a mapping graph representation feature, input the mapping graph representation feature into a classification network to obtain a category of a large-scale attributeless graph.

[0112] For the embodiment of the present invention, after capturing the first attributeless graph feature and the second attributeless graph feature of the large-scale attributeless graph, it is necessary to perform a comprehensive analysis on the above features to achieve the classification of the large-scale attributeless graph. Based on this, step 209 includes: determining the weight hyperparameters corresponding to the second attributeless graph feature output by the local-global combined Transformer network and the first attributeless graph feature output by the randomly connected stacked graph neural network; based on the weight hyperparameters, adding the second attributeless graph feature output by the local-global combined Transformer network and the first attributeless graph feature output by the randomly connected stacked graph neural network to obtain the combined feature. Finally, the combined feature is linearly mapped using a linear mapping network to obtain a mapping graph representation feature, and finally, the classification network is used to achieve the classification of the attributeless graph based on the mapping graph representation feature.

[0113] Specifically, the graph representation features obtained by the randomly connected stacked graph neural network module and the graph representation features obtained by the local-global combined Transformer network are combined as shown in the following formula, where α is a weight hyperparameter, RAN-SGNN represents a randomly connected stacked graph neural network module, (Randomly wired Stacking-based Graph Neural Network), GMEAN (Z) represents the attribute-free graph feature output by the local-global combined Transformer network, and f O It is a linear mapping network. Finally, the Softmax function in the classification network completes the graph classification task. Z0 represents the combined graph representation feature. The representation map represents the features.

[0114] Z O =(1-α)GMEAN(Z)+αRAN-SGNN(H,A),

[0115]

[0116] According to another large-scale attributeless graph classification method, device and storage medium based on graph Transformer provided by the present invention, compared with the current method using graph kernel or graph classification method based on CNN, the present invention obtains the adjacency matrix and node feature matrix of the large-scale attributeless graph to be classified; and uses a random connection generation algorithm to determine the connection mode between each operation in the basic stacking unit; at the same time, calls a neural architecture search algorithm to determine the operation mode of each operation in the basic stacking unit, and determines the number of unit layers of the basic stacking unit; and based on the connection mode, operation mode and unit layer number between each operation in the basic stacking unit, constructs a stacked graph neural network based on random connection; at the same time, determines a multi-layer perceptron module, an LLA module, a CLA module and a global mean pooling module, and constructs a local-global combined Transformer network by the multi-layer perceptron module, the LLA module, the CLA module and the global mean pooling module; then determines a classification network and a linear mapping network, and the basic A graph transformer is determined by using a randomly connected stacked graph neural network, a local-global combined Transformer network, a linear mapping network, and a classification network; finally, the adjacency matrix and the node feature matrix are input into the graph transformer for classification, feature extraction is performed on the large-scale attributeless graph to be classified by the randomly connected stacked graph neural network, and a first attributeless graph feature output by the randomly connected stacked graph neural network is obtained; feature extraction is performed on the large-scale attributeless graph to be classified by the local-global combined Transformer network, and a second attributeless graph feature output by the local-global combined Transformer network is obtained; the first attributeless graph feature is combined with the second attributeless graph feature to obtain a combined feature; the combined feature is mapped by the linear mapping network to obtain a mapping graph representation feature; the mapping graph representation feature is input into the classification network to obtain a category of a large-scale attributeless graph.Therefore, by formulating a dedicated graph Transformer that is compatible with the large-scale attributeless graph, the large-scale attributeless graph is classified, so that the present invention can improve the classification accuracy of the large-scale attributeless graph. At the same time, the stacked graph neural network based on random connections in the graph Transformer can capture the first attributeless graph feature in the large-scale attributeless graph, and the LLA module in the local-global combined Transformer network captures the fine-grained and short-distance information in the attributeless graph. The CLA module captures the long-distance and global information in the attributeless graph and finally obtains the second attributeless graph feature through global mean pooling. Finally, the large-scale attributeless graph is classified by performing a comprehensive analysis of the above feature information. During classification, not only the neighborhood node information of the attributeless graph can be mined, but also the information from the remote nodes in the large-scale attributeless graph can be effectively mined, avoiding the problem of only mining the shallow information of the attributeless graph. Therefore, the present invention can improve the classification accuracy of the large-scale attributeless graph.

[0117] Further, as Figure 1 The specific implementation of the present invention provides a large-scale attributeless graph classification device based on graph transformer, such as Figure 6 As shown, the device includes: an acquisition unit 31, a generation unit 32, a first determination unit 33, a first construction unit 34, a second construction unit 35, a second determination unit 36, and a classification unit 37.

[0118] The acquisition unit 31 may be used to acquire the adjacency matrix and node feature matrix of the large-scale attributeless graph to be classified.

[0119] The generating unit 32 may be used to determine the connection mode between the operations in the basic stacking unit by using a random connection generating algorithm.

[0120] The first determination unit 33 can be used to call a neural architecture search algorithm to determine the operation mode of each operation in the basic stacking unit, and determine the number of unit layers of the basic stacking unit.

[0121] The first construction unit 34 can be used to construct a stacked graph neural network based on random connections based on the connection mode, operation mode, and number of unit layers between the operations in the basic stacking unit.

[0122] The second construction unit 35 can be used to determine a multi-layer perceptron module, an LLA module, a CLA module, and a global mean pooling module, and construct a local-global combined Transformer network by using the multi-layer perceptron module, the LLA module, the CLA module, and the global mean pooling module.

[0123] The second determination unit 36 ​​can be used to determine the classification network and the linear mapping network, and determine the graph Transformer by the stacked graph neural network based on random connections, the local-global combined Transformer network, the linear mapping network, and the classification network.

[0124] The classification unit 37 can be used to input the adjacency matrix and the node feature matrix into the graph Transformer for classification, perform feature extraction on the large-scale attributeless graph to be classified through the stacked graph neural network based on random connections, and obtain a first attributeless graph feature output by the stacked graph neural network based on random connections, perform feature extraction on the large-scale attributeless graph to be classified through the local-global combined Transformer network, and obtain a second attributeless graph feature output by the local-global combined Transformer network, combine the first attributeless graph feature with the second attributeless graph feature to obtain a combined feature, map the combined feature through the linear mapping network to obtain a mapping graph representation feature, and input the mapping graph representation feature into the classification network to obtain a category of the large-scale attributeless graph.

[0125] In a specific application scenario, in order to extract the first attributeless graph features for the large-scale attributeless graph to be classified, the classification unit 37 can be specifically used to input the adjacency matrix and the node feature matrix into the randomly connected stacked graph neural network for feature extraction, and obtain the first attributeless graph features output by the randomly connected stacked graph neural network; input the node feature matrix into the local-global combined Transformer network for feature extraction, and obtain the second attributeless graph features output by the local-global combined Transformer network.

[0126] In specific application scenarios, in order to determine the attribute-free graph features of the local-global combined Transformer network output, such as Figure 7 As shown, the classification unit 37 includes a mapping module 371 , a division module 372 , an interaction module 373 , a capture module 374 , and a first determination module 375 .

[0127] The mapping module 371 can be used to map the node feature matrix to a preset latent space through the multi-layer perceptron module to obtain a graph node feature matrix.

[0128] The division module 372 can be used to divide the graph node feature matrix into a preset number of groups through the LLA module to obtain the intra-group graph node feature matrix corresponding to each group.

[0129] The interaction module 373 can be used to interact each graph node of the graph node feature matrix with other nodes in the corresponding group within each group using a preset linear attention function to obtain a graph node feature matrix after the interaction nodes.

[0130] The capture module 374 can be used to capture the pairwise influence information between any node pairs in the graph node feature matrix after the interaction nodes, and determine the influence information of the central node in each pairwise influence information.

[0131] The first determination module 375 can be used to determine the short-distance fine-grained features corresponding to each group based on the influence information of the central node corresponding to each group; perform subsampling operations on the short-distance fine-grained features of each group through the CLA module to obtain representative graph features of each group; respectively splice the representative graph features of each group to obtain the spliced ​​graph features of each group; use the preset linear attention function to perform cross-group feature exchange on the spliced ​​graph features of each group to obtain the exchanged spliced ​​graph features corresponding to each group; determine the long-distance global features based on the exchanged spliced ​​graph features corresponding to each group; extract features from the long-distance global features through the global mean pooling module to obtain the attributeless graph features output by the Transformer network that combines local and global features of the large-scale attributeless graph to be classified.

[0132] In a specific application scenario, in order to determine the combined features, the first determination module 375 can be specifically used to determine the weight hyperparameters corresponding to the attributeless graph features output by the local-global combined Transformer network and the attributeless graph features output by the stacked graph neural network based on random connections; based on the weight hyperparameters, the attributeless graph features output by the local-global combined Transformer network and the attributeless graph features output by the stacked graph neural network based on random connections are added to obtain the combined features.

[0133] In a specific application scenario, in order to determine the operation mode of each operation in the basic stacking unit and determine the number of unit layers of the basic stacking unit, the first determination unit 33 includes: a second determination module 331 , an acquisition module 332 , and an optimization module 333 .

[0134] The second determination module 331 may be used to determine a search space of the basic stacking unit and relax the search space using a preset probability distribution function.

[0135] The acquisition module 332 can be used to obtain the required neural architecture of the model to be searched, the model parameters in the relaxed search space, the model operation parameters, the model training function, and the model verification loss function.

[0136] The optimization module 333 can be used to optimize the required neural architecture and model operation parameters using a preset gradient descent method, and determine the operation mode of each operation in the basic stacking unit and the number of unit layers of the basic stacking unit based on the optimized required neural architecture, optimized model operation parameters, model parameters, model training function, and model verification loss function.

[0137] In a specific application scenario, in order to train the graph Transformer, the device further includes a training unit 38.

[0138] The training unit 38 can be used to obtain a sample attributeless graph data set, wherein the sample attributeless graph data set includes multiple categories of sample attributeless graphs; use a preset five-fold cross-validation method to classify each sample attributeless graph in the sample attributeless graph data set to obtain the actual category corresponding to each sample attributeless graph; determine the sample adjacency matrix and the sample node feature matrix of each sample attributeless graph; input the sample adjacency matrix and the sample node feature matrix into the graph transformer to obtain the predicted category of the sample attributeless graph; construct the loss function of the graph transformer based on the predicted category and the actual category of the same sample attributeless graph; based on the loss function, iteratively train the graph transformer to obtain the trained graph transformer.

[0139] The classification unit 37 can be specifically used to input the adjacency matrix and the node feature matrix into the trained graph Transformer for classification to obtain the category of the large-scale attribute-free graph.

[0140] In a specific application scenario, in order to obtain the node feature matrix of the large-scale attributeless graph to be classified, the generation unit 32 can also be used to use the degree as the feature representation of the graph nodes in the large-scale attributeless graph to be classified, and use the EBOH encoding format to represent it to generate the node feature matrix of the large-scale attributeless graph to be classified.

[0141] It should be noted that for other corresponding descriptions of the functional modules involved in the large-scale attributeless graph classification device based on graph transformer provided in the embodiment of the present invention, please refer to Figure 1 The corresponding description of the method shown will not be repeated here.

[0142] Based on the above Figure 1The method shown, accordingly, an embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the following steps are implemented: obtaining an adjacency matrix and a node feature matrix of a large-scale attributeless graph to be classified; using a random connection generation algorithm to determine the connection mode between each operation in a basic stacking unit; calling a neural architecture search algorithm to determine the operation mode of each operation in the basic stacking unit, and determining the number of unit layers of the basic stacking unit; based on the connection mode, operation mode, and unit layer between each operation in the basic stacking unit, constructing a stacked graph neural network based on random connections; determining a multi-layer perceptron module, an LLA module, a CLA module, and a global mean pooling module, and constructing a local-global combined Transformer network by using the multi-layer perceptron module, the LLA module, the CLA module, and the global mean pooling module; determining a classification network and a linear mapping network, and using the stacked graph neural network based on random connections, the local - A globally combined Transformer network, a linear mapping network, and a classification network are used to determine a graph Transformer; the adjacency matrix and the node feature matrix are input into the graph Transformer for classification; the features of the large-scale attributeless graph to be classified are extracted through the stacked graph neural network based on random connections to obtain a first attributeless graph feature output by the stacked graph neural network based on random connections; the features of the large-scale attributeless graph to be classified are extracted through the local-global combined Transformer network to obtain a second attributeless graph feature output by the local-global combined Transformer network; the first attributeless graph feature is combined with the second attributeless graph feature to obtain a combined feature; the combined feature is mapped through the linear mapping network to obtain a mapping graph representation feature; the mapping graph representation feature is input into the classification network to obtain a category of a large-scale attributeless graph.

[0143] Based on the above Figure 1 The method shown and Figure 6 The embodiment of the device shown in the figure, the embodiment of the present invention also provides a physical structure diagram of a computer device, such as Figure 8As shown, the computer device includes: a processor 41, a memory 42, and a computer program stored in the memory 42 and executable on the processor, wherein the memory 42 and the processor 41 are both arranged on a bus 43, and the processor 41 implements the following steps when executing the program: obtaining an adjacency matrix and a node feature matrix of a large-scale attributeless graph to be classified; determining a connection mode between operations in a basic stacking unit using a random connection generation algorithm; calling a neural architecture search algorithm to determine an operation mode of each operation in the basic stacking unit, and determining the number of unit layers of the basic stacking unit; constructing a stacking graph neural network based on random connections based on the connection mode, operation mode, and unit layer number between operations in the basic stacking unit; determining a multi-layer perceptron module, an LLA module, a CLA module, and a global mean pooling module, and constructing a local-global combined Transformer network by using the multi-layer perceptron module, the LLA module, the CLA module, and the global mean pooling module; determining a classification network and a linear mapping network, by using the A graph Transformer is determined based on a randomly connected stacked graph neural network, a local-global combined Transformer network, a linear mapping network, and a classification network; the adjacency matrix and the node feature matrix are input into the graph Transformer for classification; features are extracted from the large-scale attributeless graph to be classified by the randomly connected stacked graph neural network to obtain a first attributeless graph feature output by the randomly connected stacked graph neural network; features are extracted from the large-scale attributeless graph to be classified by the local-global combined Transformer network to obtain a second attributeless graph feature output by the local-global combined Transformer network; the first attributeless graph feature is combined with the second attributeless graph feature to obtain a combined feature; the combined feature is mapped by the linear mapping network to obtain a mapping graph representation feature; the mapping graph representation feature is input into the classification network to obtain a category of the large-scale attributeless graph.

[0144] Through the technical solution of the present invention, the present invention obtains the adjacency matrix and node feature matrix of the large-scale attributeless graph to be classified; and uses the random connection generation algorithm to determine the connection mode between each operation in the basic stacking unit; at the same time, calls the neural architecture search algorithm to determine the operation mode of each operation in the basic stacking unit, and determines the number of unit layers of the basic stacking unit; and based on the connection mode, operation mode, and unit layer number between each operation in the basic stacking unit, constructs a stacked graph neural network based on random connection; at the same time, determines a multi-layer perceptron module, an LLA module, a CLA module, and a global mean pooling module, and constructs a local-global combined Transformer network by the multi-layer perceptron module, the LLA module, the CLA module, and the global mean pooling module; then determines a classification network and a linear mapping network, and the stacked graph neural network based on random connection, the local-global combined Transfor mer network, linear mapping network, and classification network to determine a graph Transformer; finally, the adjacency matrix and the node feature matrix are input into the graph Transformer for classification, and features of the large-scale attributeless graph to be classified are extracted by the stacked graph neural network based on random connections to obtain a first attributeless graph feature output by the stacked graph neural network based on random connections; features of the large-scale attributeless graph to be classified are extracted by the local-global combined Transformer network to obtain a second attributeless graph feature output by the local-global combined Transformer network; the first attributeless graph feature is combined with the second attributeless graph feature to obtain a combined feature; the combined feature is mapped by the linear mapping network to obtain a mapping graph representation feature; the mapping graph representation feature is input into the classification network to obtain a category of a large-scale attributeless graph. Therefore, by formulating a dedicated graph Transformer that is compatible with the large-scale attributeless graph, the large-scale attributeless graph is classified, so that the present invention can improve the classification accuracy of the large-scale attributeless graph. At the same time, the stacked graph neural network based on random connections in the graph Transformer can capture the first attributeless graph feature in the large-scale attributeless graph, and the LLA module in the local-global combined Transformer network captures the fine-grained and short-distance information in the attributeless graph. The CLA module captures the long-distance and global information in the attributeless graph and finally obtains the second attributeless graph feature through global mean pooling. Finally, the large-scale attributeless graph is classified by performing a comprehensive analysis of the above feature information. During classification, not only the neighborhood node information of the attributeless graph can be mined, but also the information from the remote nodes in the large-scale attributeless graph can be effectively mined, avoiding the problem of only mining the shallow information of the attributeless graph. Therefore, the present invention can improve the classification accuracy of the large-scale attributeless graph.

[0145] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0146] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A large-scale attributeless graph classification method based on graph transformer, characterized in that: include: Obtaining an adjacency matrix and a node feature matrix of a large-scale attributeless graph to be classified, wherein the large-scale attributeless graph to be classified is applicable to the field of paper network classification, the field of social network classification, and the field of malware classification; A random connection generation algorithm is used to determine the connection pattern between operations in the basic stacking unit; Calling a neural architecture search algorithm to determine an operation mode of each operation in the basic stacking unit and to determine the number of unit layers of the basic stacking unit; Based on the connection mode, operation mode, and number of unit layers between the operations in the basic stacking unit, a stacking graph neural network based on random connection is constructed; Determine a multi-layer perceptron module, an LLA module, a CLA module, and a global mean pooling module, and construct a local-global combined Transformer network by using the multi-layer perceptron module, the LLA module, the CLA module, and the global mean pooling module; Determine a classification network and a linear mapping network, and determine a graph Transformer by using the randomly connected stacked graph neural network, the local-global combined Transformer network, the linear mapping network, and the classification network; The adjacency matrix and the node feature matrix are input into the graph Transformer for classification, and features of the large-scale attributeless graph to be classified are extracted through the stacked graph neural network based on random connections to obtain a first attributeless graph feature output by the stacked graph neural network based on random connections, and features of the large-scale attributeless graph to be classified are extracted through the local-global combined Transformer network to obtain a second attributeless graph feature output by the local-global combined Transformer network, and the first attributeless graph feature is combined with the second attributeless graph feature to obtain a combined feature, and the combined feature is mapped through the linear mapping network to obtain a mapping graph representation feature, and the mapping graph representation feature is input into the classification network to obtain a category of the large-scale attributeless graph.

2. The method according to claim 1, characterized in that The extracting features of the large-scale attributeless graph to be classified by the randomly connected stacked graph neural network to obtain a first attributeless graph feature outputted by the randomly connected stacked graph neural network includes: Inputting the adjacency matrix and the node feature matrix into the randomly connected stacked graph neural network for feature extraction, and obtaining a first attributeless graph feature output by the randomly connected stacked graph neural network; The extracting features of the large-scale attributeless graph to be classified by the local-global combined Transformer network to obtain a second attributeless graph feature output by the local-global combined Transformer network includes: The node feature matrix is ​​input into the local-global combined Transformer network for feature extraction to obtain a second attributeless graph feature output by the local-global combined Transformer network.

3. The method according to claim 2, characterized in that The step of inputting the node feature matrix into the local-global combined Transformer network for feature extraction to obtain a second attributeless graph feature output by the local-global combined Transformer network includes: Mapping the node feature matrix to a preset latent space through the multi-layer perceptron module to obtain a graph node feature matrix; Dividing the graph node feature matrix into a preset number of groups through the LLA module to obtain a graph node feature matrix within each group; In each of the groups, each graph node of the graph node feature matrix is ​​interacted with other nodes in the corresponding group using a preset linear attention function to obtain a graph node feature matrix after the interacted nodes; Capturing the pairwise influence information between any node pairs in the graph node feature matrix after the interaction nodes, and determining the influence information of the central node in each pairwise influence information; Based on the influence information of the central node corresponding to each group, the short-distance fine-grained features corresponding to each group are determined; Performing a subsampling operation on the short-distance fine-grained features of each group through the CLA module to obtain representative graph features of each group; The representative graph features of each group are spliced ​​respectively to obtain the spliced ​​graph features of each group; The preset linear attention function is used to perform cross-group feature exchange on the mosaic features of each group to obtain the corresponding mosaic features after exchange; Determining long-range global features based on the exchanged mosaic features corresponding to each group; The long-distance global features are extracted through the global mean pooling module to obtain attribute-free graph features output by a Transformer network combining local and global features of the large-scale attribute-free graph to be classified.

4. The method according to claim 2, characterized in that: The combining the first attributeless graph feature and the second attributeless graph feature to obtain a combined feature includes: Determine the weight hyperparameters corresponding to the second attributeless graph feature output by the local-global combined Transformer network and the first attributeless graph feature output by the randomly connected stacked graph neural network; Based on the weight hyperparameter, the second attributeless graph feature output by the local-global combined Transformer network and the first attributeless graph feature output by the randomly connected stacked graph neural network are added to obtain a combined feature.

5. The method according to claim 1, characterized in that: The calling of the neural architecture search algorithm to determine the operation mode of each operation in the basic stacking unit and the number of unit layers of the basic stacking unit includes: Determining a search space of the basic stacking unit, and relaxing the search space using a preset probability distribution function; Obtain the required neural architecture of the model to be searched, model parameters in the relaxed search space, model operation parameters, model training function, and model validation loss function; The required neural architecture and model operation parameters are optimized by using a preset gradient descent method, and based on the optimized required neural architecture, the optimized model operation parameters, the model parameters, the model training function, and the model verification loss function, the operation mode of each operation in the basic stacking unit is determined, as well as the number of unit layers of the basic stacking unit.

6. The method according to claim 1, characterized in that After determining the classification network and the linear mapping network, the graph Transformer is determined by the stacked graph neural network based on random connections, the local-global combined Transformer network, the linear mapping network, and the classification network, the method further includes: Acquire a sample attributeless graph data set, wherein the sample attributeless graph data set includes sample attributeless graphs of multiple categories; The preset five-fold cross validation method is used to classify each sample attributeless graph in the sample attributeless graph data set to obtain the actual category corresponding to each sample attributeless graph; Determine the sample adjacency matrix and sample node feature matrix of each sample attribute-free graph; Input the sample adjacency matrix and the sample node feature matrix into the graph transformer to obtain the predicted category of the sample attribute-free graph; Based on the predicted category and the actual category of the same sample attribute-free graph, construct a loss function of the graph Transformer; Based on the loss function, iteratively train the graph Transformer to obtain a trained graph Transformer; The adjacency matrix and the node feature matrix are input into the graph transformer for classification to obtain the categories of large-scale attribute-free graphs, including: The adjacency matrix and the node feature matrix are input into the trained graph transformer for classification to obtain the category of the large-scale attribute-free graph.

7. The method according to claim 1, characterized in that Obtain the node feature matrix of the large-scale attribute-free graph to be classified, including: The degree is used as the feature representation of the graph nodes in the large-scale attributeless graph to be classified, and the node feature matrix of the large-scale attributeless graph to be classified is generated using the EBOH encoding format.

8. A large-scale attributeless graph classification device based on graph transformer, characterized in that: include: An acquisition unit, used for acquiring an adjacency matrix and a node feature matrix of a large-scale attributeless graph to be classified, wherein the large-scale attributeless graph to be classified is applicable to the field of paper network classification, the field of social network classification, and the field of malware classification; A generation unit, used for determining a connection pattern between operations in a basic stacking unit by using a random connection generation algorithm; A first determination unit, configured to call a neural architecture search algorithm to determine an operation mode of each operation in the basic stacking unit, and to determine the number of unit layers of the basic stacking unit; A first construction unit is used to construct a stacked graph neural network based on random connections based on the connection mode, operation mode, and number of unit layers between the operations in the basic stacking unit; The second construction unit is used to determine a multi-layer perceptron module, an LLA module, a CLA module, and a global mean pooling module, and construct a local-global combined Transformer network by using the multi-layer perceptron module, the LLA module, the CLA module, and the global mean pooling module; A second determination unit is used to determine a classification network and a linear mapping network, and a graph Transformer is determined by the stacked graph neural network based on random connections, the local-global combined Transformer network, the linear mapping network, and the classification network; A classification unit is used to input the adjacency matrix and the node feature matrix into the graph Transformer for classification, perform feature extraction on the large-scale attributeless graph to be classified through the stacked graph neural network based on random connections, and obtain a first attributeless graph feature output by the stacked graph neural network based on random connections, perform feature extraction on the large-scale attributeless graph to be classified through the local-global combined Transformer network, and obtain a second attributeless graph feature output by the local-global combined Transformer network, combine the first attributeless graph feature with the second attributeless graph feature to obtain a combined feature, map the combined feature through the linear mapping network to obtain a mapping graph representation feature, and input the mapping graph representation feature into the classification network to obtain a category of the large-scale attributeless graph.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images