A graph node classification method based on partition pooling hypergraph neural network

Abstract

This invention discloses a graph node classification method based on a partitioned pooling hypergraph neural network, belonging to the field of image processing technology. It treats entities in a judicial image as nodes in a graph, models the relationships between entities as a graph, and transforms the target classification task into a graph node classification task. This fully utilizes the relationships between entities, treating the deep feature vectors extracted by Fast R-CNN as node feature vectors, constructing adjacency relationships between nodes based on the similarity of their feature vectors, and finally using graph node classification technology to complete the target recognition task in the judicial image. This invention improves the speed and accuracy of image classification and recognition through pooling technology.

Description

Technical Field

[0001] This invention relates to the field of image classification technology, and more specifically to a graph node classification method based on a partitioned pooling hypergraph neural network. Background Technology

[0002] When analyzing forensic images, it is necessary to identify entity targets in the images, such as people, cars, and daggers. Traditional target recognition methods use Convolutional Neural Networks (CNNs) to accomplish this task, such as Region-CNN (RCNN) and Fast Region-CNN (Fast RCNN). However, these techniques do not utilize the relationships between entity targets in the image; instead, they make predictions independently. A graph is a non-Euclidean dataset composed of nodes and their connections. These connections can be represented by an adjacency matrix. A large amount of data in daily life can be modeled as graphs, such as citation networks, social networks, and protein structures. Graph node classification refers to using the structural information of the graph and the feature matrix of its nodes to predict the type of each node. Graph node classification is an important research area in image recognition and classification.

[0003] The field of research utilizing deep learning methods for graph node classification has seen a surge of work. Based on the type of neural network used, these methods can be categorized into graph neural networks (GNNs) and hypergraph neural networks (HNNs). GNNs primarily include Graph Convolutional Networks (GCNs), Graph Attention Networks (GATs), and Graph Isomorphism Networks (GINs). GNNs use the feature matrices of nodes and the adjacency matrix of the graph as input, employing a "neighborhood information aggregation" approach. They update their own features using the features of neighboring nodes and finally obtain the classification prediction results for each node through a simple linear classifier. Hypergraph neural networks, on the other hand, are neural network structures based on hypergraph modeling. A hypergraph is a generalized graph structure composed of nodes and hyperedges. While an edge in a simple graph can only connect two nodes, a hyperedge can connect multiple nodes. This overcomes the limitation of simple graphs, which can only represent binary relationships between nodes. The connection relationships between nodes and hyperedges in a hypergraph can be represented using an incidence matrix.

[0004] In recent years, Hypergraph Neural Networks (HGNNs) based on hypergraph modeling have shown promising results in research areas such as citation networks, text classification, and pose estimation. HGNNs take the feature matrix and association matrix of each node as input and update the node's feature representation using the hypergraph Laplacian transformation. Since hypergraph neural networks cannot be directly used to process graph data, HGNNs transform the graph adjacency matrix of node self-connections into a hypergraph association matrix to process graph data, converting ordinary graphs into hypergraphs. The transformed hypergraph is then processed using HGNNs. By transforming graph-modeling data into hypergraph-modeling data, hypergraph neural networks can be used for graph node classification tasks.

[0005] To enhance the generalization ability of neural networks, reduce computational load, and obtain information from different levels, pooling techniques are often introduced. Pooling is a technique that coarsens the data, reduces computational load, and enhances generalization ability. In graph neural networks, there are mainly cluster-based and node-selection-based pooling methods, with DIFFPOOL and SAGPOOL being typical examples. They reduce the graph size by clustering and selecting nodes, respectively. In hypergraph neural networks, there are hyperedge-based pooling methods, such as HM-GNN, which reduces the hypergraph size by pooling hyperedges. Pooling techniques are frequently used as an important structural component of neural networks, playing a crucial role in large-scale image content understanding tasks and effectively improving the efficiency and effectiveness of forensic image analysis.

[0006] However, existing pooling methods cannot be directly applied to graph node classification tasks. Since graph node classification requires neural networks to retain all node features, existing graph pooling techniques reduce the number of nodes. Hypergraph neural network pooling techniques also lead to a reduction in the number of nodes, making them unsuitable for node classification and graph-based judicial image analysis tasks. Furthermore, hypergraph neural networks based on hyperedge pooling use hyperedges as pooling objects, pooling several hyperedges at a time, which may result in significant loss of hypergraph information. When applied to entity recognition in judicial images, this may lead to poor entity recognition performance.

[0007] Therefore, how to achieve accurate image recognition and classification based on pooling hypergraph neural networks is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0008] In view of this, the present invention provides a graph node classification method based on a partitioned pooling hypergraph neural network. This method treats entities in a judicial image as nodes in a graph, models the relationships between entities as a graph, and transforms the target classification task into a graph node classification task. This fully utilizes the relationships between entities, treats the deep feature vectors extracted by Fast R-CNN as node feature vectors, constructs adjacency relationships between nodes based on the similarity of their feature vectors, and finally uses graph node classification technology to complete the target recognition task in the judicial image.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A graph node classification method based on a partitioned pooling hypergraph neural network includes the following steps:

[0011] Step 1: Obtain the judicial image and use Fast RCNN to extract the high-order feature vectors of all entity targets in the judicial image;

[0012] Step 2: Take the entity targets in the judicial image as nodes, and the higher-order feature vectors as node feature vectors. The node feature vectors of all nodes constitute a node feature matrix. Calculate the cosine similarity between any two node feature vectors. Take the two nodes corresponding to the node feature vectors whose cosine similarity exceeds the set similarity threshold as adjacent nodes to obtain the initial adjacency matrix and construct the original graph.

[0013] Step 3: Transform the original graph into a hypergraph, construct the hypergraph based on the adjacency matrix, obtain the hypergraph incidence matrix, and initialize the number of iterations; initialize the number of iterations to 0.

[0014] Step 4: The hypergraph convolutional layer of the partitioned pooling hypergraph neural network calculates the hypergraph Laplacian matrix based on the hypergraph association matrix, and updates the node feature matrix using the hypergraph Laplacian matrix;

[0015] Step 5: The regional pooling layer of the hypergraph neural network calculates the comprehensive score of nodes based on the hypergraph association matrix and the node feature matrix. Based on the comprehensive score of nodes, it selects the nodes to be retained within the hyperedge of the hypergraph, deletes other nodes, and updates the hypergraph association matrix and the node feature matrix.

[0016] Step 6: The readout layer of the partitioned pooling hypergraph neural network obtains the hyperedge feature matrix based on the hypergraph association matrix and node feature matrix, and increments the iteration count by 1;

[0017] Step 7: Check if the number of iterations equals the maximum number of iterations. If yes, proceed to step 8; otherwise, return to step 4. The maximum number of iterations is set to 3.

[0018] Step 8: Classify the nodes in Step 2 based on the hyperedge feature matrix to achieve judicial image classification.

[0019] Preferably, the original graph constructed in step 2 is represented by a node feature matrix and an adjacency matrix composed of the node feature vectors of all nodes.

[0020] Preferably, in step 3, transforming the original graph into a hypergraph involves connecting each node in the original graph to its neighboring nodes using a hyperedge; and calculating the hypergraph incidence matrix based on the adjacency matrix, H = A + I. Let E represent the hypergraph incidence matrix, where E is the number of hyperedges and N is the number of entity targets. Initially, E = N. Let A represent the adjacency matrix and I represent the identity matrix.

[0021] Preferably, the specific implementation process of step 4 is as follows:

[0022] Step 41: Calculate the node degree and hyperedge degree of the hypergraph based on the hypergraph incidence matrix, expressed as:

[0023]

[0024]

[0025] Where d(v) represents the degree of the v-th node; d(e) represents the degree of the e-th hyperedge; the degree of all nodes forms the degree matrix, and the degree of all hyperedges forms the degree matrix; W(e) represents the weight of the e-th hyperedge (default is 1); H(v,e) represents the value in the hypergraph association matrix corresponding to the v-th node and the e-th hyperedge, H(v,e)=1 indicates that the v-th node is connected to the e-th hyperedge, and H(v,e)=0 indicates that the v-th node is not connected to the e-th hyperedge;

[0026] Step 42: Calculate the Laplacian matrix Δ of the hypergraph based on the node degree matrix and the hyperedge degree matrix, expressed as:

[0027]

[0028] in Represents the node degree matrix; Represents the hyperboundary degree matrix; This represents the weight matrix of the hyperedge; W defaults to a diagonal matrix of all 1s.

[0029] Step 43: Input the hypergraph Laplacian matrix into the hypergraph convolutional layer to perform hypergraph convolution, and obtain the updated node feature matrix, represented as:

[0030] X (k) =σ(ΔX) (k-1) Θ)

[0031] Among them, X(k-1) This represents the node feature matrix before hypergraph convolution; Let F represent the learnable parameters; U represent the dimension of the node feature vector of the entity target; σ represent the dimension of the hidden layer in the partitioned pooling hypergraph neural network; and σ is the calibrated linear unit activation function. This represents the node feature matrix after hypergraph convolution.

[0032] Preferably, the specific implementation process of step 5 is as follows:

[0033] Step 51: Calculate the node type score S1, the expression is:

[0034] S1 = XW s1 +b s1

[0035] in, Learnable parameters for scoring node types. X represents the learnable offset of the node type score; X represents the node feature matrix after hypergraph convolution update; U represents the hidden layer dimension of the partitioned pooling hypergraph neural network.

[0036] Step 52: Calculate the importance score of a node to a hyperedge, expressed as:

[0037] S2(v, e) = ((X(v)||X E (e))W s2 +b s2 )*H(v,e)

[0038]

[0039] Where S2(v, e) represents the importance of the v-th node to the e-th hyperedge; The learnable parameters represent the importance score of a node to a hyperedge. X is the learnable offset of the importance score of a node to a hyperedge; X(v) represents the node feature vector of the v-th node; X E (e) represents the hyperedge feature vector of the e-th hyperedge, which is the average of the feature vectors of all nodes in hyperedge e; concatenate the hyperedge feature vector X. E (e) and node feature vector X(v), and pass through a fully connected layer to obtain the importance score S2(v,e) of node v to hyperedge e;

[0040] Step 53: Calculate the overall node score S based on the node type score and the node's importance score to the hyperedge. The expression is:

[0041] S(v,e)=sigmoid(S1(v)+S2(v,e))*H(v,e)

[0042] Where S2(v, e) represents the overall node score of the v-th node in the e-th hyperedge; S1(v) represents the node type score of the v-th node;

[0043] Step 54: Update the node feature matrix X using the node comprehensive score, the expression is:

[0044]

[0045] Where X(v) represents the node feature vector of the v-th node; ⊙ represents Hadamard multiplication;

[0046] Step 55: Perform edge pooling on all hyperedges in the hypergraph. Based on the overall node score S, decide whether to retain or delete nodes within the hyperedges. The nodes to be retained are:

[0047] idx(e)=topk(S[:,e],keep(e)),

[0048]

[0049] Where idx(e) represents the node retained in the e-th hyperedge; pr represents the pooling rate; topk is a function to find the node number corresponding to the highest keep(e) score; S[:,e] represents the comprehensive node score of all nodes connected to the e-th hyperedge, and S[:,e] represents {S[1,e],S[2,e],…,S[n,e]};

[0050] Step 56: In the hypergraph incidence matrix, if node v∈idx(e), set H(v,e)=1, otherwise H(v,e)=0, delete the all-zero rows in the hypergraph incidence matrix H, and delete the corresponding node eigenvectors in the node eigenma matrix X, thus completing the region pooling operation of the hypergraph.

[0051] Preferably, the specific implementation process of step 6 is as follows:

[0052] Step 61: Calculate the hyperedge feature vector recorded in each iteration for each hyperedge, expressed as:

[0053]

[0054] Among them, X R (e) (k) Let X(v) represent the hyperedge feature vector read out by the readout layer in the kth iteration; N represents the number of entity targets; X(v) represents the node feature vector of the vth node; all hyperedge feature vectors constitute the hyperedge feature matrix.

[0055] Preferably, in step 8, several sets of hyperedge feature matrices are obtained through the above iterations. The sum of these sets of hyperedge feature matrices is used to obtain the final hyperedge feature matrix. The final hyperedge feature matrix is ​​then passed through the fully connected layer of the partitioned pooling hypergraph neural network to output the node classification results, corresponding to the entity target classification results, thus completing the judicial image classification. Several sets of hyperedge feature matrices are obtained through several iterations.

[0056] As can be seen from the above technical solution, compared with the prior art, this invention discloses a graph node classification method based on a partitioned pooling hypergraph neural network. It uses a partitioned pooling hypergraph neural network model to complete the graph node classification task, overcoming the shortcoming that pooling technology cannot be applied to graph node classification tasks. By applying pooling technology, the model's running speed can be accelerated, its accuracy improved, and computer memory consumption reduced, thereby improving the speed and accuracy of judicial image classification and recognition. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0058] Figure 1 The attached figure is a schematic diagram of the overall architecture of the graph node classification system based on partitioned pooling hypergraph neural network provided by the present invention;

[0059] Figure 2 The attached figure is a schematic diagram of the hypergraph construction provided by the present invention;

[0060] Figure 3 The attached figure is a schematic diagram of hypergraph convolution operation provided by the present invention;

[0061] Figure 4 The attached figure is a schematic diagram of the node comprehensive score calculation process provided by the present invention;

[0062] Figure 5 The attached figure is a schematic diagram of the node pooling process based on the node comprehensive score provided by the present invention;

[0063] Figure 6 The attached figure is a schematic diagram of updating the correlation matrix and reading out the hyperedge feature matrix provided by the present invention;

[0064] Figure 7 The attached figure is a schematic diagram of the final classification process provided by the present invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] This invention discloses a graph node classification method based on a partitioned pooling hypergraph neural network, comprising the following steps:

[0067] S1: Obtain the judicial image and use Fast RCNN to extract the high-order feature vectors of all entity targets in the judicial image;

[0068] S2: The entity targets in the judicial image are taken as nodes, the higher-order feature vectors are taken as node feature vectors, the cosine similarity between any two node feature vectors is calculated, and the two nodes corresponding to the feature vectors of nodes whose cosine similarity exceeds the set similarity threshold are taken as adjacent nodes, and the initial adjacency matrix is ​​constructed.

[0069] S3: Construct the hypergraph incidence matrix based on the adjacency matrix;

[0070] S4: Calculate the Laplacian matrix of the hypergraph based on the hypergraph incidence matrix;

[0071] S5: Update the node feature matrix based on the hypergraph Laplacian matrix;

[0072] S6: Calculate the overall node score based on the hypergraph association matrix and node feature matrix;

[0073] S7: Select the nodes to be retained within the hyperedge based on the node comprehensive score, delete other nodes, and update the hypergraph association matrix and node feature matrix;

[0074] S8: Obtain the hyperedge feature matrix based on the hypergraph incidence matrix and node feature matrix;

[0075] S9: Repeat S4 to S83 times;

[0076] S10: Classify nodes based on the hyperedge feature matrix to achieve judicial image classification.

[0077] This invention transforms the task of recognizing entities in judicial images into a graph node classification task.

[0078] Example

[0079] like Figure 1The diagram shows the overall architecture of a graph node classification method based on a partitioned pooling hypergraph neural network. The model construction includes four processing stages: image feature extraction, hypergraph construction, convolutional pooling, and node classification.

[0080] Image feature extraction stage: Fast R-CNN is used to extract feature vectors of entity targets in the image;

[0081] Hypergraph construction phase: First, the adjacency relationship between nodes is determined based on the feature vector of the entity target, thereby constructing a normal graph, and then the normal graph is transformed into a hypergraph;

[0082] Convolutional pooling stage: Perform hypergraph convolution and region-specific pooling;

[0083] Node classification stage: Calculate the final node classification result.

[0084] The convolutional pooling stage consists of three identical stacked modules, each of which can be divided into a hypergraph convolutional layer (S4 to S5), a region pooling layer (S6 to S7), and a readout layer (S8).

[0085] 1) Image feature extraction stage

[0086] Given a judicial image that needs to be classified into entities, we first use Fast-RCNN to extract the high-order feature vectors of the entities in the image. Entities are treated as nodes in the graph, and their high-order feature vectors are considered as the initial node feature vectors. The cosine similarity between any two node feature vectors is calculated. If the cosine similarity is higher than a threshold, the nodes are considered adjacent. An adjacency matrix is ​​used to represent the adjacency relationship between nodes, thus completing the construction of the initial graph.

[0087] Given a legal image that needs to be classified into entities, assuming there are N entities in the image, use Fast R-CNN to extract the high-order feature matrices corresponding to the entities in the image. Where F is the dimension of the node feature vectors of the entity objects extracted by Fast R-CNN. Each entity in the image is considered a node in the graph, and the high-order feature matrix composed of the node feature vectors of all entity objects is considered the initial node feature matrix of the node. Next, the adjacency matrix A∈{0,1} of the graph is constructed. N×NCalculate the cosine similarity between the feature vectors of any two nodes in the node feature matrix: sim(i,j) = cos_similarity(X(i), X(j)), where X(i) represents the initial feature vector of the i-th node, and sim(i,j) represents the cosine similarity between the i-th and j-th nodes. If sim(i,j) > γ, then set A(i,j) = 1; otherwise, A(i,j) = 0, where γ ∈ [0,1] is a manually set threshold. The above processing completes the construction of the original graph.

[0088] 2) Hypergraph construction stage

[0089] Given the adjacency matrix and node feature matrix of the original graph, this stage transforms the original graph into a hypergraph. First, the adjacency matrix of the original graph is transformed into a self-connected adjacency matrix. Each node in the original graph and its neighbors are considered as nodes connected by a hyperedge. Based on this, the original graph is transformed into a hypergraph, represented by an incidence matrix, and the node feature matrix of the original graph is used as the node feature matrix of the hypergraph. The process is as follows: Figure 2 As shown. After processing, the correlation matrix and node feature matrix are used as inputs for the next stage.

[0090] After image feature extraction is completed, the original image consists of an initial node feature matrix. And the adjacency matrix A∈{0,1} N×N This is represented as follows. To transform a graph into a hypergraph, each node in the original graph and its neighboring nodes are considered to be connected by a hyperedge, thus constructing a hypergraph. According to the hypergraph construction method described above, we have H = A + I, where... Let E represent the hypergraph incidence matrix, where E is the number of hyperedges, initially E = N, and I is the identity matrix.

[0091] If the original graph has 5 nodes, the connections between the nodes are as follows: Figure 2 The original graph is shown in the diagram. First, the structure of the original graph is represented by the adjacency matrix A. Then, the graph incidence matrix H is obtained by calculating H = A + I, thus completing the hypergraph construction process.

[0092] 3) Convolutional Pooling Stage

[0093] Based on the input hypergraph association matrix and node feature matrix, this stage performs hypergraph convolution and region pooling operations, updates the node feature matrix and reduces the hypergraph size, and reads out the hyperedge feature matrix for final classification.

[0094] First, the hypergraph convolutional layer calculates the hypergraph Laplacian matrix based on the correlation matrix, and then updates the node feature matrix using the hypergraph Laplacian matrix. Next, the correlation matrix and node feature matrix are input into the region pooling layer for node pooling within hyperedges. The region pooling layer selects to retain or delete nodes based on the overall score of nodes within each hyperedge. The overall node score consists of two parts: a node type score and a node importance score for the hyperedge. The node type score assesses the importance of a node based on its type; the node importance score measures the importance of the node to its hyperedge. After calculating the overall node score, the layer retains the k nodes with the highest overall scores within each hyperedge, deletes the remaining nodes, ensuring that at least one node is retained for each hyperedge, and updates the correlation matrix and node feature matrix based on the node selection results. This completes the region pooling operation. Finally, the correlation matrix and node feature matrix are input into the readout layer, which calculates and records the hyperedge feature matrix based on the correlation matrix and node feature matrix. Then the correlation matrix and node feature matrix are input into the next module, and the operations in the convolutional pooling stage 3) are continued until there is no next module.

[0095] ① Hypergraph Convolutional Layer

[0096] Calculate the hypergraph Laplacian matrix based on the hypergraph association matrix, and update the node feature matrix based on the hypergraph Laplacian matrix; after the hypergraph convolutional layer processing is completed, update the node feature matrix X. (k) The correlation matrix H is used as the input to the region pooling layer. For ease of representation, X... (k) It will still be recorded as X.

[0097] The obtained hypergraph association matrix H and the initial node feature matrix X are used as inputs to the hypergraph convolutional layer. The hypergraph convolutional layer updates the feature representation of X using the hypergraph convolution operation in the hypergraph neural network. Hypergraph convolution can be understood as the aggregation of information from nodes to hyperedges, and then the transmission of information from hyperedges back to nodes, thereby updating the node feature matrix in the hypergraph. The formula for hypergraph convolution is:

[0098] X (k) =σ(ΔX) (k-1) Θ) (1)

[0099] Where Δ represents the Laplacian matrix of the hypergraph, X (k-1) This represents the node feature matrix before hypergraph convolution. These are learnable parameters, where U represents the hidden layer dimension of the neural network, and σ is the Rectified Linear Unit (ReLU) activation function. This represents the node feature matrix after hypergraph convolution. The formula for calculating the hypergraph Laplacian matrix is:

[0100]

[0101] in Represents the node degree matrix, Represents the hypermarginality matrix. W represents the weight matrix of the hyperedge; it defaults to a diagonal matrix of all 1s. The degree of the v-th node is:

[0102]

[0103] The hyperedge degree of the e-th hyperedge is:

[0104]

[0105] For example, such as Figure 3 As shown, the input hypergraph contains 5 hyperedges and 5 nodes, the hypergraph incidence matrix is ​​H, and the input node feature matrix is ​​X. (k-1) First, calculate the node degree matrix D according to formula (3). v The hypermarginality matrix D is calculated according to formula (4). e Then, according to formula (2), the hypergraph Laplacian matrix Δ is calculated; finally, according to formula (1), the updated node feature matrix X is calculated. (k) .

[0106] ②Regional pooling layer

[0107] The region-based pooling layer operates on a per-hyperedge basis, calculating the pooling score based on the comprehensive evaluation of nodes within each hyperedge. Delete or retain nodes within the superedge. The overall score S of a node is determined by the node type score. And the importance score of nodes to hyperedges Together, we determine that the comprehensive score S(v, e) of the v-th node in the e-th superedge is:

[0108] S(v,e)=sigmoid(S1(vD+S2(v,e))*H(v,e) (5)

[0109] The overall score for a node considers both the node type score and the node's importance score to the hyperedge. The calculation process for the node type score and the node's importance score to the hyperedge will be explained below.

[0110] First, the node type score is calculated. The node type score reflects the assessment of the importance of that type of node. For example, in natural language, the semantics of a sentence often hinges on a few key words. The formula for calculating the node type score is:

[0111] S1 = XW n +b s1 (6)

[0112] in For learnable parameters, It is a learnable offset.

[0113] Calculate the importance score of a node to its hyperedge. Taking natural language as an example, the same word has different importance in different sentences; therefore, calculating the importance of a node to different hyperedges can also reflect the importance of the node. The importance S2(v, e) of the v-th node to the e-th hyperedge is calculated as follows:

[0114] S2(v, e) = ((XCv)||X E (e))W s2 +b s2 )*H(v,e) (7)

[0115]

[0116] in Represents the learnable parameters. It is a learnable offset. Equation (8) means that the average of the eigenvectors of all nodes in the hyperedge e is taken as the eigenvector X of the hyperedge. E (e) Then concatenate the hyperedge eigenvector X E (e) and node feature vector X(v), and pass through a fully connected layer to obtain the importance score S2(v,e) of node v to hyperedge e.

[0117] like Figure 3 The node feature matrix X and the correlation matrix H are used as inputs to the region pooling layer. First, the node type score S1 is calculated according to formula (6). Then, the hyperedge feature matrix X is calculated according to formula (8). E Then, the importance score S2 of the node to the hyperedge is calculated according to formula (7). Finally, the comprehensive score S of the node is calculated according to formula (5). Figure 4 The process of calculating the node comprehensive score S from the node feature matrix X and the correlation matrix H is demonstrated.

[0118] After obtaining the comprehensive score S of a node, the node feature matrix X is updated using the comprehensive score. The update process of the node feature matrix is ​​as follows:

[0119]

[0120] Where ⊙ represents Hadama multiplication.

[0121] Finally, based on the overall node score S, the decision is made to retain or delete nodes within the hyperedge. The nodes to be retained are:

[0122] idx(e)=topk(S[:,e],keep(e)),

[0123]

[0124] Where idx(e) represents the node retained in the e-th hyperedge, pr represents the pooling rate, and topk is a function to calculate the node number corresponding to the highest keep(e) scores. The edge-pooling operation shown in formula (10) is performed on all hyperedges. If node v∈idx(e), H(v,e) = 1; otherwise, H(v,e) = 0. Finally, all zero rows in the hypergraph association matrix H are deleted, and the corresponding eigenvectors in the node feature matrix X are also deleted, completing the region-based pooling operation on the hypergraph.

[0125] Update the feature matrix of the node according to formula (9). Calculate the number of nodes to be retained for each hyperedge according to formula (10), and select the node idx to be retained for each hyperedge. Figure 5 The process of updating the node feature matrix X based on the node's comprehensive score and selecting keep(e) nodes to retain is demonstrated.

[0126] The partitioned pooling hypergraph neural network consists of three convolutional pooling modules. Each module includes a hypergraph convolutional layer and a partitioned pooling layer, performing a hypergraph convolution-partitioned pooling operation.

[0127] The convolutional pooling module performs three hypergraph convolution-region pooling operations.

[0128] ③ Readout layer

[0129] The readout layer records the hierarchical structure information of the hypergraph, which is ultimately used for node classification. The hyperedge e recorded in the k-th convolutional pooling module is X. R (e) (k) The calculation formula is:

[0130]

[0131] This completes the hypergraph convolution-region pooling operation for one convolutional pooling module in the convolutional pooling stage. The same operation is performed on all convolutional pooling modules in the convolutional pooling stage until all modules have been traversed. Figure 6 As shown.

[0132] 4) Node classification stage

[0133] The module processing stage records three sets of hyperedge feature matrices. Since the hypergraph construction method used in the image feature extraction stage (1) allows a hyperedge to be considered a node in the original graph, classifying hyperedges can be seen as classifying nodes in the original graph. First, the three sets of hyperedge feature matrices are summed to obtain the final hyperedge feature matrix. Then, a fully connected layer outputs the final node classification prediction result. In this application, the node classification result is the entity target classification result in the judicial image, such as... Figure 7 As shown.

[0134] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0135] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A graph node classification method based on a partitioned pooling hypergraph neural network, characterized in that, The following steps are involved: Step 1: Obtain the judicial image and use Fast RCNN to extract the high-order feature vectors of all entity targets in the judicial image; Step 2: Take the entity targets in the judicial image as nodes, and the higher-order feature vectors as node feature vectors. The node feature vectors of all nodes constitute a node feature matrix. Calculate the cosine similarity between any two node feature vectors. Take the two nodes corresponding to the node feature vectors whose cosine similarity exceeds the set similarity threshold as adjacent nodes to obtain the initial adjacency matrix and construct the original graph. Step 3: Transform the original graph into a hypergraph, construct the hypergraph based on the adjacency matrix, obtain the hypergraph incidence matrix, and initialize the number of iterations; Step 4: The hypergraph convolutional layer of the partitioned pooling hypergraph neural network calculates the hypergraph Laplacian matrix based on the hypergraph association matrix, and updates the node feature matrix using the hypergraph Laplacian matrix; Step 5: The regional pooling layer of the hypergraph neural network calculates the comprehensive score of nodes based on the hypergraph association matrix and the node feature matrix. Based on the comprehensive score of nodes, it selects the nodes to be retained within the hyperedge of the hypergraph, deletes other nodes, and updates the hypergraph association matrix and the node feature matrix. Step 6: The readout layer of the partitioned pooling hypergraph neural network obtains the hyperedge feature matrix based on the hypergraph association matrix and node feature matrix, and increments the iteration count by 1; Step 7: Is the number of iterations equal to the maximum number of iterations? If yes, proceed to step 8; otherwise, return to step 4. Step 8: Classify the nodes in Step 2 based on the hyperedge feature matrix to achieve judicial image classification.

2. The graph node classification method based on partitioned pooling hypergraph neural network according to claim 1, characterized in that, The original graph constructed in step 2 is represented by a node feature matrix and an adjacency matrix composed of the node feature vectors of all nodes.

3. The graph node classification method based on a partitioned pooling hypergraph neural network according to claim 1, characterized in that, In step 3, transforming the original graph into a hypergraph involves connecting each node in the original graph to its neighboring nodes using a hyperedge; the hypergraph incidence matrix is ​​then calculated based on the adjacency matrix, H = A + I. Let E represent the hypergraph incidence matrix, where E is the number of hyperedges and N is the number of entity targets. Initially, E = N; A represents the adjacency matrix. I is the identity matrix.

4. The graph node classification method based on a partitioned pooling hypergraph neural network according to claim 1, characterized in that, The specific implementation process of step 4 is as follows: Step 41: Calculate the node degree and hyperedge degree of the hypergraph based on the hypergraph incidence matrix, expressed as: Where d(v) represents the node degree of the v-th node; d(e) represents the hyperedge degree of the e-th hyperedge; the node degrees of all nodes form a node degree matrix, and the hyperedge degrees of all hyperedges form a hyperedge degree matrix; W(e) represents the hyperedge weight of the e-th hyperedge; H(v,e) represents the value in the hypergraph association matrix corresponding to the v-th node and the e-th hyperedge; E represents the number of hyperedges; and N is the number of entity targets. Step 42: Calculate the Laplacian matrix Δ of the hypergraph based on the node degree matrix and the hyperedge degree matrix, expressed as: in, Represents the node degree matrix; Represents the hypermarginality matrix; The weight matrix represents the hyperedge weights; Step 43: Input the hypergraph Laplacian matrix into the hypergraph convolutional layer to perform hypergraph convolution, and obtain the updated node feature matrix, represented as: X (k) =σ(ΔX (k-1) I) Among them, X (k-1) This represents the node feature matrix before hypergraph convolution; The parameters to be learned are: F represents the dimension of the node feature vector of the entity target; U represents the dimension of the hidden layer of the partitioned pooling hypergraph neural network; and σ is the calibrated linear unit activation function. This represents the node feature matrix after hypergraph convolution.

5. The graph node classification method based on a partitioned pooling hypergraph neural network according to claim 1, characterized in that, The specific implementation process of step 5 is as follows: Step 51: Calculate the node type score S1, the expression is: S1=XW s1 +b s1 in, Learnable parameters for scoring node types. X represents the learnable offset of the node type score; X represents the node feature matrix after hypergraph convolution update; U represents the hidden layer dimension of the partitioned pooling hypergraph neural network. Step 52: Calculate the importance score of a node to a hyperedge, expressed as: S2(v,e)=((X(v)||X E (e))W s2 +b s2 )*H(v,e) Where S2(v, e) represents the importance of the v-th node to the e-th hyperedge; Learnable parameters representing the importance score of a node to a hyperedge. X is the learnable offset of the importance score of a node to a hyperedge; X(v) represents the node feature vector of the v-th node; X E (e) represents the hyperedge feature vector of the e-th hyperedge; d(e) represents the hyperedge degree of the e-th hyperedge; H(v,e) represents the value in the hypergraph association matrix corresponding to the v-th node and the e-th hyperedge; N is the number of entity targets; Step 53: Calculate the overall node score S based on the node type score and the node's importance score to the hyperedge. The expression is: S(v,e)=sigmoid(S1(v)+S2(v,e))*H(v,e) Where S(v, e) represents the overall node score of the v-th node in the e-th hyperedge; S1(ν) represents the node type score of the v-th node; Step 54: Update the node feature matrix using the node comprehensive score, the expression is: Where X(v) represents the node feature vector of the v-th node; ⊙ represents Hadamard multiplication; E represents the number of hyperedges; and d(v) represents the node degree of the v-th node. Step 55: Perform edge pooling on all hyperedges in the hypergraph. Based on the overall node score S, decide whether to retain or delete nodes within the hyperedges. The nodes to be retained are: idx(e)=topk(S[:,e],keep(e)), Where idx(r) represents the node retained in the e-th hyperedge; pr represents the pooling rate; topk is a function to find the node number corresponding to the highest keep(e); S[:,e] represents the comprehensive node score of all nodes connected to the e-th hyperedge; Step 56: In the hypergraph incidence matrix, if node v∈idx(e), set H(v,e)=1, otherwise H(v,e)=0, delete the row of all zeros in the hypergraph incidence matrix, and delete the corresponding node eigenvector in the node eigenma matrix to complete the region pooling operation of the hypergraph.

6. The graph node classification method based on a partitioned pooling hypergraph neural network according to claim 1, characterized in that, The specific implementation process of step 6 is as follows: The eigenvector of each hyperedge recorded in each iteration is calculated using the following expression: Among them, X R (e) (k) Let X(v) represent the hyperedge feature vector read out by the readout layer in the k-th iteration; N represents the number of entity targets; X(v) represents the node feature vector of the v-th node; all hyperedge feature vectors constitute the hyperedge feature matrix; d(e) represents the hyperedge degree of the e-th hyperedge; H(v,e) represents the value in the hypergraph association matrix corresponding to the v-th node and the e-th hyperedge.

7. The graph node classification method based on a partitioned pooling hypergraph neural network according to claim 1, characterized in that, In step 8, several sets of hyperedge feature matrices are obtained through iteration. The sum of these sets of hyperedge feature matrices is used to obtain the final hyperedge feature matrix. The final hyperedge feature matrix is ​​then used to output the node classification results and corresponding entity target classification results through the fully connected layer of the partition pooling hypergraph neural network, thus completing the judicial image classification.

Images