A multi-label classification method for disciplinary inspection clues based on heterogeneous graph neural network

Abstract

The present application discloses a multi-label classification method for disciplinary clues based on a heterogeneous graph neural network, constructs and utilizes a text multi-label classification model based on a text heterogeneous graph for classification, and improves classification efficiency. The text is preprocessed and a heterogeneous graph structure is constructed. The vector representations of different types of nodes in the heterogeneous graph structure are initialized, the obtained node vector representation and the heterogeneous graph structure are input into the heterogeneous graph neural network, and the obtained sentence node vector representation is input into the BILSTM neural network. The obtained content vector representation and sentence node vector representation are input into the fully connected network after passing through the attention mechanism, and the label classification result is output, thereby realizing node updates for different edge information and node type information. It solves the technical problems of low classification efficiency, limited text representation ability, and inability to update nodes according to different edge information and node type information in the existing multi-label classification method for disciplinary clues.

Description

Technical Field

[0001] The present application belongs to the technical field of text multi-label classification, and in particular relates to a multi-label classification method for disciplinary inspection clues based on heterogeneous graph neural networks. Background Art

[0002] Disciplinary inspection clues are clues and materials accepted by disciplinary inspection and supervision agencies that reflect suspected violations of party discipline, administrative discipline and national laws and regulations by party organizations, party members and administrative supervision targets. They mainly include: clues of petition cases forwarded by leaders; clues of suspected disciplinary violations discovered by disciplinary inspection and supervision organizations at all levels in the process of handling cases by relevant units or individuals; clues reported by towns (districts, streets, central communities), district-level departments, and judicial organs; information related to the case collected and mastered by other parties, etc. The management method of disciplinary inspection clues is usually to set up a case clue reserve in the case management office of the district disciplinary committee, and appoint a dedicated person to manage it, who is specifically responsible for the collection, registration, classification, and archiving of case clues. In the existing technology, the multi-label classification method for disciplinary inspection clues mostly adopts manual multi-label classification of text, which is inefficient and consumes a lot of manpower of disciplinary inspection and supervision cadres. Existing graph neural network-based classification methods perform multi-label text classification, but do not comprehensively consider the syntactic, semantic, and structural information of the text. The vector representation of the text does not fully integrate multiple types of information, which limits the representation ability of the text. At the same time, there is a lack of node update methods for different edge information and node type information, resulting in unsatisfactory text multi-label classification results. Summary of the Invention

[0003] This application provides a multi-label classification method for disciplinary inspection clues based on heterogeneous graph neural networks, which is used to solve the technical problems of low classification efficiency, limited text representation capabilities, and inability to update nodes according to different edge information and node type information in existing multi-label classification methods for disciplinary inspection clues.

[0004] In view of this, this application provides a multi-label classification method for disciplinary inspection clues based on heterogeneous graph neural networks, including:

[0005] Build a text multi-label classification model based on text heterogeneous graphs. The text multi-label classification model based on text heterogeneous graphs includes a heterogeneous graph neural network, a BILSTM neural network, a text representation module based on an attention mechanism, and a fully connected network.

[0006] Preprocess the text, construct a heterogeneous graph structure of the text, obtain the nodes, edges and edge weights of the heterogeneous graph structure, and establish an adjacency matrix based on the relationship between nodes and edges;

[0007] Initialize the vector representations of different types of nodes in the heterogeneous graph structure to obtain initial vector representations;

[0008] Input the node vector representation and heterogeneous graph structure into the heterogeneous graph neural network to obtain the sentence node vector representation;

[0009] Input the sentence node vector representation into the BILSTM neural network in the order of the sentences to obtain the content vector representation of the text;

[0010] Through the attention mechanism-based text representation module of content vector representation and sentence node vector representation, the weight of the sentence in the text representation is obtained, and the text vector representation is obtained;

[0011] The text vector representation is input into the fully connected network and the label classification result is output.

[0012] Optionally, preprocess the text, including:

[0013] The text is segmented into words, sentences, stop words are removed, the text is syntactically analyzed, and the semantic similarity of words in the text is calculated to obtain the word set and sentence set of the text.

[0014] Optionally, the text is segmented into words and sentences, stop words are removed, the text is syntactically analyzed, and the semantic similarity of words in the text is calculated to obtain a word set and a sentence set of the text, including:

[0015] The text is segmented into words and sentences, and stop words are removed. Each sentence in the text is syntactically analyzed using syntactic dependency tools. The words in each sentence are structurally analyzed based on co-occurrence information. The semantic distance between words in the text is calculated using the word vector of Emebedding to obtain the word set and sentence set of the text.

[0016] Optionally, the nodes include sentence nodes converted from sentences in the sentence set and word nodes converted from words in the word set.

[0017] Optionally, initializing vector representations of different types of nodes in the heterogeneous graph structure to obtain initial vector representations includes:

[0018] Embedded Emedding is used to obtain word vector representations for the words in the word set. The word vector representations are input into the BILSTM model and the CNN model with different convolution kernels respectively, and the output results of the two models are spliced ​​to obtain the initial vector representation of the sentence.

[0019] Optionally, the node vector representation and the heterogeneous graph structure are input into a heterogeneous graph neural network to obtain a sentence node vector representation, including:

[0020] The initial vector representation and heterogeneous graph structure are input into the heterogeneous graph neural network, and the sentence node vector representation is obtained after two stages of updating: updating adjacent nodes of the same type of edges and fusing information of different types of edges.

[0021] Optionally, obtaining the weight of the sentence in the text representation and obtaining the text vector representation by using an attention mechanism-based text representation module of the content vector representation and the sentence node vector representation includes:

[0022] The content vector is used as the query vector, and the sentence node vector is represented as the attention of key and value to obtain the text vector representation based on the attention mechanism.

[0023] Optionally, the text vector representation is input into a fully connected network, and the output label classification results include:

[0024] The text vector representation is input into the fully connected network and the label classification result is output through the Sigmod function.

[0025] Optionally, the text vector representation is fed into a fully connected network to output the label classification result, followed by:

[0026] The heterogeneous graph neural network in the text multi-label classification model based on text heterogeneous graph is iteratively trained to extract data features, and the loss function is calculated. The stochastic gradient descent method is used to iteratively train the model with the purpose of reducing the value of the loss function until the expected threshold is met. The text is input according to the parameters obtained after the training to obtain the label classification result.

[0027] Optionally, the heterogeneous graph neural network in the text multi-label classification model based on the text heterogeneous graph is iteratively trained to extract data features, and a loss function is calculated. The stochastic gradient descent method is used to iteratively train the data for the purpose of reducing the value of the loss function until the expected threshold is met. The text is input according to the parameters obtained after the training to obtain a label classification result, including:

[0028] The heterogeneous graph neural network in the text multi-label classification model based on text heterogeneous graph is iteratively trained to extract data features until the loss function value no longer decreases. The accuracy, recall rate and F value of the model are calculated, and the performance of the model is verified by evaluation indicators. If the F value is lower than the expected threshold, it is re-trained after manual intervention and iterated repeatedly until the F value is higher than the expected threshold. The text is input according to the parameters obtained after the training to obtain the label classification result.

[0029] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0030] The present application provides a multi-label classification method for disciplinary inspection clues based on a heterogeneous graph neural network. By constructing a text multi-label classification model based on a text heterogeneous graph and using the text multi-label classification model based on a text heterogeneous graph for classification, the classification time can be reduced and the classification efficiency can be improved. The text is preprocessed and a heterogeneous graph structure with structural information, semantic information and syntactic information of the text is constructed. The syntactic information, semantic information and structural information of the text are comprehensively considered so that the vector representation of the text fully integrates multiple information and improves the representation ability of the text. The vector representations of different types of nodes in the heterogeneous graph structure are initialized, the obtained node vector representation and the heterogeneous graph structure are input into the heterogeneous graph neural network, and the obtained sentence node vector representation is input into the BILSTM neural network in the order of the sentences. Then, a text representation module based on the attention mechanism of the content vector representation and the sentence node vector representation is obtained to obtain a text vector representation based on the attention mechanism. Finally, the text vector representation is input into a fully connected network and the label classification result is output, thereby realizing node update for different edge information and node type information. The paper solves the technical problems of low classification efficiency, limited text representation ability and inability to update nodes according to different edge information and node type information in the existing multi-label classification method of disciplinary inspection clues. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0032] Figure 1 A flowchart of a multi-label classification method for disciplinary inspection clues based on a heterogeneous graph neural network provided in an embodiment of the present application;

[0033] Figure 2 This is a schematic diagram of the structure of a text multi-label classification model based on a text heterogeneous graph provided in an embodiment of the present application;

[0034] Figure 3 A schematic diagram of the structure of the heterogeneous graph provided in the embodiments of the present application;

[0035] Figure 4 This is a schematic diagram of the process of initializing the node vector representation in the heterogeneous graph structure provided in the embodiments of the present application. DETAILED DESCRIPTION

[0036] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0037] For easier understanding, see Figures 1 to 4 The present application provides an embodiment of a multi-label classification method for disciplinary clues based on a heterogeneous graph neural network, comprising:

[0038] Step 101: Build a text multi-label classification model based on a text heterogeneous graph. The text multi-label classification model based on a text heterogeneous graph includes a heterogeneous graph neural network, a BILSTM neural network, an attention mechanism, and a fully connected network.

[0039] Step 102: Preprocess the text to construct a heterogeneous graph structure. Specifically, the nodes, edges, and edge weights of the heterogeneous graph structure are obtained, and an adjacency matrix is ​​established based on the relationships between the nodes and edges, thereby obtaining the heterogeneous graph structure. Step 103: Initialize the vector representations of different types of nodes in the heterogeneous graph structure to obtain initial vector representations.

[0040] Step 104: Input the node vector representation (i.e., the initial vector representation) and the heterogeneous graph structure into the heterogeneous graph neural network to obtain the sentence node vector representation.

[0041] Step 105: Input the sentence node vector representation into the BILSTM neural network in the order of the sentences to obtain the content vector representation of the text.

[0042] Step 106: Obtain a text vector representation based on the attention mechanism through the attention mechanism of the vector table of the content vector representation and the sentence node vector representation.

[0043] Step 107: Input the text vector representation into the fully connected network and output the label classification result.

[0044] Step 108: Iteratively train the heterogeneous graph neural network in the text multi-label classification model based on the text heterogeneous graph to extract data features, calculate the loss function, and use the stochastic gradient descent method to iteratively train with the purpose of reducing the value of the loss function until the expected threshold requirement is met. According to the parameters obtained after the training, the text is input to obtain the label classification result.

[0045] It should be noted that nodes include sentence nodes converted from sentences in the sentence set and word nodes converted from words in the word set. Stochastic gradient descent (SGD) is a simple but very effective method, often used to learn linear classifiers under convex loss functions such as support vector machines and logistic regression. SGD has been successfully applied to large-scale and sparse machine learning problems often encountered in text classification and natural language processing. SGD can be used for both classification and regression calculations.

[0046] Construct a text multi-label classification model based on a text heterogeneous graph, segment the text into words, sentences, remove stop words, perform syntactic analysis on the text, calculate the semantic similarity of words in the text, perform structural analysis on the words in each sentence based on co-occurrence information, calculate the semantic distance between words in the text based on the word vector of Emebedding, and obtain the word set and sentence set of the text. Then construct a heterogeneous graph structure with the structural information, semantic information and syntactic information of the text, obtain the nodes, edges and edge weights of the heterogeneous graph structure, and establish an adjacency matrix based on the relationship between nodes and edges. For text, it can be represented as: S = {s1, s2, ..., s n}, S represents the set of sentences in the text; W = {w1,w2,…,w m}, W represents the set of words in the text. Heterogeneous graph structure G = {V, E}, where V represents the nodes in the heterogeneous graph structure, and E represents the set of edges in the heterogeneous graph structure, including the connection relationship and weight of the edges. V = S∪W, E = {E syn ,E stu ,E sem ,E sw ,E ss}, where E syn The edges representing the syntactic relationship between words, have:

[0047]

[0048] E stu The edges represent the structural relationships between words. have:

[0049]

[0050] where p i Indicated in the word w i Position in the sentence. If two words co-occur in multiple sentences, the values ​​are added together.

[0051] E sem Indicates the semantic relationship between words. have:

[0052] eij =|cos(H i ,H j )|, if |cos(H i ,H j )|>α, where α is the threshold parameter.

[0053] E sw Indicates whether the word appears in the sentence, if it appears, it is 1, otherwise it is 0;

[0054] E ss Indicates the order relationship between sentences.

[0055] Therefore, we finally obtain the text heterogeneous graph G.

[0056] Suppose there is a text with two sentences S1 and S2, where the word sequence contained in S1 is W1W2W3W4, and the word sequence contained in S2 is W5. Then each word is converted into a word node, the set of word nodes is W = {W1, W2, W3, W4, W5}, and the sentence node set is S = {S1, S2}.

[0057] The constructed heterogeneous graph is as follows Figure 3 In the example shown, the document contains two sentences S1 and S2. S1 contains words W1, W2, W3, and W4, and S2 contains word W5. When the co-occurrence distance is 2, W1 is adjacent to W2; W2 is adjacent to W1 and W3; W3 is adjacent to W2 and W4; and W4 is adjacent to W3. There is a semantic relationship between W1 and W4, a syntactic relationship between W2 and W1 and W4, and a syntactic relationship between W4 and W3. Assuming the size of the co-occurrence window is 2, the constructed heterogeneous graph structure is as follows: Figure 3 As shown. The weight of the structural relationship between W2 and W3 is: The weights of W2 and W2 itself are:

[0058] Step 103:

[0059] After constructing the heterogeneous graph structure, the vector representations of different types of nodes in the heterogeneous graph structure are initialized, and the words in the word set are embedded using Emedding to obtain word vector representations. Representing word vectors Input the BILSTM model and the CNN model with different convolution kernels respectively, and concatenate the output results of the two models to obtain the initial vector representation of the sentence Such as the specific process Figure 4 As shown in the figure, BILSTM is a bidirectional long short-term memory network, CNN is a convolutional neural network, and Emedding is a way to convert discrete variables into continuous vectors.

[0060] Step 104-Step 106:

[0061] The initial vector is represented as The heterogeneous graph structure is input into the heterogeneous graph neural network, and the sentence node vector representation is obtained after the update method of updating the adjacent nodes of the same type of edges and fusing the information of different types of edges. Then, the sentence nodes are sequentially input into the BILSTM neural network to obtain the vector representation H of the text content. d . With H d is the query vector, the sentence node vector represents The attention of key and value is obtained, and the text vector representation based on the attention mechanism is obtained. The adjacent nodes of the same type of edge are given attention weights according to the attention mechanism. And get the vector representation of this type of information transfer.

[0062]

[0063] W ′ is the learning parameter; Represents the vector representation of node i at the l-1 layer; N t Represents an adjacent node of type t, and [] represents a connection operation.

[0064]

[0065] W represents the adjacent information representation of node i in the l-layer heterogeneous graph neural network of type t, t represents the parameter matrix, and σ() represents the nonlinear activation function.

[0066] For the information obtained by aggregating different types of edges, the attention weight of each type is obtained according to the attention mechanism, and finally the vector representation of the node is obtained.

[0067]

[0068] α t is the weight obtained by the attention mechanism, W l is the parameter matrix.

[0069] The vector representation of the sentence and the content vector representation are used to obtain the attention weight of each sentence through the attention mechanism, and finally the vector representation of the text is obtained

[0070] α ds represents the attention weight, and W″ is the learning parameter.

[0071] The vector representation of text is:

[0072]

[0073] Where W s is the parameter matrix.

[0074] Steps 107 and 108: Finally, the text vector representation is input into the fully connected network and the label classification result is output through the Sigmoid function. Next, the heterogeneous graph neural network in the text multi-label classification model based on the text heterogeneous graph is iteratively trained to extract data features, calculate the loss function, and iteratively train using stochastic gradient descent to reduce the loss function value until the expected threshold is met. The text is then input according to the parameters obtained after training to obtain the label classification result. Using the trained model for text multi-label classification can achieve more accurate label classification results.

[0075] A multi-label classification method for disciplinary inspection clues based on a heterogeneous graph neural network provided in the embodiment of the present application is constructed. A text multi-label classification model based on a text heterogeneous graph is constructed, and the disciplinary inspection clue text is segmented, sentence-separated, and stop words are removed. The word segmentation tool jieba in Python is used to segment the text, sentence-separated, and stop words are removed. The SpaCy tool is used to perform syntactic analysis on words to obtain syntactic dependencies. SpaCy is the world's fastest industrial-grade natural language processing tool that supports a variety of basic natural language processing functions. The main functions of SpaCy include word segmentation, part-of-speech tagging, stemming, named entity recognition, noun phrase extraction, etc. The semantic similarity of words is calculated using the Glove model. The semantic word vector similarity represented by the Glove model is as close as possible to the statistical similarity in the statistical co-occurrence matrix, and different co-occurring words have different weights. After pre-processing the text, a word set and a sentence set of the text are obtained. Then a heterogeneous graph structure with structural information, semantic information, and syntactic information of the text is constructed to obtain the nodes, edges, and edge weights of the heterogeneous graph structure, and an adjacency matrix is ​​established according to the relationship between the nodes and edges. Initialize the vector representations of different node types in the heterogeneous graph structure to obtain an initial vector representation. Then, input the node vector representation and the heterogeneous graph structure into a heterogeneous graph neural network to obtain a sentence node vector representation. The sentence node vector representation is then input into a BILSTM neural network in sentence order to obtain a content vector representation of the disciplinary clue text. Finally, the disciplinary clue text vector representation is input into a fully connected network, and the label classification result is output. The model is iteratively trained to extract data features until the loss function value no longer decreases. The model's accuracy, recall, and F-value are calculated, and the model's performance is verified using evaluation metrics. If the F-value is below the expected threshold, retraining is performed after manual intervention. Repeated iterations are performed until the F-value exceeds the expected threshold to obtain a trained model. The trained model is then used to perform the disciplinary clue multi-label classification task and output the label classification results corresponding to the document. The label classification results are shown in Table 1.

[0076] Table 1 Label types of disciplinary inspection clues

[0077]

[0078]

[0079] The paper solves the technical problems of low classification efficiency, limited text representation ability and inability to update nodes according to different edge information and node type information in the existing multi-label classification method of disciplinary inspection clues.

[0080] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-label classification method for disciplinary clues based on heterogeneous graph neural networks, characterized by: include: Constructing a text multi-label classification model based on a text heterogeneous graph, wherein the text multi-label classification model based on a text heterogeneous graph includes a heterogeneous graph neural network, a BILSTM neural network, a text representation module based on an attention mechanism, and a fully connected network; Preprocessing the text, constructing a heterogeneous graph structure of the text, obtaining nodes, edges, and edge weights of the heterogeneous graph structure, and establishing an adjacency matrix according to the relationship between the nodes and the edges; the preprocessing of the text includes: segmenting the text into words, segmenting the text into sentences, removing stop words, performing syntactic analysis on the text, and calculating the semantic similarity of words in the text to obtain a word set and a sentence set of the text; the nodes include sentence nodes converted from the sentences in the sentence set and word nodes converted from the words in the word set; Initializing vector representations of different types of nodes in the heterogeneous graph structure to obtain initial vector representations; inputting the node vector representations and the heterogeneous graph structure into the heterogeneous graph neural network to obtain sentence node vector representations, including: inputting the initial vector representations and the heterogeneous graph structure into the heterogeneous graph neural network, and obtaining the sentence node vector representation after updating in a two-stage updating method of updating adjacent nodes of the same type of edges and fusing information of different types of edges; Inputting the node vector representation and the heterogeneous graph structure into the heterogeneous graph neural network to obtain a sentence node vector representation; Inputting the sentence node vector representation into the BILSTM neural network in the order of the sentences to obtain the content vector representation of the text; Obtaining the weight of the sentence in the text representation and obtaining the text vector representation by using the attention mechanism-based text representation module of the content vector representation and the sentence node vector representation; Input the text vector representation into the fully connected network and output a label classification result; Initializing the vector representations of the different types of nodes in the heterogeneous graph structure to obtain initial vector representations includes: Using embedded emedding to obtain word vector representations for the words in the word set, inputting the word vector representations into a BILSTM model and a CNN model with different convolution kernels, respectively, and concatenating the output results of the two models to obtain the initial vector representation of the sentence; The text representation module based on the attention mechanism through the content vector representation and the sentence node vector representation obtains the weight of the sentence in the text representation and obtains the text vector representation, including: taking the content vector as the query vector and the sentence node vector representation as the attention of the key and value, and obtaining the text vector representation based on the attention mechanism.

2. The multi-label classification method for disciplinary clues based on heterogeneous graph neural network according to claim 1 is characterized in that: The step of segmenting the text into words, dividing the text into sentences, removing stop words, performing syntactic analysis on the text, and calculating the semantic similarity of words in the text to obtain a word set and a sentence set of the text includes: The text is segmented into words, sentences, and stop words are removed; each sentence in the text is subjected to syntactic analysis using a syntactic dependency tool; the words in each sentence are structurally analyzed according to co-occurrence information; the semantic distance between the words in the text is calculated according to the word vector of Emebedding, to obtain the word set and the sentence set of the text.

3. The multi-label classification method for disciplinary inspection clues based on heterogeneous graph neural network according to claim 1 is characterized in that: Inputting the text vector representation into the fully connected network and outputting a label classification result includes: The text vector representation is input into the fully connected network and the label classification result is output through the Sigmod function.

4. The multi-label classification method for disciplinary inspection clues based on heterogeneous graph neural network according to claim 1 is characterized in that: The step of inputting the text vector representation into the fully connected network and outputting the label classification result further includes: The heterogeneous graph neural network in the text multi-label classification model based on the text heterogeneous graph is iteratively trained to extract data features, and the loss function is calculated. The stochastic gradient descent method is used to iteratively train the text to reduce the value of the loss function until the expected threshold requirement is met. The text is input according to the parameters obtained after the training to obtain the label classification result.

5. The multi-label classification method for disciplinary inspection clues based on heterogeneous graph neural network according to claim 4 is characterized in that: The iterative training of the heterogeneous graph neural network in the text multi-label classification model based on the text heterogeneous graph extracts data features, calculates a loss function, and iteratively trains the text using a stochastic gradient descent method for the purpose of reducing the value of the loss function until an expected threshold value is met, and inputs the text according to the parameters obtained after the training to obtain the label classification result, including: The heterogeneous graph neural network in the text multi-label classification model based on the text heterogeneous graph is iteratively trained to extract data features until the value of the loss function no longer decreases, the accuracy, recall rate and F value of the model are calculated, and the performance of the model is verified by evaluation indicators. If the F value is lower than the expected threshold, re-training is performed after manual intervention, and iterations are repeated until the F value is higher than the expected threshold. The text is input according to the parameters obtained after the training to obtain the label classification result.

Images