An interpretable text classification system based on dual-path encoder

Abstract

The present invention provides an interpretable text classification system based on a dual-path encoder, comprising: a preprocessing module for characterizing text; an encoder module for acquiring semantic information from different perspectives of the text; a classifier module for outputting the text classification results; and an explanatory information generator module for outputting a textual explanation of the text classification results. This technical solution, by fusing the semantic representations of multi-head attention with bidirectional gated recurrent units, can learn multi-level semantic information from text. It also uses a secondary attention mechanism to address the mismatch between queries and attention results, enhancing the encoded semantic information and thus achieving better interpretable text classification.

Description

Technical Field

[0001] The present invention relates to the technical field of interpretability of text classification, and in particular to an interpretable text classification system based on a dual-path encoder. Background Art

[0002] Explainable text classification aims to simultaneously generate explanations for document classification, serving as the basis for model decisions. This helps people understand how machine learning models learn, what they learn from the data, why they make the decisions they do for each input, and whether these decisions are reliable. Explainable text classification enhances model interpretability and transparency, eliminating security risks associated with model deployment in real-world applications. This allows users to trust the model's decisions and promotes the widespread application of text classification methods based on deep neural network models in security-sensitive fields such as finance and healthcare. This is of far-reaching significance.

[0003] At present, the research on the interpretability of machine learning models can be divided into three categories:

[0004] Based on the ante-hoc interpretability analysis method. Ex ante interpretability refers to the process of training a model with a simple structure and good interpretability or integrating interpretability into a specific model structure to make the model itself interpretable. The main ex ante interpretation methods are as follows: (1) Self-interpretation model, which is a type of model with transparent structure or derivable decision results, such as decision tree, naive Bayes model, etc. This type of model itself has its own interpretability, but this type of method requires that the model structure should not be too complex, which inevitably limits the fitting ability of this type of model. For example, for decision tree models, simple and shallow decision trees are very intuitive visually and easy to interpret, but deep decision trees are more difficult to visually and interpret. (2) Generalized additive model, generalized additive model can not only improve the accuracy of simple linear models, but also retain the good built-in interpretability of linear models. The general form of generalized additive model is:

[0005]

[0006] Where f0 is a constant, f i Represents feature x iThe nonlinear shape function of the feature, the additive model g(y) is composed of linear shape functions of the features. The shape function can be a linear function or a nonlinear function. The flexible choice makes the generalized additive model have a higher degree of freedom to fit the data, combined with the excellent interpretability of the additive model. Although the generalized additive model provides an excellent model interpretation solution, its accuracy is still lacking compared with the existing deep network model. (3) Attention mechanism. The neural network model has poor interpretability due to its complex model structure and low algorithm transparency. Therefore, the interpretability of the neural network model itself can only be achieved by introducing an additional interpretability module. An effective method is to introduce the attention mechanism. In the case of limited computing power, the attention mechanism is an effective means to solve the problem of information overload. By determining the input part that needs attention, limited information processing resources are allocated to more important tasks. In addition, the attention mechanism has good interpretability, and the attention weight matrix directly reflects the area of ​​interest of the model in the decision-making process. Attention helps people understand the contribution of each sentence and each word to the text classification results.

[0007] Based on post-hoc interpretability analysis methods. Depending on the object and purpose of interpretation, this method can be divided into global interpretability and local interpretability. Global interpretability aims to help people understand the complex working mechanism behind the model from a holistic perspective. Typical global interpretation methods include interpretation model / rule extraction, model distillation, activation maximization interpretation, etc. Local interpretability aims to help people understand the decision-making process and decision basis of the learning model for each specific input sample. Unlike global interpretability, the local interpretability of the model is achieved by analyzing the contribution of each dimension of the input sample to the final result. In fact, due to the opacity of the model algorithm and the complexity of the application scenario, providing a global explanation of the model is generally more difficult than a local explanation. Therefore, most current research focuses on the local interpretability of the model. Classic local interpretability analysis methods include sensitivity analysis, local approximation, gradient backpropagation, and class activation mapping.

[0008] Explainability analysis methods based on text summarization. While classifying the original text, these methods use extractive or generative text summarization methods to extract key paragraphs and sentences from the document or generate explanatory summaries to serve as the basis for model decision-making. However, existing model methods do not fully mine the semantic information of the original text, and the generated text representations contain insufficient semantic information, resulting in poor text classification performance and poor explanatory information generation performance. Summary of the Invention

[0009] In view of this, the purpose of the present invention is to provide an interpretable text classification system based on a dual-path encoder, which can learn the multi-level semantic information of the text by fusing the semantic representation of multi-head attention and bidirectional gated recurrent units, and use a secondary attention mechanism to solve the mismatch problem between the query and the attention results, enhance the encoded semantic information, and thus better realize interpretable text classification.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: an interpretable text classification system based on a dual-path encoder, comprising:

[0011] A preprocessing module for characterizing text;

[0012] An encoder module to obtain semantic information from different perspectives of the text;

[0013] A classifier module for outputting text classification results;

[0014] An explanatory information generator module that outputs textual explanations of text classification results.

[0015] In a preferred embodiment, the preprocessing module first performs word segmentation on the source text; and then converts the text data from text form to vector form through GloVe.

[0016] In a preferred embodiment, the encoder module introduces a multi-head attention mechanism and a bidirectional gated recurrent unit mechanism to extract multi-level semantic information of the text from different perspectives, introduces secondary attention to alleviate the mismatch between the query and the attention results, and integrates the original text semantic features with the semantic information encoded by the encoder;

[0017] The encoder module is composed of a multi-head attention module and a bidirectional gated recurrent unit neural network. The text data in vector form is divided into two paths. One path is input into the multi-head attention module and then passes through a feedforward neural network, and the other path enters the bidirectional gated recurrent unit neural network. The self-attention model is regarded as establishing the interaction between different forms in the input vector in a linear projection space. The multi-head attention establishes different projection information in multiple different projection spaces. The input matrix is ​​projected differently, and after obtaining many output matrices, they are spliced ​​together. Focusing on different positions in the text, the specific formula is as follows:

[0018]

[0019]

[0020] MultiHead(Q,K,V)=Concat(head i ,…,headh )W O

[0021] Where Q represents the query vector, K represents the key information vector suitable for receiving the search, and V represents the content vector;

[0022] The Bidirectional Gated Recurrent Unit (BiGRU) neural network consists of two GRUs: one is a forward GRU model that accepts forward input; the other is a reverse GRU model that learns reverse input and is used to model the global semantic information of the text to prevent the GRU from being unable to encode information from back to front. The specific formula for updating the GRU model at time t is as follows:

[0023] h t =(1-z)⊙h t-1 +z⊙h′.

[0024] The interpretable text classification system based on the dual-path encoder is characterized by further comprising an encoding fusion module, which fuses the two-path encoding results of the encoder module with the original semantic information encoding features after being processed by the secondary attention module; the formula is as follows:

[0025] y=concat(x1,x2,x3)

[0026] Among them, x1, x2, and x3 are the output of the multi-head attention of the AOA module and then the output of a feedforward neural network, the output of the BiGRU of the AOA module, and the original text semantic features.

[0027] In a preferred embodiment, the classifier module first uses a three-layer fully connected network to reduce the dimension of the semantic vector encoded by the encoder, and then uses the softmax function to process the obtained text representation vector to predict the category of each text;

[0028] The classifier module consists of a three-layer fully connected network and a softmax function. The fused text semantic features output by the encoding fusion module are encoded through a three-layer fully connected network, and then the softmax function is used to process the resulting text representation vector to predict the category of the text. The formula is as follows:

[0029] P classified =softmax(f C (W C ·e C +b C )).

[0030] The present invention relates to an interpretable text classification system based on a dual-path encoder, characterized in that an interpretable information generator module uses a conditional variational autoencoder (CVAE) to generate text-type interpretable information, and the interpretable information generator module is composed of five parts: an interpretable information encoder, a posterior network, a priori network, a decoder, and a latent variable supervision module; wherein the encoder and the decoder use a transformer encoder based on multi-head attention, the posterior network and the priori network both use a multi-layer perceptron, and the latent variable supervision module is a classifier of a three-layer fully connected network followed by a softmax function; during the training process, the interpretable information generator module uses a vector representation v of the interpretable information and the original text e And the vector representation v of the text classification result c As input;

[0031] Conditional variational autoencoder CVAE needs to maximize the given text information and text classification result information during training [v e ;v c ], v c The conditional likelihood variation lower bound of the text explanation information x corresponding to the text classification result is as follows:

[0032] pxc)=∫pxz,cp(zc)dz

[0033] Where z is a latent variable, c=[v e ;v c ];

[0034] After encoding the embedded representation of the explanation information, the encoder maps it to a vector representation x, which is used to estimate p(x|z,c), denoted as p D (x|z,c); the prior network is based on v e and v c The concatenation of is input, and the prior probability distribution p(z|c) of the latent variable z′ under the condition of unknown target text is fitted, denoted as p P (z|c); by maximizing the variational lower bound of the prior probability distribution, the prior network is trained to approximate the posterior network and output a better hidden variable z′; the posterior network is based on vectors x, v e and v c The concatenation of is input, fitting the posterior probability distribution q of the latent variable z under the condition of known target text R (z|x,c), which estimates the true posterior distribution; the variational lower bound of logp(x|c) is as follows:

[0035]

[0036] Where θ is the network parameter;

[0037] The decoder uses the posterior distribution q R The hidden variable z obtained by sampling in (z|x,c) initializes the hidden state with v e The target text embedding representation x is taken as input, and the word probability distribution at each time step is output. The training is performed by optimizing the cross entropy loss with the actual target review summary. The latent variable supervision module is introduced to classify the text through the latent variable z. If the correct classification is obtained, it means that the latent variable z encodes enough source information to generate explanation information. Therefore, the loss function of the recognition network and the prior network in CVAE is:

[0038]

[0039] Among them L Z is the classification loss of text through latent variable z;

[0040] During testing, the generator module generates a vector representation v of the text. e And the text classification result vector representation v c As input; [v e ;v c ]After inputting the prior network, the latent variable z′ is sampled from the prior distribution. The decoder uses the latent variable z′ to add the target explanatory information embedding representation, outputs the target word step by time, and generates the final explanatory information.

[0041] Compared with the existing technology, the present invention has the following beneficial effects: it can learn the multi-level semantic information of the text by fusing the semantic representation of multi-head attention and bidirectional gated recurrent unit, and use the secondary attention mechanism to solve the mismatch problem between query and attention results, enhance the encoded semantic information, and thus better realize interpretable text classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a schematic configuration diagram of an interpretable text classification system based on a dual-path encoder according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0044] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0045] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form, and it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0046] A dual-path encoder interpretable text classification system, reference Figure 1 , specifically including: a preprocessing module 1, which is used to feature the text; an encoder module 2, which is used to model the interaction between texts and capture the global semantic features of the text; a secondary attention module 3, which is used to alleviate the problem of mismatch between query and attention results; an encoding fusion module 4, which concatenates and fuses the output of the multi-head attention module with the output of the bidirectional gated recurrent unit neural network module, and also fuses the original text semantic features to avoid excessive loss of semantic information of the encoded text itself; a text classifier module 5, which is used to process the fused text feature representation to obtain the final text classification result; an explanatory information generator module 6, which is used to model the latent variable z in the explanation generation process for generating explanations.

[0047] The configuration of each module is described in detail below.

[0048] 1) Preprocessing module 1

[0049] First, it is described how the text preprocessing module 1 obtains the initial text vector.

[0050] Since neural network input data is generally vectorized to facilitate end-to-end model training, text data needs to be vectorized. To facilitate data processing and analysis, in the data preprocessing module of this invention, we first segment the source text; then, we use GloVe to convert the text data from text form into vector form.

[0051] 2) Encoder module 2

[0052] The following describes how the encoder module 2 processes the data obtained from the previous module. This module is a combination of a multi-head attention module (Multi-head-attention) and a bidirectional gated recurrent unit neural network (BiGRU). The text data in vector form is divided into two paths, one is input into the multi-head attention module and then passes through a feedforward neural network, and the other enters the bidirectional gated recurrent unit neural network. The self-attention model can be regarded as establishing an interactive relationship between different forms in the input vector in a linear projection space. Multi-head attention is to establish different projection information in multiple different projection spaces. The input matrix is ​​projected differently, and after obtaining many output matrices, they are spliced ​​together. In this way, different positions in the text can be focused on to avoid the loss of key information. The specific formula is as follows:

[0053]

[0054]

[0055] MultiHead(Q,K,V)=Concat(head i ,…,head h )W O

[0056] Where Q represents the query vector, K represents the key information vector suitable for receiving the search, and V represents the content vector.

[0057] The Bidirectional Gated Recurrent Unit (BiGRU) neural network consists of two GRUs: a forward GRU model that accepts forward inputs; and a reverse GRU model that learns reverse inputs. This model is used to model the global semantic information of the text, preventing the GRU from being unable to encode information from the back to the front. The specific formula for the GRU update at time t is as follows:

[0058] h t =(1-z)⊙h t-1 +z⊙h′

[0059] 3) Secondary Attention (AOA) Module 3

[0060] The classic attention mechanism is Attention(Q, K, V), where Q represents the query vector, K represents the key information vector suitable for receiving the search, and V represents the content vector. Regardless of whether Q is related to K, V or not, the attention module will generate weight information for each dimension of K. Therefore, there may be a result vector that is unrelated to the query vector, and noise information may also be generated. In order to solve this problem, the AOA mechanism is used to solve the problem of mismatch between attention results and queries. A relevance score is calculated for each attention result and query. First, the query vector and the result vector are spliced, and then passed through two separate linear layers to generate the information vector i and the attention gate vector g respectively. The specific calculation method is as follows:

[0061]

[0062]

[0063] in b g ∈R D are training parameters, D is the dimension of q and v is the attention result, and σ is the sigmoid activation function.

[0064] Then, AOA obtains the weight information of each dimension of the attention result through the gating network to filter the result and obtain the final attention result.

[0065]

[0066] where ⊙ is the bit-by-bit multiplication.

[0067] The final AOA formula is as follows:

[0068]

[0069] 4) Coding fusion module 4

[0070] This module combines the two-way encoding results of module 2 with the original semantic information encoding features after processing them through the secondary attention module to avoid excessive loss of semantic information of the text itself. The formula is as follows:

[0071] y = concat(x1, x2, x3)

[0072] Among them, x1, x2, and x3 are the output of the multi-head attention of the AOA module and then the output of a feedforward neural network, the output of the BiGRU of the AOA module, and the original text semantic features.

[0073] 5) Text Classifier Module 5

[0074] This module consists of a three-layer fully connected network and a softmax function. After encoding the fused text semantic features output by the encoding fusion module 4 through a three-layer fully connected network, the softmax function is used to process the resulting text representation vector to predict the category of the text. The formula is as follows:

[0075] P classified =softmax(f C (W C ·e C +b C ))

[0076] 6) Explanatory Information Generator Module 6

[0077] The following describes how the explanatory information generator module 6 generates explanatory information for the text classification results obtained by the previous module.

[0078] The explanatory information generator module uses the conditional variational autoencoder (CVAE) to generate text-based explanatory information. The module mainly consists of five parts: the explanatory information encoder, the posterior network, the prior network, the decoder, and the latent variable supervision module. The encoder and decoder both use the transformer encoder based on multi-head attention, the posterior network and the prior network both use the multi-layer perceptron (MLP), and the latent variable supervision module is a classifier with a three-layer fully connected network followed by a softmax function. During the training process, the module uses the vector representation v of the explanatory information and the original text. e And the vector representation vc of the text classification result is taken as input.

[0079] Conditional variational autoencoder CVAE needs to maximize the given text information and text classification result information during training [v e ;v c ], v c The conditional likelihood variation lower bound of the text explanation information x corresponding to the text classification result is as follows:

[0080] p(x|c)=∫p(x|z,c)p(z|c)dz

[0081] Where z is a latent variable, c=[v e ;v c ].

[0082] After encoding the embedded representation of the explanation information, the encoder maps it to a vector representation x, which is used to estimate p(x|z, c), denoted as p D (x|z,c). The prior network is based on v e and v cThe concatenation of is input, fitting the prior probability distribution p(z|c) of the latent variable z′ under the condition of unknown target text, denoted as p P (z|c). By maximizing the variational lower bound of the prior probability distribution, the prior network is trained to approximate the posterior network and output a better hidden variable z′. The posterior network takes vectors x, v e and v c The concatenation of is input, fitting the posterior probability distribution q of the latent variable z under the condition of known target text R (z|x, c), which estimates the true posterior distribution. The variational lower bound for log p(x|c) is as follows:

[0083]

[0084] Where θ is the network parameter;

[0085] The decoder uses the posterior distribution q R The hidden variable z obtained by sampling in (z|x, c) initializes the hidden state with v e The target text embedding representation x is taken as input, and the word probability distribution at each time step is output. Training is performed by optimizing the cross-entropy loss with the actual target review summary. In order to enable the latent variable z to encode sufficient source information, a latent variable supervision module is introduced. The text is classified using the latent variable z. If the classification is correct, it means that the latent variable z encodes sufficient source information and can generate better explanation information. Therefore, the loss function for the recognition network and the prior network in CVAE is:

[0086]

[0087] Among them L Z is the classification loss of the text through the latent variable z.

[0088] During testing, the generator module generates a vector representation v of the text. e And the text classification result vector representation v c As input. e ;v c ]After inputting the prior network, the latent variable z′ is sampled from the prior distribution. The decoder uses the latent variable z′ to add the target explanatory information embedding representation, outputs the target word step by time, and generates the final explanatory information.

Claims

1. An interpretable text classification system based on a dual-path encoder, characterized in that include: A preprocessing module for characterizing text; An encoder module to obtain semantic information from different perspectives of the text; A classifier module for outputting text classification results; An explanatory information generator module that outputs textual explanations of text classification results; The explanatory information generator module uses the conditional variational autoencoder CVAE to generate text-based explanatory information. The explanatory information generator module consists of five parts: an explanatory information encoder, a posterior network, a priori network, a decoder, and a latent variable supervision module. The encoder and decoder use a transformer encoder based on multi-head attention, and both the posterior network and the priori network use a multi-layer perceptron. The latent variable supervision module is a classifier with a three-layer fully connected network followed by a softmax function. During the training process, the explanatory information generator module uses the vector representation v of the explanatory information and the original text. e And the vector representation v of the text classification result c As input; Conditional variational autoencoder CVAE needs to maximize the given text information and text classification result information during training [v e ;v c ], v c The conditional likelihood variation lower bound of the text explanation information x corresponding to the text classification result is as follows: Where z is a latent variable, c=[v e ;v c ]; After encoding the embedded representation of the explanation information, the encoder maps it to a vector representation x, which is used to estimate p(x|z,c), denoted as p D (x|z,c); the prior network is based on v e and v c The concatenation of is input and the latent variable z is fitted. ' The prior probability distribution p(z|c) under the condition of unknown target text is denoted as P P(Z∣C) ; By maximizing the variational lower bound of the prior probability distribution, the prior network is trained to approximate the posterior network and output a better latent variable z ' ; The posterior network takes vectors x, v e and v c The concatenation of is input, fitting the posterior probability distribution q of the latent variable z under the condition of known target text R (z|x,c), which estimates the true posterior probability distribution; the variational lower bound of logp(x|c) is as follows: Where θ is the network parameter; The decoder uses the posterior distribution q R The hidden variable z obtained by sampling in (z|x,c) initializes the hidden state with v e The target text embedding representation x is taken as input, and the word probability distribution at each time step is output. The training is performed by optimizing the cross entropy loss with the actual target review summary. The latent variable supervision module is introduced to classify the text through the latent variable z. If the correct classification is obtained, it means that the latent variable z encodes enough source information to generate explanation information. Therefore, the loss function of the recognition network and the prior network in CVAE is: Among them L Z is the classification loss of text through latent variable z; During testing, the generator module generates a vector representation v of the text. e And the text classification result vector representation v c As input; [v e ;v c ]After inputting the prior network, the latent variable z is sampled from the prior distribution ' , the decoder uses this latent variable z ' It is added to the target explanation information embedding representation, and the target word is output time-step by time-step to generate the final explanatory information.

2. The interpretable text classification system based on a dual-path encoder according to claim 1, characterized in that: The preprocessing module first segments the source text; then uses GloVe to convert the text data from text form into vector form.

3. The interpretable text classification system based on a dual-path encoder according to claim 1, characterized in that: The encoder module introduces a multi-head attention mechanism and a bidirectional gated recurrent unit mechanism to extract multi-level semantic information from different perspectives. It also introduces secondary attention to alleviate the mismatch between the query and the attention results, and integrates the semantic features of the original text with the semantic information encoded by the encoder. The encoder module is composed of a multi-head attention module and a bidirectional gated recurrent unit neural network. The text data in vector form is divided into two paths. One path is input into the multi-head attention module and then passes through a feedforward neural network, and the other path enters the bidirectional gated recurrent unit neural network. The self-attention model is regarded as establishing the interaction between different forms in the input vector in a linear projection space. The multi-head attention establishes different projection information in multiple different projection spaces. The input matrix is ​​projected differently, and after obtaining many output matrices, they are spliced ​​together. Focusing on different positions in the text, the specific formula is as follows: MultiHead(Q,K,V)=Concat(head i ,…,head h )W O Where Q represents the query vector, K represents the key information vector suitable for receiving the search, and V represents the content vector; The Bidirectional Gated Recurrent Unit (BiGRU) neural network consists of two GRUs: one is a forward GRU model that accepts forward input; the other is a reverse GRU model that learns reverse input and is used to model the global semantic information of the text to prevent the GRU from being unable to encode information from back to front. The specific formula for updating the GRU model at time t is as follows: h t =(1-z)⊙h t-1 +z⊙h'。 4. The interpretable text classification system based on a dual-path encoder according to claim 1, characterized in that: The encoding fusion module is further included, which processes the two-way encoding results of the encoder module through the secondary attention module and fuses them with the original semantic information encoding features; The formula is as follows: y=concat(x1,x2,x3) Among them, x1, x2, and x3 are the output of the multi-head attention of the AOA module and then the output of a feedforward neural network, the output of the BiGRU of the AOA module, and the original text semantic features.

5. The interpretable text classification system based on a dual-path encoder according to claim 1, characterized in that: The classifier module first uses a three-layer fully connected network to reduce the dimension of the semantic vector encoded by the encoder, and then uses the softmax function to process the resulting text representation vector to predict the category of each text; The classifier module consists of a three-layer fully connected network and a softmax function. The fused text semantic features output by the encoding fusion module are encoded through a three-layer fully connected network, and then the softmax function is used to process the resulting text representation vector to predict the category of the text. The formula is as follows: P classified =softmax(f C (W C ·e C +b C ))。

Images