A deep multi-instance learning text classification method based on variational autoencoder

Abstract

The application discloses a kind of deep multi-instance learning text classification method based on variational autoencoder, comprising: first, text is extracted and normalized, the embedding feature of instance in package is extracted by variational autoencoder extraction package, each instance weight is obtained using Dropout layer and attention network, and the weighted sum of instance in package is used as package embedding representation, the package prediction result is obtained after linear neural network is inputted to package embedding and is normalized, loss function is established to train network, and text classification is completed using the text to be classified and the network after training.The application introduces variational autoencoder, so that the network can fully exploit the information hidden in unlabelled examples, extract smooth and continuous instance embedding, thereby improving model robustness, avoiding overfitting, and improving the accuracy of text classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of text classification, and particularly relates to a deep multi-instance learning text classification method based on a variational autoencoder. BACKGROUND

[0002] Text classification is an important task in natural language processing (NLP) and is widely used in spam detection, sentiment analysis, and news classification. Traditional text classification methods mainly rely on machine learning models such as support vector machines (SVM), decision trees, and random forests. These methods are usually based on the bag-of-words (BoW) or TF-IDF feature extraction methods. However, as the size and complexity of data increase, traditional methods are insufficient in capturing semantic relationships and handling long texts.

[0003] Deep learning methods such as convolutional neural networks (CNN) and recurrent neural networks (RNN) can automatically extract semantic features and achieve good results on large-scale data. However, their dependence on a large amount of labeled data makes them costly in practical applications. Therefore, weakly supervised learning models have become an important research direction for handling complex text classification tasks.

[0004] Multi-instance learning (MIL) is a typical weakly supervised learning method that provides a new and effective solution for text classification. In MIL, data is organized in bags, which contain multiple instances (such as sentences or paragraphs in a document), but only the bag labels are known, and the instance labels are unknown. The goal of MIL is to learn the bag-level labels and infer the contribution of each instance to complete the classification task.

[0005] In recent years, some researchers have tried to combine deep neural networks (DNN) with MIL and have made great improvements. Among them, attention-based deep multi-instance learning has shown excellent performance and has received widespread attention from researchers. The instance embedding network in this algorithm (used to extract instance features) may receive meaningful gradient signals from some instances, which have a considerable contribution to the bag embedding. However, in the real world, labels are difficult to obtain. Unlike traditional machine learning, multi-instance learning has two levels of features: bag and instance. Current deep multi-instance learning methods focus on extracting effective bag representations, ignoring the objective fact that bag embeddings are aggregated from instance features. When the number of samples is limited, the supervision signal of MIL is weak, making it difficult for the instance embedding network to learn effective representations, resulting in a significant overfitting trend in the overall model. In addition, although the number of labeled bags is limited, there are still a large number of unlabeled instances in the bags. The number of instances is much larger than the number of data bags, and there is a lot of information in the instances, but this information is largely ignored. SUMMARY

[0006] In view of the deficiencies of the above multi-instance learning in the text classification task, the application provides a deep multi-instance learning text classification method based on a variational autoencoder.

[0007] To achieve the above purpose, the application provides a deep multi-instance learning text classification method based on a variational autoencoder, which comprises the following steps:

[0008] The purpose of the application is achieved by the following technical scheme: a deep multi-instance learning text classification method based on a variational autoencoder, comprising the following steps:

[0009] Step 1: The document is regarded as a bag, and each sentence or paragraph in the bag is regarded as an instance. The text is preprocessed by feature extraction.

[0010] Step 2: Normalizing the data set;

[0011] Step 3: Constructing a multi-instance learning text classification learning network, comprising: a variational autoencoder extracting embedding features of instances in the bag, using a Dropout layer and an attention network to obtain weights of each instance, weighting and summing the instances in the bag as a bag embedding representation, inputting the bag embedding into a linear neural network and normalizing to obtain a bag prediction result,

[0012] Step 4: Establishing a loss function, calculating the reconstruction loss of the variational autoencoder and the loss between the bag prediction result and the true value, and updating the weight parameters of the variational autoencoder, the attention mechanism and the multi-layer linear neural network by optimizing the loss function until convergence;

[0013] Step 5: Preprocessing and normalizing the document to be classified and inputting it into the trained multi-instance learning text classification learning network to obtain a text classification result.

[0014] Further, the preprocessing of the text comprises: using a predefined / pretrained feature extraction method for processing, specifically using TF-IDF, and selecting the top n features as instance vectors.

[0015] Further, the variational autoencoder is composed of an encoder, a hidden variable layer and a decoder, specifically as follows:

[0016] The encoder consists of three fully connected layers, the first two layers have 256 and 128 neurons respectively, and both layers use ReLU activation function; the third layer has two branches, each with 64 neurons, respectively output the mean μ and standard deviation σ of the latent variable;

[0017] The latent variable is obtained by reparameterization, and the calculation formula is as follows:

[0018] z = μ + σ☉∈

[0019] Where z is the latent variable corresponding to the instance, μ and σ correspond to the mean and standard deviation respectively, ☉ represents the product between elements, and ∈ is an auxiliary noise variable subject to standard normal distribution;

[0020] The decoder consists of three fully connected layers, with 64, 128 and 256 neurons respectively, the first two layers use ReLU activation function, and the third layer directly outputs the reconstruction vector.

[0021] Further, the use of Dropout layer and attention network to obtain the weight of each instance specifically includes:

[0022] The instance embedding is processed by the Dropout layer with a dropout rate of 0.5, and then the attention network is used to learn each instance embedding to obtain the weight of each instance:

[0023]

[0024] Where i and j are the indexes of the corresponding package and instance, And are parameters learned by neural network, M is the dimension of the latent variable, L is set to 128, and n i is the number of instances in the package.

[0025] Further, the package embedding representation is obtained by the following formula:

[0026]

[0027] Where b i is the package embedding representation corresponding to the package with index i, n i is the number of instances in the package, z ij and a ij are the instance vector and its weight with index j in the package respectively.

[0028] Further, the loss function calculation formula is as follows:

[0029]

[0030] Where λ is the weight coefficient, is the reconstruction loss of the variational autoencoder, It is the loss between the predicted result and the true value.

[0031] Furthermore, the reconstruction loss of the variational autoencoder is calculated as follows:

[0032]

[0033] Among them, n i is the number of instances in the package, x ij 、z ij are instance vectors and their corresponding latent variables, p θ Corresponding to the encoder and decoder respectively.

[0034] Represents the loss between the instance vector obtained by the decoder and the original instance, specifically the mean square error:

[0035] represents the approximate posterior probability and the prior distribution p θ (z), specifically: Where M is the mean vector μ ij and the standard deviation vector σ ij The dimension of μ ijm , σ ijm Represent μ ij and σ ij The value of the mth dimension of .

[0036] Furthermore, the loss between the packet prediction result and the true value adopts the cross entropy loss function:

[0037]

[0038] Among them, S i is the predicted packet score, Y i is the true value of the bag label.

[0039] On the other hand, the specification also provides a deep multi-instance learning text classification device based on a variational autoencoder, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the deep multi-instance learning text classification method based on a variational autoencoder.

[0040] On the other hand, the specification also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, it implements the deep multi-instance learning text classification method based on variational autoencoder.

[0041] The beneficial effects of the present application are: 1) a variational autoencoder is introduced to extract deep features of instances in an unsupervised learning manner, and the reconstruction error is added to the loss function; 2) in addition, the variational autoencoder can also constrain the distribution of the hidden space where the extracted features are located, so that the extracted features are smoother and more robust. It has been proved in practice that the method proposed in the present application can improve the robustness of the model, avoid overfitting, and effectively improve the classification accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The network structure of the present application is shown in the figure;

[0043] Figure 2 The figure shows the package embedding obtained by attention mechanism pooling;

[0044] Figure 3 The figure shows a deep multi-instance learning text classification device based on a variational autoencoder provided by an embodiment of the present application. DETAILED DESCRIPTION

[0045] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0046] As shown in the figure, the present application proposes a deep multi-instance learning text classification method based on a variational autoencoder, which is used to improve the robustness of the multi-instance learning (MIL) model under limited supervision signals and avoid overfitting problems. The following specific embodiments detail the working principle and implementation steps of the present application: Figure 1 Text feature extraction: the document is regarded as a package, and each sentence or paragraph in it is regarded as an instance. The text is preprocessed by a pre-defined / pre-trained feature extraction method;

[0047] TF-IDF is used to extract the text features of each paragraph, and the top 200 features are selected as the instance vector;

[0048] Data set normalization

[0049] The following formula is used to normalize the data set:

[0050]

[0051]

[0052] Where μ and σ are the mean and standard deviation of all instances in the data set.

[0053] The variational autoencoder algorithm is used to extract the embedding features of the instances in the package;

[0054] ​The variational autoencoder (VAE) of the application is composed of an encoder, a latent variable layer and a decoder, the encoder encodes to obtain a latent variable, and the latent variable is decoded to obtain a reconstructed instance vector, which is as follows:

[0055] The encoder is composed of three fully connected layers. The first two layers have 256 and 128 neurons respectively, and both layers use ReLU activation function. The third layer has two branches, each with 64 neurons, which respectively output the mean μ and standard deviation σ of the latent variable;

[0056] The latent variable is obtained by a reparameterization trick, and the calculation formula is as follows:

[0057] z = μ + σ☉∈

[0058] Wherein, z is the latent variable corresponding to the instance, μ and σ correspond to the mean and standard deviation respectively, ⊙ represents the product between elements, and ∈ is an auxiliary noise variable obeying standard normal distribution;

[0059] The decoder is composed of three fully connected layers, with 64, 128 and 256 neurons respectively, and the first two layers use ReLU activation function, and the third layer directly outputs the reconstructed vector.

[0060] After the instance embedding is processed through the Dropout layer, an attention network is constructed to learn the weight of each instance;

[0061] The dropout rate of the Dropout layer is 0.5 to prevent model overfitting. The latent variable output by the encoder is processed through the Dropout layer, and then enters the attention network to learn the contribution of each instance to the package embedding. The weight calculation formula is as follows:

[0062]

[0063] Wherein, i and j are the indexes of the corresponding package and instance, And are parameters learned by neural network, M is the dimension of the latent variable (i.e. 64), L is the dimension of the hidden layer in the attention mechanism network, which is set to 128, and n i is the number of instances in the package.

[0064] The weighted sum of the instances in the package is calculated to obtain the package embedding representation;

[0065] The weighted sum of the instances is calculated by the following formula to obtain the package embedding representation:

[0066]

[0067] Wherein, b i is the package embedding representation corresponding to the package with index i, n i is the number of instances in the package, and zij with a ij are the instance vector and its weight in the bag with index j respectively.

[0068] The bag embedding is input into a linear neural network with one output, normalized by a Sigmoid layer, and the output predicts the bag label.

[0069] The loss function is established to calculate the reconstruction loss of the variational autoencoder and the loss between the bag prediction result and the true value, and the weight parameters of the variational autoencoder, the attention mechanism and the multi-layer linear neural network are updated by optimizing the loss function until convergence.

[0070] The present application defines a double loss function for optimizing the VAE and the bag classifier:

[0071]

[0072] where λ is a weight coefficient, is the reconstruction loss of the variational autoencoder, is the loss between the bag prediction result and the true value.

[0073] For the reconstruction loss of the variational autoencoder, the calculation formula is as follows:

[0074]

[0075] where n i is the number of instances in the bag, x ij , z ij are the instance vector and its corresponding latent variable respectively, p θ correspond to the encoder and the decoder respectively.

[0076] The loss between the instance vector obtained by the decoder and the original instance is represented as L(x, z), which is the mean square error:

[0077] represents the KL divergence between the approximate posterior probability and the prior distribution p θ (z), which is: where M is the dimension of the mean vector μ ij and the standard deviation vector σ ij , μ ijm , σ ijm represent the mth dimension value of μ ij and σ ij .

[0078] For the loss between the bag prediction result and the true value, the cross-entropy loss function is adopted:

[0079]

[0080] where S i is the predicted bag score, Y i is the bag label ground truth.

[0081] The optimizer is an Adam optimizer, the learning rate is 5x10 -4 , the weight decay coefficient is 1x10 -4 , and the first and second momentum estimation decay rates are set to 0.8 and 0.999, respectively. During the training of the model, the BatchSize is 16, and based on the above loss function, the parameters of each layer network are updated by gradient descent until convergence.

[0082] Hardware and software configuration

[0083] The hardware configuration realized by the present application is: NVIDIA RTX 4090, 24GB video memory GPU, 16vCPU AMD EPYC965496 core processor, 60GB memory CPU, and the software platform configuration thereof is: Ubuntu 22.04, Python 3.10.8, PyTorch2.1.2.

[0084] Experimental results

[0085] In order to better illustrate the technical effects of the present application, specific examples are used to experimentally verify the present application. The experimental example uses a publicly available text classification dataset: 20 text classification datasets derived from the 20NewsGroups corpus. In each dataset, each class consists of 50 positive bags and 50 negative bags. In the positive bag, 3% of the paragraphs come from the target newsgroup, while in the negative bag, the instances are randomly selected paragraphs from other newsgroups. This dataset already contains preprocessed TF-IDF features. In this case, the present application can directly use the first 200 features extracted in the dataset during actual operation, without repeating the above preprocessing steps. Although the processed data is used in the experiment, the method of the present application is still universally applicable and can be applied to raw text data that has not been preprocessed.

[0086] Table 1 Comparison of average accuracy of the present application and other algorithms in five ten-fold cross-validation experiments on the 20NewsGroup dataset

[0087]

[0088] As can be seen from Table 1, the performance of the present application is better than that of other methods, and the highest classification accuracy is achieved on the 20NewsGroup dataset.

[0089] Corresponding to the foregoing embodiment of the method for text classification based on deep multi-instance learning of variational autoencoder, the present application also provides an embodiment of a device for text classification based on deep multi-instance learning of variational autoencoder.

[0090] Referring to Figure 3 The embodiment of the device for text classification based on deep multi-instance learning of variational autoencoder provided by the present application comprises a memory and one or more processors, the memory stores executable codes, and the processor executes the executable codes to implement the method for text classification based on deep multi-instance learning of variational autoencoder in any of the foregoing embodiments.

[0091] The embodiment of the device for text classification based on deep multi-instance learning of variational autoencoder provided by the present application can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking the software implementation as an example, as a logically meaningful device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory for execution by the processor of the device with data processing capability where the device is located. From the hardware level, as shown in Figure 3 As shown in the figure, in addition to the processor, the memory, the network interface, and the non-volatile memory shown in the figure, the device with data processing capability where the device embodiment is located usually comprises other hardware according to the actual functions of the device with data processing capability, and no further description is given. Figure 3

[0092] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, and no further description is given here.

[0093] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the part of the method embodiment. The device embodiment described above is only illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the present application scheme. Those skilled in the art can understand and implement without creative labor.

[0094] ​The embodiment of the present application further provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the deep multi-instance learning text classification method based on the variational autoencoder.

[0095] The computer readable storage medium can be an internal storage unit of any data processing capable device, such as a hard disk or a memory, of any of the preceding embodiments. The computer readable storage medium can also be an external storage device of any data processing capable device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both an internal storage unit and an external storage device of any data processing capable device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing capable device, and can also be used to temporarily store data that has been output or will be output.

[0096] The present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the deep multi-instance learning text classification method based on the variational autoencoder.

[0097] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The application is intended to cover any variations, uses or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice in the art to which the application pertains. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the application is indicated by the appended claims.

[0098] It should be understood that the general description above and the following detailed description are only exemplary and explanatory, and are not limiting to the present application. The present application is not limited to the precise structures described and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is only limited by the appended claims.

Claims

1. A deep multi-instance learning text classification method based on variational autoencoder, characterized by: The following steps are involved: Step 1: Treat the document as a package and each sentence or paragraph as an instance, and preprocess the text by feature extraction; Step 2: Normalize the data set; Step 3: Construct a multi-instance learning text classification learning network, including: extracting the embedded features of the instances in the package using a variational autoencoder, obtaining the weights of each instance using a dropout layer and an attention network, taking the weighted sum of the instances in the package as the package embedding representation, inputting the package embedding into a linear neural network, and obtaining the package prediction result after normalization. Step 4: Establish a loss function, calculate the reconstruction loss of the variational autoencoder and the loss between the packet prediction result and the true value, and update the weight parameters of the variational autoencoder, attention mechanism, and multi-layer linear neural network by minimizing the loss function through an optimization algorithm until convergence; the variational autoencoder consists of an encoder, a latent variable layer, and a decoder. The encoder encodes the latent variable, and the latent variable is decoded to obtain the reconstructed instance vector, as follows: The encoder consists of three fully connected layers. The first two layers have 256 and 128 neurons respectively, and both layers use ReLU activation function. The third layer has two branches, each with 64 neurons, which output the mean of the latent variable. and standard deviation ; The latent variables are obtained by reparameterization, and the calculation formula is as follows: ； in, is the hidden variable corresponding to the instance, and Corresponding to their mean and standard deviation respectively, represents the product between elements, is an auxiliary noise variable that obeys the standard normal distribution; The decoder consists of three fully connected layers with 64, 128, and 256 neurons respectively. The first two layers use the ReLU activation function, and the third layer directly outputs the reconstructed vector; Step 5: Preprocess and normalize the documents that need to be classified and input them into the trained multi-instance learning text classification learning network to obtain the text classification results.

2. A method for text classification based on deep multi-instance learning of variational autoencoders according to claim 1, characterized in that: The preprocessing of the text includes: processing using a predefined / pretrained feature extraction method, specifically using TF-IDF, and selecting the first n features as instance vectors.

3. A method for text classification based on deep multi-instance learning of variational autoencoders according to claim 1, characterized in that: The method of using the Dropout layer and the attention network to obtain the weight of each instance specifically includes: The instance embedding is processed by a Dropout layer with a dropout ratio of 0.5, and then the attention network is used to learn the embedding of each instance to obtain the weight of each instance: ； in, 、 is the index of the corresponding package and instance, and are the parameters learned through the neural network, is the dimension of the latent variable, is the dimension of the hidden layer in the attention mechanism network, is the number of instances in the package, is the hidden variable corresponding to the instance.

4. A method for text classification based on deep multi-instance learning of variational autoencoders according to claim 1, characterized in that: The packet embedding representation is obtained by the following formula: ； in, The index is The package embedding representation corresponding to the package of is the number of instances in the package, and The indexes in the package are The instance latent variables and their weights.

5. The method for text classification based on deep multi-instance learning of variational autoencoders according to claim 1, characterized in that: The loss function calculation formula is as follows: ； in, is the weight coefficient, is the reconstruction loss of the variational autoencoder, It is the loss between the predicted result and the true value.

6. A method for text classification based on deep multi-instance learning of variational autoencoders according to claim 5, characterized in that: The reconstruction loss of the variational autoencoder is calculated as follows: ； in, is the number of instances in the package, 、 are instance vectors and their corresponding latent variables, 、 Corresponding to the encoder and decoder respectively; Represents the loss between the instance vector obtained by the decoder and the original instance, specifically the mean square error: ; represents the approximate posterior probability and the prior distribution The KL divergence between is: ,in is the mean vector and the standard deviation vector The dimension of 、 Respectively represent and No. The value of the dimension.

7. A method for text classification based on deep multi-instance learning of variational autoencoders according to claim 5, characterized in that: The loss between the predicted result and the true value of the package adopts the cross entropy loss function: ； in, is the predicted packet score, is the true value of the bag label.

8. A deep multi-instance learning text classification device based on variational autoencoder, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a deep multi-instance learning text classification method based on a variational autoencoder is implemented as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a deep multi-instance learning text classification method based on a variational autoencoder is implemented as described in any one of claims 1 to 7.

Images