Depth multi-instance learning text classification method based on variational automatic encoder

Abstract

The invention discloses a deep multi-instance learning text classification method based on a variational automatic encoder, and the method comprises the steps: firstly, carrying out the feature extraction and normalization of a text, enabling the variational automatic encoder to extract the embedded features of instances in a package, obtaining the weight of each instance through a Dropout layer and an attention network, enabling the weighted sum of the instances in the package to serve as a package embedded expression, and carrying out the feature extraction and normalization of the text; embedding and inputting the packet into the linear neural network, normalizing the packet to obtain a packet prediction result, establishing a loss function to train the network, and completing text classification by using the to-be-classified text and the trained network. According to the method, by introducing the variational automatic encoder, the network can fully mine information hidden in unmarked examples and extract smooth and continuous example embedding, so that the model robustness is improved, overfitting is avoided, and the text classification accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of text classification, and in particular to a deep multi-instance learning text classification method based on variational autoencoder. Background Art

[0002] Text classification is an important task in natural language processing (NLP) and is widely used in fields such as spam detection, sentiment analysis, and news classification. Traditional text classification methods mainly rely on machine learning models such as support vector machines (SVM), decision trees, and random forests, which are usually based on manual feature extraction methods such as bag-of-words (BoW) or TF-IDF. However, as the scale and complexity of data increase, traditional methods are insufficient in capturing semantic relationships and processing long texts.

[0003] Deep learning methods such as convolutional neural networks (CNN) and recurrent neural networks (RNN) can automatically extract semantic features and achieve good results on large-scale data. However, their reliance on large amounts of labeled data makes them expensive in practical applications. Therefore, weakly supervised learning models have become an important research direction for handling complex text classification tasks.

[0004] As a typical weakly supervised learning method, multiple instance learning (MIL) provides a new and effective solution for text classification research. In MIL, data is organized in the form of bags, which contain multiple instances (such as sentences and paragraphs in a document), but only the labels of the bags are known, and the labels of the instances are unknown. The goal of MIL is to learn bag-level labels and infer the contribution of each instance to complete the classification task.

[0005] In recent years, some researchers have tried to combine deep neural networks (DNNs) with MIL and have achieved great improvements. Among them, attention-based deep multi-instance learning has shown excellent performance and has received widespread attention from researchers. The instance embedding network (used to extract instance features) in this algorithm may receive meaningful gradient signals from some instances, which contribute significantly to bag embedding. In the real world, labels are difficult to obtain. Unlike traditional machine learning, multi-instance learning has features at both bag and instance levels. Current deep multi-instance learning methods focus on extracting effective bag representations, ignoring the objective fact that bag embeddings are aggregated from instance features. When the number of samples is limited, the supervision signal of MIL is weak, making it difficult for the instance embedding network to learn effective representations, resulting in a significant overfitting trend in the overall model. In addition, although the number of annotated bags is limited, the bags still contain a large number of unannotated instances. The number of instances greatly exceeds the number of data packets, and there is a lot of information in the instances, but this information is largely ignored. Summary of the invention

[0006] In view of the shortcomings of the above-mentioned multi-instance learning in text classification tasks, the present invention proposes a deep multi-instance learning text classification method based on variational autoencoder. In view of the problem that the existing attention-based deep multi-instance learning algorithm is difficult to learn effective instance embedding, resulting in overfitting of the overall model, the present invention introduces a variational autoencoder, so that the network can fully mine the information hidden in the unlabeled instances and extract smooth and continuous instance embedding, thereby improving the model robustness, avoiding overfitting, and improving the performance of downstream tasks.

[0007] In order to achieve the above object, the present invention provides a deep multi-instance learning text classification method based on variational autoencoder, and the steps are as follows:

[0008] The object of the present invention is to achieve the following technical solution: a deep multi-instance learning text classification method based on variational autoencoder, comprising the following steps:

[0009] Step 1: Treat the document as a package, each sentence or paragraph in it as an instance, and preprocess the text by feature extraction;

[0010] Step 2: Normalize the data set;

[0011] Step 3: Construct a multi-instance learning text classification learning network, including: extracting the embedding features of the instances in the package using a variational autoencoder, using the Dropout layer and the attention network to obtain the weights of each instance, taking the weighted sum of the instances in the package as the package embedding representation, inputting the package embedding into a linear neural network, and obtaining the package prediction result after normalization.

[0012] Step 4: Establish a loss function, calculate the reconstruction loss of the variational autoencoder and the loss between the packet prediction result and the true value, and use the optimization algorithm to minimize the loss function to update the weight parameters of the variational autoencoder, attention mechanism, and multi-layer linear neural network until convergence;

[0013] Step 5: Preprocess and normalize the documents that need to be classified and input them into the trained multi-instance learning text classification learning network to obtain the text classification results.

[0014] Furthermore, the preprocessing of the text includes: processing using a predefined / pretrained feature extraction method, specifically using TF-IDF, and selecting the first n features as instance vectors.

[0015] Furthermore, the variational autoencoder is composed of an encoder, a latent variable layer and a decoder, as follows:

[0016] The encoder consists of three fully connected layers. The first two layers have 256 and 128 neurons respectively, and both layers use ReLU activation function. The third layer has two branches, each with 64 neurons, which output the mean μ and standard deviation σ of the latent variable respectively.

[0017] The latent variables are obtained by reparameterization, and the calculation formula is as follows:

[0018] z=μ+σ☉∈

[0019] Among them, z is the hidden variable corresponding to the instance, μ and σ correspond to its mean and standard deviation respectively, ☉ represents the product between elements, and ∈ is the auxiliary noise variable that obeys the standard normal distribution;

[0020] The decoder consists of three fully connected layers with 64, 128, and 256 neurons respectively. The first two layers use the ReLU activation function, and the third layer directly outputs the reconstructed vector.

[0021] Furthermore, the method of using the Dropout layer and the attention network to obtain the weight of each instance specifically includes:

[0022] The instance embedding is processed through a Dropout layer with a dropout ratio of 0.5, and then the attention network is used to learn the embedding of each instance to obtain the weight of each instance:

[0023]

[0024] Among them, i and j are the indexes of the corresponding packages and instances. and is the parameter learned by the neural network, M is the dimension of the latent variable, L is set to 128, and n i is the number of instances in the package.

[0025] Furthermore, the packet embedding representation is obtained by the following formula:

[0026]

[0027] Among them, b i is the package embedding representation corresponding to the package with index i, n i is the number of instances in the package, z ij with a ij are the instance vector with index j in the bag and its weight respectively.

[0028] Furthermore, the loss function calculation formula is as follows:

[0029]

[0030] Among them, λ is the weight coefficient, is the reconstruction loss of the variational autoencoder, It is the loss between the predicted result and the true value of the bag.

[0031] Furthermore, the reconstruction loss of the variational autoencoder is calculated as follows:

[0032]

[0033] Among them, n i is the number of instances in the package, x ij 、z ij are instance vectors and their corresponding latent variables, p θ Corresponding to the encoder and decoder respectively.

[0034] Represents the loss of the instance vector obtained by the decoder and the original instance, specifically the mean square error:

[0035] represents the approximate posterior probability and the prior distribution p θ (z), specifically: Where M is the mean vector μ ij and the standard deviation vector σ ij The dimension of ijm , σ ijm Represent μ ij and σ ij The value of the mth dimension of .

[0036] Furthermore, the loss between the packet prediction result and the true value adopts the cross entropy loss function:

[0037]

[0038] Among them, S i is the predicted packet score, Y i is the true value of the bag label.

[0039] On the other hand, the specification also provides a deep multi-instance learning text classification device based on a variational autoencoder, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the deep multi-instance learning text classification method based on a variational autoencoder.

[0040] On the other hand, the specification also provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the deep multi-instance learning text classification method based on a variational autoencoder is implemented.

[0041] The beneficial effects of the present invention are as follows: 1) The variational autoencoder is introduced to extract the deep features of the instance in an unsupervised learning manner, and the reconstruction error is added to the loss function. 2) In addition, the variational autoencoder can also constrain the distribution of the latent space where the extracted features are located, so that the extracted features are smoother and more robust. Practice has proved that the method proposed in the present invention can improve the robustness of the model, avoid overfitting, and effectively improve the classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the network structure of the present invention;

[0043] Figure 2 Schematic diagram of bag embedding obtained through pooling of attention mechanism;

[0044] Figure 3 A schematic diagram of a deep multi-instance learning text classification device based on a variational autoencoder provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0046] like Figure 1 As shown, the present invention proposes a deep multiple instance learning text classification method based on variational autoencoder, which is used to improve the robustness of the multiple instance learning (MIL) model under limited supervision signals and avoid overfitting problems. The following specific embodiments describe in detail the working principle and implementation steps of the present invention:

[0047] Text feature extraction: treat the document as a bag, each sentence or paragraph in it as an instance, and preprocess the text through predefined / pretrained feature extraction methods;

[0048] Use TF-IDF to extract text features of each paragraph and select the first 200 features as instance vectors;

[0049] Dataset Normalization

[0050] The dataset is normalized using the following formula:

[0051]

[0052] Here, μ and σ are the mean and standard deviation of all instances in the dataset, respectively.

[0053] Extract embedding features of instances in the bag using a variational autoencoder algorithm;

[0054] The variational autoencoder (VAE) of the present invention is composed of an encoder, a latent variable layer and a decoder. The encoder encodes to obtain latent variables, and the latent variables are decoded to obtain a reconstructed instance vector as follows:

[0055] The encoder consists of three fully connected layers. The first two layers have 256 and 128 neurons respectively, and both layers use the ReLU activation function. The third layer has two branches, each with 64 neurons, which output the mean μ and standard deviation σ of the latent variable respectively;

[0056] The latent variables are obtained through the reparameterization technique, and the calculation formula is as follows:

[0057] z=μ+σ☉∈

[0058] Among them, z is the hidden variable corresponding to the instance, μ and σ correspond to its mean and standard deviation respectively, ⊙ represents the product between elements, and ∈ is the auxiliary noise variable that obeys the standard normal distribution;

[0059] The decoder consists of three fully connected layers with 64, 128, and 256 neurons respectively. The first two layers use the ReLU activation function, and the third layer directly outputs the reconstructed vector.

[0060] After the instance embedding is processed by the Dropout layer, an attention network is constructed to learn it and obtain the weights of each instance;

[0061] The dropout ratio of the Dropout layer is 0.5 to prevent the model from overfitting. After the latent variables output by the encoder are processed by the Dropout layer, they enter the attention network to learn the contribution of each instance to the package embedding. The weight calculation formula is as follows:

[0062]

[0063] Among them, i and j are the indexes of the corresponding packages and instances. and is the parameter learned by the neural network, M is the dimension of the hidden variable (i.e. 64), L is the dimension of the hidden layer in the attention mechanism network, which is set to 128, and n i is the number of instances in the package.

[0064] Calculate the weighted sum of instances in the bag to get the bag embedding representation;

[0065] The weighted sum of instances is calculated using the following formula to get the bag embedding representation:

[0066]

[0067] Among them, b i is the package embedding representation corresponding to the package with index i, n i is the number of instances in the package, zij with a ij are the instance vector with index j in the bag and its weight respectively.

[0068] The package embedding representation is input into a single-output linear neural network, normalized by a Sigmoid layer, and outputs the predicted package label.

[0069] Establish a loss function, calculate the reconstruction loss of the variational autoencoder and the loss between the packet prediction result and the true value, and use the optimization algorithm to minimize the loss function to update the weight parameters of the variational autoencoder, attention mechanism, and multi-layer linear neural network until convergence;

[0070] The present invention defines a dual loss function for optimizing VAE and packet classifier:

[0071]

[0072] Among them, λ is the weight coefficient, is the reconstruction loss of the variational autoencoder, It is the loss between the predicted result and the true value of the bag.

[0073] For the reconstruction loss of the variational autoencoder, the calculation formula is as follows:

[0074]

[0075] Among them, n i is the number of instances in the package, x ij 、z ij are instance vectors and their corresponding latent variables, p θ Corresponding to the encoder and decoder respectively.

[0076] Represents the loss of the instance vector obtained by the decoder and the original instance, specifically the mean square error:

[0077] represents the approximate posterior probability and the prior distribution p θ (z), specifically: Where M is the mean vector μ ij and the standard deviation vector σ ij The dimension of ijm , σ ijm Represent μ ij and σ ij The value of the mth dimension of .

[0078] For the loss between the package prediction result and the true value, the cross entropy loss function is adopted:

[0079]

[0080] Among them, S i is the predicted packet score, Y i is the true value of the bag label.

[0081] The optimizer is Adam optimizer, and the learning rate is 5×10 -4 , the weight decay coefficient is 1×10 -4 , the first-order and second-order momentum estimation decay rates are set to 0.8 and 0.999 respectively. When the model is trained, the BatchSize is 16, and based on the above loss function, the parameters of each layer of the network are updated by gradient descent until convergence.

[0082] Software and hardware configuration

[0083] The hardware configuration implemented by the present invention: NVIDIA RTX 4090, 24GB video memory GPU, 16vCPU AMD EPYC965496-core processor, 60GB memory CPU, and its software platform configuration: Ubuntu 22.04, Python 3.10.8, PyTorch2.1.2.

[0084] Experimental Results

[0085] In order to better illustrate the technical effect of the present invention, the present invention is experimentally verified by using specific examples. This experimental example uses a public text classification data set: 20 text classification data sets derived from the 20NewsGroups corpus. In each data set, each category consists of 50 positive bags and 50 negative bags. In the positive bag, 3% of the paragraphs come from the target news group, while in the negative bag, the examples are paragraphs randomly selected from other news groups. The data set already contains pre-processed TF-IDF features. In view of this situation, the present invention can directly use the first 200 features extracted from the data set in actual operation, without repeating the above pre-processing steps. Although the processed data is used in the experiment, the method of the present invention is still generally applicable and can be applied to raw text data that has not been pre-processed.

[0086] Table 1 Comparison results of the average accuracy of the present invention and other algorithms in five ten-fold cross validation experiments on the 20NewsGroup dataset

[0087]

[0088] It can be seen from Table 1 that the performance of the present invention is better than other methods, and the highest classification accuracy is achieved on the 20NewsGroup dataset.

[0089] Corresponding to the aforementioned embodiment of a deep multi-instance learning text classification method based on a variational autoencoder, the present invention also provides an embodiment of a deep multi-instance learning text classification device based on a variational autoencoder.

[0090] See also Figure 3 An embodiment of the present invention provides a deep multi-instance learning text classification device based on a variational autoencoder, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a deep multi-instance learning text classification method based on a variational autoencoder in the above embodiment.

[0091] The embodiment of the deep multi-instance learning text classification device based on variational autoencoder provided by the present invention can be applied to any device with data processing capability, and the device with data processing capability can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capability in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From the hardware level, if Figure 3 As shown, it is a hardware structure diagram of a device with data processing capability for deep multi-instance learning text classification device based on variational autoencoder provided by the present invention, except Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0092] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0093] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0094] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a deep multi-instance learning text classification method based on a variational autoencoder in the above embodiment is implemented.

[0095] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0096] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the deep multi-instance learning text classification method based on variational autoencoder.

[0097] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0098] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.

Claims

1. A deep multi-instance learning text classification method based on variational autoencoder, characterized in that: The following steps are involved: Step 1: Treat the document as a package, each sentence or paragraph in it as an instance, and preprocess the text by feature extraction; Step 2: Normalize the data set; Step 3: Construct a multi-instance learning text classification learning network, including: extracting the embedding features of the instances in the package using a variational autoencoder, using the Dropout layer and the attention network to obtain the weights of each instance, taking the weighted sum of the instances in the package as the package embedding representation, inputting the package embedding into a linear neural network, and obtaining the package prediction result after normalization. Step 4: Establish a loss function, calculate the reconstruction loss of the variational autoencoder and the loss between the packet prediction result and the true value, and use the optimization algorithm to minimize the loss function to update the weight parameters of the variational autoencoder, attention mechanism, and multi-layer linear neural network until convergence; Step 5: Preprocess and normalize the documents that need to be classified and input them into the trained multi-instance learning text classification learning network to obtain the text classification results.

2. A method for text classification based on deep multi-instance learning of variational autoencoders according to claim 1, characterized in that: The preprocessing of the text includes: processing using a predefined / pretrained feature extraction method, specifically using TF-IDF, and selecting the first n features as instance vectors.

3. The method for deep multi-instance learning text classification based on variational autoencoder according to claim 1, characterized in that: The variational autoencoder consists of an encoder, a latent variable layer and a decoder. The encoder encodes the latent variable, and the latent variable is decoded to obtain a reconstructed instance vector, as follows: The encoder consists of three fully connected layers. The first two layers have 256 and 128 neurons respectively, and both layers use ReLU activation function. The third layer has two branches, each with 64 neurons, which output the mean μ and standard deviation σ of the latent variable respectively. The latent variables are obtained by reparameterization, and the calculation formula is as follows: z=μ+σ⊙∈ Among them, z is the hidden variable corresponding to the instance, μ and σ correspond to its mean and standard deviation respectively, ⊙ represents the product between elements, and ∈ is the auxiliary noise variable that obeys the standard normal distribution; The decoder consists of three fully connected layers with 64, 128, and 256 neurons respectively. The first two layers use the ReLU activation function, and the third layer directly outputs the reconstructed vector.

4. The method for deep multi-instance learning text classification based on variational autoencoder according to claim 1, characterized in that: The method of using the Dropout layer and the attention network to obtain the weights of each instance specifically includes: The instance embedding is processed through a Dropout layer with a dropout ratio of 0.5, and then the attention network is used to learn the embedding of each instance to obtain the weight of each instance: Among them, i and j are the indexes of the corresponding packages and instances. and is the parameter learned by the neural network, M is the dimension of the hidden variable, L is the dimension of the hidden layer in the attention mechanism network, and n i is the number of instances in the package.

5. The method for deep multi-instance learning text classification based on variational autoencoder according to claim 1, characterized in that: The packet embedding representation is obtained by the following formula: Among them, b i is the package embedding representation corresponding to the package with index i, n i is the number of instances in the package, z ij with a ij are the instance vector with index j in the bag and its weight respectively.

6. The method for deep multi-instance learning text classification based on variational autoencoder according to claim 1, characterized in that: The loss function calculation formula is as follows: Among them, λ is the weight coefficient, is the reconstruction loss of the variational autoencoder, It is the loss between the predicted result and the true value of the bag.

7. A method for deep multi-instance learning text classification based on variational autoencoder according to claim 6, characterized in that: The reconstruction loss of the variational autoencoder is calculated as follows: Among them, n i is the number of instances in the package, x ij 、z ij are instance vectors and their corresponding latent variables, pθ corresponds to the encoder and decoder respectively; Represents the loss of the instance vector obtained by the decoder and the original instance, specifically the mean square error: represents the approximate posterior probability and the prior distribution p θ (z), specifically: Where M is the mean vector μ ij and the standard deviation vector σ ij The dimension of ijm , σ ijm Represent μ ij and σ ij The value of the mth dimension of .

8. The method for deep multi-instance learning text classification based on variational autoencoder according to claim 6, characterized in that: The loss between the predicted result and the true value of the package adopts the cross entropy loss function: Among them, S i is the predicted packet score, Y i is the true value of the bag label.

9. A deep multi-instance learning text classification device based on variational autoencoder, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a deep multi-instance learning text classification method based on a variational autoencoder as described in any one of claims 1-8 is implemented.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a deep multi-instance learning text classification method based on a variational autoencoder as described in any one of claims 1 to 8 is implemented.

Images