Text classification method of capsule network fusing mobile flipping bottleneck convolutional network and adaptive feature channel

Abstract

The invention relates to the field of text classification methods, in particular to a text classification method fusing a mobile flip bottleneck convolution network and a capsule network of an adaptive feature channel, the text classification method is realized through an MBConv-CapsNet model, and the MBConv-CapsNet model comprises a mobile flip bottleneck convolution module used for preprocessing a text matrix and obtaining a mobile flip bottleneck convolution model; comprising a standard convolution layer, a channel-by-channel convolution layer, an SE network layer and a point-by-point convolution layer. The improved capsule network module is used for the text classification process and comprises an N-gram convolution layer, a selective kernel network layer, a main capsule layer, a convolution capsule layer and a full-connection capsule layer, sequence information, word vectors and a context dependency relationship of the text can be fully considered, local and global information of the text can be captured more effectively, and the text classification efficiency is improved. And the influence of noise and redundant information is reduced, and an original text matrix is converted into more compact and more representative feature representation.

Description

Technical Field

[0001] The present invention relates to the field of text classification methods, and specifically to a text classification method of a capsule network that integrates a moving flip bottleneck convolutional network and an adaptive feature channel. Background Art

[0002] The rapid development of the Internet has generated a large amount of data in the form of text, including web pages, news, papers, e-books, chat content, emails, user comments, etc. Faced with such a large amount of text data, how to classify and organize it so as to quickly and accurately obtain the required information is an important research issue. In text classification tasks, we usually encounter three main types: binary classification, multi-classification, and multi-label classification. The binary classification task aims to classify the text into two mutually exclusive categories, such as positive and negative emotions, positive and negative opinions, etc. The multi-classification task divides the text into multiple mutually exclusive categories, and each text can only belong to one category, such as news topic classification, question classification, etc. The multi-label classification task allows a text to belong to multiple categories at the same time. For example, a news document may involve multiple topics or multiple emotional aspects. These different types of text classification tasks each have their own unique application scenarios and challenges. Understanding their differences is crucial to building accurate and applicable text classification models.

[0003] In the past, many mainstream text classification models were generally implemented based on convolutional neural networks (CNN) and recurrent neural networks (RNN). CNN-based models mainly extract multivariate text features by using convolution windows of different scales. These text features contain rich contextual information, which can help the model better understand the text semantics. Therefore, how to accurately and comprehensively capture grammatical features is a key point to improve model performance. RNN-based models mainly perform recursive calculations along the sequence direction and propagate features in a chain manner, so that they can effectively capture sequence information. The LSTM model is an extension of the recurrent neural network. It combines a gating mechanism to more effectively capture sequential information and alleviate the problem of gradient disappearance. Although models represented by CNN and RNN can capture semantic and syntactic information, their pooling operations also bring about the problem of semantic loss, which limits the effect of text classification. In recent years, Capsule Network (Capsule Network) is an emerging deep learning model. Its core idea is to use capsules to replace neurons in convolutional neural networks, so that the network can retain detailed posture information and spatial hierarchical relationships between objects, which is expected to solve the problems of information loss and classification limitations caused by the above-mentioned traditional text classification methods. However, the application of existing capsule networks in the field of text classification still faces many challenges. First, it is difficult for capsule networks to fully consider the long-distance dependency and position dependency in text in text classification, which is particularly important in text classification tasks; second, it lacks consideration of the importance of feature channels in the text and cannot focus on capturing the key information in the text; in addition, the dynamic routing algorithm will generate a lot of noise capsules when fully connecting high-level capsules and low-level capsules, which will affect the training efficiency of the capsule network. In view of the limitations of existing text classification models, the present invention proposes a text classification method that integrates a mobile flip bottleneck convolutional network and a capsule network with adaptive feature channels. Summary of the invention

[0004] The purpose of the present invention is to provide a text classification method that integrates a mobile flip bottleneck convolutional network and a capsule network with an adaptive feature channel to solve the problems raised in the above background technology.

[0005] To achieve the above purpose, a text classification method is provided by integrating a mobile flip bottleneck convolutional network and a capsule network with an adaptive feature channel. The text classification method is implemented by an MBConv-CapsNet model. The MBConv-CapsNet model includes:

[0006] The mobile flip bottleneck convolution module is used for preprocessing of text matrices, including standard convolution layers, channel-by-channel convolution layers, SE network layers, and point-by-point convolution layers;

[0007] The improved capsule network module is used in the text classification process, including N-gram convolution layer, selective kernel network layer, main capsule layer, convolutional capsule layer and fully connected capsule layer.

[0008] Preferably, the standard convolution layer integrates a 1×1 convolution kernel, a BN layer, and a Swish activation function, and its input adopts a sentence matrix:

[0009] X=[x1,x2,K,x n-1, x n ]∈R L×d (1)

[0010] Among them, x i ∈R C is the word vector of the i-th word in the sentence, n is the sentence length, and C is the word vector dimension.

[0011] Preferably, the channel-by-channel convolution layer integrates a 3×3 depth-separable convolution kernel, a BN layer, and a Swish activation function, and the SE network layer uses a SENet attention mechanism in the channel dimension.

[0012] Preferably, the point-by-point convolution layer integrates a 1×1 convolution kernel, a BN layer, and a Dropout layer.

[0013] Preferably, the N-gram convolution layer uses B convolution filters of the same N-gram size to perform convolution operation on the sentence matrix X to obtain the N-gram feature matrix M:

[0014]

[0015] Among them, L is the length of the sentence, and K1 is the size of the N-gram when sliding on the sentence to detect features at different positions.

[0016] Preferably: the selective kernel network layer uses two convolution kernels with kernel sizes of 3 and 5 respectively for convolution, and the feature matrix V is obtained through three steps of segmentation, fusion and selection:

[0017]

[0018] Preferably, the main capsule layer uses C conversion matrices to convert the N-gram feature matrix M into a primary capsule matrix P:

[0019]

[0020] Where d is the dimension of the capsule and (L-K1+1)×C×d is the dimension of the primary capsule matrix P.

[0021] Preferably: the convolutional capsule layer updates the coupling coefficient through a dynamic routing algorithm, updates the connection weight according to the prediction consistency between capsules, and uses D transformation matrices to perform capsule convolution operation on P to obtain the capsule matrix U:

[0022]

[0023] Among them, each capsule in this layer corresponds to K2×C of the spatial area of ​​the previous layer, and (L-K1-K2+2)×D×d is the dimension of the capsule matrix U.

[0024] Preferably: the fully connected capsule layer is responsible for converting the output of the convolutional capsule layer into the final capsule vector representation, mapping the capsule vector to the classification label space through the fully connected operation, and outputting the final classification result. By calculating the activation degree of each capsule, the probability of which category the input text belongs to can be determined, y E ∈R d The capsule representing the Eth class is used to form the capsule Y representing the category:

[0025] Y=[y1,y2,...,y E ]∈R E×d (6).

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. It can fully consider the sequence information, word vectors and contextual dependencies of the text, more effectively capture the local and global information of the text, reduce the impact of noise and redundant information, and convert the original text matrix into a more compact and representative feature representation;

[0028] 2. A capsule network is proposed that can adaptively adjust the importance of different feature channels. The network includes an N-gram convolution layer, a selective kernel network layer, a main capsule layer, a convolutional capsule layer, and a fully connected capsule layer, which improves the model's ability to capture text semantic information, thereby more effectively capturing key information in the text;

[0029] 3. Use the sparsemax function instead of the softmax function for dynamic routing, so that the capsule network pays more attention to the capsules that contribute more to the final output during the routing process, and ignores those capsules that contribute less. This effectively reduces the model's dependence on noise information, improves the model's robustness, and further improves the model's classification performance.

[0030] 4. Through experimental verification on multiple public text classification datasets, the method of the present invention has achieved significant performance improvement in binary classification, multi-classification and multi-label text classification tasks compared with previous text classification models, and has better generalization ability and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A network model structure diagram of the method of the present invention;

[0032] Figure 2 This is a diagram of the mobile flip bottleneck convolution module of the present invention;

[0033] Figure 3 This is the CapsNet network structure diagram of the present invention;

[0034] Figure 4 This is a schematic diagram of the SKNet module of the present invention;

[0035] Figure 5 It is a 2D comparison diagram of the softmax function and the sparsemax function of the present invention;

[0036] Figure 6 It is a 3D comparison diagram of the softmax function and the sparsemax function of the present invention;

[0037] Figure 7 , Figure 8 , Fig. 9 , Fig.10 and Fig.11 This is a comparison of the model performance of different data sets in the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the implementation regulations described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Example

[0040] See also Figure 1 , the figure is a network model structure diagram of the present invention, a text classification method of a capsule network integrating a mobile flip bottleneck convolutional network and an adaptive feature channel, the text classification method is implemented by an MBConv-CapsNet model, and the MBConv-CapsNet model includes:

[0041] The mobile flip bottleneck convolution module is used for preprocessing of the text matrix, including standard convolution layer, channel-by-channel convolution layer, SE network layer and point-by-point convolution layer; the improved capsule network module is used for the text classification process, including N-gram convolution layer, selective kernel network layer, main capsule layer, convolution capsule layer and fully connected capsule layer.

[0042] In text classification tasks, text data usually exists in a high-dimensional and sparse form, which makes feature extraction and representation complicated. Traditional text classification methods often find it difficult to effectively capture contextual information and semantic relationships in texts when processing complex texts. In addition, the sparsity and high dimensionality of text data also bring challenges to feature extraction. In order to overcome the above problems, the present invention proposes to use moving flip bottleneck convolution (MBConv) as a preprocessing step for the text matrix. By considering the sequence information, word vector relationship or context dependency of the text, the local and global information of the text can be effectively captured, and the influence of noise and redundant information can be reduced, so as to convert the original text matrix into a more compact and representative feature representation.

[0043] MBConv, the full name of which is MobileNetV2 Convolutional Bloc, adopts the optimized version of MBConv in the EfficientNet lightweight network proposed by Google in 2019. The design of MBConv follows the principle of "lightweight and efficient" and constructs a network structure that can efficiently extract input features. Specifically, MBConv is mainly composed of four parts: standard convolution layer, channel-by-channel convolution layer (DCM), SE network layer, and point-by-point convolution layer (PCM). The depth-separable convolution k of the present invention is set to 3, and its network structure is shown in the figure below. Figure 2 shown.

[0044] The standard convolutional layer integrates a 1×1 convolution kernel, a BN layer (Batch Normalization), and a Swish

[20] activation function. The input is a sentence matrix:

[0045] X=[x1,x2,K,x n-1, x n ]∈R L×d (1)

[0046] Among them, x i ∈R C is the word vector of the i-th word in the sentence, n is the sentence length, and C is the word vector dimension. First, the input feature is dimensionally upgraded through a 1×1 convolution kernel, and the number of feature channels is flexibly adjusted. Then, the BN layer normalizes the convolved features to effectively eliminate internal covariate shifts, thereby greatly accelerating the network training process. Finally, the Swish activation function is introduced to bring nonlinear characteristics to the network, significantly improving its fitting ability, so that the low-dimensional sentence matrix can smoothly complete the feature dimension upgrade operation while maintaining the resolution.

[0047] The channel-by-channel convolution layer integrates a 3×3 depth-separable convolution kernel, a BN layer, and a Swish activation function. Its core concept is to divide the feature map after dimensionality increase into group = C groups, and use a 3×3 convolution kernel to perform channel-by-channel depth convolution on each group, that is, a single filter convolution is performed on each channel, aiming to collect the spatial features of each channel. This structure retains spatial information, helps to improve model performance, and can effectively reduce the number of trainable parameters.

[0048] The SE network layer also uses the SENet attention mechanism in the channel dimension. By assigning corresponding weights to different channels and multiplying them with feature maps, the model can focus more on the channel features with the richest information, while ignoring those unimportant channel features. It mainly includes three operations: the first is the Squeeze operation, which compresses features along the spatial dimension and converts each two-dimensional feature channel into a real number. This real number has a global receptive field to some extent, and the output dimension matches the number of input feature channels. It represents the global distribution of responses on the feature channel and enables the layer close to the input to obtain the global receptive field. The second is the Excitation operation, which is a mechanism similar to the gate in a recurrent neural network. The parameter w is used to generate weights for each feature channel, where the parameter w is learned to explicitly model the correlation between feature channels. Finally, the Scale operation regards the weight of the Excitation output as the importance of each feature channel after feature selection, and then weights it to the previous feature channel by channel through multiplication to complete the recalibration of the original feature in the channel dimension.

[0049] The point-by-point convolution layer integrates a 1×1 convolution kernel, a BN layer, and a Dropout layer. The 1×1 convolution kernel is responsible for dimensionality reduction, the BN layer is responsible for batch normalization, and the introduction of Dropout aims to prevent overfitting and accelerate the convergence of the network by randomly discarding some parameters.

[0050] The model uses the MBConv module as the initial layer of the model to extract features from the input text data. The MBConv module can effectively extract local and global features in the text and reduce redundant information through its efficient depth-separable convolution and inverted residual structure, generating a more compact and expressive feature representation. This preprocessing method not only improves the ability of the subsequent capsule network to process text data, but also helps the model better capture the contextual information and semantic relationships in the text.

[0051] The capsule network in the MBConv-CapsNet model proposed in the present invention is composed of five parts: N-gram convolution layer, selective kernel network layer, main capsule layer, convolution capsule layer, and fully connected capsule layer. In the MBConv-CapsNet model, a selective kernel network layer (SKNet) is introduced before entering the main capsule layer, so that the model can adaptively adjust the importance of different feature channels, thereby more effectively capturing key information in the text. In addition, the sparsemax function is used instead of the softmax function in the dynamic routing algorithm, making the routing weight distribution more sparse, thereby enhancing the model's ability to capture key information. The proposed capsule network structure is as follows: Figure 3 shown.

[0052] The first layer is the N-gram convolution layer, which is similar to the convolution layer in the traditional CNN, but focuses on capturing the N-gram information in the text and generating a series of local feature maps. These maps not only consider the direct adjacent relationship between words, but also contain a wider range of local statistical characteristics, providing rich local information for the subsequent capsule layer. This layer uses B convolution filters of the same N-gram size to perform convolution operations on the sentence matrix X to obtain the N-gram feature matrix M:

[0053]

[0054] Among them, L is the length of the sentence, and K1 is the size of the N-gram when sliding on the sentence to detect features at different positions.

[0055] The second layer is the selective kernel network layer, which is responsible for screening and enhancing features. It dynamically selects different convolution kernels to adapt to different input data, thereby achieving adaptive feature extraction. This layer uses two convolution kernels with kernel sizes of K1 and K2 for convolution. The feature matrix V is obtained through three steps of segmentation, fusion and selection:

[0056]

[0057] The third layer is the main capsule layer, which is responsible for converting the features output by the N-gram convolution layer into capsule representations. Unlike neurons in traditional CNNs, capsules can encode richer information, including the existence, direction, size, etc. of features. The main capsule layer converts low-level features into high-level capsule representations through a series of transformation matrices, providing more expressive features for subsequent layers. This layer uses C transformation matrices to convert the N-gram feature matrix M into the primary capsule matrix P:

[0058]

[0059] Where d is the dimension of the capsule and (L-K1+1)×C×d is the dimension of the primary capsule matrix P.

[0060] The fourth layer is the convolutional capsule layer. This layer updates the coupling coefficient through a dynamic routing algorithm and updates the connection weights according to the prediction consistency between capsules. It allows the network to automatically learn which low-level capsules are more valuable for predicting high-level capsules and adjust the connection weights accordingly. This layer uses D transformation matrices to perform capsule convolution operations on P to obtain the capsule matrix U:

[0061]

[0062] Among them, each capsule in this layer corresponds to K2×C of the spatial area of ​​the previous layer, and (L-K1-K2+2)×D×d is the dimension of the capsule matrix U.

[0063] The fifth layer is the fully connected capsule layer, which is responsible for converting the output of the convolutional capsule layer into the final capsule vector representation, mapping the capsule vector to the classification label space through the fully connected operation, and outputting the final classification result. Each capsule in this layer represents a possible category. By calculating the activation degree of each capsule, the probability of which category the input text belongs to can be determined, y E ∈R d The capsule representing the Eth class is used to form the capsule Y representing the category:

[0064] Y=[y1,y2,…,y E ]∈R E×d (6).

[0065] Although capsule networks perform well in processing structured data such as images, in text classification tasks, traditional capsule networks may not be able to fully adapt to the dynamics and complexity of text data. The key information in the text data may be distributed in different words or phrases, requiring the model to have the ability to adaptively adjust the focus. This paper adds a selective kernel network layer to the capsule network, and the model can adaptively adjust the importance of different feature channels. The SKNet layer calculates the weights of different feature channels based on the feature representation of the input text, and generates a new feature representation by weighted summation. This mechanism enables the model to adaptively adjust the importance of feature channels according to the actual situation of the text data, thereby improving the classification performance.

[0066] SKNet, the full name of Selective Kernel Net, is a kernel network that dynamically selects feature information of different scales to enhance the representation ability of the model. By adding SKNet before entering the main capsule layer, the model can better capture the key information in the text and improve the accuracy of classification by weighted fusion of feature maps of different scales. The SKNet module is mainly implemented in three steps: Split, Fuse and Select. The model structure is as follows: Figure 4 shown.

[0067] Split: Split the input text features Figure X ∈R H×W×C By dynamically selecting convolution kernels with kernel sizes K1 and K2, we can obtain feature maps of two different scales. and

[0068] Fuse: Enables neurons to adaptively adjust the size of their receptive fields according to the stimulus content. The basic idea is to use gates to control the flow of information from multiple branches to the next layer of neurons. These branches carry information of different scales. To achieve this goal, it is necessary to integrate the information of all branches. First, the results of two branches are fused by element-wise summation:

[0069] F=F1+F2 (7)

[0070] Then, the global information is embedded by simply using global average pooling to generate channel-dependent statistics s∈R C , specifically, the cth element of s is computed by shrinking F by the spatial dimensions H×W:

[0071]

[0072] In addition, there is a compact functional component z∈R d×1 , is to achieve accurate and adaptive selection of guidance, which is achieved by a simple fully connected (fc) layer to improve efficiency by reducing the dimensionality:

[0073] z=F fc (s) = δ(β(Ws)) (9)

[0074] Where δ is the ReLU function, β represents batch normalization, and W∈R d×C , in order to study the effect of d on the model efficiency, the reduction rate r is used to control its value:

[0075] d=max(C / r,L) (10)

[0076] Here, L represents the minimum value of d (L=32 is a typical setting in our experiments).

[0077] Select: Cross-channel soft attention is used to adaptively select different spatial scales of information, which are guided by the compact feature descriptor z. Specifically, a softmax operator is applied on the channel-wise numbers:

[0078]

[0079] Where A, B ∈ R C×d, a, b represent the soft attention vectors of F1 and F2 respectively, A c ∈R 1×d is the c-th row of A, ac is the c-th element of a, and so are Bc and bc. In the case of two branches, the matrix B is redundant because a c +b c =1, the final feature map Y is obtained by the attention weights on different kernels, integrating the information of multiple receptive fields and having multi-scale information:

[0080] Y c =a c ×F 1c +b c ×F 2c , a c +b c =1 (12)

[0081] where Y=[Y1,Y2,...,Y c ],Y c ∈R H×W .

[0082] In capsule networks, the basic idea of ​​dynamic routing is to construct a nonlinear mapping in an iterative manner, which is responsible for transferring the information of low-level capsules to high-level capsules and determining the activation state of capsules. The original dynamic routing algorithm uses the softmax function to ensure the non-negativity and normalization of routing weights, which helps the model learn the relative importance of each capsule during training. Its formula is defined as:

[0083]

[0084] From the formula, it can be found that the element weights calculated by the softmax function are not equal to 0, that is, even elements irrelevant to the query will be assigned a very small weight, causing all elements to have more or less influence on the prediction, which may cause some unimportant information to be given a higher weight, and the contribution of the key capsule cannot be effectively highlighted. In this invention, it is proposed to use the sparsemax function instead of the softmax function to improve the dynamic routing algorithm. The sparsemax function can generate a sparse weight vector, so that the routing weights are more concentrated on a few key capsules, thereby better highlighting their contribution to the classification task. The formula is defined as:

[0085]

[0086] A linear-time algorithm can be used to obtain the Euclidean projection of the input vector z onto the simplex, using the formula [K] = {1, K, K} and [t] +=max{0,t}, the specific implementation method of formula (9) is:

[0087] sparsemax i (z)=[z i -τ(z)] + (15)

[0088]

[0089] Among them, for each satisfying the formula ∑ j [z i -τ(z)] + = 1 for all z: K →R, the algorithm process is as shown above.

[0090] The proposed sparsemax function returns the Euclidean projection of the input vector z on the K-1 dimensional probability simplex. These projections are likely to touch the boundary of the simplex, causing some probability values ​​to drop to 0, thus obtaining a sparse probability distribution. The comparison between the softmax function and the sparsemax function is shown in the figure below. Figure 5 and Figure 6 shown.

[0091] It can be found that sparsemax is piecewise linear, but the asymptotic curve is similar to softmax. This sparsity helps the capsule network pay more attention to those capsules that contribute more to the final output during the routing process, while ignoring those capsules that contribute less. At the same time, the use of the sparsemax function can also improve the computational efficiency of the model and reduce redundant calculations, which helps to reduce the model's dependence on noise information, improve the model's robustness, and improve the performance of the capsule network in tasks such as text classification, and promote the development of the capsule network in the field of deep learning. The improved dynamic routing algorithm process is as follows:

[0092]

[0093] Example

[0094] This paper adopts four well-known public text classification datasets to evaluate the proposed MBConv-CapsNet model on text classification tasks on single-label and multi-label datasets, verifying the effectiveness of the model. The experimental model input uses 300-dimensional Word2Vec pre-trained word vectors. The training process uses the Adam optimizer with a learning rate of 1e-3 to minimize the objective function. The capsule dimension is set to 16, the training batch epoch is 50, and the number of dynamic routing algorithm iterations is 3, because it can optimize the loss faster and eventually converge to a lower loss.

[0095] The data sets used in the present invention include: movie reviews (MR), Subjectivity dataset (Subj), TREC question dataset (TREC), and Reuters-21578.

[0096] In the Reuters-21578 dataset of British Reuters financial news text, 10,788 news documents under 8 category labels related to economic and financial topics are selected for multi-label text classification tasks, and divided into three sub-datasets: Reuters-Single, Reuters-Full, and Reuters-Multi. In the Reuters-Single dataset, only single-label texts are retained as test sets, in Reuters-Full, all texts are saved as test sets, and in Reuters-Multi, only multi-label texts are retained as test sets. MR and Subj are binary classification datasets composed of movie sentiment reviews, which are used for single-label binary classification tasks. The TREC dataset comes from the corpus of the Text Retrieval Conference and is mainly used for question-answering systems and information retrieval tasks. TREC and Reuters-Single are used for single-label multi-classification tasks. Reuters-Full and Reuters-Multi are used for multi-label text classification tasks. The specific description of the experimental data is shown in Table 1.

[0097] Table 1 Experimental data description table

[0098]

[0099] In order to evaluate the classification results more accurately, the evaluation indicators used in this invention include: Accuracy (Acc), Exact Match Rate (ER), Micro-Precision (Pre), Micro-Recall (Recall), Micro-F1 (F1), and Margin Loss (ML). The specific explanations of each indicator are as follows:

[0100] Accuracy (in multi-label classification, the calculation method is the same as that of precision ER) refers to the ratio of the number of samples correctly predicted by the model to the total number of samples. Accuracy is used as the evaluation indicator in single-label text classification, while precision is used as the evaluation indicator in multi-label text classification. :

[0101] Micro-average precision refers to the ratio of true positive examples of all categories to the number of samples predicted as positive by all models:

[0102]

[0103] The micro-average recall refers to the proportion of true positive examples of all categories to all true positive examples:

[0104]

[0105] The micro-average F1 value is the harmonic mean of precision and recall, which is used to comprehensively evaluate the performance of the model:

[0106]

[0107] Among them, n represents the number of category labels, TP represents true positive examples, that is, the number of samples correctly predicted by the model as positive; TN represents true negative examples, that is, the number of samples correctly predicted by the model as negative; FP represents false positive examples, that is, the number of samples where the model incorrectly predicts negative samples as positive; FN represents false negative examples, that is, the model incorrectly predicts positive samples as negative; TPi represents the number of true positive examples of the i-th category; TNi represents the number of samples that the model correctly predicts do not belong to the i-th category as other categories; FPi represents the number of false positive examples of the i-th category; FNi represents the number of false negative examples of the i-th category.

[0108] Margin loss aims to improve the model's ability to distinguish between different categories by adjusting model parameters so that correctly classified capsules have larger activation values ​​than other capsules:

[0109] ML=y·max(0,m-preds) 2 +0.25·(1-y)·max(0,preds-m) 2 (20)

[0110] Among them, y is the true label, preds is the predicted value of the model, and m is a pre-set margin value. In the formula, for correctly classified samples, a positive margin value is used, hoping that the predicted value can be as large as possible to reduce the loss; and for incorrectly classified samples, a negative margin value is used, hoping that the predicted value can be as small as possible to increase the loss. Generally speaking, the smaller the marginal loss, the better the model's ability to represent the data and the better the classification accuracy.

[0111] In order to further verify the performance of the MBConv-CapsNet model, the present invention uses text classification models based on KimCNN, C-LSTM and Capsule-A as benchmark models for comparative experiments. KimCNN is a classic text classification model based on CNN; C-LSTM first uses CNN to extract semantic features at the phrase level, and then sends them to LSTM to determine the contextual dependencies between words; Capsule-A is a text classification model based on capsule networks. The "+" in Acc, Pre, Recall and F1 means that the higher the value, the better the effect of the model; the "-" in the ML column means that the smaller the value, the better the effect of the model. The best result of the experiment is represented by bold font. The performance comparison is carried out while ensuring that the configuration environment and the initial training parameters are consistent. From the experimental results, we can see that the MBConv-CapsNet model proposed in the present invention has obvious performance improvement in multiple evaluation indicators of binary classification, multi-classification and multi-label text classification. The experimental results are shown in Table 2, Table 3 and Table 4 respectively.

[0112] Table 2 Comparison of text classification performance on binary classification dataset

[0113]

[0114] Table 3 Comparison of text classification performance on multiple classification datasets

[0115]

[0116] Table 4 Comparison of text classification performance on multi-label datasets

[0117]

[0118] The experimental results show that the MBConv-CapsNet model proposed in this paper has achieved significant performance improvements in binary classification, multi-classification and multi-label classification compared with the performance indicators of other benchmark text classification models, and a line graph is drawn to observe the classification effect of each model. Figure 7-Figure 11 shown.

[0119] To verify the effectiveness of the mobile flip bottleneck convolution module, the improved capsule network, and the optimized dynamic routing algorithm for binary classification, multi-classification, and multi-label text classification, ablation experiments were carried out under the same experimental conditions to evaluate the impact of different modules on the performance. The experimental results are shown in Tables 5, 6, and 7, respectively.

[0120] Table 5 Ablation experiment on binary classification dataset

[0121]

[0122]

[0123] Table 6 Ablation experiments on multi-classification datasets

[0124]

[0125] Table 7 Ablation experiments on multi-label datasets

[0126]

[0127] It can be seen from the ablation experiment that in single-label text classification, by integrating the advantages of the mobile flip bottleneck convolutional network and the capsule network, the model can capture text information more comprehensively. At the same time, the use of the sparsemax function for dynamic routing makes the model of the present invention have a strong resistance to noise information. In multi-classification tasks, the proposed capsule network for adaptively adjusting the importance of feature channels is particularly effective. The model can better capture the key information in the text, thereby improving the accuracy of multi-classification tasks. Experimental results show that the model has strong robustness and versatility when processing various types of text data.

[0128] The capsule network generates text capsules through iterative updates of the dynamic routing algorithm. In order to verify the influence of the number of dynamic routing iterations on the experimental results, experiments are conducted on the Retuers-Full dataset and learning curves are drawn to show the property indicators of different routing iterations. As shown in Table 8, when the number of routing times is small, the capsule network cannot fully learn the input representation of the sequence model. As the number of iterations increases, the algorithm tends to converge when the number of routing times reaches 3 times. However, as the number of iterations continues to increase, the training process is overfitted, which reduces the classification effect. Therefore, the number of routing iterations is selected as 3 in the present invention.

[0129] Table 8 Performance comparison of different routing iteration times

[0130]

[0131] The above content is a further detailed description of the present invention in combination with specific implementation methods. It cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, some simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the scope of protection determined by the claims submitted for the present invention.

Claims

1. A text classification method that integrates a mobile flip bottleneck convolutional network and a capsule network with an adaptive feature channel, wherein the text classification method is implemented by an MBConv-CapsNet model, and is characterized in that: The MBConv-CapsNet model includes: The mobile flip bottleneck convolution module is used for preprocessing of text matrices, including standard convolution layers, channel-by-channel convolution layers, SE network layers, and point-by-point convolution layers; The improved capsule network module is used in the text classification process, including N-gram convolution layer, selective kernel network layer, main capsule layer, convolutional capsule layer and fully connected capsule layer.

2. The text classification method of the capsule network integrating the mobile flip bottleneck convolution network and the adaptive feature channel according to claim 1 is characterized in that: The standard convolution layer integrates a 1×1 convolution kernel, a BN layer, and a Swish activation function, and its input uses a sentence matrix: X =[x1,x2,K,x n-1, x n ]∈R L×d (1) Among them, x i ∈R C is the word vector of the i-th word in the sentence, n is the sentence length, and C is the word vector dimension.

3. The text classification method of the capsule network integrating the mobile flip bottleneck convolution network and the adaptive feature channel according to claim 2 is characterized by: The channel-by-channel convolution layer integrates a 3×3 depth-separable convolution kernel, a BN layer, and a Swish activation function, and the SE network layer uses the SENet attention mechanism in the channel dimension.

4. The text classification method of the capsule network integrating the mobile flip bottleneck convolution network and the adaptive feature channel according to claim 3 is characterized by: The point-by-point convolution layer integrates a 1×1 convolution kernel, a BN layer, and a Dropout layer.

5. The text classification method of the capsule network integrating the mobile flip bottleneck convolution network and the adaptive feature channel according to claim 4 is characterized in that: The N-gram convolution layer uses B convolution filters of the same N-gram size to perform convolution operations on the sentence matrix X to obtain the N-gram feature matrix M: Among them, L is the length of the sentence, and K1 is the size of the N-gram when sliding on the sentence to detect features at different positions.

6. The text classification method of the capsule network integrating the mobile flip bottleneck convolution network and the adaptive feature channel according to claim 5 is characterized by: The selective kernel network layer uses two convolution kernels with kernel sizes of 3 and 5 respectively for convolution, and the feature matrix V is obtained through three steps of segmentation, fusion and selection:

7. The text classification method of the capsule network integrating the mobile flip bottleneck convolution network and the adaptive feature channel according to claim 6 is characterized by: The main capsule layer uses C transformation matrices to convert the N-gram feature matrix M into a primary capsule matrix P: Where d is the dimension of the capsule and (L-K1+1)×C×d is the dimension of the primary capsule matrix P.

8. The text classification method of the capsule network integrating the mobile flip bottleneck convolution network and the adaptive feature channel according to claim 7 is characterized by: The convolutional capsule layer updates the coupling coefficient through a dynamic routing algorithm, updates the connection weight according to the prediction consistency between capsules, and uses D transformation matrices to perform capsule convolution operations on P to obtain the capsule matrix U: Among them, each capsule in this layer corresponds to K2×C of the spatial area of ​​the previous layer, and (L-K1-K2+2)×D×d is the dimension of the capsule matrix U.

9. According to claim 8, the fully connected capsule layer is responsible for converting the output of the convolutional capsule layer into a final capsule vector representation, mapping the capsule vector to the classification label space through a fully connected operation, and outputting the final classification result. By calculating the activation degree of each capsule, the probability of which category the input text belongs to can be determined, y E ∈R d The capsule representing the Eth class is used to form the capsule Y representing the category: Y = [y1, y2, ..., y E ]∈R E×d (6).

Images