A scientific data asset multi-label text classification method based on a capsule network

Abstract

The application discloses a scientific data asset multi-label text classification method based on a capsule network, which comprises the following steps: firstly, preprocessing academic paper data sets to obtain text data; secondly, extracting and fusing features of the text data through a hybrid feature module tAL-HPYP of a multi-level hierarchical capsule network model M-CapsNet; then, transmitting the fused features to a main network M-Caps j Module of the multi-level hierarchical capsule network model, outputting a classification prediction vector to a sub-decoder network, and dynamically adjusting a global loss by using an improved hinge loss; and finally, connecting classification results of the sub-decoder network to a final decoder network, and obtaining a final multi-label prediction classification by summarizing all output labels. The application can extract deep and multi-dimensional multi-label features from scientific data texts, and improve the effect of a multi-label text classification task.

Description

Technical Field

[0001] This invention relates to the fields of deep learning, natural language processing, and text classification, and more specifically, to a multi-label text classification method for scientific data assets based on capsule networks. Background Technology

[0002] With the rapid growth of scientific research data and the increasing specialization of scientific research fields, the classification of scientific data assets has become increasingly important. In this context, the semantics and objects carried by a piece of text information can have multiple meanings, and a single label can no longer fully summarize the content; often, multiple labels are needed to summarize a piece of text information. Furthermore, scientific data assets contain a long-tailed distribution of low-frequency features, a phenomenon commonly referred to as the "long-tail effect." This phenomenon is prevalent in many real-world datasets, especially complex and diverse scientific data, and common text classification models are ineffective in handling it. Most existing multi-label classification algorithms use Word2Vec, which projects text features from a dictionary-dimensional vector space to a lower-dimensional space to obtain a dense vector representation. It is trained on unlabeled text using contextual information. For the same word, its vector representation is fixed, but the meaning of the same word often differs in different contexts. Word2vec cannot express the complete semantics of the text and loses positional features. The BERT model is currently the most widely used model in natural language processing. It introduces two tasks: the random masking task MLM (Masked Language Model), which randomly masks certain words in the text and uses contextual information for prediction; and the text pairing task NSP (NextSentence Prediction), which determines whether two sentences are contextual. These two tasks achieve information extraction at both the word and sentence dimensions. The attention mechanism and random masking in the model ensure that the generated word vectors are dynamic word vector representations that incorporate contextual information. The trained model can be fine-tuned to complete specific downstream tasks such as text classification and sequence labeling. However, due to its large number of parameters, large model size, and slow inference speed, it is not widely adopted.

[0003] Therefore, adapting to the sparsity common in scientific data and improving the accuracy and efficiency of multi-label text classification of scientific data assets has become a problem that needs to be solved. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a multi-label text classification method for scientific data assets based on capsule networks, comprising the following steps:

[0005] Step one involves preprocessing the arXiv Academic Papers Dataset (AAPD) to obtain text data. This mainly includes removing invalid symbols, normalizing numbers, converting all uppercase English characters to lowercase, indentation, etc., merging the titles and abstracts of the papers into a single input, and forming a text sequence set after preprocessing.

[0006] Step two: For the text data, semantic and topic features are extracted using the tAL-HPYP feature extraction function of the multi-level hierarchical capsule network model M-CapsNet. The tAL-HPYP module uses an ALBERT-based pre-trained model for semantic feature extraction. The input to ALBERT is the sum of token embeddings, segment embeddings, and position embeddings. After passing through a multi-layer Transformer encoder, the semantic feature vector corresponding to each word is output. First, the summary text sequence G = [g1, g2, ... g...] obtained after step one is processed... n ], where n is the sentence length of the summary, g i Let the i-th word in the abstract be the word in the title text sequence H = [h1, h2, ... h2]. m ], m is the title length, h i Add the [SEP] separator to the i-th word in the title and concatenate them to get E. input =[h1,h2,...h m [SEP], g1, g2, ... g n After E input Word embedding is performed to obtain E t , for E input Segment embedding is performed to obtain E s , for E input E is obtained by position embedding. p Finally, the formula E=E t +E s +E p The corresponding word embedding E is obtained. t Segment embedding E s and position embedding E p The sum of the three encodings, E, is input into ALBERT. In each Transformer encoder layer, the semantic expression of the text sequence is first enhanced through a multi-head self-attention mechanism. Then, the attention result is mapped to a larger-dimensional semantic space through a feedforward neural network. The non-linear expression capability of the model is enhanced through the Gelu non-linear activation function. This operation is repeated and processed through multiple encoder layers to obtain text block-level semantic feature vectors and word-level semantic feature vectors.

[0007] Token embedding: Token embedding maps each input word to a low-dimensional, dense real-valued vector space. The positions of different words in the vector space represent their semantic similarity.

[0008] Segment Embedding: Since ALBERT needs to process two segments, the title and the summary, segment embedding is used to distinguish different segments in the input sequence. It assigns a specific vector to each segment, enabling the model to distinguish whether a word belongs to the title or the summary. All words in the title are assigned the same segment embedding vector, while all words in the summary are assigned a different segment embedding vector.

[0009] Position Embedding: Transformer models themselves lack the ability to capture sequence order information, therefore position embedding is needed to explicitly encode the positional information of words within the sequence. Position embedding assigns a unique vector to each position in the sequence, enabling the model to distinguish words at different positions and thus understand the sequential relationships between words. ALBERT uses learned position embeddings—that is, it learns the vector representation corresponding to each position through training data—allowing the model to better capture long-range dependencies.

[0010] The tAL-HPYP module employs the Hierarchical Pitman-Yor Process (HPYP) model for topic feature extraction. It is a hierarchical representation of the Pitman-Yor Process (PYP), a network of PYP nodes. All distributions on the probability vector are modeled by PYP, giving it the advantages of a generative process and effectively capturing long-tailed data structures. D represents the number of text blocks in the dataset, i represents the sequence number of the text block, and Z... i Represents the text block-level topic distribution, each Z i The formula for the topic of a text block is Z. i ~PYP(α) Zi ,β Zi ,ν), i=1,...,D. In terms of vocabulary, for the topic of each word, K represents the number of words in that text block, j represents the word's index, and w j This represents the word-level topic distribution, w. j Given the PYP distribution of the parent node γ, its formula is w j ~PYP(α) wj ,β wj, γ), j = 1, ..., K. All the above α and β are the discount parameter and the concentration parameter respectively, which are the hyperparameters of HPYP, and the text block-level topic feature vector and the word-level topic feature vector are obtained respectively. Finally, the semantic feature vector and the topic feature vector are obtained through the feature extraction function of the tAL-HPYP module respectively.

[0011] Step 3: Feature fusion is performed on the extracted semantic feature vector and topic feature vector through the feature fusion function of the hybrid feature module tAL-HPYP of the multi-level hierarchical capsule network model M-CapsNet, and the text block-level information fusion strategy S1, the word-level information fusion strategy S2, and the hybrid information fusion strategy S3 are carried out. S3 is used as the input of the multi-level hierarchical capsule network and as the feature for multi-label text classification prediction.

[0012] Step 4: S3 is passed to the backbone network of the multi-level hierarchical capsule network model M-CapsNet. The M-Caps j module itself is a capsule network. Each module is responsible for processing the categories at the j-th hierarchical level in the label tree. In this way, M-Caps1 represents the coarsest level, and M-Caps N represents the finest level among the n hierarchical levels. M-Caps1, M-Caps2...M-Caps N , and a "coarse-to-fine" multi-label prediction scheme is carried out. These branches follow the coarse-to-fine paradigm, that is, all branches are used for the reconstruction term of the total loss. This enables the fine branches to utilize the features learned by the coarse branches, thereby improving the prediction accuracy. Then the logit function is used to output the classification prediction probability P, and its formula is The value range is 0 < P < 1. θ is the parameter vector of the model, representing the weight coefficient (including the bias term), and X is the classification prediction vector output by the capsule block. Finally, the prediction value provided by the current prediction layer and the output of the activation vector from the M-Caps block at the current level are passed to the sub-decoder network.

[0013] Step 5: The classification results of each sub-decoder network are connected to the final decoder network. All decoder networks use a three-layer structure. The first layer contains 512 neurons, the second layer contains 1024 neurons, and both layers use the ReLu activation function. The sigmoid function is used as the output function of the fully connected layer of the multi-level hierarchical capsule network model M-CapsNet in the third layer of the decoder network, and its formula is X H is the linear weighted summation vector of the final input, P ij is the probability of the j-th class in the i-th text block, and the value range is 0 < P ij < 1. When P ijA value of ≥0.5 indicates that the label is one of the output labels of the current text block. All output labels are combined to obtain the final predicted classification using multiple labels.

[0014] The beneficial effects of this invention are as follows:

[0015] M-CapsNet, a multi-level hierarchical capsule network model, extracts and fuses features through the hybrid feature module tAL-HPYP. It employs a text block-level information fusion strategy (S1), a word-level information fusion strategy (S2), and a hybrid information fusion strategy (S3) to achieve complementarity between fine-grained semantic and topical information, enhancing interpretability and improving adaptability to specific scientific domains. It also better handles the long-tail distribution characteristics of scientific data texts and adapts to data sparsity issues. Through a backbone multi-level hierarchical capsule network, it processes the hierarchical information of the label tree from coarse to fine. An improved loss function, utilizing an improved hinge loss, dynamically and globally adjusts the loss. This loss considers the number of classes in each hierarchical structure and the relationships between labels within the label tree. Dynamic weight adjustment modulates the contribution of each hierarchical level to the loss, while classification error is used to balance the contribution of each level, significantly improving the model's classification performance. Finally, it achieves excellent classification results on the Academic Papers Dataset (AAPD), demonstrating the superior performance of this model in multi-label text classification of scientific data assets. Attached Figure Description

[0016] Figure 1 Flowchart of multi-label text classification for scientific data assets in this invention;

[0017] Figure 2 Schematic diagram of the HPYP model of this invention;

[0018] Figure 3 The multi-level hierarchical capsule network model structure diagram of this invention;

[0019] Figure 4 Comparison of experimental results of this invention (AAPD dataset). Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the application. Rather, these embodiments are provided to make the disclosure more thorough and complete, and to fully convey the scope of the disclosure to those skilled in the art. Those skilled in the art will recognize that embodiments of the present application can be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present application can be specifically implemented as entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0021] This invention provides a multi-label text classification method for scientific data assets based on capsule networks. The process is as follows: Figure 1 As shown. This method is a multi-level hierarchical capsule network model, M-CapsNet, which extracts and fuses features through the hybrid feature module tAL-HPYP. It employs a text block-level information fusion strategy S1, a word-level information fusion strategy S2, and a hybrid information fusion strategy S3 to achieve complementarity between fine-grained semantic and topical information of features, enhancing interpretability and strengthening the adaptability to information in specific scientific domains. It also better handles the long-tail distribution characteristics of scientific data texts and adapts to data sparsity issues. Through the backbone capsule network, the hierarchical information of the label tree is processed from coarse to fine. An improved loss function, specifically an improved hinge loss, is used to dynamically and globally adjust the loss. This loss considers the number of classes in each hierarchical structure and the relationships between labels in the label tree. By dynamically adjusting the weights, the contribution of each hierarchical level to the loss is adjusted, while the classification error is used to balance the contribution of each hierarchical level, significantly improving the model's classification performance. Finally, it demonstrates superior multi-label text classification capabilities on the Academic Papers Dataset (AAPD). The specific implementation steps of this invention are as follows:

[0022] Step one involves preprocessing the arXiv Academic Papers Dataset (AAPD) to obtain text data. This mainly includes removing invalid symbols, normalizing numbers, converting all uppercase English characters to lowercase, indentation, etc., merging the titles and abstracts of the papers into a single input, and forming a text sequence set after preprocessing.

[0023] Step two: For the text data, semantic and topic features are extracted using the tAL-HPYP feature extraction function of the multi-level hierarchical capsule network model M-CapsNet. The tAL-HPYP module uses an ALBERT-based pre-trained model for semantic feature extraction. The input to ALBERT is the sum of token embeddings, segment embeddings, and position embeddings. After passing through a multi-layer Transformer encoder, the semantic feature vector corresponding to each word is output. First, the summary text sequence G = [g1, g2, ... g...] obtained after step one is processed... n ], where n is the sentence length of the summary, g i Let the i-th word in the abstract be the word in the title text sequence H = [h1, h2, ... h2]. m ], m is the title length, h i Add the [SEP] separator to the i-th word in the title and concatenate them to get E. input =[h1,h2,...h m[SEP], g1, g2, ... g n After E input Word embedding is performed to obtain E t For E input Segment embedding is performed to obtain E s For E input E is obtained by position embedding. p Finally, the formula E=E t +E s +E p The corresponding word embedding E is obtained. t Segment embedding E s and position embedding E p The sum of the three encodings, E, is input into ALBERT. In each Transformer encoder layer, the semantic expression of the text sequence is first enhanced through a multi-head self-attention mechanism. Then, the attention result is mapped to a larger-dimensional semantic space through a feedforward neural network. The non-linear expression capability of the model is enhanced through the Gelu non-linear activation function. This operation is repeated and processed through multiple encoder layers to obtain text block-level semantic feature vectors and word-level semantic feature vectors.

[0024] Obtaining text block-level semantic feature vectors: Extract each text block D from the CLS labels of the last layer of ALBERT. i The semantic feature vector C (less than 512 words in length) i , i is the sequence number of the text block, R d It is a d-dimensional real space, where d is 768, representing the internal hidden size of ALBERT. See formula (1):

[0025] C i =ALBERT(D i )∈R d (1)

[0026] Word-level semantic feature vector acquisition: For each text block D i The semantic feature vector v of each word is obtained from the tags of the last layer of ALBERT. ij , i is the index of the text block, ij represents the j-th word in the i-th text block, R d It is a d-dimensional real space, where d is 768, representing the internal hidden size of ALBERT. See formula (2):

[0027] v ij =ALBERT(D i )∈R d (2)

[0028] The tAL-HPYP module employs the Hierarchical Pitman-Yor Process (HPYP) model for topic feature extraction. It is a hierarchical representation of the Pitman-Yor Process (PYP), a network of PYP nodes where all distributions on the probability vector are modeled by PYP. It leverages the advantages of a generative process, effectively capturing long-tailed data structures. Long-tail features include high-frequency characteristics (head) such as common terms and concepts like "temperature," "humidity," and "pressure," and low-frequency characteristics (long tail) such as rare, proprietary terms or features appearing under specific conditions, like "potassium ion concentration in soil" and "electron density in the ionosphere." Topic feature extraction using HPYP is as follows: Figure 2 As shown.

[0029] At the root level, μ and γ are distributed in the form of PYP, as shown in formulas (3) and (4):

[0030] μ~PYP(α μ ,β μ H μ (3)

[0031] γ~PYP(α γ ,β γ H γ (4)

[0032] In HPYP, the variable μ is the root node of a topic, and γ is the root node of a word. To allow learning an arbitrary number of topics, μ and H... μ The base distribution of γ can be a continuous distribution or a discrete distribution with an infinite number of samples. A discrete uniform distribution of γ is chosen based on the vocabulary size of the dataset, allowing words not observed in the training set to be assigned a very small probability. Therefore, Where |ν| is the set of all words in the AAPD dataset.

[0033] For the subject aspect of HPYP, ν is a child node of μ. It follows a PYP given ν, which serves as its base distribution, as shown in Equation (5):

[0034] ν~PYP(α ν ,β ν ,μ)(5)

[0035] Text block-level topic feature vector extraction: Z i This represents the text block-level topic distribution, i.e., the text block-level topic feature vector, where each Z... i The topic corresponding to a text block is shown in formula (6), where D represents the number of text blocks in the dataset and i represents the sequence number of the text block.

[0036]

[0037] Word-level topic feature vector extraction: For the topic of each word, K represents the number of words in the text block, j represents the word index, and w j The word-level topic distribution, w, represents the word-level topic feature vector. j Given the PYP distribution of the parent node γ, see formula (7):

[0038]

[0039] All of the above α and β are hyperparameters of HPYP, with the discount parameter set to 0.5 and the concentration parameter set to 0.01, respectively.

[0040] Finally, the semantic feature vector and topic feature vector were obtained through the feature extraction function of the tAL-HPYP module.

[0041] Step 3: The extracted semantic feature vectors and topic feature vectors are fused using the feature fusion function of the tAL-HPYP hybrid feature module of the multi-level hierarchical capsule network model M-CapsNet, employing text block-level information fusion strategy S1, word-level information fusion strategy S2, and hybrid information fusion strategy S3.

[0042] 1. Text block-level information fusion strategy S1: Concatenate text block-level topic feature vectors Z line by line. i and text block-level semantic feature vector C i k is a vector Z i The dimension d is 768, representing the internal hidden size of ALBERT, k = d. See formula (8):

[0043] S1 = [Z i C i ]∈R k+d (8)

[0044] 2. Word-level information fusion strategy S2: Concatenate the word-level topic feature vectors w of each word in the text block column by column. j and word-level semantic feature vector v ij These two vectors have the same dimension, i.e., k = d, and L is the length of the vector group. See formula (9);

[0045]

[0046] 3. Hybrid Information Fusion Strategy S3: Concatenate the word-level topic feature vectors w of each word column-wise. j Text block-level topic feature vector Z i Text block-level semantic feature vector C i and word-level semantic feature vector vij The four vectors have the same dimension, i.e., k = d; see formula (10):

[0047]

[0048] The hybrid information fusion strategy S3 is used as input to a multi-level hierarchical capsule network and serves as a feature for multi-label text classification prediction.

[0049] Step four, pass S3 to the backbone network of the multi-level hierarchical capsule network model M-CapsNet, as follows: Figure 3 As shown in the figure, the M-Caps correspond to the hierarchical level. j The modules themselves are capsule networks, with each module responsible for processing the category at the j-th level in the label tree. In this way, M-Caps1 represents the coarsest level, and M-Caps... N This represents the finest level among n hierarchical levels. Therefore, each hierarchical level, from coarse to fine, has its corresponding capsule network modules, and the number of these modules is the same as the number of hierarchical levels. This is achieved through M-Caps1, M-Caps2...M-Caps... N A coarse-to-fine multi-label prediction scheme is employed, constructing multiple capsule network layers at each level of the hierarchical label tree. These branches follow a coarse-to-fine paradigm, where all branches are used for the reconstruction of the total loss. This allows finer branches to utilize features learned by coarser branches, thus improving prediction accuracy. Each M-Caps block has its own prediction layer, and the decoder used in the model combines the predictions provided by this layer with the output of the activation vectors from each M-Caps block, thereby improving classification performance.

[0050] In the model, all M-Caps blocks have the same structure. Each M-Caps block has a feature processing block to process features at the corresponding level in the following hierarchical tree. The coarser-level M-Caps block in the feature processing block has two convolutional layers, containing 128 filters. The medium-level M-Caps block consists of two convolutional layers, containing 128 and 256 filters respectively. The finer-level M-Caps block consists of three convolutional layers, containing 128, 256, and 512 filters respectively. There is a max pooling layer between each layer, and the stride between each layer is set to 2×2. All kernel sizes are set to 3×3 and the ReLu activation function is used. In addition, there is a batch normalization layer with TensorFlow default settings after each convolutional layer. Then these processed features are passed to the primary capsules in the M-Caps block. Therefore, the feature map for hierarchical level j is reshaped to fit the input shape of the primary capsule network P j The primary capsule used here consists of a convolutional capsule layer with an 8-dimensional vector. The vector output from the primary capsule P j is passed to the secondary capsule S j The secondary capsule S j is 16-dimensional, and each S j contains K j capsules, and each capsule corresponds to a class label at the j-th layer of the label tree. In the considered hierarchical level, the number of capsules is equal to the number of classes at that level. Each capsule network K j outputs a classification prediction vector for the j-th layer of the label tree. The classification prediction probability P is output using the logit function, and its formula is The value range is 0 < P < 1, θ is the parameter vector of the model, representing the weight coefficients (including the bias term), and X is the classification prediction vector output by K j

[0051] In addition, we dynamically globally adjust the loss through an improved loss function. Specifically, an improved hinge loss is used, which takes into account the number of classes in each hierarchical structure and the relationship between them in the label tree. By dynamically adjusting the weights, the contribution of each hierarchical level to the loss is regulated, and at the same time, the classification error is used to balance the contribution of each hierarchical level, significantly improving the performance of the model.

[0052] The loss function of the multi-level hierarchical capsule network model M-CapsNet is the weighted sum of all classification losses and reconstruction losses in the label tree. Therefore, the total loss of the model is defined as shown in formula (11):

[0053] L T = λL R + L C (11) ​

[0054] Where λ is a constant used to control the impact of classification loss on the total loss, L R L represents the reconstruction loss. C For details on the classification loss, please refer to formula (12):

[0055]

[0056] In formula (12), γ j It is a weight used to adjust the contribution of the j-th layer in the hierarchy to the total loss. The hinge loss of the j-th layer M-Caps block is shown in formula (13):

[0057]

[0058] In formula (13), if category k belongs to the j-th level in the hierarchy, T k,j =1; otherwise T k,j =0. m + Set to 0.9 and m - Set to 0.1, these are hyperparameters; η, set to 0.5, is a hyperparameter used to adjust the loss weights; v k,j It is the output vector of the secondary capsule corresponding to category k.

[0059] In this way, the classification loss becomes a linear combination of the hinge losses of the N-layer structure in the label tree. Furthermore, the weight γ... j λ is used to balance the hinge loss and reconstruction loss. In the reconstruction loss, the input instance x and the instance reconstructed by the final decoder are used. The L2 norm between them, and the reconstruction loss expression are shown in formula (14):

[0060]

[0061] In the multi-level hierarchical capsule network model M-CapsNet, these weights control the contribution of the classification loss to the total loss, and are therefore used to calculate the loss function L. C weight γ j This can be used to improve the accuracy of the model. All weights γ j The sum of λ and λ equals 1. First, calculate the number of categories K in the j-th layer. j The ratio to the total number of all categories in the label tree is shown in formula (15):

[0062]

[0063] Based on the above ratio, the influence of hinge loss in each layer is determined by the training accuracy, as shown in formula (16):

[0064] τ j=(1-Acc) j )·e j (16)

[0065] In formula (16), Acc j It is the training accuracy of the j-th layer in the label tree, when Acc j At lower levels, τ j The value of γ will be relatively large. The final weights for calculating the classification loss are given in formula (17), where (1-λ) represents all weights γ. j The sum of λ and λ equals 1. Used for normalization.

[0066]

[0067] Finally, the predicted values ​​provided by the current prediction layer and the output of the activation vectors from the M-Caps blocks of the current layer are passed to the sub-decoder network.

[0068] Step 5: Connect the classification results of each sub-decoder network to the final decoder network. All decoder networks use a three-layer structure. The first layer contains 512 neurons, the second layer contains 1024 neurons, and both layers use the ReLU activation function. In the third layer of the decoder network, the sigmoid function is used as the output function of the fully connected layer of the multi-level hierarchical capsule network model M-CapsNet, and its formula is: X H P is the final linear weighted summation vector of the input. ij It represents the probability of the j-th class in the i-th text block, with a value ranging from 0 to 1. <P ij <1, when P ij A value of ≥0.5 indicates that the label is one of the output labels of the current text block, such as Deep learning (label 1), Graph neural network (label 2), ..., Multi-head attention (label n), which summarizes all the output labels to obtain the final predicted classification multi-label.

[0069] Comparative Experiment: Using the same dataset (AAPD), the M-CapsNet model's multi-label text classification results were compared with those of other common network models with similar structures on this dataset. The comparative experimental results are as follows: Figure 4 As shown.

[0070] The effectiveness of the method is evaluated using precision (P), recall (R), F1 score (F1), and Hamming loss (HL).

[0071] Precision reflects the average percentage of correctly predicted labels across all samples. Its value ranges from 0 to 1; a higher value indicates greater accuracy when predicting positive examples. The calculation formula is as follows: Where TP represents true positives, the number of samples that are predicted to be positive and are actually positive, and FP represents false positives, the number of samples that are predicted to be positive but are actually negative.

[0072] Recall reflects the average percentage of correctly predicted labels and true labels among all samples. Its value ranges from 0 to 1; a higher value indicates a stronger ability of the model to capture actual positive samples. The calculation formula is as follows: Where TP represents true positives, the number of samples that are predicted to be positive and are actually positive, and FN represents false negatives, the number of samples that are predicted to be negative but are actually positive.

[0073] The F1 score is a comprehensive metric that combines precision and recall. Its value ranges from 0 to 1; a higher value indicates better model performance. The calculation formula is as follows: Where P represents precision and R represents recall.

[0074] The Hamming loss measures the proportion of misclassified labels, the proportion of incorrectly predicted labels, and the proportion of incorrectly predicted labels. Its value ranges from 0 to 1. The smaller the Hamming loss value, the better the model performs on multi-label text classification tasks. The calculation formula is as follows: Where N represents the total number of samples, y i Indicates the actual label, Indicates the predicted label, This indicates that if the predicted label and the actual label are different, the value is 1; otherwise, it is 0.

[0075] In comparative experiments, such as Figure 4 In this paper, the model M-CapsNet outperforms current common text classification models in terms of precision (P), recall (R), F1 score (F1), and Hamming loss (HL).

[0076] Finally, it should be noted that the above embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the technical scope disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

[0077] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

Claims

1. A multi-label text classification method for scientific data assets based on capsule networks, characterized in that, Includes the following steps: Step 1: Preprocess the academic paper dataset to obtain text data; Step two: For text data, feature extraction and feature fusion are performed using the tAL-HPYP hybrid feature module of the multi-level hierarchical capsule network model M-CapsNet. The specific implementation process is as follows: Step 2.1: The tAL-HPYP module uses an ALBERT-based pre-trained model for semantic feature extraction and an HPYP model for topic feature extraction. The specific implementation is as follows: ALBERT takes the sum of word embeddings, segment embeddings, and position embeddings as input, and outputs a semantic feature vector for each word after passing through a multi-layer Transformer encoder. The preprocessed summary text sequence is then used as the input. , where n is the sentence length of the summary. This represents the i-th word in the abstract, and it is compared with the title text sequence. m is the title length. Add the [SEP] separator to the i-th word in the title and concatenate them to get ,right Word embedding is performed to obtain ,right Segment embedding is performed to obtain ,right Position embedding is performed to obtain Finally, using the formula Obtain the corresponding word embedding Segment embedding and position embedding The sum of the three encodings, E, is input into ALBERT. In each Transformer encoder layer, a multi-head self-attention mechanism is used to enhance the semantic representation of the text sequence. Then, a feedforward neural network maps the attention results to the semantic space, and the Gelu nonlinear activation function enhances the nonlinear expressive power of the model. Each text block D is obtained from the CLS labels of the last layer of ALBERT. i semantic feature vector For each text block D i The semantic feature vector of each word is obtained from the tags in the last layer of ALBERT. This process is repeated through multiple encoders to obtain text block-level semantic feature vectors and word-level semantic feature vectors. The tAL-HPYP module uses the HPYP model for topic feature extraction. The HPYP model is a hierarchical version of the Pitman-Yor process (PYP), and is a network of PYP nodes. All distributions on the probability vector are modeled by PYP. D represents the number of text blocks in the dataset, and i represents the sequence number of the text block. Represents the text block-level topic distribution, each The formula corresponding to the topic of a text block is: , i=1,...,D; In terms of vocabulary, for the topic of each word, For each child node, K represents the number of words in the text block, and j represents the word index. Represents word-level topic distribution, word-level topic distribution Given the PYP distribution of the parent node γ, its formula is: j=1,...,K; all and The discount parameter and concentration parameter are used to obtain the text block-level topic feature vector and the word-level topic feature vector, respectively. The process of obtaining text block-level topic feature vectors and word-level topic feature vectors is as follows: All distributions on the probability vector of the HPYP model are modeled by PYP, which has the advantage of a generative process. At the root level, μ and Distributed in the form of PYP: The variable μ is the root node of a topic in HPYP, while It is the root node of the word, μ. The base distribution is either a continuous distribution or a discrete distribution with infinite samples; the discrete uniform distribution of γ is chosen based on the vocabulary size of the dataset. , where |ν| is the set of all words in the AAPD dataset; For a topic in HPYP, ν is a child node of μ, following a given PYP, which serves as its base distribution: Text block-level topic feature vector extraction: This represents the text block-level topic distribution, i.e., the text block-level topic feature vector, where each... The topic corresponding to a text block: ,i=1,...,D Word-level topic feature vector extraction: For the topic of each word, This represents the word-level topic distribution, also known as the word-level topic feature vector. Given the PYP distribution of the parent node γ: ,j=1,...,K All of the above and These are the discount parameter and the concentration parameter; Step 2.2: The semantic feature vector and the topic feature vector are fused using the feature fusion function of the hybrid feature module tAL-HPYP of the multi-level hierarchical capsule network model M-CapsNet; The feature fusion includes a text block-level information fusion strategy. Word-level information fusion strategy Hybrid information fusion strategy ,Will As input to a multi-level hierarchical capsule network; The specific implementation process of step 2.2 is as follows: Text block-level information fusion strategy : Connect text blocks line-by-line topic feature vectors and text block-level semantic feature vector ; Word-level information fusion strategy : Word-level topic feature vectors of each word in a text block, concatenated column by column. and word-level semantic feature vectors These two vectors have the same dimension. Hybrid Information Fusion Strategy : Word-level topic feature vectors concatenated column by column Text block-level topic feature vector Text block-level semantic feature vector and word-level semantic feature vectors The four types of vectors have the same dimension; Step 3: Pass the fused features to the backbone network M-CapsNet of the multi-level capsule network model. j The module outputs the classification prediction vector and passes it to the sub-decoder network; Step four: Connect the classification results of each sub-decoder network to the final decoder network. All decoder networks use a three-layer structure and the sigmoid function as the output function. Summarize all output labels to obtain the final predicted multi-label classification.

2. The multi-label text classification method for scientific data assets based on capsule networks according to claim 1, characterized in that, The preprocessing includes removing invalid symbols, normalizing numbers, converting all uppercase English characters to lowercase, indentation, merging the title and abstract of the paper into a single input, and forming a text sequence set, i.e., text data, after preprocessing.

3. The multi-label text classification method for scientific data assets based on capsule networks according to claim 2, characterized in that, The specific implementation process of step three is as follows: The fused features are then passed to the backbone network M-CapsNet of the multi-level capsule network model. j Modules, each module is responsible for processing the category at the j-th level in the label tree, M-Caps1 represents the coarsest level, M-Caps N This represents the finest level among n hierarchical levels, M-Caps1, M-Caps2...M-Caps N A multi-label prediction scheme is implemented, with all branches used for the reconstruction of the total loss; then, the logit function is used to output the classification prediction probability P, and finally, the prediction value provided by the current prediction layer and the output of the activation vector from the M-Caps block of the current layer are passed to the sub-decoder network; the M-Caps blocks have the same structure. The loss is dynamically and globally adjusted through an improved loss function: an improved hinge loss is used, which considers the number of classes in each hierarchy and the relationships between them in the label tree. By dynamically adjusting the weights, the contribution of each hierarchy level to the loss is adjusted, and the contribution of each hierarchy level is balanced by the classification error. The loss function of the multi-level hierarchical capsule network model M-CapsNet is the weighted sum of all classification and reconstruction losses in the label tree. The total loss is defined as: ,in, It is a constant. Indicates the reconstruction loss. Representing classification loss: It is a weight. Indicates the hinge loss of the j-th layer M-Caps block: In the above formula, category k belongs to the j-th level in the hierarchical structure. =1; otherwise =0; and It's a hyperparameter. It is a hyperparameter that adjusts the loss weights. It is the output vector of the secondary capsule corresponding to category k; In reconstructing loss In the middle, the input example is used. and instances of final decoder reconstruction Between Norm; All weights and The sum equals 1, so calculate the number of categories in the j-th layer. Ratio to the total number of all categories in the label tree The impact of hinge loss on each layer is determined by the training accuracy: It represents the training accuracy of the j-th layer in the label tree; the weights for calculating the classification loss are: Finally, the predicted values ​​provided by the current prediction layer and the output of the activation vectors from the M-Caps blocks of the current layer are passed to the sub-decoder network.

Images