A Chinese Short Text Classification Method and System Based on Multi-Scale Features and Association Features

Abstract

This invention discloses a Chinese short text classification method and system based on multi-scale features and association features, belonging to the field of natural language processing. The method includes: acquiring and preprocessing a text dataset and a keyword dataset; extracting text features using a BERT model; extracting multi-scale semantic features of the text using multiple different convolutional layers; calculating the association information of the preprocessed text and constructing a text graph; extracting text association features using a graph neural network; fusing the multi-scale semantic features and text association features using a cross-attention mechanism to obtain fused features; constructing a text classifier; inputting the fused features into the text classifier to obtain classification results; evaluating the classification results; calculating evaluation metrics and iteratively training the text classifier to optimize its parameters. This invention enhances text features by combining multi-scale semantic features and text association features, thereby further improving the accuracy of Chinese short text classification.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and more specifically, to a method and system for classifying short Chinese texts based on multi-scale features and association features. Background Technology

[0002] Text classification, as a technique within these research fields, aims to categorize text data into predefined categories, enabling rapid identification of unknown document types and thus improving the efficiency of processing large amounts of information. Applying this technology can save significant information processing costs and effectively filter out the desired information. Specific applications include news type identification, spam SMS detection, and text sentiment analysis.

[0003] Currently, most text classification methods employ deep learning-based algorithms, which can be broadly categorized into several types: Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Transformers, Large Language Models (GLLMs), and Graph Neural Networks (GNNs). RNN-based algorithms effectively capture contextual features but struggle with processing locally important features. CNNs, while capable of capturing locally important features, fail to extract contextual features. Transformers simultaneously capture both contextual and locally important features, while GLLMs offer strong adaptability. However, in these algorithms, text features are input as vectors, each existing independently, neglecting the relationships between features. For text classification, the relationships between features provide valuable information. To further utilize these relationships, GNNs were developed. In text classification, GNNs consider the relationships between different features, but their input data requires the construction of a text graph, which is quite complex.

[0004] Therefore, existing text classification methods struggle to capture contextual features while simultaneously acquiring text features at different scales and effectively utilizing the correlation information of these features, thus hindering further improvements in text classification accuracy. Summary of the Invention

[0005] To overcome the shortcomings of the existing technology, which is difficult to capture contextual features while acquiring text features at different scales and effectively utilizing the correlation information of features, thus making it difficult to further improve the accuracy of text classification, this invention provides a Chinese short text classification method based on multi-scale features and correlation features.

[0006] To achieve the above-mentioned objectives, the technical solution of the present invention includes the following steps:

[0007] A set of labeled text data and a preset set of keywords are obtained and preprocessed to obtain a text set and a vocabulary. The preprocessing includes text segmentation, stop word filtering, vocabulary construction, text encoding, keyword encoding, tag extraction, and tag one-hot encoding.

[0008] The BERT model is used to extract text features from the text set, and then multiple different convolutional layers are used to extract multi-scale semantic features of the text features.

[0009] Calculate word association information in the text set and the vocabulary, construct a text graph based on the association information, and then use a graph neural network to extract text association features from the text graph;

[0010] The extracted multi-scale semantic features of the text and the text association features are fused using a cross-attention mechanism to obtain fused features;

[0011] A text classifier is constructed, and the fused features are input into the text classifier to obtain the classification result.

[0012] Furthermore, the preprocessing method specifically includes:

[0013] The jieba library, a Chinese word segmentation model, was used to segment the tagged text dataset, resulting in segmented text. This segmented text was then compared with the Harbin Institute of Technology stop word list, and stop words were removed to obtain a text set in the following form:

[0014] Text=[token1,token2,token3,…,token N ]

[0015] Among them, token N This represents the Nth word of the text;

[0016] Construct a vocabulary list by adding each word from the text set to the vocabulary list and removing duplicate parts of duplicate words in the vocabulary list;

[0017] The text in the text set is vectorized using an encoder to obtain text vectors;

[0018] Then, the encoder is used to vectorize all the word segments in the vocabulary to obtain word vectors;

[0019] The topic terms in the topic term set are vectorized using an encoder to obtain topic term vectors.

[0020] Extract the tags from the tagged text set and use the tags as topic tags. Then, use a one-hot encoder to vectorize the topic tags to obtain the one-hot vectors of the topic tags.

[0021] Furthermore, the steps for extracting multi-scale semantic features of the text are as follows:

[0022] First, the text vector is input into the BERT model, and the output of the BERT model is obtained as the text semantic features.

[0023] The text semantic features are input into the multiple different convolutional layers, and the outputs of the different convolutional layers are obtained to obtain text semantic features at multiple scales.

[0024] A cross-attention mechanism is used to fuse the text semantic features at multiple scales to obtain multi-scale text semantic features.

[0025] Furthermore, the multiple scales of text semantic features specifically include:

[0026] Small-scale text semantic features:

[0027]

[0028] Mesoscale text semantic features:

[0029]

[0030] Large-scale text semantic features:

[0031]

[0032] The small, medium, and large-scale text semantic features are obtained by convolving the text semantic features with convolutional layers of kernel sizes of 3*768, 4*768, and 5*768, respectively, where h s h m h l represents the dimensions of text semantic features at small, medium, and large scales, respectively, and fs = 256 represents the number of convolution kernels.

[0033] Furthermore, the specific steps for calculating the associated information are as follows:

[0034] The word vectors are input into the BERT model, and the output of the BERT model is obtained as the word features. Then, the topic word vectors are input into the BERT model, and the output of the BERT model is obtained as the topic word features.

[0035] A word embedding model is used to obtain word embedding vectors of the word features, the topic word features, and the text semantic features. Based on the word embedding vectors, association information is calculated, including text-word association information, word-topic word association information, word-word association information, and text-text association information.

[0036] The text-word association information, as a feature of the edge between text node and word node, is measured by the TF-IDF value of the word. The specific formula for calculating the TF-IDF value of the word is as follows:

[0037]

[0038] TF-IDF (word i ,doc j ) = TF * IDF

[0039] Where TF represents word frequency, Sum(word i ,doc j Sum(word) represents the number of times word i appears in text j. i The dataset represents the total number of times word i appears in the entire corpus, IDF represents inverse document frequency, N represents the total number of texts in the corpus, and df t This indicates the number of texts containing the word 'i'.

[0040] The word-topic term association information is used as a feature of the edge between word nodes and topic term nodes, determined by the similarity S between words and topic terms. word The specific formula for calculating the similarity between the stated words and the topic words is as follows:

[0041]

[0042] Among them, S word Indicates the similarity between words and topic words. wordi V is the word embedding vector of the i-th word in the vocabulary. topicj It is the word embedding vector of the j-th topic word;

[0043] The word-word association information, as a feature of the edges between word nodes, is measured by the mutual information between words. The specific formula for calculating the mutual information between words is as follows:

[0044]

[0045] Among them, PMI (word i ,word j P(word) represents the mutual information between word i and word j. i ,wordj P(word) represents the probability that word i and word j appear simultaneously. i P represents the probability of word i appearing. j () represents the probability of word j appearing;

[0046] The text-to-text association information, as a feature of the edges between text nodes, is determined by the cosine similarity S between two texts in the text set. text The cosine similarity S between the two texts is measured. text The specific calculation formula is as follows:

[0047]

[0048] Among them, V i V j Let i and j represent the word embedding vectors of text i and text j, respectively.

[0049] Furthermore, the specific steps for constructing a text graph based on association information and then using a graph neural network to extract text association features from the text graph are as follows:

[0050] A text graph G is constructed based on the calculated TF-IDF values ​​of the words, the similarity between words and topic words, the mutual information between words, and the cosine similarity between texts.

[0051] G = (V, E)

[0052] Where V is the vertex set, representing the nodes in the text graph, containing words, keywords, and text; E is the edge set, representing the association information between different nodes; the method for constructing the text graph can be represented as:

[0053]

[0054] Among them, TF-IDF ij PMI represents the TF-IDF matrix composed of the TF-IDF values ​​of each word in the vocabulary. wordi-wordj Similarity represents the word mutual information matrix composed of the mutual information between words in the vocabulary. wordi-topicj Cos Similarity(doci,docj) represents the word-topic similarity matrix composed of the similarity between each word in the vocabulary and the topic word, while Cos Similarity(doci,docj) represents the text similarity matrix composed of the cosine similarity between each text.

[0055] Using the method described above for constructing the text graph, an adjacency matrix can be obtained.

[0056]

[0057] Where s2 represents the total number of nodes, and the adjacency matrix A is the text graph;

[0058] The text semantic features are used as text node features, the word features are used as word node features, and the topic word features are used as topic word node features;

[0059] The text node features, word node features, topic word node features, and the text graph are input into a graph neural network to obtain the output of the graph neural network. And as a text association feature, where s2 represents the total number of nodes, d G Dimensions representing text association features.

[0060] Furthermore, the specific steps for fusing the extracted multi-scale semantic features of the text with the text association features using a cross-attention mechanism to obtain the fused features are as follows:

[0061] A linear transformation is performed on the multi-scale semantic features and the text association features to calculate the query matrix Q based on the multi-scale semantic features ω. ω Based on text association features G f key matrix Sum matrix The calculation formula is as follows:

[0062] Q ω =W Q

[0063]

[0064] in, This represents the query matrix obtained based on the multi-scale semantic features ω of the text. G represents text association features f The obtained key matrix, G represents text association features f The resulting value matrix. d represents the weight matrices needed to calculate the query matrix, key matrix, and value matrix, respectively. t d represents the dimension of the multi-scale semantic features of the text. G Dimensions representing text association features;

[0065] The multi-scale semantic features ω and text association features G are calculated based on the obtained query matrix, key matrix, and value matrix. f The cross-attention weight matrix CA(,G) f The calculation formula is as follows:

[0066]

[0067] in, Key matrix Dimensions That is, the multi-scale semantic features ω and the text association features G of the text. f The fused features are obtained after feature fusion through cross-attention mechanism.

[0068] Furthermore, the specific steps for constructing the text classifier, which involve inputting the fused features of the text to be classified into the text classifier to obtain the classification result, are as follows:

[0069] Construct a text classifier that uses fully connected layers to convert input features into probability distributions for each category and initialize the text classifier;

[0070] The fusion features of the text to be classified are input into the text classifier to obtain the output of the text classifier. The output is the probability distribution of each category. The topic word with the highest probability in each category probability distribution is selected as the classification result of the text to be classified.

[0071] Furthermore, after obtaining the classification result, the method further includes the following steps:

[0072] The classification result is evaluated by encoding it into a one-hot vector. The one-hot vector of the classification result is compared with the one-hot vector of the topic label. If they are the same, the prediction result is recorded as a correct prediction; otherwise, it is recorded as a wrong prediction.

[0073] All texts to be classified are classified using the text classifier and evaluation metrics are calculated based on the prediction results. The evaluation metrics include accuracy, precision, recall, and F1 score.

[0074] The text classifier is iteratively trained based on the evaluation results and evaluation metrics to optimize its parameters.

[0075] To achieve the objective of the invention, another aspect of the present invention provides a Chinese short text classification system based on multi-scale features and joint features, the system comprising:

[0076] The text preprocessing module is used to preprocess the labeled text data set and the keyword set. The preprocessing includes text segmentation, stop word filtering, vocabulary construction, text encoding, keyword encoding, tag extraction, and tag one-hot encoding.

[0077] The multi-scale semantic feature extraction module is used to extract multi-scale semantic features from the preprocessed text.

[0078] The text graph construction and association feature extraction module is used to construct text graphs based on text sets and vocabularies and extract the text association features contained in the text graphs.

[0079] The feature fusion module is used to fuse multi-scale semantic features of text with text association features;

[0080] The text classification and evaluation module is used to build a classifier, classify text, evaluate the classification results, calculate evaluation metrics, and iteratively train and optimize the text classifier based on the evaluation results and metrics.

[0081] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0082] This invention combines multi-scale semantic features and text association features to enhance text features, thereby further improving the accuracy of Chinese short text classification. This can not only mine text features at multiple scales but also take into account the global context information, local feature information, and the association information between texts. Attached Figure Description

[0083] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0084] Figure 1 This is a schematic diagram of the implementation process of the text classification method of the present invention and a schematic diagram of the classification system modules;

[0085] Figure 2 This is a schematic diagram of the text preprocessing module of the text classification system of the present invention;

[0086] Figure 3 This is a schematic diagram of the multi-scale semantic feature extraction module of the text classification system of the present invention;

[0087] Figure 4 This is a schematic diagram of the text graph construction and associated feature extraction module of the text classification system of the present invention;

[0088] Figure 5 This is a schematic diagram of the feature fusion module of the text classification system of the present invention;

[0089] Figure 6 This is a schematic diagram of the text classification and evaluation module of the text classification system of the present invention. Detailed Implementation

[0090] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0091] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0092] Example 1

[0093] Please see Figure 1 An embodiment of the present invention provides a Chinese short text classification method based on multi-scale features and association features, comprising the following steps:

[0094] S1: Obtain the labeled text data set and the topic term set and perform preprocessing. The preprocessing results in a preprocessed text set and vocabulary. The preprocessing includes text segmentation, stop word filtering, vocabulary construction, text encoding, topic term encoding, tag extraction, and tag one-hot encoding.

[0095] S2: After preprocessing, the BERT model is used to extract the text features of the preprocessed text set, and then multiple different convolutional layers are used to extract the multi-scale semantic features of the text features.

[0096] S3: Calculate the association information between the words in the preprocessed text set and vocabulary, construct a text graph based on the association information, and then use a graph neural network to extract text association features from the text graph;

[0097] S4: The extracted multi-scale semantic features and text association features are fused using a cross-attention mechanism to obtain fused features;

[0098] S5: Construct a text classifier, input the fused features into the text classifier, obtain the classification result, evaluate the classification result, calculate the evaluation index, and iteratively train the text classifier based on the evaluation result and the evaluation index to optimize the parameters of the text classifier.

[0099] Further, please refer to Figure 2 In this embodiment of the invention, the preprocessing in step S1 specifically includes the following steps:

[0100] S11: Text tokenization and stop word filtering. Since there is useless information in the text data set to be classified, such as modal particles, auxiliary words, conjunctions, etc., it is necessary to preprocess the text data set to be classified to remove these useless words and achieve sample noise removal. In the embodiment of the present invention, the denoising process includes text tokenization and stop word filtering. Text tokenization refers to splitting a text into a set of text words using a tokenization model, and stop word filtering refers to removing words with no actual meaning in the text, such as modal particles, quantifiers, numbers, etc.;

[0101] In this embodiment, the text to be classified, that is, the text data set with labels, is "China Women's University: Only 1 major recruits male students at the undergraduate level" and "Parallel filing scores for liberal arts and history in the second batch of undergraduate admissions in Hainan in 2011". After using the Chinese tokenization model jieba library to perform Chinese tokenization on it, the following text sets can be obtained:

[0102] {China, Women's, University, :, undergraduate, level, only, 1, major, recruit, male students}

[0103] {Hainan, 2011, year, college entrance examination, undergraduate, second, batch, liberal arts and history, parallel, filing, scores},

[0104] Among them, there are some useless information, such as {:, only, 1, recruit}, {2011, year, second, batch}, which will not affect the semantics of the sentence and have no actual meaning, belonging to noise and need to be removed. Then, by comparing with the Harbin Institute of Technology stop word list and passing through stop word filtering, the preprocessed text set can be obtained:

[0105] {China, Women's, University, undergraduate, level, major, male students}, {Hainan, college entrance examination, undergraduate, liberal arts and history, parallel, filing, scores};

[0106] S12: Construct a vocabulary. Add each token in the preprocessed text set into the vocabulary to obtain a vocabulary with duplicate words:

[0107] {China, Women's, University, undergraduate, level, major, male students, Hainan, college entrance examination, undergraduate, liberal arts and history, parallel, filing, scores},

[0108] There is a duplicate word "undergraduate" in the vocabulary. Perform a duplicate removal operation on the vocabulary to obtain a duplicate-removed vocabulary:

[0109] {China, Women's, University, undergraduate, level, major, male students, Hainan, college entrance examination, liberal arts and history, parallel, filing, scores};

[0110] S13: Text vector encoding. Use an encoder to vectorize the texts in the preprocessed text set obtained in step S11 to obtain 3 text vectors, namely text word embedding vectors TEtext Text segment embedding vector SE text and text position embedding vector PE text ;

[0111] S14: Vocabulary word encoding. An encoder is used to vectorize all words in the vocabulary constructed in step S12, resulting in three word vectors: word embedding vector TE. word Word segment embedding vector SE word and word position embedding vector Pe word ;

[0112] S15: Topic word encoding, using an encoder to vectorize the topic words in the initially obtained topic word set, resulting in topic word vectors;

[0113] S16: Tag extraction and one-hot encoding of tags. First, the tags of the text in the tagged text dataset are extracted and used as topic tags. Then, the topic tags are encoded using a one-hot encoder to obtain the one-hot vector of the text topic tags.

[0114] In this embodiment, the preset keyword set is:

[0115] {Finance, Reality, Stocks, Education, Science, Society, Politics, Sports, Games, Entertainment}

[0116] The extracted text's topic tags are "education" and "politics," and the one-hot vector corresponding to the topic tag "education" is...

[0117] [0, 0, 0, 1, 0, 0, 0, 0, 0, 0],

[0118] The one-hot vector corresponding to the hashtag "politics" is

[0119] [0, 0, 0, 0, 0, 1, 0, 0, 0].

[0120] Further, please refer to Figure 3 Step S2, which involves extracting multi-scale semantic features of text, includes the following steps:

[0121] S21: Input the text vector obtained in step S13 into the BERT model, and obtain the output of the BERT model as the text semantic features. Where s represents the length of the text sequence, d t =768 represents the feature dimension, and the text semantic feature ω has the following form:

[0122]

[0123] Where h represents the global semantic vector of the text, serving as a text-level feature, and h represents the semantic vector of each word segment in the text, serving as a word-level feature;

[0124] S22: Input the text semantic features into multiple different convolutional layers, using convolutional layers with kernel sizes of 3*768, 4*768, and 5*768 respectively, to obtain text semantic features at multiple scales, namely small-scale text semantic features. Mesoscale text semantic features and large-scale text semantic features Among them, h s h m h l These represent the dimensions of features at different scales, and fs = 256 represents the number of convolution kernels.

[0125] S23: Use a cross-attention mechanism to fuse the text semantic features at multiple scales obtained in step S22 to obtain multi-scale text semantic features.

[0126] Further, please refer to Figure 4 Step S3, extracting text association features, specifically includes the following steps:

[0127] S31: Input the word vectors obtained in step S14 into the BERT model, obtain the output of the BERT model and use it as word features, then input the topic word vectors obtained in step S15 into the BERT model, obtain the output of the BERT model and use it as topic word features;

[0128] S32: Use a word embedding model to obtain word embedding vectors of word features, topic word features and text semantic features;

[0129] S33: Based on word embedding vectors, calculate association information, which includes text-word association information, word-topic word association information, word-word association information, and text-text association information;

[0130] First, the text-word association information is calculated. This information is used as a feature of the edges between text nodes and word nodes, and is measured by the TF-IDF value of the word. The specific formula for calculating the TF-IDF value of a word is as follows:

[0131]

[0132] TF-IDF (word i ,doc j ) = TF * IDF

[0133] Where TF represents word frequency, Sum(word i ,docj Sum(word) represents the number of times word i appears in text j. i The dataset represents the total number of times word i appears in the entire corpus, IDF represents inverse document frequency, N represents the total number of texts in the corpus, and df t This indicates the number of texts containing the word 'i'.

[0134] S34: Calculate the word-topic term association information, which is used as the feature of the edge between word node and topic term node, based on the similarity S between words and topic terms. word The specific formula for calculating the similarity between words and topic words is as follows:

[0135]

[0136] Among them, S word Indicates the similarity between words and topic words. wordi V is the word embedding vector of the i-th word in the vocabulary. topicj It is the word embedding vector of the j-th topic word;

[0137] S35: Calculate word-word association information. Word-word association information is used as a feature of the edges between word nodes, and is measured by the mutual information between words. The specific formula for calculating the mutual information between words is as follows:

[0138]

[0139] Among them, PMI (word i ,word j P(word) represents the mutual information between word i and word j. i ,word j P(word) represents the probability that word i and word j appear simultaneously. i P(word) represents the probability of word i appearing. j () represents the probability of word j appearing;

[0140] S36: Calculate text-to-text association information, which is used as the feature of the edges between text nodes, and is calculated from the cosine similarity S between two texts in the preprocessed text set. text The cosine similarity S between two texts is measured. text The specific calculation formula is as follows:

[0141]

[0142] Among them, V i V j Let i and j represent the word embedding vectors of text i and text j, respectively.

[0143] S37: Construct a text graph G based on the TF-IDF values ​​of words, the similarity between words and topic words, the mutual information between words, and the cosine similarity between texts obtained from the above steps;

[0144] In this embodiment of the invention, a text heterogeneous graph construction method is used, and new topic word nodes and text similarity edges are added to construct a text graph G:

[0145] G = (V, E)

[0146] Where V is the vertex set, representing the nodes in the text graph, including words, keywords, and text, and E is the edge set, representing the relationships between different nodes. The text graph construction method of this invention is as follows:

[0147]

[0148] Among them, TF-IDF ij PMI represents the TF-IDF matrix composed of the TF-IDF values ​​of each word in the vocabulary. wordi-wordj Similarity represents the word mutual information matrix composed of the mutual information between words in the vocabulary. eordi-topicj Cos Similarity(doci,docj) represents the word-topic similarity matrix composed of the similarities between each word in the vocabulary and the topic words, while Cos Similarity(doci,docj) represents the text similarity matrix composed of the cosine similarities between each text.

[0149] This invention adds a topic term node and similarity-related edges between text words and topic terms to capture higher-order relevance between text, words, and topics. The similarity between text and topic terms is calculated. word-topic This method captures the similarity between text words and a certain topic. The higher the similarity, the greater the probability that the words belong to that topic. By capturing the topic distribution of text words through similarity, the topic distribution of the text can be revealed.

[0150] Using the text graph construction method described above, an adjacency matrix can be obtained.

[0151]

[0152] Where s2 represents the total number of nodes, and the adjacency matrix A is the text graph;

[0153] S38: Use text semantic features as text node features, word features as word node features, and topic word features as topic word node features;

[0154] Text node features, word node features, topic word node features, and text graph are input into a graph neural network to obtain the output of the graph neural network. And as a text association feature, where s2 represents the total number of nodes, d G Dimensions representing text association features.

[0155] Further, please refer to Figure 5 In this embodiment, feature fusion in step S4 includes the following steps:

[0156] S41: Perform a linear transformation on the multi-scale semantic features and text association features of the text, and calculate the query matrix Q based on the multi-scale semantic features ω of the text. ω Based on text association features G f key matrix Sum matrix The calculation formula is as follows:

[0157] Q ω =W Q

[0158]

[0159] in, This represents the query matrix obtained based on the multi-scale semantic features ω of the text. G represents text association features f The obtained key matrix, G represents text association features f The resulting value matrix. d represents the weight matrices needed to calculate the query matrix, key matrix, and value matrix, respectively. t d represents the dimension of the multi-scale semantic features of the text. G Dimensions representing text association features;

[0160] The multi-scale semantic features ω and text association features G are calculated based on the obtained query matrix, key matrix, and value matrix. f The cross-attention weight matrix CA(,G) f The calculation formula is as follows:

[0161]

[0162] in, Key matrix Dimensions That is, the multi-scale semantic features ω and the text association features G of the text. f The fused features are obtained after feature fusion through cross-attention mechanism.

[0163] Further, please refer to Figure 6 Step S5 specifically includes the following steps:

[0164] S51: Construct and initialize a text classifier. The text classifier uses a fully connected layer, which converts the input features into probability distributions for each category. In this embodiment, the input size of the fully connected layer is set to s1×h, where s1 represents the sequence length of the features, h represents the feature dimension, and s1×h is the length of the fused features flattened into a one-dimensional vector. The output size of the fully connected layer is set to the number of categories of the topic words in the preset topic word set. The preset number of topic word categories is 10. The weights of each node in the fully connected layer adopt the default initialization strategy.

[0165] S52: Input the fused features of the text to be classified into the text classifier, and obtain the output of the text classifier. The output is the probability distribution of each category, in the form of:

[0166] P = [P] 金融 ,P 现实 ,P 股票 ,P 教育 ,P 科学 ,P 社会 ,P 政治 ,P 运动 ,P 游戏 ,P 娱乐 ]

[0167] Among them, P 金融 P 现实 Equation represents the probability that the text to be classified belongs to the topic. The topic word with the highest probability in the probability distribution is selected as the classification result of the text to be classified.

[0168] S53: Encode the classification result into a one-hot vector, evaluate the one-hot vector of the classification result with the one-hot vector of the topic label obtained in step S16, if the two are the same, the prediction result is recorded as correct, otherwise it is recorded as incorrect.

[0169] Specifically, the preset number of topic categories is 10, and the classification result of the i-th text to be classified is result. i =Education, after one-hot encoding, the one-hot vector of this classification result is pred i = [0,0,0,1,0,0,0,0,0,0], if the true topic label of this sample is education, its true topic label one-hot vector is True. i =[0,0,0,1,0,0,0,0,0,0], if the classification result is the same as the true topic label, it is recorded as a correct prediction; otherwise, it is recorded as a wrong prediction.

[0170] S54: All texts to be classified are classified by using a text classifier and evaluation metrics are calculated based on the prediction results. The evaluation metrics used in this embodiment are accuracy, precision, recall and F1 score.

[0171] The formula for calculating accuracy is as follows:

[0172]

[0173] The formula for calculating precision is as follows:

[0174]

[0175] The formula for calculating recall is as follows:

[0176]

[0177] The formula for calculating the F1 score is as follows:

[0178]

[0179] Where TP, FP, TN, and FN represent the number of true positives, false positives, true negatives, and false negatives, respectively.

[0180] The text classifier is optimized by classifying all texts to be classified, calculating evaluation metrics, and iteratively training the text classifier based on the evaluation results and metrics.

[0181] Compared with existing technologies, Embodiment 1 of the present invention combines multi-scale semantic features and text association features of text, which can not only perform multi-scale mining of text features, but also take into account the global context information, local feature information and the association information between texts, thereby enhancing the text features and further improving the accuracy of Chinese short text classification.

[0182] Example 2

[0183] Please see Figure 1-6 This embodiment also provides a Chinese short text classification system based on multi-scale features and joint features, which includes the following modules:

[0184] The text preprocessing module is used to preprocess the labeled text dataset and the keyword dataset. The preprocessing includes text segmentation, stop word filtering, vocabulary construction, text encoding, keyword encoding, tag extraction, and tag one-hot encoding.

[0185] The multi-scale semantic feature extraction module is used to extract multi-scale semantic features from the preprocessed text.

[0186] The text graph construction and association feature extraction module is used to construct text graphs based on the preprocessed text set and vocabulary and extract the text association features contained in the text graphs.

[0187] The feature fusion module is used to fuse multi-scale semantic features of text with text association features;

[0188] The text classification and evaluation module is used to build a classifier, classify text, evaluate the classification results, calculate evaluation metrics, and iteratively train and optimize the text classifier based on the evaluation results and metrics.

[0189] Compared to existing technologies, Embodiment 2 of this invention fuses the features extracted by the multi-scale semantic feature module and the text association feature module through a feature fusion module. This not only enables multi-scale mining of text features but also takes into account the global context information, local feature information, and the association information between texts, thereby enhancing the text features and further improving the accuracy of Chinese short text classification.

[0190] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A Chinese short text classification method based on multi-scale features and association features, characterized in that, Includes the following steps: Obtain a set of labeled text data and a preset set of keywords, and perform preprocessing to obtain a text set and a vocabulary. The preprocessing includes at least one of the following: text segmentation, stop word filtering, vocabulary construction, text encoding, keyword encoding, tag extraction, and tag one-hot encoding. The BERT model is used to extract text features from the text set, and then multiple different convolutional layers are used to extract multi-scale semantic features of the text features. Calculate the association information between the text set and the words in the vocabulary, construct a text graph based on the association information, and then use a graph neural network to extract text association features from the text graph; The extracted multi-scale semantic features of the text and the text association features are fused using a cross-attention mechanism to obtain fused features; Construct a text classifier, input the fused features into the text classifier, and obtain the classification result; The preprocessing method specifically includes: The jieba library, a Chinese word segmentation model, is used to perform word segmentation on the tagged text dataset, resulting in segmented text. This segmented text is then compared with the Harbin Institute of Technology stop word list, and stop words are removed to obtain a text set in the following form: Text=[token1,token2,token3,…,token N ] Among them, token N This represents the Nth word of the text; Construct a vocabulary list by adding each word from the text set to the vocabulary list and removing duplicate parts of duplicate words in the vocabulary list; The text in the text set is vectorized using an encoder to obtain text vectors; Then, the encoder is used to vectorize all the word segments in the vocabulary to obtain word vectors; The topic terms in the topic term set are vectorized using an encoder to obtain topic term vectors. Extract the tags from the tagged text set and use the tags as topic tags. Then, use a one-hot encoder to vectorize the topic tags to obtain the one-hot vectors of the topic tags. The specific steps for calculating the associated information are as follows: The word vectors are input into the BERT model, and the output of the BERT model is obtained as the word features. Then, the topic word vectors are input into the BERT model, and the output of the BERT model is obtained as the topic word features. A word embedding model is used to obtain word embedding vectors of the word features, the topic word features, and the text semantic features. Based on the word embedding vectors, association information is calculated, including text-word association information, word-topic word association information, word-word association information, and text-text association information.

2. The Chinese short text classification method based on multi-scale features and joint features according to claim 1, characterized in that, The steps for extracting multi-scale semantic features of text are as follows: First, the text vector is input into the BERT model, and the output of the BERT model is obtained as the text semantic features. The text semantic features are input into the multiple different convolutional layers, and the outputs of the different convolutional layers are obtained to obtain text semantic features at multiple scales. A cross-attention mechanism is used to fuse the text semantic features at multiple scales to obtain multi-scale text semantic features.

3. The Chinese short text classification method based on multi-scale features and joint features according to claim 2, characterized in that, The specific text semantic features at multiple scales are as follows: Small-scale text semantic features: Mesoscale text semantic features: Large-scale text semantic features: The small, medium, and large-scale text semantic features are obtained by convolving the text semantic features with convolutional layers of kernel sizes of 3×768, 4×768, and 5×768, respectively, where h s h m h l represents the dimensions of text semantic features at small, medium, and large scales, respectively, and fs = 256 represents the number of convolution kernels.

4. The Chinese short text classification method based on multi-scale features and joint features according to claim 1, characterized in that, The text-word association information, as a feature of the edge between text node and word node, is measured by the TF-IDF value of the word. The specific formula for calculating the TF-IDF value of the word is as follows: TF-IDF(word i ,doc j )=TF*IDF Where TF represents word frequency, Sum(word i ,doc j Sum(word) represents the number of times word i appears in text j. i The dataset represents the total number of times word i appears in the entire corpus, IDF represents inverse document frequency, N represents the total number of texts in the corpus, and df t This indicates the number of texts containing the word 'i'. The word-topic term association information is used as a feature of the edge between word nodes and topic term nodes, determined by the similarity S between words and topic terms. word The specific formula for calculating the similarity between the stated words and the topic words is as follows: Among them, S word V represents the similarity between words and topic words. wordi V is the word embedding vector of the i-th word in the vocabulary. topicj It is the word embedding vector of the j-th topic word; The word-word association information, as a feature of the edges between word nodes, is measured by the mutual information between words. The specific formula for calculating the mutual information between words is as follows: Among them, PMI (word i ,word j P(word) represents the mutual information between word i and word j. i ,word j P(word) represents the probability that word i and word j appear simultaneously. i P(word) represents the probability of word i appearing. j () represents the probability of word j appearing; The text-to-text association information, as a feature of the edges between text nodes, is determined by the cosine similarity S between two texts in the text set. text The cosine similarity S between the two texts is measured. text The specific calculation formula is as follows: Among them, V i V j Let i and j represent the word embedding vectors of text i and text j, respectively.

5. The Chinese short text classification method based on multi-scale features and association features according to claim 4, characterized in that, The specific steps for constructing a text graph based on association information and then using a graph neural network to extract text association features from the text graph are as follows: A text graph G is constructed based on the calculated TF-IDF values ​​of the words, the similarity between words and topic words, the mutual information between words, and the cosine similarity between texts. G = (V, E) Where V is the vertex set, representing the nodes in the text graph, containing words, keywords, and text; E is the edge set, representing the association information between different nodes; the method for constructing the text graph can be represented as: Among them, TF-IDF ij PMI represents the TF-IDF matrix composed of the TF-IDF values ​​of each word in the vocabulary. wordi-wordj Similarity represents the word mutual information matrix composed of the mutual information between words in the vocabulary. wordi-topicj Cos Similarity(doci,docj) represents the word-topic similarity matrix composed of the similarities between each word in the vocabulary and the topic words, while Cos Similarity(doci,docj) represents the text similarity matrix composed of the cosine similarities between each text. Using the method described above for constructing the text graph, an adjacency matrix can be obtained: Where s2 represents the total number of nodes, and the adjacency matrix A is the text graph; The text semantic features are used as text node features, the word features are used as word node features, and the topic word features are used as topic word node features; The text node features, word node features, topic word node features, and the text graph are input into a graph neural network to obtain the output of the graph neural network. And as a text association feature, where s2 represents the total number of nodes, d G Dimensions representing text association features.

6. The Chinese short text classification method based on multi-scale features and association features according to claim 1, characterized in that, The specific steps for fusing the extracted multi-scale semantic features of the text with the text association features using a cross-attention mechanism to obtain the fused features are as follows: A linear transformation is performed on the multi-scale semantic features and the text association features to calculate the query matrix Q based on the multi-scale semantic features ω. ω Based on text association features G f key matrix Sum matrix The calculation formula is as follows: Q ω =ωW Q K Gf =G f W K V Gf =G f W V in, This represents the query matrix obtained based on the multi-scale semantic features ω of the text. G represents text association features f The obtained key matrix, G represents text association features f The resulting value matrix, d represents the weight matrices needed to calculate the query matrix, key matrix, and value matrix, respectively. t d represents the dimension of the multi-scale semantic features of the text. G Dimensions representing text association features; The multi-scale semantic features ω and text association features G are calculated based on the obtained query matrix, key matrix, and value matrix. f The cross-attention weight matrix CA(ω,G) f The calculation formula is as follows: in, Key matrix Dimensions That is, the multi-scale semantic features ω and the text association features G of the text. f The fused features are obtained after feature fusion through cross-attention mechanism.

7. The Chinese short text classification method based on multi-scale features and association features according to claim 6, characterized in that, The specific steps for constructing a text classifier, by inputting the fused features of the text to be classified into the text classifier, and obtaining the classification result are as follows: Construct a text classifier that uses fully connected layers to convert input features into probability distributions for each category and initialize the text classifier; The fused features of the text to be classified are input into the text classifier to obtain the output of the text classifier. The output is the probability distribution of each category. The topic word with the highest probability in each category probability distribution is selected as the classification result of the text to be classified.

8. The Chinese short text classification method based on multi-scale features and association features according to claim 1, characterized in that, After obtaining the classification result, the following steps are also included: The classification result is evaluated by encoding it into a one-hot vector. The one-hot vector of the classification result is compared with the one-hot vector of the topic label. If they are the same, the prediction result is recorded as a correct prediction; otherwise, it is recorded as a wrong prediction. All texts to be classified are classified using the text classifier and evaluation metrics are calculated based on the prediction results. The evaluation metrics include accuracy, precision, recall, and F1 score. The text classifier is iteratively trained based on the evaluation results and evaluation metrics to optimize its parameters.

9. A Chinese short text classification system based on multi-scale features and joint features, applied to the Chinese short text classification method based on multi-scale features and association features as described in any one of claims 1-8, characterized in that, The system includes: The text preprocessing module is used to preprocess the labeled text data set and the topic word set to obtain a text set and a vocabulary. The preprocessing includes at least any one of the following: text segmentation, stop word filtering, vocabulary construction, text encoding, topic word encoding, tag extraction, and tag one-hot encoding. The multi-scale semantic feature extraction module is used to extract multi-scale semantic features from a text collection. The text graph construction and association feature extraction module is used to construct text graphs based on text sets and vocabularies and extract the text association features contained in the text graphs. The feature fusion module is used to fuse multi-scale semantic features of text with text association features; The text classification and evaluation module is used to build a classifier, classify text, evaluate the classification results, calculate evaluation metrics, and iteratively train and optimize the text classifier based on the evaluation results and metrics.

Images