A text classification method and apparatus based on multi-semantic feature fusion

Abstract

This invention discloses a text classification method and apparatus based on multi-semantic feature fusion. The method includes: acquiring text to be classified; preprocessing the text to be classified to obtain processed text; inputting word-level granular vectors into a trained word-level semantic feature extraction model to obtain word-level semantic features; inputting sentence-level granular vectors into a trained sentence-level semantic feature extraction model to obtain sentence-level semantic features; extracting features from the processed text based on article-level vectors to obtain text-level semantic features; concatenating and fusing the word-level, sentence-level, and text-level semantic features to obtain fused features; and using a classifier to classify the fused features. This method performs fine-grained semantic modeling of text at multiple levels, including word-level, sentence-level, and article-level, and utilizes multi-semantic fusion features of text for text classification, thereby improving the accuracy of text classification.

Description

Technical Field

[0001] This invention relates to the field of text classification technology, and specifically to a text classification method and apparatus based on multi-semantic feature fusion. Background Technology

[0002] Text classification is a crucial component of many natural language processing tasks, applicable to sentiment analysis, public opinion monitoring, and web page retrieval. Currently, text classification primarily employs deep learning-based neural network models, such as CNNs, RNNs, and attention mechanisms. In the text preprocessing involved in deep text classification models, text vectorization is typically required. However, a current challenge in text classification lies in effectively representing text using word vectors. The varying importance of individual word vectors in text representation leads to inaccurate representations of short text vectors.

[0003] Using SBERT pre-trained word vector representation models, semantic information at the "word-level" or "sentence-level" of text can be extracted; meanwhile, topic models such as LDA can obtain global semantic information of the text. If this semantic information can be fully utilized for text classification, the accuracy of text classification will be significantly improved. Summary of the Invention

[0004] In view of this, the present invention provides a text classification method and apparatus based on multi-semantic feature fusion, which can solve the technical problem of inaccurate text classification caused by different weights of word vectors at the "word granularity", "sentence granularity" and global semantic information of the text.

[0005] To solve the above-mentioned technical problems, the present invention is implemented as follows.

[0006] A text classification method based on multi-semantic feature fusion, the method comprising:

[0007] Step S1: Obtain the text to be classified, preprocess the text to be classified, and obtain the processed text;

[0008] Step S2: Based on the pre-trained SBert model, perform word embedding and sentence embedding on the processed text to obtain word-level granular vectors and sentence-level granular vectors; input the word-level granular vectors into the pre-trained word-level semantic feature extraction model to obtain word-level semantic features; input the sentence-level granular vectors into the pre-trained sentence-level semantic feature extraction model to obtain sentence-level semantic features.

[0009] Step S3: Construct an article-level semantic feature extraction model file based on the LDA topic model, input the text after word processing into the trained article-level semantic feature extraction model file to obtain article-level vectors, and extract features from the processed text based on the article-level vectors to obtain text-level semantic features;

[0010] Step S4: The word-level semantic features, sentence-level semantic features, and text-level semantic features are concatenated and fused to obtain fused features, and a classifier is used to classify the fused features.

[0011] Preferably, in step S2, the SBert model vectorizes all words in the dictionary to obtain a word-granularity vector file; loads the word-granularity vector file and a dictionary file generated based on the dictionary; associates the word-granularity vector file and the dictionary file; concatenates the vectorized words in the word-granularity vector file; and embeds word vectors into the concatenated word-granularity vector file using the Embedding layer of the SBert model to obtain a word vector matrix; the word vector matrix is ​​then input into the trained word-level semantic feature extraction model to obtain word-level semantic features; wherein, the structure of the word-level semantic feature extraction model is as follows:

[0012] The word-level semantic feature extraction model includes a pre-trained SBert upstream text word vector model and a first long and short word memory LSTM model connected in sequence. The pre-trained SBert upstream text word vector model is used to obtain text word granular vector features from the received word vector matrix. The long and short word memory LSTM model is used to semantically associate the obtained text word granular vector features with contextual information. After association, text word granular vector features containing contextual information and / or post-contextual features are obtained. Then, attention weighting is obtained and applied to the text word granular vector features containing contextual information and / or post-contextual features to obtain word-level semantic features.

[0013] Preferably, the SBert model performs vectorization processing on the sentence set to obtain a sentence granularity vector file corresponding to the sentence granularity vector. The sentence vectors in the sentence granularity vector file are embedded in the Embedding layer of the SBert model to obtain a sentence vector matrix. The sentence vector matrix is ​​then input into the trained sentence-level semantic feature extraction model to obtain sentence-level semantic features. The structure of the sentence-level semantic feature extraction model is as follows:

[0014] The sentence-level semantic feature extraction model includes a pre-trained SBert upstream text sentence vector model and a second long short word memory LSTM model connected in sequence. The pre-trained SBert upstream text sentence vector model is used to perform vectorization operations on the received sentence vector matrix to obtain sentence-level semantic features. The second long short word memory LSTM model is used to semantically associate the context of the sentences above and below. After association, sentence-granular vector features containing the pre- and / or post-features of the context of the sentences above and below are obtained. Then, attention weights are obtained and applied to the sentence-granular vector features containing the pre- and / or post-features of the context of the sentences above and below to obtain sentence-level semantic features.

[0015] Preferably, step S3 includes: constructing an article-level semantic feature extraction model based on an LDA topic model; inputting the word-processed text into the trained article-level semantic feature extraction model to obtain article-level vectors; extracting features from the article-level vectors to obtain article-level extracted features; representing the word-processed text with topic vectors based on the article-level extracted features; and using the feature vectors corresponding to different topic levels as text-level semantic features; wherein, the article-level semantic feature extraction model is a three-layer Bayesian structure including a document layer, a topic layer, and a word layer, used to extract feature vectors from the word-processed text to obtain article-level vectors. The article-level semantic feature extraction model is trained by obtaining LDA clustering parameters.

[0016] Preferably, step S1, obtaining the processed text, includes: removing stop words from the text, performing word segmentation to obtain word-processed text, generating a dictionary based on the word-processed text, and segmenting the text into a set of sentences based on the punctuation marks in the text.

[0017] The present invention provides a text classification device based on multi-semantic feature fusion, the device comprising:

[0018] Preprocessing module: configured to acquire the text to be classified, preprocess the text to be classified, and obtain the processed text;

[0019] The first module is configured to perform word embedding and sentence embedding on the processed text based on the pre-trained SBert model, to obtain word-level granular vectors and sentence-level granular vectors; the word-level granular vectors are input into the pre-trained word-level semantic feature extraction model to obtain word-level semantic features; the sentence-level granular vectors are input into the pre-trained sentence-level semantic feature extraction model to obtain sentence-level semantic features.

[0020] The second module is configured to construct an article-level semantic feature extraction model file based on the LDA topic model. The text after word processing is input into the trained article-level semantic feature extraction model file to obtain article-level vectors. Based on the article-level vectors, feature extraction is performed on the processed text to obtain text-level semantic features.

[0021] The classification module is configured to concatenate and fuse the word-level semantic features, sentence-level semantic features, and text-level semantic features to obtain fused features, and then use a classifier to classify the fused features.

[0022] The present invention provides a computer-readable storage medium storing a plurality of instructions; the plurality of instructions are used by a processor to load and execute the method as described above.

[0023] The present invention provides an electronic device, characterized in that the electronic device comprises:

[0024] A processor is used to execute multiple instructions;

[0025] Memory, used to store multiple instructions;

[0026] The plurality of instructions are to be stored in the memory and loaded and executed by the processor as described above.

[0027] The beneficial technical effects of this invention are as follows:

[0028] (1) This invention performs fine semantic modeling of text at multiple levels, such as word granularity, sentence granularity and article granularity, and uses the multi-semantic fusion features of text for text classification, thereby improving the accuracy of text classification.

[0029] (2) The present invention utilizes the LDA topic model to obtain high-weight sensitive keywords, then represents the text segmentation with word vectors, and calculates the weighted average of all words to improve the accuracy of short text vector construction.

[0030] (3) The present invention can use an attention mechanism to weight the fusion features of the text, highlighting important features, and is suitable for text classification with rich semantics.

[0031] (4) This invention uses the SBERT model instead of the original BERT model, which improves the performance of extracting vector feature values. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the text classification method based on multi-semantic feature fusion according to the present invention;

[0033] Figure 2This is a flowchart illustrating the text classification method based on multi-semantic feature fusion of the present invention.

[0034] Figure 3 A flowchart illustrating the feature construction process of the text classification method based on multi-semantic feature fusion provided by this invention;

[0035] Figure 4 This is a schematic diagram of the text classification device based on multi-semantic feature fusion according to the present invention. Detailed Implementation

[0036] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0037] like Figures 1-2 As shown, this invention proposes a text classification method based on multi-semantic feature fusion, the method comprising:

[0038] Step S1: Obtain the text to be classified, preprocess the text to be classified, and obtain the processed text;

[0039] Step S2: Based on the pre-trained SBert model, perform word embedding and sentence embedding on the processed text to obtain word-level granular vectors and sentence-level granular vectors; input the word-level granular vectors into the pre-trained word-level semantic feature extraction model to obtain word-level semantic features; input the sentence-level granular vectors into the pre-trained sentence-level semantic feature extraction model to obtain sentence-level semantic features.

[0040] Step S3: Construct an article-level semantic feature extraction model file based on the LDA topic model, input the text after word processing into the trained article-level semantic feature extraction model file to obtain article-level vectors, and extract features from the processed text based on the article-level vectors to obtain text-level semantic features;

[0041] Step S4: The word-level semantic features, sentence-level semantic features, and text-level semantic features are concatenated and fused to obtain fused features, and a classifier is used to classify the fused features.

[0042] This invention integrates multi-granularity semantic features at the word, sentence, and article levels for text classification.

[0043] Step S1, obtaining the processed text, includes: removing stop words from the text, performing word segmentation to obtain word-processed text, generating a dictionary based on the word-processed text, and segmenting the text into a set of sentences based on the punctuation marks in the text.

[0044] In step S2, the SBert model vectorizes all words in the dictionary to obtain a word-granular vector file. It loads the word-granular vector file and a dictionary file generated based on the dictionary, associates the word-granular vector file and the dictionary file, concatenates the vectorized words in the word-granular vector file, and embeds the word vectors into the concatenated word-granular vector file using the Embedding layer of the SBert model to obtain a word vector matrix. The word vector matrix is ​​then input into the trained word-level semantic feature extraction model to obtain word-level semantic features. The structure of the word-level semantic feature extraction model is as follows:

[0045] The word-level semantic feature extraction model includes a pre-trained SBert upstream text word vector model and a first long and short word memory LSTM model connected in sequence. The pre-trained SBert upstream text word vector model is used to obtain text word granular vector features from the received word vector matrix. The long and short word memory LSTM model is used to semantically associate the obtained text word granular vector features with contextual information. After association, text word granular vector features containing contextual information and / or post-contextual features are obtained. Then, attention weighting is obtained and applied to the text word granular vector features containing contextual information and / or post-contextual features to obtain word-level semantic features.

[0046] In step S2, the SBert model vectorizes the sentence set to obtain a sentence granularity vector file corresponding to the sentence granularity vector. The SBert model's Embedding layer embeds sentence vectors into this file to obtain a sentence vector matrix. This sentence vector matrix is ​​then input into the trained sentence-level semantic feature extraction model to obtain sentence-level semantic features. The structure of the sentence-level semantic feature extraction model is as follows:

[0047] The sentence-level semantic feature extraction model includes a pre-trained SBert upstream text sentence vector model and a second long short word memory LSTM model connected in sequence. The pre-trained SBert upstream text sentence vector model is used to perform vectorization operations on the received sentence vector matrix to obtain sentence-level semantic features. The second long short word memory LSTM model is used to semantically associate the context of the sentences above and below. After association, sentence-granular vector features containing the pre- and / or post-features of the context of the sentences above and below are obtained. Then, attention weights are obtained and applied to the sentence-granular vector features containing the pre- and / or post-features of the context of the sentences above and below to obtain sentence-level semantic features.

[0048] The difference between the word-level semantic feature extraction model and the sentence-level semantic feature extraction model lies in the fact that word-level feature extraction extracts local word-level information, while sentence-level feature extraction extracts global semantic information. Combining the two improves the accuracy of the model.

[0049] In this invention, the embedding layer uses an SBERT-based pre-trained language model to embed word and sentence vectors, generating word and sentence vector matrices. Within the model, an attention mechanism is used to weight all temporal vectors, thereby optimizing the accuracy of the semantic feature vectors extracted by the model. A classification model is established by combining SBERT, LSTM, and Attention, and the semantic feature vector (Y) generated by these models is connected to a softmax layer to extract local word-level information from sentence-level features, thus completing the extraction of word-level and sentence-level features.

[0050] Step S3: Construct an article-level semantic feature extraction model based on the LDA topic model. Input the word-processed text into the trained article-level semantic feature extraction model to obtain article-level vectors. Extract features from the article-level vectors to obtain article-level extracted features. Based on the article-level extracted features, represent the word-processed text with topic vectors. Use the feature vectors corresponding to different topic levels as text-level semantic features. The article-level semantic feature extraction model is a three-layer Bayesian structure including a document layer, a topic layer, and a word layer, used to extract feature vectors from the word-processed text to obtain article-level vectors. The article-level semantic feature extraction model is trained using LDA clustering parameters.

[0051] In step S4, the classifier is an XgBoost classifier.

[0052] This invention provides a specific embodiment of a text classification method based on multi-semantic feature fusion.

[0053] The text classification method based on multi-semantic feature fusion is logically divided into five processing steps: constructing a word-level semantic feature extraction model, constructing a sentence-level semantic feature extraction model, constructing an article-level semantic feature extraction model, semantic feature fusion, and constructing an XgBoost text classification model.

[0054] The construction of a word-level semantic feature extraction model mainly involves three parts: text corpus preprocessing, word vector generation, text word embedding, and training the word-level semantic feature extraction model. The specific implementation method is as follows:

[0055] Step S1: Text Corpus Preprocessing

[0056] Step S1-1: Use the natural language processing package HanLP to perform operations such as converting traditional Chinese text to simplified Chinese text, word segmentation, and removing stop words to complete the data preprocessing.

[0057] Step S1-2: Generate the corresponding dictionary using the preprocessed text corpus.

[0058] Step S2: Word Vector Generation

[0059] Step S2-1: Vectorize the words in the S1 dictionary using the vector generation service bert-as-service.

[0060] Step S2-2: Use NumPy to instantiate the word vectors of all words in the dictionary into a local file.

[0061] Step S3: Text word embedding

[0062] Step S3-1: Load word vector files and dictionary files

[0063] Step S3-2: Vectorize the preprocessed words in the text by associating dictionary files and word vector files.

[0064] Step S3-3: Concatenate the vectorized words in the text into a word vector matrix.

[0065] Step S4: Train the word-level semantic feature extraction model

[0066] Step S4-1: Train the sample data using SBERT+LSTM+Attention.

[0067] Step S4-2: After training, save the word-level semantic feature extraction model.

[0068] The construction of a sentence-level semantic feature extraction model involves two parts: text preprocessing, text sentence embedding, and training the sentence-level semantic feature extraction model. The specific implementation method is as follows:

[0069] Step S1: Segment the text into sentence sets based on the punctuation marks to complete the text preprocessing.

[0070] Step S2: Use the vector generation service bert-as-service to vectorize the segmented sentence set to obtain the sentence vector matrix of the text, thus completing the text sentence embedding.

[0071] Step S3: Train the word-level semantic feature extraction model

[0072] Step S3-1: Train the sample data using SBERT+LSTM+Attention.

[0073] Step S3-2: After training, save the sentence-level semantic feature extraction model.

[0074] The construction of an article-level semantic feature extraction model involves two parts: text preprocessing and training the article-level semantic feature extraction model. The specific implementation method is as follows:

[0075] Step S1: Use the natural language processing package HanLP to perform operations such as converting traditional Chinese text to simplified Chinese text, word segmentation, and removing stop words to complete text preprocessing.

[0076] Step S2: Train the article-level semantic feature extraction model.

[0077] Step S2-1: Define the LDA clustering parameters to be trained

[0078] Step S2-2: Train the preprocessed corpus using the LDA topic model.

[0079] Step S2-3: Save the trained LDA topic model

[0080] Multi-semantic feature fusion involves two parts: multi-semantic feature generation and multi-semantic feature fusion. The specific implementation method is as follows:

[0081] Step S1: Generation of multiple semantic features.

[0082] Step S1-1: Load the word-level semantic feature extraction model file, the sentence-level semantic feature extraction model file, and the LDA topic model file.

[0083] Step S1-2: The input text content is processed by the word-level semantic feature extraction model file to generate word-level feature vectors.

[0084] Steps S1-3: The input text content is processed by the sentence-level semantic feature extraction model file to generate "sentence" granular feature vectors.

[0085] Steps S1-4: The input text content is processed by the LDA topic model file to generate "article" granular feature vectors.

[0086] Step S2: Generate multi-semantic feature vectors by concatenation.

[0087] The construction of the XgBoost text classification model involves three parts: inputting a multi-semantic feature matrix, training the XgBoost classification model, and saving the classification model. The specific implementation method is as follows:

[0088] Step S1: Input of multi-semantic feature matrix

[0089] Step S1-1: After preprocessing the training text, divide it into training set and validation set.

[0090] Step S1-2: Generate the corresponding multi-semantic feature vectors and convert them into matrix objects, which are then input into the xGboost classification model.

[0091] Step S2: Train the XgBoost classification model

[0092] Step S3: Save the trained XgBoost classification model file.

[0093] Based on the aforementioned trained word-level, sentence-level, and article-level semantic feature extraction models, this invention extracts semantic features from the input text content at the "word," "sentence," and "article" levels, respectively, to obtain corresponding word-level, sentence-level, and article-level semantic feature vectors. Then, the word-level, sentence-level, and article-level semantic feature vectors are concatenated to obtain a fused multi-semantic feature vector, which is then used as the input to the Xgboost model to obtain the final text classifier.

[0094] like Figure 4 As shown, the present invention provides a text classification device based on multi-semantic feature fusion, the device comprising:

[0095] Preprocessing module: configured to acquire the text to be classified, preprocess the text to be classified, and obtain the processed text;

[0096] The first module is configured to perform word embedding and sentence embedding on the processed text based on the pre-trained SBert model, to obtain word-level granular vectors and sentence-level granular vectors; the word-level granular vectors are input into the pre-trained word-level semantic feature extraction model to obtain word-level semantic features; the sentence-level granular vectors are input into the pre-trained sentence-level semantic feature extraction model to obtain sentence-level semantic features.

[0097] The second module is configured to construct an article-level semantic feature extraction model file based on the LDA topic model. The text after word processing is input into the trained article-level semantic feature extraction model file to obtain article-level vectors. Based on the article-level vectors, feature extraction is performed on the processed text to obtain text-level semantic features.

[0098] The classification module is configured to concatenate and fuse the word-level semantic features, sentence-level semantic features, and text-level semantic features to obtain fused features, and then use a classifier to classify the fused features.

[0099] The specific embodiments described above only illustrate the design principles of the present invention. The shapes and names of the components in this description may differ and are not limited. Therefore, those skilled in the art can modify or make equivalent substitutions to the technical solutions described in the foregoing embodiments; and these modifications and substitutions do not depart from the inventive spirit and technical solutions of the present invention, and should all fall within the protection scope of the present invention.

Claims

1. A text classification method based on multi-semantic feature fusion, characterized in that, include: Step S1: Obtain the text to be classified, preprocess the text to be classified, and obtain the processed text; Step S2: Based on the pre-trained SBert model, perform word embedding and sentence embedding on the processed text to obtain word-level and sentence-level granular vectors; input the word-level granular vectors into the pre-trained word-level semantic feature extraction model to obtain word-level semantic features; input the sentence-level granular vectors into the pre-trained sentence-level semantic feature extraction model to obtain sentence-level semantic features. The pre-trained SBert upstream text sentence vector model in the sentence-level semantic feature extraction model is used to perform vectorization operations on the received sentence vector matrix to obtain the vectorized sentence vector matrix. The second short word memory LSTM model in the sentence-level semantic feature extraction model performs semantic association on the context of the sentences above and below. After association, sentence-granular vector features containing the pre- and / or post-features of the context of the sentences above and below are obtained. Then, attention weights are obtained and applied to the sentence-granular vector features containing the pre- and / or post-features of the context of the sentences above and below to obtain sentence-level semantic features. Step S3: Construct an article-level semantic feature extraction model based on the LDA topic model. Input the word-processed text into the trained article-level semantic feature extraction model to obtain article-level vectors. Extract features from the article-level vectors to obtain article-level extracted features. Based on the article-level extracted features, represent the word-processed text with topic vectors. Use the feature vectors corresponding to different topic levels as text-level semantic features. The article-level semantic feature extraction model is a three-layer Bayesian structure including a document layer, a topic layer, and a word layer, used to extract feature vectors from the word-processed text to obtain article-level vectors. The article-level semantic feature extraction model is trained by obtaining LDA clustering parameters. Step S4: The word-level semantic features, sentence-level semantic features, and text-level semantic features are concatenated and fused to obtain fused features, and a classifier is used to classify the fused features.

2. The method as described in claim 1, characterized in that, In step S2, the SBert model vectorizes all words in the dictionary to obtain a word-granular vector file. It loads the word-granular vector file and a dictionary file generated based on the dictionary, associates the word-granular vector file and the dictionary file, concatenates the vectorized words in the word-granular vector file, and embeds the word vectors into the concatenated word-granular vector file using the Embedding layer of the SBert model to obtain a word vector matrix. The word vector matrix is ​​then input into the trained word-level semantic feature extraction model to obtain word-level semantic features. The structure of the word-level semantic feature extraction model is as follows: The word-level semantic feature extraction model includes a pre-trained SBert upstream text word vector model and a first long and short word memory LSTM model connected in sequence. The pre-trained SBert upstream text word vector model is used to obtain text word granular vector features from the received word vector matrix. The long and short word memory LSTM model is used to semantically associate the obtained text word granular vector features with contextual information. After association, text word granular vector features containing contextual information and / or post-contextual features are obtained. Then, attention weighting is obtained and applied to the text word granular vector features containing contextual information and / or post-contextual features to obtain word-level semantic features.

3. The method according to any one of claims 1-2, characterized in that, Step S1, obtaining the processed text, includes: removing stop words from the text, performing word segmentation to obtain word-processed text, generating a dictionary based on the word-processed text, and segmenting the text into a set of sentences based on the punctuation marks in the text.

4. A text classification device based on multi-semantic feature fusion, characterized in that, The device includes: Preprocessing module: Configured to obtain the text to be classified, preprocess the text to be classified, and obtain the processed text; The first module is configured to perform word embedding and sentence embedding on the processed text based on the pre-trained SBert model, obtaining word-level and sentence-level granular vectors respectively; the word-level granular vectors are input into the pre-trained word-level semantic feature extraction model to obtain word-level semantic features; the sentence-level granular vectors are input into the pre-trained sentence-level semantic feature extraction model to obtain sentence-level semantic features. The pre-trained SBert upstream text sentence vector model in the sentence-level semantic feature extraction model is used to perform vectorization operations on the received sentence vector matrix to obtain the vectorized sentence vector matrix. The second short word memory LSTM model in the sentence-level semantic feature extraction model performs semantic association on the context of the sentences above and below. After association, sentence-granular vector features containing the pre- and / or post-features of the context of the sentences above and below are obtained. Then, attention weights are obtained and applied to the sentence-granular vector features containing the pre- and / or post-features of the context of the sentences above and below to obtain sentence-level semantic features. The second module is configured to build an article-level semantic feature extraction model based on the LDA topic model. The word-processed text is input into the trained article-level semantic feature extraction model to obtain article-level vectors. Feature extraction is performed on the article-level vectors to obtain article-level extracted features. Based on these features, topic vectors are represented on the word-processed text. The feature vectors corresponding to different topic levels are used as text-level semantic features. The article-level semantic feature extraction model is a three-layer Bayesian structure including a document layer, a topic layer, and a word layer. It is used to extract feature vectors from the word-processed text to obtain article-level vectors. The article-level semantic feature extraction model is trained using LDA clustering parameters. The classification module is configured to concatenate and fuse the word-level semantic features, sentence-level semantic features, and text-level semantic features to obtain fused features, and then use a classifier to classify the fused features.

5. A computer-readable storage medium storing a plurality of instructions; the plurality of instructions being loaded by a processor and executing the method as claimed in any one of claims 1-3.

6. An electronic device, characterized in that, The electronic device includes: A processor is used to execute multiple instructions; Memory, used to store multiple instructions; The plurality of instructions are to be stored in the memory and loaded by the processor and executed as described in any one of claims 1-3.

Images