Method, equipment and medium for short text classification of questions on China's agricultural question-answering platform
By combining ERNIE with multi-scale convolutional CNNs model, using channel attention and class residual reconstruction technology, the problems of typos, semantic complexity and data imbalance in short text classification of Chinese agricultural question questions were solved, and classification accuracy and Generalization ability.
Patent Information
- Application Number
- CN202411035631.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-31
AI Technical Summary
The existing technology has problems such as many typos, complex semantics, sparse features and unbalanced data in the classification of short texts of Chinese agricultural question questions, resulting in poor classification effect and low accuracy, and the performance of pre-trained language models in Chinese text classification. Loss, it is difficult to completely extract the semantics between phrases and sentences.
The pre-trained language model ERNIE combined with the multi-scale convolution CNNs model to obtain context-related dynamic word embedding information through the embedding layer of ERNIE, and extract global semantic information from Transformer Encoder, combined with channel attention ECANet in multi-scale convolution neural network , use class residual reconstruction to reconstruct the output semantic information, add the maximum pooling layer and ECA channel attention mechanism, and perform feature extraction and classification.
It improves the accuracy and generalization ability of short text classification in Chinese agricultural question questions, enhances the integrity of word vectors, solves the problems of sparse features and unbalanced data, and achieves better semantic information acquisition and classification performance.
Smart Images

Figure CN118568262B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data classification, and in particular relates to a method, device and medium for classifying short texts of questions on a Chinese agricultural question-and-answer platform. Background Art
[0002] Question classification belongs to the research scope of short text classification. Although traditional machine learning methods have certain classification effects, they rely on shallow learning algorithms, resulting in weak generalization ability of the model. In addition, these methods are usually difficult to capture the complex semantics and contextual information present in natural language texts, especially in tasks involving large-scale and diverse data sets, where performance is limited.
[0003] The single-layer convolutional neural network model for text classification treats text as a fixed-length image input into the model, constructs a TextCNN model, and improves the text classification effect, but the feature extraction of short text is still not accurate enough, and the impact of sequence information in text information is not considered. The classic TextRNN model integrates RNN into the multi-task framework and uses the information sharing mechanism to improve the LSTM method. The model integrates all related tasks into a joint training system. The specific task layer learns single-task information, and the shared layer shares information for multiple tasks. Although sequence information is taken into account, it is difficult for the model to learn the internal dependencies and structural information such as grammar of short texts. Some literature has combined CNN and RNN to study short text classification, and further extract text features while obtaining context information, but this type of model is very difficult to train and has poor domain generalization. The Transformers method is used to classify short texts, and the self-attention mechanism is used to capture the relationship between different positions in the input sequence, thereby processing long-distance dependencies, and achieving very good results. However, this paper is a study on English data sets, while Chinese semantics are complex and the structure is diverse. This method still finds it difficult to quickly learn accurate text features.
[0004] Most of the above models use static word vectors such as word2vec and TF-IDF as the model's lower word vector representation, which cannot solve the problem of polysemy. Bert was proposed by Google in 2018 and has been widely used in natural language processing (NLP) tasks, and a large number of researchers have used it for text classification tasks. Some literature has classified academic papers based on BERT, BiGRU and BiLSTM models, achieving good classification results, but has not considered the impact of the receptive field of feature extraction on classification. BERT-CNN combines the Bert encoding layer with CNN to propose a text classification model that takes into account multiple classification granularities while taking pre-training into account, expands the receptive field of feature extraction through the convolution layer, and learns more feature information, proving that splicing the pre-trained model with the convolution can improve classification accuracy. The above models all use the Bert pre-trained model as the word vector encoding layer of the hybrid model combined with RNN and CNN. Although it can improve the classification effect, the Bert model mainly focuses on [Mask] learning for character granularity or English word granularity, and does not model based on lexical structure and syntactic structure. It is difficult to give a standard vector representation for new words, and this problem is more obvious in the Chinese corpus. Therefore, the Bert model still has a certain performance loss in Chinese text classification.
[0005] The ERNIE pre-training model improves the [Mask] mechanism on the basis of Bert. By masking Chinese entities and characters, the lexical structure, grammatical structure, and semantic information in the training data are uniformly modeled, which greatly enhances the general semantic representation ability of Chinese. It can potentially learn knowledge dependencies and longer semantic dependencies to make the model more generalized, and has achieved results that significantly surpass BERT in multiple tasks. Some studies have been conducted on the classification of Chinese news texts based on the ERNIE model, which has verified that the model has an advantageous position in Chinese classification tasks. Some have combined the ERNIE model with a recurrent neural network. Although the long-term dependency of the context is taken into account, the input of each time step needs to be processed sequentially, which affects its efficiency in processing long sequences. Wang et al. proposed the ERNIE-TextCNN model to study the classification of Chinese short texts. It extracts feature information by sliding the convolution kernel at different positions, which means that CNN can capture the local relationship between words and phrases and better capture the local information in short texts. At the same time, the use of a shared parameter mechanism helps to reduce the number of parameters of the model, thereby achieving better classification results. However, CNNs rely on the input word vector information. If the word vector is not standard and comprehensive enough, it will affect the CNNs feature extraction. The above methods all regard each part as equally important, which is easy to lose key information and pay too much attention to non-key information. Some literature introduces the attention mechanism in the model, assigns different weights to words according to their importance to the classification results in the data set, enhances the word dependency relationship within the text, and thus greatly improves the classification accuracy. There is a multi-channel hybrid model for Chinese text classification. This hybrid multi-channel model will greatly increase the complexity of the model and the amount of parameter calculation, which may cause unnecessary resource consumption and lead to insufficient resources. Therefore, for the problem of Chinese short text classification, it is necessary to select the appropriate number and complexity of models and seek optimization from multiple aspects such as performance and parameter quantity.
[0006] Data imbalance is a common problem in text classification tasks, and model performance is also affected. In recent years, researchers have also been committed to solving the problem of data imbalance. There is a method of adversarial training to solve the classification problem of imbalanced text. Shaikn et al. used the recently proposed text generation model based on GPT-2 and LSTM to introduce balance in highly imbalanced text datasets. Good results were achieved on three highly imbalanced datasets from different fields. Although the generated text is brand new and retains the correct grammatical structure, context, and semantic information related to the original text, it also requires an increase in the number of parameters and memory space. Fernandez et al. summarized the solutions to the problem of data imbalance and proposed that resampling may lead to information loss and ignore the correlation between samples.
[0007] In the field of agriculture, due to the lack of large-scale available data sets, related research is still in its infancy, and there is little verification of classification on multi-category agricultural question data sets. Existing research provides reference and feasibility basis for deep learning in agricultural text classification. However, these models are either not applicable to Chinese and cannot solve the problems of poor model classification and low accuracy caused by many typos and complex semantics in the Chinese agricultural farmer question corpus; or they cannot solve the problems caused by sparse features and unbalanced data. Pre-trained language models have better generalization, but there is also the problem that the semantics between phrases and sentences cannot be fully extracted, which reduces the accuracy of the results. At present, there is little research on the classification method of short agricultural question text based on the ERNIE model. Summary of the invention
[0008] The purpose of the present invention is to solve the problems in the prior art and to propose a method, device and medium for classifying short texts of questions on the Chinese agricultural question-and-answer platform.
[0009] The present invention is implemented by the following technical scheme. The present invention proposes a classification method for short texts of questions on the China Agriculture Question and Answer Platform, and the method comprises the following steps:
[0010] Step 1: Establish a short text database of agricultural questions;
[0011] Step 2: Build a short text classification model and use the database to train the short text classification model;
[0012] The short text classification model adopts the form of a pre-trained language model ERNIE combined with a multi-scale convolutional CNNs model, specifically: using the pre-trained language model ERNIE as an embedding layer and an encoding layer, obtaining context-related dynamic word embedding information of text data through the embedding layer, including tag embedding, segment embedding, and position embedding information; extracting and transmitting the obtained embedding information to the Transformer Encoder part, extracting the output results of the last three encoding layers of the embedded global semantic information, and reconstructing the text vector as the final output, passing it to the multi-scale CNNs convolutional layer for feature extraction, adding channel attention ECANet at different positions in the multi-scale convolutional neural network, and extracting feature vectors from the local semantic information of the sentence through convolution kernels of different scales;
[0013] Step 3: Use the trained short text classification model to classify short texts of agricultural questions. Connect it to the softmax layer in a fully connected manner, normalize the fused values containing diverse information, and complete the short text classification of agricultural questions of multiple categories.
[0014] Furthermore, a maximum pooling layer is added after the convolution layer to perform the maximum pooling operation on the feature vector output by the convolution layer, converting the high-dimensional feature matrix into a low-dimensional feature matrix to achieve dimensionality reduction, and using the Relu activation function to truncate the negative value part to 0, thereby enhancing the nonlinear characteristics of the neural network.
[0015] Furthermore, in the feature extraction process, the output local semantic feature vector is connected with the global semantic feature vector of the [CLS] position containing global semantic information output by the ERNIE encoding layer as the output of the feature extraction layer.
[0016] Furthermore, an ECA channel attention mechanism is added after the convolution layers with kernel sizes of 2 and 4 in the multi-scale convolution, so that the classification model can quickly capture keyword information that is beneficial to classification.
[0017] Furthermore, ERNIE's word vector representation method adopts a context-based word vector representation method, that is, for a word, its context information is taken into account; first, after the sentence is input into the classification model, for each embedding layer of the input sentence, the [CLS] tag is used as a classification tag at the beginning of the sentence, and the tag [SEP] is added at the end of the sentence; the labeled sentence is input into the embedding layer to obtain the semantic representation of the sentence, which consists of tag embedding, segment embedding and position embedding; the tag embedding part provides an embedding representation for each word, and the segment embedding is used to embed the information of the global semantic input text and merged with the tag embedding, and the position embedding is used to distinguish word vectors at different positions; the embedding information obtained by summing the three embedding parts is used as the output of the entire embedding layer.
[0018] Furthermore, the pre-trained language model ERNIE has a total of 12 encoding layers. The result of the last layer is combined with the output of the first two layers to perform a residual-like superposition reconstruction to complete the calculation of the entire encoding layer. The text vector representation of the output semantic information effect is reconstructed as the final word vector representation of the encoding layer of the pre-trained language model, which is used as the input of the multi-scale CNNs convolutional layer for feature extraction.
[0019] The superposition reconstruction formula of the residual-like method is: H(x)=(F(x)+G(x)+R(x));
[0020] Where x is the input, F(x) represents the output after the transformation of the tenth layer, G(x) represents the output after the transformation of the eleventh layer, R(x) represents the output after the transformation of the twelfth layer, and H(x) is the final output of the twelfth layer after residual reconstruction, as the final vector result.
[0021] Furthermore, in the process of extracting features in the convolution layer, multiple convolution kernels are used, each of which extracts a feature and is set to multi-scale. Convolution kernels of different scales are used to obtain word-related content in different ranges to form vertical difference information, that is, different information brought by the appearance of words in different ranges in a sentence.
[0022] Furthermore, the number of neurons in the maximum pooling layer is equal to the number of filters in the convolutional layer, and the vector carrying the most important feature information in each convolution kernel is obtained through the maximum pooling operation.
[0023] The present invention proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a method for classifying short texts of questions on a Chinese agricultural question-and-answer platform are implemented.
[0024] The present invention proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of a method for classifying short texts of questions on a Chinese agricultural question-and-answer platform.
[0025] Beneficial effects of the present invention:
[0026] The present invention proposes a method, device and medium for classifying short texts of questions on the Chinese agricultural question-answering platform. The method can be applied to Chinese, and can solve the problems of poor model classification effect and low precision caused by many typos and complex semantics in the Chinese agricultural farmers' question corpus; it can also solve the problems caused by sparse features and unbalanced data. The semantic loss problem is avoided, the integrity of word vector acquisition is enhanced, and richer semantic information can be obtained. Compared with the existing methods, this method is superior to the original ERNIE and other comparative models in classification performance. The language model in the method of the present invention has better generalization, significantly enhanced generalization ability, and improved the accuracy of the results. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0028] Figure 1 Schematic diagram of the short text classification model.
[0029] Figure 2 The following is a graph showing the changing trends of accuracy and loss values on the validation set during model training. The left graph shows the changing trend of loss values, and the right graph shows the changing trend of accuracy.
[0030] Figure 3 This is a diagram of the confusion matrix of the model before modification on the agricultural question data. The horizontal axis is the true label and the vertical axis is the predicted label.
[0031] Figure 4 This is a diagram of the confusion matrix of the modified model on the agricultural question data. The horizontal axis is the true label and the vertical axis is the predicted label.
[0032] Figure 5 This is a schematic diagram of the accuracy comparison results of the model described in the present invention and other deep learning models, with the horizontal axis representing the category and the vertical axis representing the accuracy.
[0033] Figure 6 This is a schematic diagram of the recall rate comparison results of the model described in the present invention and other deep learning models, where the horizontal axis is the category and the vertical axis is the recall rate.
[0034] Figure 7 This is a schematic diagram of the F1 score comparison results of the model described in the present invention and other deep learning models, with the horizontal axis representing the category and the vertical axis representing the F1 score. DETAILED DESCRIPTION
[0035] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0036] Combination Figure 1-Figure 7 The present invention proposes a classification method for short texts of questions on the China Agriculture Question and Answer Platform, the method comprising the following steps:
[0037] Step 1: Establish a short text database of agricultural questions;
[0038] Step 2: Build a short text classification model and use the database to train the short text classification model;
[0039] The short text classification model adopts the form of a pre-trained language model ERNIE combined with a multi-scale convolutional CNNs model, specifically: using the pre-trained language model ERNIE as an embedding layer and an encoding layer, obtaining context-related dynamic word embedding information of text data through the embedding layer, including tag embedding, segment embedding, and position embedding information; extracting and transmitting the obtained embedding information to the Transformer Encoder part, extracting the output results of the last three encoding layers of the embedded global semantic information, and reconstructing the text vector as the final output, passing it to the multi-scale CNNs convolutional layer for feature extraction, adding channel attention ECANet at different positions in the multi-scale convolutional neural network, and extracting feature vectors from the local semantic information of the sentence through convolution kernels of different scales;
[0040] Step 3: Use the trained short text classification model to classify agricultural question short texts. Connect it to the softmax layer in a fully connected manner, normalize the fused values containing diverse information, and complete the short text classification of agricultural questions of multiple categories.
[0041] A maximum pooling layer is added after the convolution layer to perform the maximum pooling operation on the feature vector output by the convolution layer, converting the high-dimensional feature matrix into a low-dimensional feature matrix to achieve dimensionality reduction, and using the Relu activation function to truncate the negative value part to 0 to enhance the nonlinear characteristics of the neural network.
[0042] In the feature extraction process, the output local semantic feature vector is connected with the global semantic feature vector of the [CLS] position containing global semantic information output by the ERNIE encoding layer as the output of the feature extraction layer.
[0043] In the multi-scale convolution, an ECA channel attention mechanism is added after the convolution layers with kernel sizes of 2 and 4 to enable the classification model to quickly capture keyword information that is beneficial to classification.
[0044] ERNIE's word vector representation method adopts a context-based word vector representation method, that is, for a word, its context information is taken into account; first, after the sentence is input into the classification model, for each embedding layer of the input sentence, the [CLS] tag is used as a classification tag at the beginning of the sentence, and the tag [SEP] is added at the end of the sentence; the labeled sentence is input into the embedding layer to obtain the semantic representation of the sentence, which consists of tag embedding, segment embedding and position embedding; the tag embedding part provides an embedding representation for each word, and the segment embedding is used to embed the information of the global semantic input text and merge it with the tag embedding, and the position embedding is used to distinguish word vectors at different positions; the embedding information obtained by summing the three embedding parts is used as the output of the entire embedding layer.
[0045] The pre-trained language model ERNIE has a total of 12 encoding layers. The result of the last layer is combined with the output of the first two layers to perform a residual-like superposition reconstruction to complete the calculation of the entire encoding layer. The text vector representation of the reconstructed output semantic information is used as the final word vector representation of the encoding layer of the pre-trained language model, which is used as the input of the multi-scale CNNs convolutional layer for feature extraction.
[0046] The superposition reconstruction formula of the residual-like method is: H(x)=(F(x)+G(x)+R(x));
[0047] Where x is the input, F(x) represents the output after the transformation of the tenth layer, G(x) represents the output after the transformation of the eleventh layer, R(x) represents the output after the transformation of the twelfth layer, and H(x) is the final output of the twelfth layer after residual reconstruction, as the final vector result.
[0048] In the process of extracting features in the convolution layer, multiple convolution kernels are used, each of which extracts a feature and is set to multi-scale. Convolution kernels of different scales are used to obtain word-related content in different ranges to form vertical difference information, that is, different information brought by the appearance of words in different ranges in a sentence.
[0049] The number of neurons in the maximum pooling layer is equal to the number of filters in the convolution layer. The maximum pooling operation is used to obtain the vector carrying the most important feature information in each convolution kernel.
[0050] Example
[0051] The present invention proposes a classification method for short texts of questions on a Chinese agricultural question-answering platform, the method comprising the following steps:
[0052] Step 1: Establish a short text database of agricultural questions;
[0053] Step 2: Construct a short text classification model and use the database to train the short text classification model; the constructed model includes input layer, embedding layer, encoding layer, feature extraction layer, pooling layer, full connection and softmax layer (classification layer), and output layer. In the process of model construction, the pre-trained language model ERNIE is used in combination with the multi-scale convolution (CNNs) model, and its structure is improved to obtain the final classification model. First, the pre-trained language model ERNIE is used as the embedding layer and encoding layer, and the context-related dynamic word embedding information of the text data is obtained through the embedding layer, including tag embedding, segment embedding and position embedding information. The obtained embedding information is extracted and transmitted to the Transformer Encoder part, and the output results of the last three encoding layers are extracted from the embedded global semantic information and reconstructed as the final output text vector, which is passed to the multi-scale CNNs convolution layer for feature extraction. Channel attention (ECANet) is added at different positions in the multi-scale convolutional neural network, and the local semantic information of the sentence is extracted through convolution kernels of different scales to extract feature vectors. In order to reduce the parameters and network complexity, a maximum pooling layer is added after the convolution layer, and the feature vector output by the convolution layer is subjected to the maximum pooling operation, and the high-dimensional feature matrix is converted into a low-dimensional feature matrix to achieve dimensionality reduction. In order to avoid incomplete feature extraction, the output local semantic feature vector is connected with the [CLS] position vector containing global semantic information output by the ERNIE encoding layer as the output of the feature extraction layer. Finally, it is connected to the softmax layer in a fully connected manner, and the fused value containing diversified information is normalized to complete the short text classification of agricultural questions of multiple categories. In order to solve the problem of data imbalance and increase the robustness of the model, it is found that the accuracy of category classification of a few samples is not significantly improved after using the adversarial training method, but at the same time the amount of parameter calculation is greatly increased. Therefore, the present invention uses data enhancement technology to process the data into a slightly unbalanced state and then uses the model adaptive weights to improve the weight convergence. Allows each task in the training to be closer to the optimal weight learning, and better learns the classification task of unbalanced data.
[0054] Step 3: Use the trained short text classification model to classify agricultural question short texts. Connect it to the softmax layer in a fully connected manner, normalize the fused values containing diverse information, and complete the short text classification of agricultural questions of multiple categories.
[0055] The method of the present invention uses the language pre-training model ERNIE as the embedding layer and encoding layer of the model, and embeds characters, words, entities and other different granularities. The encoding layer (transformer) is improved, and a total of 12 layers are connected in series, and the last layer is taken for output. The text reconstructs the last three layers in a residual-like manner to reduce the information loss caused during the transmission process, and can capture the corpus embedding representation information in a larger range. The improved ERNIE of the encoding layer is combined with multi-scale convolution (CNNs), and CNNs are used as the feature extraction layer to extract text classification feature information. In the multi-scale convolution, the ECA channel attention mechanism is added after the convolution layer with a convolution kernel size of 2 and 4 to enable the model to quickly capture keyword information that is beneficial to classification. It uses a local cross-channel interaction strategy without dimensionality reduction, involving only a few parameters. Appropriate cross-channel interaction can significantly reduce the complexity of the model while maintaining performance. The feature extraction layer of CNNs + ECA attention mechanism extracts local features. The output of ERNIE encoding layer contains a special tag [CLS], which contains the classification information of each sentence, which is equivalent to the global feature of a sentence representation. It is reused here. After the feature extraction layer outputs, the local features are combined with the global features of [CLS] as the final feature representation and sent to the classifier.
[0056] ERNIE-based word vector representation: By introducing three levels of knowledge masking, namely token level (Basic-Level), phrase level (Phrase-Level) and entity level (Entity-Level). For the plain text in the corpus, the lexical analysis tool is first used to segment the text data into different granularities such as characters, words, and entities, and then the segmented data is tokenized to obtain the token sequence and segmentation boundary of the text, and then it is mapped to id data according to the dictionary. During the training process, the model will randomly mask the continuous tokens according to the segmentation boundary, so that the model can learn the corresponding relationships at different levels such as semantics and knowledge, effectively improve the model's knowledge understanding of words and phrases, and learn more accurate semantic information for Chinese agricultural questions. ERNIE's word vector representation method adopts a context-based word vector representation method, that is, for a word, its context information is taken into account. First, after the sentence is input into the model, for each embedding layer of the input sentence, the [CLS] tag is used as a classification tag at the beginning of the sentence, and the tag [SEP] is added at the end of the sentence. The tokenized sentences are input into the embedding layer to obtain the semantic representation of the sentences, which are composed of tag embedding, segment embedding and position embedding. The tag embedding part provides an embedding representation of each word, and the segment embedding is used to embed the information of the global semantic input text and merge with the tag embedding. In order to solve the problem of the difference in semantic information carried by words in different positions in Chinese, position embedding is added as a method to distinguish word vectors in different positions. And the embedding information obtained by summing the three embedding parts is used as the output of the entire embedding layer. In the coding layer stack of the model, except for the bottom coding layer, the inputs of other coding layers are the outputs of their adjacent coding layers, and the output text vector representation of the last layer containing [CLS] is usually obtained as the final output. Due to the inevitable information loss during the transmission of data in the coding layer, and the ERNIE model selected by the present invention has a total of 12 coding layers, and the data will produce a certain amount of loss when passing through each layer. In order to reduce the information loss caused during the transmission process and to capture semantic information in a larger range, the present invention combines the results of the last layer with the outputs of the first two layers to perform a superposition reconstruction similar to the residual method to complete the calculation of the entire transformer encoder. The text vector representation of the reconstructed output semantic information effect is used as the final word vector representation of the model encoding layer, and is used as the input of the multi-scale CNN channel for feature extraction.
[0057] Multi-scale convolution feature extraction: The word vector representation obtained by superimposing the coding layer is input into CNN for information feature extraction. A sequence is used as the vertical axis and the embedding of the token is used as the horizontal axis. In order to enable the convolution layer to extract multiple features, multiple convolution kernels are used in parallel, each of which extracts a feature and is set to multi-scale. Convolution kernels of different scales can obtain word-related content in different ranges to form vertical difference information, that is, different information brought by the appearance of words in different ranges in a sentence. The present invention uses the convolution kernel scale of "2, 3, 4", that is, Figure 1 2*dim, 3*dim and 4*dim in . Each convolution kernel has its own focus. In order to learn multiple different information, the present invention sets 256 convolution kernels. A convolution kernel of one scale obtains a vector after convolution operation. The results of 256 convolution kernels are combined together to form the extracted feature vector at each scale.
[0058] Adding ECA attention mechanism in multiple locations: In the process of model feature extraction, since the data set of the present invention is for problems such as planting in the agricultural field raised by farmers, each piece of data has a decisive part as the category feature of the data. In order to make the model capture key features more quickly and accurately, the present invention adds attention mechanisms in two places in the model to locate and capture key features. Avoid the impact of dimensionality reduction on learning effects, while reducing the computational cost and memory usage of the model. ECANet is an efficient channel attention module for convolutional networks. It adopts a local cross-channel interaction strategy without dimensionality reduction, involving only a few parameters. Appropriate cross-channel interaction can significantly reduce the complexity of the model while maintaining performance. It avoids the negative effects of the dimensionality reduction mechanism of the incentive layer in SENet and SKNet on attention prediction, and the problem of inefficient capture of dependencies between all channels, while better controlling computing power and model complexity.
[0059] The fusion sentence indicates that the vector data is subjected to a multi-scale convolution operation containing the ECANet attention mechanism and then an activation function to remove redundant features in the data, but this will cause the sparsity of the network. In order to solve the sparsity problem caused by the activation function on the network, the present invention adds a maximum pooling layer after the convolution layer. The number of neurons in the pooling layer is equal to the number of filters in the convolution layer, and the vector carrying the most important feature information in each convolution kernel is obtained through the maximum pooling operation. The pooling operation can effectively reduce the size of the parameter matrix, thereby reducing the number of parameters in the last connection layer, which is convenient for use in the fully connected layer. The results obtained at multiple scales are merged as the output of the feature extraction layer.
[0060] The text input into the classification model adopts the word-splitting operation, taking a Chinese character as the smallest splitting unit. In addition, two special word units, namely the flags [SEP] and [CLS], are inserted at the beginning and the end, which are used to distinguish the two sentences of the sentence pair in the input and the identification of the classification task of the corpus, respectively. Among them, [CLS] is the sentence representation, which corresponds to the classification vector representation in the final hidden layer, represented as global feature information, and the vector information under the identification can be transmitted backward to obtain the final classification result. In order to capture the sentence representation in a larger range and avoid the incomplete feature information extracted by the feature extraction layer, the present invention reuses the [CLS] class residual method containing global feature information, and combines it with the local feature representation obtained by the feature extraction layer to obtain the final feature representation result. The multi-layer perceptron (MLP) of the ECA attention mechanism will enhance the weight of the information covering important semantics, so as to achieve the model's learning of a single feature word. As the sentence representation after the original semantic embedding, [CLS] can capture the semantic information of the entire input sequence. Combined with [CLS], it just completes the sequence information after the ECA assigns weights and fills in the semantic relationship connection. The two have a synergistic effect, using their respective advantages to strengthen each other, so that the overall model has better classification performance. The results of the ErcNet experiment (parameter settings during the experiment are shown in Table 1) show that during the transmission process of the ERNIE model encoding layer, the encoding reorganization using the residual method can enhance the word vector and improve the model performance. Finally, the feature representation is connected to the softmax function through a full connection method for classification, and the prediction label is given to achieve agricultural question short text classification.
[0061] The superposition reconstruction formula of the residual-like method is: H(x)=(F(x)+G(x)+R(x));
[0062] Where x is the input, F(x) represents the output after the transformation of the tenth layer, G(x) represents the output after the transformation of the eleventh layer, R(x) represents the output after the transformation of the twelfth layer, and H(x) is the final output of the twelfth layer after residual reconstruction, as the final vector result.
[0063] Table 1 Parameter settings
[0064]
[0065] from Figure 3 and Figure 4 It can be seen from the confusion matrix that the model proposed in this invention has a significant improvement in classification effect compared with the optimal result of the baseline model, especially in the planting management category, where the number of correctly classified questions increased by 6, accounting for about 2% of the total number of questions in this category. The number of correctly classified questions in the policy subsidy category increased by about 3% of the total number of questions in this category, and the misclassification rates of other categories were reduced.
[0066] The present invention compares the proposed model with several common multi-classification models on a subset of an agricultural question dataset and a THUCNews news text classification dataset. This includes FastText, TextCNN, TextRNN, BiLSTM-Attention, Transformer, Bert and ERNIE models. The present invention also compares the model proposed by the present invention with text multi-classification fusion models such as BERT_CNN and BERT-BiLSTM. The model proposed by the present invention is improved on the basis of ERNIE to obtain the ErcNet model, which achieves 96.67%, 96.74%, 96.70%, 0.13 and 96.66% in precision, recall rate, F1 score, loss value and accuracy, respectively. Compared with ERNIE, the precision, recall rate and F1 score are improved by 1.88%, 1.09% and 1.51 respectively, and the loss value is reduced by 0.06. Therefore, the model proposed by the present invention has the best performance in the comparative experiment. The ErcNet model has a slightly low verification accuracy at the beginning, but as the training progresses, the model can efficiently learn key features, surpass other comparison models at the fastest speed, reach a higher accuracy and converge quickly, and basically remain stable when the step is 120, with strong stability. Compared with other models, ErcNet has the highest accuracy and the lowest loss value at a faster convergence speed. From the comprehensive comparison of convergence speed, model stability, accuracy and loss value, it is concluded that the proposed ErcNet model performs best and has a strong training effect.
[0067] Comparison of the precision, recall and F1 score of each question category in the dataset shows that the ErcNet model has relatively stable results in the seven question categories, and the scores in precision, recall and F1 score can be maintained above 94%, 92% and 93%. The classification accuracy of agricultural machinery is slightly lower than that of FastText, but the accuracy of other categories is the highest compared with other models. In addition, other models have poor classification effects on categories with a small number of samples, but the ErcNet model shows better classification performance in categories with a small number of samples, especially in the agricultural machinery management category with the least number of samples. The F1 score of the ErcNet model reaches 98.68%. Comparative data show that the ErcNet model is very effective in the classification tasks of short texts of agricultural questions in seven categories, has better robustness and can solve classification tasks with unbalanced samples.
[0068] Experimental results on public datasets in other fields also show that the ErcNet model can also obtain better classification results on the THUCNews dataset, reaching 91.78%, 91.75%, 91.73%, 0.24 and 91.70% in precision, recall, F1 score, loss value and accuracy respectively. Except for the loss value, all other values are above 90%, reaching the optimal value among the deep learning comparison models.
[0069] The present invention proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a method for classifying short texts of questions on a Chinese agricultural question-and-answer platform are implemented.
[0070] The present invention proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of a method for classifying short texts of questions on a Chinese agricultural question-and-answer platform.
[0071] The memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory of the method described in the present invention is intended to include, but is not limited to, these and any other suitable types of memory.
[0072] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disc (SSD)).
[0073] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.
[0074] It should be noted that the processor in the embodiment of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment may be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0075] The above is a detailed introduction to the short text classification method, device and medium for the Chinese agricultural question-and-answer platform proposed in the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A classification method for short texts of questions on the China Agriculture Question and Answer Platform, characterized in that: The method comprises the following steps: Step 1: Establish a short text database of agricultural questions; Step 2: Build a short text classification model and use the database to train the short text classification model; The short text classification model adopts the form of a pre-trained language model ERNIE combined with a multi-scale convolutional CNNs model, specifically: using the pre-trained language model ERNIE as an embedding layer and an encoding layer, obtaining context-related dynamic word embedding information of text data through the embedding layer, including tag embedding, segment embedding and position embedding information; extracting and transmitting the obtained embedding information to the Transformer Encoder part, extracting the output results of the last three encoding layers of the embedded global semantic information, reconstructing the text vector as the final output, and passing it to the multi-scale CNNs convolutional layer for feature extraction, adding channel attention ECANet at different positions in the multi-scale convolutional neural network, and extracting feature vectors from the local semantic information of the sentence through convolution kernels of different scales; Step 3: Use the trained short text classification model to classify agricultural question short texts. Connect it to the softmax layer in a fully connected manner, normalize the fused values containing diverse information, and complete the classification of short texts of agricultural questions of multiple categories. ERNIE's word vector representation method adopts a context-based word vector representation method, that is, for a word, its context information is taken into account; first, after the sentence is input into the classification model, for each embedding layer of the input sentence, the [CLS] tag is used as a classification tag at the beginning of the sentence, and the tag [SEP] is added at the end of the sentence; the labeled sentence is input into the embedding layer to obtain the semantic representation of the sentence, which consists of tag embedding, segment embedding and position embedding; the tag embedding part provides an embedding representation for each word, and the segment embedding is used to embed the information of the global semantic input text and merge it with the tag embedding, and the position embedding is used to distinguish the word vectors at different positions; the embedding information obtained by summing the three embedding parts is used as the output of the entire embedding layer; The pre-trained language model ERNIE has a total of 12 encoding layers. The result of the last layer is combined with the output of the first two layers to perform a residual-like superposition reconstruction to complete the calculation of the entire encoding layer. The text vector representation of the reconstructed output semantic information is used as the final word vector representation of the encoding layer of the pre-trained language model, which is used as the input of the multi-scale CNNs convolutional layer for feature extraction. The superposition reconstruction formula of the residual-like method is: H(x) = (F(x) + G(x) + R(x)); Where x is the input, F(x) represents the output after the transformation of the tenth layer, G(x) represents the output after the transformation of the eleventh layer, R(x) represents the output after the transformation of the twelfth layer, and H(x) is the final output of the twelfth layer after residual reconstruction, as the final vector result.
2. The method according to claim 1, characterized in that A maximum pooling layer is added after the convolution layer to perform the maximum pooling operation on the feature vector output by the convolution layer, converting the high-dimensional feature matrix into a low-dimensional feature matrix to achieve dimensionality reduction, and using the Relu activation function to truncate the negative value part to 0 to enhance the nonlinear characteristics of the neural network.
3. The method according to claim 1, characterized in that In the feature extraction process, the output local semantic feature vector is connected with the global semantic feature vector of the [CLS] position containing global semantic information output by the ERNIE encoding layer as the output of the feature extraction layer.
4. The method according to claim 1, characterized in that: In the multi-scale convolution, an ECA channel attention mechanism is added after the convolution layers with kernel sizes of 2 and 4 to enable the classification model to quickly capture keyword information that is beneficial to classification.
5. The method according to claim 1, characterized in that In the process of extracting features in the convolution layer, multiple convolution kernels are used, each of which extracts a feature and is set to multi-scale. Convolution kernels of different scales are used to obtain word-related content in different ranges to form vertical difference information, that is, different information brought by the appearance of words in different ranges in a sentence.
6. The method according to claim 2, characterized in that The number of neurons in the maximum pooling layer is equal to the number of filters in the convolution layer. The maximum pooling operation is used to obtain the vector carrying the most important feature information in each convolution kernel.
7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
8. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
Citation Information
Cited By
Unmanned production field planting harvesting consistency adjusting and optimizing system based on hybrid large language model and quantitative model
CN121581274A
