Agricultural question classification method based on bidirectional coding recurrent neural network
Through the bidirectional encoding recursive neural network model, the problems of sparseness and specialization of agricultural text data are solved, the accuracy and stability of agricultural question classification are improved, and efficient text feature extraction and classification are achieved.
Patent Information
- Application Number
- CN202411967991.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-25
AI Technical Summary
The high dimension, sparseness and specialization of agricultural text data lead to insufficient accuracy and stability of existing text classification methods, especially the lack of comprehensive text corpus and sparse proper nouns in the agricultural field, which leads to complex feature extraction, affecting classification accuracy.
The bidirectional encoding recursive neural network (BERNN) model is adopted, including the AgQsBERT layer, BiLSTM layer and fully connected layer, and the text context dependencies are extracted through a bidirectional encoder, and sentence-level attributes and context information are integrated with the multi-headed attention mechanism and the bidirectional long and short-term memory network to generate the classification results of agricultural questions.
Effectively extracting the complex semantic features of agricultural short texts improves the accuracy of agricultural text classification, and realizes efficient text feature extraction and classification, especially in agricultural question classification.
Smart Images
Figure CN119884370A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of agricultural text classification, and in particular to an agricultural question classification method of a bidirectional encoding recursive neural network. Background Art
[0002] With the rapid development of smart agriculture, information technology using the Internet, big data, and artificial intelligence is revolutionizing the way people acquire agricultural knowledge. These knowledge platforms are rich in resources, have large amounts of information, and are updated quickly. However, the integration of agricultural information is still low. In addition, the high dimensionality, sparsity, and specialization of agricultural text data make it impossible to effectively manage the processing of massive texts through manual feature extraction and shallow learning models. Traditional machine learning-based text classification methods have significant advantages in accuracy and stability, but their effectiveness is heavily dependent on the labor-intensive feature extraction process. In contrast, deep learning methods based on neural networks are favored for their powerful feature extraction capabilities. Deep learning excels in extracting local, contextual, and key features. However, agricultural texts are usually short and contain a large number of specialized and sparse domain-specific nouns. Relying on a single neural network model for feature extraction is not enough. Therefore, a variety of fusion model techniques have been proposed to extract richer and more comprehensive text features. However, the short length and domain-specific sparse nouns commonly found in agricultural queries usually make it difficult for popular word embedding models to be effectively applied. In summary, although many scholars have made great progress in agricultural text classification, there are still some problems: (1) There is a lack of comprehensive text corpora in the agricultural field, which makes downstream research difficult; (2) Agricultural texts often contain synonyms in different expressions, which complicates feature extraction and leads to class imbalance, further hindering efforts to improve classification accuracy. Summary of the invention
[0003] In view of the problems in the prior art, the present invention provides an agricultural question classification method based on a bidirectional encoding recursive neural network, which aims to effectively extract complex semantic features from agricultural short texts and improve the accuracy of agricultural text classification.
[0004] A method for classifying agricultural questions using a bidirectional encoding recurrent neural network comprises the following steps:
[0005] Step 1: Build an agricultural question-answer corpus and preprocess it through data cleaning, data marking and data labeling to obtain the input vector;
[0006] Step 2: Build a semantic extraction and classification model. The semantic extraction and classification model includes an input layer, an AgQsBERT layer, a BiLSTM layer, a fully connected layer, and an output layer. The AgQsBERT layer includes a bidirectional encoder, which extracts the contextual dependency of the text in the input vector and forms a fusion vector.
[0007] Step 3: Pass the fused vector to the bidirectional long short-term memory network, integrate the sentence-level attributes and context information in the vector sequence through the bidirectional long short-term memory network, and output the integrated vector;
[0008] Step 4: Reduce the dimension of the integrated vector through the fully connected layer and output the agricultural label category;
[0009] Step 5: The agricultural label category is normalized by Softmax to obtain the classification results of agricultural text.
[0010] Further: the bidirectional encoder includes an embedding layer and a six-layer Transformer encoder. The embedding layer first decomposes the input sentence into tokens, and then generates an initial word embedding vector for each token. This vector includes: Token embedding, Segment embedding and Position embedding. The three embedding vectors are added together to obtain the initial representation vector of each token. The initial representation vector is sequentially passed through the six-layer Transformer encoder to obtain the fusion vector.
[0011] Further: The Transformer encoder includes a multi-head attention mechanism layer, a first residual network and a normalization layer, a feedforward neural network layer, and a second residual network and a normalization layer. The processing process of the input vector in the Transformer encoder is:
[0012] First, the input vector E is input into the encoder, and through linear transformation, the Q matrix representing the word, the K matrix representing the context of each word, and the original matrix V representing the target word and each word in the previous context are obtained; the calculation formulas of Q, K, and V are:
[0013] Q=Linear(E)=EW Q
[0014] K=Linear(E)=EW K
[0015] V=Linear(E)=EW V
[0016] Among them, Q, K, V are the linear mapping matrices of the input vector E, W Q , W K , W V is the weight matrix of the assignment;
[0017] Secondly, the score value Attention(Q,K,V) of the self-attention mechanism is obtained by calculating the scaled dot product. The specific calculation process is:
[0018]
[0019] Then, the self-attention mechanism score values calculated i times are concatenated and linearly transformed to finally obtain an enhanced semantic vector with the same length as the original word vector as the output MultiHead(Q,K,V) of the multi-head attention mechanism layer, specifically:
[0020] MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W 0
[0021] head i =Attention(QW i Q ,KW i K ,VW i V )
[0022] Among them, Q, K, V represent the input word vector matrix, d k represents the input dimension, W i Q , W i K , W i V Indicates head i The weight matrix, W 0 Represents an additional weight matrix; head i is the attention weight of the i-th head, and Concat is to concatenate multiple heads;
[0023] Finally, the output of the multi-head attention mechanism layer is connected and normalized through residual connection to obtain the new character representation fusion vector T.
[0024] Further: the bidirectional long short-term memory network includes two LSTM models with opposite time sequences. The two LSTM models capture text information from both the forward and reverse directions of the sequence. The separate hidden layers in the two LSTM models are combined into a bidirectional layer. The text feature vectors captured in the two directions are fused respectively and then processed by the activation function to output the integrated vector.
[0025] The beneficial effects of the present invention are as follows: a bidirectional encoder is used to capture lexical-level semantic information in agricultural texts and generate hidden vectors with sentence-level features; subsequently, a bidirectional long short-term memory network is used to effectively integrate contextual information to learn hidden complex semantic information, thereby effectively extracting complex semantic features from agricultural short texts. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 is a flow chart of the present invention;
[0027] Figure 2 It is a structural diagram of the semantic extraction and classification model in the present invention;
[0028] Figure 3 Embedded process diagram;
[0029] Figure 4 This is the accuracy curve of 10 cycles on the agricultural problem training set;
[0030] Figure 5 The loss curve for 10 cycles on the agricultural problem training set. DETAILED DESCRIPTION
[0031] The present invention is described in detail below in conjunction with the accompanying drawings. The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention. The directional terms such as left, middle, right, top, and bottom in the embodiments of the present invention are only relative concepts or are based on the normal use state of the product, and should not be considered as restrictive.
[0032] A bidirectional encoding recurrent neural network method for agricultural question classification, such as Figure 1 As shown, the following steps are included:
[0033] Step 1: Build an agricultural question-answer corpus and preprocess it through data cleaning, data marking and data labeling to obtain the input vector;
[0034] In order to solve the problem of few public datasets for agricultural question classification, this paper uses the China Agricultural Technology Extension Information Platform, China Agricultural Information Network and the First Agricultural Economics Network as the corpus source, and crawls 284 questions from v100 as the basic corpus. The obtained agricultural question-answer corpus is preprocessed by data cleaning, data marking and data labeling;
[0035] There are a lot of noise data in the agricultural question-and-answer text, such as special web page tags, carriage returns, special characters, etc. These useless characters may affect the results of model feature extraction. Before building the corpus, these useless characters need to be removed to ensure the data quality of the corpus. Data cleaning mainly completes preprocessing operations such as denoising, deduplication, and missing information processing. The cleaned data will be marked with "[CLS]" and "[SEP]" for questions and answers. The text between "[CLS]" and "[SEP]" is the question, and the text between "[SEP]" and "[SEP]" is the answer to the question. Finally, the text will be marked and classified into six different categories: breeding, planting, processing, fishery, edible fungi, and technical consultation. The data distribution and subject terms in these categories are detailed in Table 1.
[0036] Table 1 Agricultural question text data
[0037]
[0038]
[0039] The question dataset is divided as follows: for each category, 80% of the questions are sampled as the training set (containing 80,482 questions), 10% as the test set (9,901 questions), and the rest as the validation set (9,901 questions).
[0040] Step 2: Build a semantic extraction and classification model (BERNN model), such as Figure 2 As shown, the semantic extraction and classification model includes an input layer, an AgQsBERT layer, a BiLSTM layer, a fully connected layer, and an output layer. The input layer is designed to perform preprocessing operations on text data, including data cleaning and denoising, converting text data into tokens, adding [CLS] and [SEP] tags, padding or truncating text to achieve a uniform length, and converting text data into a matrix or tensor form as an input vector;
[0041] The AgQsBERT layer includes a bidirectional encoder, which extracts the contextual dependency of the text in the input vector and forms a fusion vector. The bidirectional encoder includes a Token unit, a Segment unit, a Position unit, and a six-layer Transformer encoder. The Token unit, the Segment unit, and the Position unit form an embedding layer. The embedding layer first decomposes the input sentence into tokens, and then generates an initial word embedding vector for each token. This vector includes: Token embedding (the basic vector representation of each token), Segment embedding (the vector that distinguishes different sentences), and Position embedding (the position of the token in the sentence). The three embedding vectors are added together to obtain the initial representation vector of each token. Specifically, combined with Figure 3 As shown, the input vector of the training set is e=(e1,e2,e3,…,e n ) The input vector is embedded into words, paragraphs and positions respectively by the Token unit, Segment unit and Position unit, and the initial representation vector E = (E1, E2, E3, ..., E n ), the initial representation vector is passed through the six-layer Transformer encoder to obtain the fusion vector T = (t1, t2, t3, ..., t n ), which is the final output of the semantic extraction and classification model;
[0042] The Transformer encoder includes a multi-head attention mechanism layer, a first residual network and normalization layer, a feedforward neural network layer, and a second residual network and normalization layer. The input vector is processed in the Transformer encoder as follows:
[0043] First, the input vector E is input into the encoder, and through linear transformation, the Q matrix representing the word, the K matrix representing the context of each word, and the original matrix V representing the target word and each word in the previous context are obtained; the calculation formulas of Q, K, and V are:
[0044] Q=Linear(E)=EW Q
[0045] K=Linear(E)=EW K
[0046] V=Linear(E)=EW V
[0047] Among them, Q, K, V are the linear mapping matrices of the input vector E, W Q , W K , W V is the weight matrix of the assignment;
[0048] Secondly, the score value Attention(Q,K,V) of the self-attention mechanism is obtained by calculating the scaled dot product. The specific calculation process is:
[0049]
[0050] Then, the self-attention mechanism score values calculated i times are concatenated and linearly transformed to finally obtain an enhanced semantic vector with the same length as the original word vector as the output MultiHead(Q,K,V) of the multi-head attention mechanism layer, specifically:
[0051] MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W 0
[0052] head i =Attention(QW i Q ,KW i K ,VW i V )
[0053] Among them, Q, K, V represent the input word vector matrix, d k represents the input dimension, W i Q , W i K , W i V Indicates head i The weight matrix, W 0 Represents an additional weight matrix; head i is the attention weight of the i-th head, and Concat is to concatenate multiple heads;
[0054] Finally, the output of the multi-head attention mechanism layer is connected and normalized through residuals to obtain a new character representation fusion vector T;
[0055] Step 3: Pass the fused vector to the bidirectional long short-term memory network, integrate the sentence-level attributes and context information in the vector sequence through the bidirectional long short-term memory network, and output the integrated vector; the bidirectional long short-term memory network includes two LSTM models with opposite time sequences, and the two LSTM models capture text information from the forward and reverse directions of the sequence. The separate hidden layers in the two LSTM models are combined into a bidirectional layer, and the text feature vectors captured in the two directions are fused and then processed by the activation function to output the integrated vector;
[0056] Step 4: Reduce the dimension of the integrated vector through the fully connected layer and output the agricultural label category;
[0057] Step 5: The agricultural label category is normalized by the Softmax of the output layer to obtain the classification result of agricultural text.
[0058] In order to evaluate the contribution of each component in the agricultural text classifier, an ablation experiment was designed. The specific settings of the experiment are as follows: (1) AgQsBERT model, (2) BiLSTM model, (3) AgQsBERT-BiLSTM. The ablation experiment results on the agricultural question test set are shown in Table 2:
[0059] Table 2 Ablation experiment results
[0060]
[0061] As can be seen from Table 2, the BERNN model (semantic extraction and classification model) shows that combining AgQsBERT with BiLSTM generally performs better than using either model alone. AgQsBERT's ability to generate contextual embeddings enables the model to understand the meaning of words based on the context in the sentence, thereby enhancing the overall text understanding ability; in addition, BiLSTM performs well when processing long text sequences because it can retain information over longer distances compared to traditional LSTM.
[0062] In order to comprehensively evaluate different models for text classification tasks, the model experimental results are compared:
[0063]
[0064] As shown in Table 3 above, the BERNN model achieves a classification accuracy of 99.34% on the agricultural problem test dataset with a loss of 0.02, outperforming all other models. Compared with BiLSTM, BiGRU, BiLSTM-Att, and BiGRU-Att, the DPCNN model shows superior classification performance due to its enhanced spatial modeling capabilities. Specifically, the F1 score of DPCNN is 1.93 and 1.54 percentage points higher than that of BiLSTM-Att and BiGRU-Att, respectively, indicating that spatial modeling is more effective than temporal modeling of BiLSTM for problem classification. In addition, the accuracy of the fusion models BiLSTM-CNN and BiGRU-CNN is 0.72 and 0.94 percentage points higher than that of BiLSTM and BiGRU, respectively, highlighting the positive impact of introducing CNN on classification performance. The F1 score of the BERT-DPCNN model reaches 99.08%, surpassing the two fusion models, proving that the pre-trained model using word embedding vectors significantly improves classification performance. The BERNN model uses AgQsBERT to generate embedded latent vectors for words, which are then processed in chronological order, thereby improving the spatial modeling ability of the model. This approach effectively solves the problem of insufficient features of short texts and difficulty in learning semantic information. The F1 score of the BERNN model on the test dataset reached 99.35%, which is 3.26, 2.80, and 0.27 percentage points higher than the fusion models BiLSTM-CNN, BiGRU-CNN, and BERT-DPCNN, respectively.
[0065] In order to evaluate the convergence efficiency of the BERNN model relative to its comparison model, the following detailed analysis is made of the training loss value and the accuracy fluctuations during the training rounds; Figure 4 and Figure 5 The accuracy and loss of 10 cycles on the agricultural problem training set are compared in detail. Figure 4 and Figure 5 As shown in Figure 2, the BERNN model exhibits significantly faster convergence speed while maintaining the same classification accuracy. In addition, at the same convergence speed, BERNN achieves higher classification accuracy. These findings highlight the excellent convergence efficiency of BERNN in question text classification.
[0066] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.
Claims
1. A method for agricultural question classification based on a bidirectional encoding recursive neural network, characterized in that: The following steps are involved: Step 1: Build an agricultural question-answer corpus and preprocess it through data cleaning, data marking and data labeling to obtain the input vector; Step 2: Build a semantic extraction and classification model. The semantic extraction and classification model includes an input layer, an AgQsBERT layer, a BiLSTM layer, a fully connected layer, and an output layer. The AgQsBERT layer includes a bidirectional encoder, which extracts the contextual dependency of the text in the input vector and forms a fusion vector. Step 3: Pass the fused vector to the bidirectional long short-term memory network, integrate the sentence-level attributes and context information in the vector sequence through the bidirectional long short-term memory network, and output the integrated vector; Step 4: Reduce the dimension of the integrated vector through the fully connected layer and output the agricultural label category; Step 5: The agricultural label category is normalized by Softmax to obtain the classification results of agricultural text.
2. The agricultural question classification method using a bidirectional encoding recursive neural network according to claim 1 is characterized in that: The bidirectional encoder includes an embedding layer and a six-layer Transformer encoder. The embedding layer first decomposes the input sentence into tokens, and then generates an initial word embedding vector for each token. This vector includes: Token embedding, Segment embedding, and Position embedding. The three embedding vectors are added together to obtain the initial representation vector of each token. The initial representation vector is sequentially passed through the six-layer Transformer encoder to obtain the fused vector.
3. The agricultural question classification method using a bidirectional encoding recursive neural network according to claim 2 is characterized in that: The Transformer encoder includes a multi-head attention mechanism layer, a first residual network and normalization layer, a feedforward neural network layer, and a second residual network and normalization layer. The input vector is processed in the Transformer encoder as follows: First, the input vector E is input into the encoder, and through linear transformation, the Q matrix representing the word, the K matrix representing the context of each word, and the original matrix V representing the target word and each word in the previous context are obtained; the calculation formulas of Q, K, and V are: Q=Linear(E)=EW Q K=Linear(E)=EW K V=Linear(E)=EW V Among them, Q, K, V are the linear mapping matrices of the input vector E, W Q , W K , W V is the assigned weight matrix; Secondly, the score value Attention(Q,K,V) of the self-attention mechanism is obtained by calculating the scaled dot product. The specific calculation process is: Then, the self-attention mechanism score values calculated i times are concatenated and linearly transformed to finally obtain an enhanced semantic vector with the same length as the original word vector as the output MultiHead(Q,K,V) of the multi-head attention mechanism layer, specifically: MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W 0 Among them, Q, K, V represent the input word vector matrix, d k represents the input dimension, W i Q , Indicates head i The weight matrix, W 0 Represents an additional weight matrix; head i is the attention weight of the i-th head, and Concat is to concatenate multiple heads; Finally, the output of the multi-head attention mechanism layer is connected and normalized through residual connection to obtain the new character representation fusion vector T.
4. The agricultural question classification method using a bidirectional encoding recursive neural network according to claim 1 is characterized in that: The bidirectional long short-term memory network includes two LSTM models with opposite time sequences. The two LSTM models capture text information from both the forward and reverse directions of the sequence. The separate hidden layers in the two LSTM models are combined into a bidirectional layer. The text feature vectors captured in the two directions are fused and then processed by the activation function to output the integrated vector.