Intelligent classification method for power grid opinion work orders
Through the combination of Bert model and BiLSTM and the attention mechanism, the problem of incomplete learning of text features of work orders for power grid power supply service is solved, more efficient classification effect and accuracy is achieved, manual classification time is reduced, and intelligent is provided for the power grid. to obtain information.
Patent Information
- Application Number
- CN202410944514.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-12-03
AI Technical Summary
The existing intelligent classification method does not fully learn the text characteristics of power grid power supply service work orders, resulting in insufficient classification effect and accuracy, and is not very generalized, which cannot meet the needs of efficient and fast information extraction.
The Bert model is used for pre-training and fine-tuning, combined with BiLSTM and attention mechanism, the text data is extracted and vectorized through the Jieba word segmentation and word2vector tool, dynamically obtain the semantic information before and after, calculate the word weight, and finally classified through the softmax function.
It improves the classification effect and accuracy of power grid power supply service work orders, reduces manual classification time, provides high flexibility in acquisition of feature information, and improves intelligent guarantee of customer power consumption satisfaction.
Smart Images

Figure CN119066540A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of text detection and processing in the electric power industry, and in particular relates to an intelligent classification method for power grid opinion work orders. Background Art
[0002] With the rapid development of informatization, the power grid has accumulated more and more data, and the number of power supply service work orders has increased dramatically every year. Manual extraction of effective data can no longer meet current needs. It is very important to extract effective information from these work orders efficiently and quickly. The power supply service work orders of the power grid include nine categories: fault reporting, inquiry consultation, business application, complaint, report, opinion and suggestion, customer urging, distribution inspection and verification, and suspected power outage warning analysis. Accurate and rapid classification of each work order can provide strong work support for power supply service workers in the power industry to quickly understand customer demands. With the advent of the information age, most of the current intelligent classification methods use recurrent neural network RNN model classification to obtain important information by learning and extracting features from the input data text. The representation space dimension of the text vector of this method is large, and the generalization is not strong. It does not guarantee the integrity of feature acquisition and has certain limitations. Therefore, it is very necessary to provide a power grid opinion work order intelligent classification method with high flexibility, more feature information acquisition, improved classification effect and accuracy, and reduced manual classification time. Summary of the invention
[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and to provide an intelligent classification method for power grid opinion work orders which has high flexibility, obtains more feature information, improves classification effect and accuracy, and reduces manual classification time.
[0004] The object of the present invention is achieved by: a method for intelligently classifying power grid opinion work orders, the method comprising the following steps:
[0005] Step 1: Collect power supply service work order data, build a power supply service work order data set, and pre-process the collected data;
[0006] Step 2: Use the Jieba word segmentation tool to segment the text data in the power supply service work order dataset, and use word2vector to convert it into a vector representation;
[0007] Step 3: Bert layer: Use the binary cross entropy loss function to pre-train and fine-tune the BERT module, dynamically obtain the contextual semantic information of the text to be classified, and output a text vector containing the semantic relationship;
[0008] Step 4: BiLSTM layer: Use BiLSTM to concatenate the text vector output by the Bert layer and the text input word vector for feature extraction, and output a text vector containing more features and semantic information;
[0009] Step 5: Attention layer: Calculate the weight of each word in the text output by the BiLSTM layer. The more important the word is to the sentence classification result, the greater its weight;
[0010] Step 6: Output layer: The weighted output is mapped to the category space of these 9 categories through the fully connected layer, and the probability distribution of each category of the text is calculated through softmax function normalization. The category with the largest probability value is the category to which the text data is finally classified.
[0011] The step 1 specifically comprises the following steps:
[0012] Step 1.1: Collect grid power supply service work orders and construct a data set. The work order categories include nine categories: fault repair, inquiry and consultation, business application, complaint, report, opinion and suggestion, customer reminder, distribution inspection and verification, and suspected power outage warning analysis;
[0013] Step 1.2: Clean the collected data, including removing duplicate data, normalizing the data, and eliminating invalid and erroneous data operations to obtain a standardized power supply service work order data set.
[0014] The step 2 is specifically as follows: the data in the power supply service work order dataset consists of texts, each text contains one or more sentences, Si represents the i-th sentence, the sentence Si is segmented using the jieba word segmentation tool to obtain the word sequence {C1, C2, ..., Cj} corresponding to the sentence, and the word2vector tool is used to vectorize these words to obtain the corresponding word vectors
[0015] The binary cross entropy loss function in step 3 is specifically: the cross entropy loss function L is used to calculate the difference between the true classification result and the model prediction result as the guiding goal of model training, Where n is the total number of samples; x i ,y i are the true value and predicted value of the i-th text classification respectively.
[0016] The step 3 specifically comprises the following steps:
[0017] Step 3.1: The WordPiece word segmentation tool in the Bert model divides the text Si into individual words, and obtains a word set {W1, W2, ..., Wi}, where Wi represents the i-th word in the sentence Si;
[0018] Step 3.2: Vectorize each word {W1, W2, ..., Wi} of the ith sentence Si as the input of Bert. Each word Wi contains three pieces of information: word vector Text Vector and the position vector in Represents the word vector of the i-th word; The text vector representing the i-th sentence; Represents the position vector of the i-th word;
[0019] Step 3.3: Pre-train the Bert model: Perform vector sum encoding on the word vector, text vector, and position vector of each word to obtain a text vector representation containing three pieces of information: Input the Bert model for pre-training. The pre-training tasks include text masking and text prediction, and train a general Bert model with learning semantics.
[0020] Step 3.4: Fine-tune the BERT model: On the pre-trained Bert model, use the data of a specific classification task to train the model. That is, let the model adaptively adjust the model learning parameters under the specific task of power supply service work order classification, train a Bert model suitable for this classification task, and save the model with the best performance for subsequent classification use;
[0021] Step 3.5: After fine-tuning, the Bert model suitable for this classification task is obtained. After feature learning of the text to be classified, a vector representation containing semantic information is output.
[0022] The step 4 specifically comprises the following steps:
[0023] Step 4.1: Transform the word vector of sentence Si and the text vector output by the Bert layer After concatenation, we get the new text vector corresponding to the sentence Si
[0024] Step 4.2: Based on the input word ms at time s and the hidden vector output at the previous time s(t-1) The forget gate r is calculated by the sigmoid function s , r s It is an n-dimensional vector, and the value of each dimension is between (0,1). As a result, the information of a dimension close to 0 is forgotten, and the information of a dimension close to 1 is retained.
[0025] Step 4.3: Calculate r s After that, new information is input at time s, according to ms and The input gate p is calculated by the sigmoid function s The value of gets the new input information, Some of the input information from the previous moment is forgotten;
[0026] Step 4.4: Input gate p s Determine the information input at time s, and then obtain x after nonlinear transformation through tanh function s′ , stores important information of the current input state at time s;
[0027] Step 4.5: Get x by adding the information of the previous time s(t-1) and the information of the current time s s , which is used to decide which information is remembered and output to the next time s(t+1), and according to ms and The output gate q is calculated by the sigmoid function s , get the corresponding output information, for x s After nonlinear transformation, s Multiply to get the vector output of the hidden state That is, the output at the current time s.
[0028] In step 4.4, the nonlinear transformation by the tanh function is used to obtain x s′ Specifically: In the formula, x s′ It is a memory unit that stores important information of the current input state at time s after nonlinear transformation.
[0029] In step 4.5, x is obtained by adding the information of the previous time s(t-1) and the information of the current time s. s For: x s =r s *x (s-1) +p s *x s′ ; Calculate the output gate q through the sigmoid function s for: Output at the current time s for: Where M r 、M p 、M x 、M q Respectively represent the weight parameters that need to be learned in different states; a r 、a p 、a x 、a q They represent the bias parameters in different states respectively.
[0030] The step 5 specifically comprises the following steps:
[0031] Step 5.1: Hidden representation based on the output of the BiLSTM layer Calculate the weight I of each word in sentence Si at time ss : In the formula, M and a are weight and bias respectively;
[0032] Step 5.2: Get the weight coefficient through normalization: In the formula, q is a randomly initialized value;
[0033] Step 5.3: Set the weight coefficient Z s and the hidden vector representation Weighted summation is performed to obtain the output representation O of the attention layer s :
[0034] The step 6 specifically comprises the following steps:
[0035] Step 6.1: Represent the output of the attention layer as O s , which is mapped to a vector dimension containing 9 categories of power supply service orders in the fully connected layer In the output matrix A of the fully connected layer, A = L*O s +B, where L1, L2, ..., L9 represent the nine category labels of the power supply service work order; B is the bias parameter;
[0036] Step 6.2: Use the softmax function to normalize the output of the fully connected layer and convert it into a probability distribution, that is, the probability value of the category to which each sentence belongs. The category with the highest probability is the category to which the sentence is finally classified: Y′=softmax(MA+a), where M and a are weight and bias respectively; Y′ represents the probability value output by the softmax function.
[0037] Beneficial effects of the invention: The invention is a method for intelligently classifying power grid opinion work orders. The invention makes full use of the advantages of the pre-trained model Bert and the recurrent neural network BiLSTM model to avoid the disadvantages of incomplete text feature learning. In use, under the specific classification task of power supply service work orders, Bert uses data pre-training and model fine-tuning to obtain a model suitable for the classification task. Compared with other models, it has great flexibility and outputs a vector representation containing text features. After splicing the data input and the Bert output vector representation, the BiLSTM model learns the previous and next semantic information to extract important features in the text. Other unidirectional RNN models obtain more feature information; integrating the attention mechanism, calculating the word similarity, and assigning different weights to different words can well distinguish the importance of each word to the sentence, thereby improving the classification effect; the combination of Bert, BiLSTM and attention mechanism greatly improves the accuracy of classification of various categories of data in power grid power supply service work orders, reduces manual classification time, provides strong support for quickly obtaining effective information of various categories of work orders, and provides intelligent technical guarantee for improving customer electricity satisfaction; the present invention has the advantages of high flexibility, obtaining more feature information, improving classification effect and accuracy, and reducing manual classification time. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 It is a flow chart of the present invention.
[0039] Figure 2 It is the Bert model diagram of the present invention.
[0040] Figure 3 This is the Bert-BiLSTM model diagram of the present invention. DETAILED DESCRIPTION
[0041] The present invention will be further described below in conjunction with the accompanying drawings.
[0042] Example 1
[0043] like Figure 1-3 As shown, a method for intelligently classifying power grid opinion work orders comprises the following steps:
[0044] Step 1: Collect power supply service work order data, build a power supply service work order data set, and pre-process the collected data;
[0045] Step 2: Use the Jieba word segmentation tool to segment the text data in the power supply service work order dataset, and use word2vector to convert it into a vector representation;
[0046] Step 3: Bert layer: Use the binary cross entropy loss function to pre-train and fine-tune the BERT module, dynamically obtain the contextual semantic information of the text to be classified, and output a text vector containing the semantic relationship;
[0047] Step 4: BiLSTM layer: Use BiLSTM to concatenate the text vector output by the Bert layer and the text input word vector for feature extraction, and output a text vector containing more features and semantic information;
[0048] Step 5: Attention layer: Calculate the word similarity of the output of BiLSTM, weight each word, and give a larger weight to important words. The larger the weight, the greater the impact of the word on the accuracy of sentence classification.
[0049] Step 6: Output layer: The weighted output is mapped to the category space of these 9 categories through the fully connected layer, and the probability distribution of each category of the text is calculated through softmax function normalization. The category with the largest probability value is the category to which the text data is finally classified.
[0050] In this embodiment, the relevant representative meanings of the present invention are explained: ①jieba word segmentation tool: jieba is an open source word segmentation tool for Chinese text processing, which can divide continuous Chinese text into words; ②word2vector: word2vec is an open source word vector calculation tool, which can convert words in the text into vector representations that can be understood by computers; ③The Bert-BiLSTM classification model constructed by the present invention includes five layers: input layer, Bert layer, BiLSTM layer, attention layer and output layer. Bert and BiLSTM are deep learning models widely used in text classification tasks; the input layer divides the text sentences into single words, which are vectorized and input into the Bert model for training.
[0051] The present invention is a method for intelligently classifying power grid opinion work orders. The present invention makes full use of the advantages of the pre-trained model Bert and the recurrent neural network BiLSTM model to avoid the disadvantage of incomplete text feature learning. In use, under the specific classification task of power supply service work orders, Bert uses data pre-training and model fine-tuning to obtain a model adapted to the classification task. Compared with other models, it has great flexibility and outputs a vector representation containing text features. After splicing the data input and the Bert output vector representation, the BiLSTM model learns the previous and next semantic information to extract important features in the text. Compared with other one-way The RNN model obtains more feature information; the attention mechanism is integrated to calculate the word similarity and assign different weights to different words, which can well distinguish the importance of each word to the sentence, thereby improving the classification effect; the combination of Bert, BiLSTM and attention mechanism greatly improves the accuracy of classification of various types of data of power grid power supply service work orders, reduces the time of manual classification, provides strong support for quickly obtaining effective information of various types of work orders, and provides intelligent technical guarantee for improving customer satisfaction with electricity consumption; the present invention has the advantages of high flexibility, obtaining more feature information, improving classification effect and accuracy, and reducing manual classification time.
[0052] Example 2
[0053] like Figure 1-3 As shown, a method for intelligently classifying power grid opinion work orders comprises the following steps:
[0054] Step 1: Collect power supply service work order data, build a power supply service work order data set, and pre-process the collected data;
[0055] In this embodiment, step 1 includes the following steps:
[0056] Step 1.1: Collect grid power supply service work orders and construct a data set. The work order categories include nine categories: fault repair, inquiry and consultation, business application, complaint, report, opinion and suggestion, customer reminder, distribution inspection and verification, and suspected power outage warning analysis;
[0057] Step 1.2: Clean the collected data, including removing duplicate data, normalizing the data, and eliminating invalid and erroneous data operations to obtain a standardized power supply service work order data set.
[0058] Step 2: Use the Jieba word segmentation tool to segment the text data in the power supply service work order dataset, and use word2vector to convert it into a vector representation;
[0059] In this embodiment, step 2 is specifically as follows: the data in the power supply service work order dataset consists of texts, each text contains one or more sentences, Si represents the i-th sentence, and the sentence Si is segmented using the jieba word segmentation tool to obtain the word sequence {C1, C2, ..., Cj} corresponding to the sentence, and these words are vectorized using the word2vector tool to obtain the corresponding word vectors
[0060] Step 3: Bert layer: Use the binary cross entropy loss function to pre-train and fine-tune the BERT module, dynamically obtain the contextual semantic information of the text to be classified, and output a text vector containing the semantic relationship; the cross entropy loss function L is used to calculate the difference between the actual classification result and the model prediction result as the guiding goal of model training. Where n is the total number of samples; x i ,y i are the true value and predicted value of the i-th text classification respectively;
[0061] In this embodiment, step 3 includes the following steps:
[0062] Step 3.1: The WordPiece word segmentation tool in the Bert model divides the text Si into individual words (WordPiece is a built-in word segmentation tool in the Bert model), and obtains a word set {W1, W2, ..., Wi}, where Wi represents the i-th word in the sentence Si;
[0063] Step 3.2: Vectorize each word {W1, W2, ..., Wi} of the ith sentence Si as the input of Bert. Each word Wi contains three pieces of information: word vector Text Vector and the position vector in Represents the word vector of the i-th word; The text vector representing the i-th sentence; represents the position vector of the i-th word; since {W1, W2, ..., Wi} all belong to the words of sentence Si, their corresponding text vectors are
[0064] Step 3.3: Pre-train the Bert model: Perform vector sum encoding on the word vector, text vector, and position vector of each word to obtain a text vector representation containing three pieces of information: The pre-training task includes text masking and text prediction, and a general Bert model with learning semantics is trained. Text masking and text prediction mainly involve randomly masking certain words in the input sentence, predicting the masked content, and training the network to learn the semantic information between the words in the sentence. At the same time, in the input text, for a certain sentence, predict the next sentence of the sentence, and train the network sentence-level contextual semantic information.
[0065] Step 3.4: Fine-tune the BERT model: On the pre-trained Bert model, use the data of a specific classification task to train the model. That is, let the model adaptively adjust the model learning parameters under the specific task of power supply service work order classification, train a Bert model suitable for this classification task, and save the model with the best performance for subsequent classification use;
[0066] Step 3.5: After fine-tuning, the Bert model suitable for this classification task is obtained. After feature learning of the text to be classified, a vector representation containing semantic information is output.
[0067] Step 4: BiLSTM layer: Use BiLSTM to concatenate the text vector output by the Bert layer and the text input word vector for feature extraction, and output a text vector containing more features and semantic information;
[0068] In this embodiment, step 4 includes the following steps:
[0069] Step 4.1: Transform the word vector of sentence Si and the text vector output by the Bert layer After concatenation, we get the new text vector corresponding to the sentence Si
[0070] Step 4.2: Based on the input word ms at time s and the hidden vector output at the previous time s(t-1) The forget gate r is calculated by the sigmoid function s , r s It is an n-dimensional vector, and the value of each dimension is between (0,1). As a result, the information of a dimension close to 0 is forgotten, and the information of a dimension close to 1 is retained.
[0071] Step 4.3: Calculate r s After that, new information is input at time s, according to ms and The input gate p is calculated by the sigmoid function s The value of gets the new input information, Some of the input information from the previous moment is forgotten;
[0072] Step 4.4: Input gate p s Determine the information input at time s, and then obtain x after nonlinear transformation through tanh function s′ , stores important information of the current input state at time s, In the formula, x s′ It is a memory unit that stores important information of the current input state at time s after nonlinear transformation;
[0073] Step 4.5: Get x by adding the information of the previous time s(t-1) and the information of the current time s s :x s =r s *x (s-1) +p s *x s′ , which is used to decide which information is remembered and output to the next time s(t+1), and according to ms and The output gate q is calculated by the sigmoid function s : Get the corresponding output information for x s After nonlinear transformation, s Multiply to get the vector output of the hidden state That is, the output at the current time s, where M r 、M p 、M x 、M q Respectively represent the weight parameters that need to be learned in different states; a r 、a p 、a x 、a q They represent the bias parameters in different states respectively.
[0074] Step 5: Attention layer: Calculate the weight of each word in the text output by the BiLSTM layer. The more important the word is to the sentence classification result, the greater its weight;
[0075] In this embodiment, step 5 includes the following steps:
[0076] Step 5.1: Hidden representation based on the output of the BiLSTM layer Calculate the weight I of each word in sentence Si at time s s : In the formula, M and a are weight and bias respectively;
[0077] Step 5.2: Get the weight coefficient through normalization: In the formula, q is a randomly initialized value;
[0078] Step 5.3: Set the weight coefficient Zs and the hidden vector representation Weighted summation is performed to obtain the output representation O of the attention layer s :
[0079] Step 6: Output layer: The weighted output is mapped to the category space of these 9 categories through the fully connected layer, and the probability distribution of each category of the text is calculated through softmax function normalization. The category with the largest probability value is the category to which the text data is finally classified.
[0080] In this embodiment, step 6 specifically includes the following steps:
[0081] Step 6.1: Represent the output of the attention layer as O s , which is mapped to a vector dimension containing 9 categories of power supply service orders in the fully connected layer In the output matrix A of the fully connected layer, A = L*O s +B, where L1, L2, ..., L9 represent the nine category labels of the power supply service work order; B is the bias parameter;
[0082] Step 6.2: Use the softmax function to normalize the output of the fully connected layer and convert it into a probability distribution, that is, the probability value of the category to which each sentence belongs. The category with the highest probability is the category to which the sentence is finally classified: Y′=softmax(MA+a), where M and a are weight and bias respectively; Y′ represents the probability value output by the softmax function.
[0083] The present invention is a method for intelligently classifying power grid opinion work orders. The present invention makes full use of the advantages of the pre-trained model Bert and the recurrent neural network BiLSTM model to avoid the disadvantage of incomplete text feature learning. In use, under the specific classification task of power supply service work orders, Bert uses data pre-training and model fine-tuning to obtain a model adapted to the classification task. Compared with other models, it has great flexibility and outputs a vector representation containing text features. After splicing the data input and the Bert output vector representation, the BiLSTM model learns the previous and next semantic information to extract important features in the text. Compared with other one-way The RNN model obtains more feature information; the attention mechanism is integrated to calculate the word similarity and assign different weights to different words, which can well distinguish the importance of each word to the sentence, thereby improving the classification effect; the combination of Bert, BiLSTM and attention mechanism greatly improves the accuracy of classification of various types of data of power grid power supply service work orders, reduces the time of manual classification, provides strong support for quickly obtaining effective information of various types of work orders, and provides intelligent technical guarantee for improving customer satisfaction with electricity consumption; the present invention has the advantages of high flexibility, obtaining more feature information, improving classification effect and accuracy, and reducing manual classification time.
Claims
1. A method for intelligent classification of power grid opinion work orders, characterized by: The method comprises the following steps: Step 1: Collect power grid service work order data, build a power supply service work order dataset, and preprocess the collected data; Step 2: Use the Jieba word segmentation tool to segment the text data in the power supply service work order dataset and convert it into a vector representation using word2vector; Step 3: BERT layer: Use the binary cross entropy loss function to pre-train and fine-tune the BERT module, dynamically obtain the contextual semantic information of the text to be classified, and output a text vector containing the semantic relationship; Step 4: BiLSTM layer: Use BiLSTM to concatenate the text vector output by the Bert layer and the text input word vector for feature extraction, and output a text vector containing more features and semantic information; Step 5: Attention layer: Calculate the weight of each word in the text output by the BiLSTM layer. The more important the word is to the sentence classification result, the greater its weight. Step 6: Output layer: The weighted output is mapped to the category space of these 9 categories through the fully connected layer. The softmax function is used for normalization to calculate the probability distribution of each category of the text. The category with the largest probability value is the category to which the text data is finally classified.
2. The method for intelligently classifying power grid opinion work orders according to claim 1, characterized in that: The step 1 specifically includes the following steps: Step 1.1: Collect grid power supply service work orders and construct a data set. The work order categories include nine categories: fault repair, inquiry, business application, complaint, report, opinion and suggestion, customer follow-up, distribution inspection and verification, and suspected power outage warning analysis; Step 1.2: Clean the collected data, including removing duplicate data, normalizing the data, and eliminating invalid and erroneous data operations to obtain a standardized power supply service work order data set.
3. The method for intelligent classification of power grid opinion work orders according to claim 1, characterized in that: The step 2 is specifically as follows: the data in the power supply service work order dataset consists of texts, each text contains one or more sentences, Si represents the i-th sentence, the sentence Si is segmented using the jieba word segmentation tool to obtain the word sequence {C1, C2, ..., Cj} corresponding to the sentence, and the word2vector tool is used to vectorize these words to obtain the corresponding word vectors 4. The method for intelligently classifying power grid opinion work orders according to claim 1, wherein: The binary cross entropy loss function in step 3 is specifically: the cross entropy loss function L is used to calculate the difference between the true classification result and the model prediction result as the guiding goal of model training, Where n is the total number of samples; x i 、y i are the true value and predicted value of the i-th text classification respectively.
5. The method for intelligent classification of power grid opinion work orders according to claim 4, characterized in that: The step 3 specifically includes the following steps: Step 3.1: The WordPiece word segmentation tool in the Bert model splits the text Si into individual words, obtaining a word set {W1, W2, ..., Wi}, where Wi represents the i-th word in the sentence Si; Step 3.2: Vectorize each word {W1, W2, ..., Wi} of the i-th sentence Si as the input of Bert. Each word Wi contains three pieces of information: word vector Text Vector and position vector in Represents the word vector of the i-th word; The text vector representing the i-th sentence; Represents the position vector of the i-th word; Step 3.3: Pre-train the BERT model: Perform vector sum encoding on the word vector, text vector, and position vector of each word to obtain a text vector representation containing three pieces of information: Input into the BERT model for pre-training. The pre-training tasks include text masking and text prediction, and a general BERT model with learning semantics is trained. Step 3.4: Fine-tune the BERT model: Based on the pre-trained BERT model, use data from a specific classification task to perform model training. This allows the model to adaptively adjust its learning parameters for the specific task of power service work order classification. A BERT model suitable for this classification task is trained, and the best performing model is saved for subsequent classification. Step 3.5: After fine-tuning, a Bert model suitable for this classification task is obtained. After feature learning of the text to be classified, a vector representation containing semantic information is output.
6. The method for intelligent classification of power grid opinion work orders according to claim 1, characterized in that: The step 4 specifically includes the following steps: Step 4.1: Transform the word vector of sentence Si And the text vector output by the Bert layer After splicing, we get the new text vector corresponding to the sentence Si Step 4.2: Based on the input word ms at time s and the hidden vector output at the previous time s(t-1) Calculate the forget gate r through the sigmoid function s , r s It is an n-dimensional vector, and the value of each dimension is between (0,1). As a result, the information of the dimension close to 0 is forgotten, and the information close to 1 is retained. Step 4.3: Calculate r s After that, new information is input at time s, according to ms and The input gate p is calculated by the sigmoid function s The value of , get the new input information, Some of the input information from the previous moment is forgotten; Step 4.4: Input gate p s The information input at time s is determined by the nonlinear transformation of the tanh function to obtain x s′ , stores important information at the current input state s; Step 4.5: Get x by adding the information of the previous moment s(t-1) and the information of the current moment s s , which is used to decide which information is remembered and output to the next moment s(t+1), and at the same time, according to ms and The output gate q is calculated by the sigmoid function s , get the corresponding output information, for x s After nonlinear transformation, s Multiply to get the vector output of the hidden state That is, the output at the current time s.
7. The method for intelligently classifying power grid opinion work orders according to claim 6, characterized in that: The nonlinear transformation of the tanh function in step 4.4 yields x s′ Specifically: Where x s′ It is a memory unit that stores important information of the current input state at time s after nonlinear transformation.
8. The method for intelligently classifying power grid opinion work orders according to claim 6, characterized in that: In step 4.5, the information of the previous moment s(t-1) and the information of the current moment s are added together to obtain x s is: x s =r s *x (s-1) +p s *x s′ ; Calculate the output gate q through the sigmoid function s for: Output at the current time s for: Where M r 、M p 、M x 、M q Respectively represent the weight parameters that need to be learned in different states; a r 、a p 、a x 、a q They represent the bias parameters in different states respectively.
9. The method for intelligent classification of power grid opinion work orders according to claim 1, characterized in that: The step 5 specifically includes the following steps: Step 5.1: Hidden representation based on the output of the BiLSTM layer Calculate the weight I of each word in sentence Si at time s s : Where M and a are weight and bias respectively; Step 5.2: Obtain the weight coefficient through normalization: In the formula, q is a randomly initialized value; Step 5.3: Set the weight coefficient Z s and hidden vector representation Weighted summation is performed to obtain the output representation O of the attention layer s :
10. The method for intelligent classification of power grid opinion work orders according to claim 1, characterized in that: The step 6 specifically includes the following steps: Step 6.1: Represent the output of the attention layer as O s , mapped to a vector dimension containing 9 categories of power supply service orders in the fully connected layer In the example, we get the output matrix A of the fully connected layer, A=L*O s +B, where L1, L2, ..., L9 represent the nine category labels of the power supply service work order; B is the bias parameter; Step 6.2: Use the softmax function to normalize the output of the fully connected layer and convert it into a probability distribution, that is, the probability value of the category to which each sentence belongs. The category with the highest probability is the category to which the sentence is finally classified: Y′ = softmax(MA+a), where M and a are the weight and bias, respectively; Y′ represents the probability value output by the softmax function.
Citation Information
Cited By
Automatic generation method and system for periodic maintenance work order of equipment
CN121766727A
