Power marketing business key element extraction method and system based on natural language processing technology

Abstract

The invention discloses a power marketing business key element extraction method and system based on a natural language processing technology. The method provided by the invention comprises the following steps: determining the classification of power marketing business work orders; different types of entities are automatically identified and marked; the method comprises the following steps: constructing a multi-level element classification model by utilizing context embedding representation of RoBERTa and combining a preset service level classification structure, realizing fine-grained extraction of key elements of the power marketing service, and constructing a deep neural network entity relationship extraction model based on an attention mechanism to extract an association relationship of the key elements; and calculating the similarity between the key elements of the power marketing business and the key elements of the policy document, extracting keywords in the key elements of the policy document, generating a policy basis abstract based on an improved Seq2Seq model, and improving the abstract quality in combination with an optimization mechanism. According to the method, the problems of poor key element recognition accuracy, insufficient policy document generation summary information extraction and incomplete semantic expression of the existing power marketing business are solved.

Description

Technical Field

[0001] This invention belongs to the field of information processing technology for power marketing business work orders, and in particular, it is a method and system for extracting key elements of power marketing business based on natural language processing technology. Background Technology

[0002] With the widespread adoption of smart grids and the launch of marketing systems, the complexity of electricity marketing operations has increased significantly. The volume of work order data in electricity marketing operations has increased, and the key business elements and policy rules involved in different professional categories have become more complex, making it difficult for traditional marketing work order processing requirements and corresponding policy rule descriptions to keep pace with the development of business needs.

[0003] In the current electricity marketing business, the following challenges are mainly faced: (1) Diversity and complexity of electricity marketing data: Various marketing work orders cover multiple aspects such as electricity price, metering and data collection, business expansion and installation, and customer service. The work order processing process under different professional categories is complex. The technical details and policy rules involved are difficult to handle accurately by humans, which affects the efficiency of business work order processing. (2) Complex content structure of marketing business work orders and high identification difficulty: Electricity marketing business work orders contain a large number of technical terms and rule details, and different types of work orders contain different types of entities and key business elements. Traditional document processing technology cannot effectively understand its deep semantics and business logic, and it is difficult to accurately identify and extract key business elements and the relationship between elements, resulting in the content not being systematic and structured, and thus unable to achieve automated extraction and intelligent analysis. (3) High difficulty in extracting policy basis: The clauses in policy documents are relatively complex, with many professional terms and a large amount of implicit information, which is difficult for non-professionals to understand accurately and comprehensively; at the same time, the extraction of policy document summaries cannot be accurately and effectively matched with the business needs in electricity marketing business, and manual generation is required, which leads to reduced efficiency and increased workload in the business processing process. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for extracting key elements of marketing business based on natural language processing technology. It aims to improve the efficiency of entity identification and the accuracy of key element extraction in marketing business by applying natural language processing technology. It can also intelligently generate corresponding policy documents based on the needs of marketing business, thereby improving the processing efficiency and accuracy of work orders in power marketing business, and helping power companies respond to customer needs and handle audit issues more promptly.

[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solution.

[0006] In a first aspect, the present invention provides a method for extracting key elements of marketing operations based on natural language processing technology, comprising:

[0007] Step 1: Define category labels for electricity marketing business, label the business work order text with category labels, and build a labeled work order dataset; use the TF-IDF method to extract the feature vector of the work order text, input it into the support vector machine for training, and realize the automatic recognition of work order business type;

[0008] Step 2: Collect and preprocess the work order text of electricity marketing business. Use Word2Vec to convert each word in the work order text into a fixed-dimensional vector. Use the BiLSTM model to obtain the context information of the words. Use Conditional Random Field (CRF) to label the word sequence output by the BiLSTM model and automatically identify and label different categories of entities.

[0009] Step 3: Utilize the context embedding representation of the RoBERTa model and combine it with the preset business hierarchy classification structure to construct a multi-level element classification model, thereby achieving fine-grained extraction of key elements in the power marketing business. Based on the attention mechanism, construct a deep neural network entity relationship extraction model to extract the association relationships of key elements.

[0010] Step 4: Calculate the similarity between key elements of electricity marketing business and key elements of policy documents using a similarity algorithm, extract keywords from key elements of policy documents using the TextRank algorithm, generate policy basis summaries based on an improved Seq2Seq model, and improve the quality of the summaries by combining optimization mechanisms.

[0011] Furthermore, the specific process of step 1 includes:

[0012] Step 1.1: Collect historical work order text data, define category labels for electricity marketing business, manually label each work order text with category labels to form a labeled work order dataset for training.

[0013] Step 1.2: Preprocess the labeled work order texts by using the TF-IDF (Term Frequency-Inverse Document Frequency) method to calculate the term frequency features of the work order texts, and obtain the feature vector of each work order in the vocabulary space, which is used as the input for subsequent model training.

[0014] Step 1.3: Using the feature vector as input and the work order category label as output, train a support vector machine to build the model; after training, classify and predict new work orders in the power marketing business to achieve automatic identification of work order business types.

[0015] Furthermore, the specific process of step 1.2 includes:

[0016] Step 1.2.1: Preprocess each marked work order text, including word segmentation, removal of stop words, and standardization of capitalization;

[0017] Step 1.2.2: Use the TF-IDF method to calculate the word frequency features of the text, quantify the importance of each term, and obtain the feature vector of each work order text in the lexical space.

[0018] Furthermore, the specific process of step 1.3 includes:

[0019] Step 1.3.1: Using the aforementioned feature vector as input and the work order category label as output, train a support vector machine, determine the optimal classification hyperplane by maximizing the class margin, and form a text classifier;

[0020] Step 1.3.2: Apply the trained text classifier to predict new work orders for the power marketing business. Input the feature vector extracted from the new work order text into the text classifier, and determine the business category to which the work order belongs through support vectors and decision boundaries to achieve automatic classification.

[0021] Furthermore, the specific process of step 2 includes:

[0022] Step 2.1: Obtain the original power marketing business work order text and preprocess it to remove irrelevant information, filter out irrelevant noise, special symbols, numbers, stop words and specific format information, use the jieba Chinese word segmentation tool to segment the work order text, extract its part-of-speech features, and form a word sequence with grammatical information.

[0023] Step 2.2: Define the category labels for each entity, manually or semi-automatically label the entity categories in the business work order, and convert the labeled data into a format suitable for training.

[0024] Step 2.3: Use Word2Vec to map each word in the work order text to a fixed-dimensional word vector; based on the word vector, combine the aforementioned entity category labels to annotate and align each word, and more accurately quantify the words containing entity content;

[0025] Step 2.4: Use the BiLSTM model to extract the contextual semantic features of words, and input the output of the BiLSTM model into the conditional random field layer to perform entity recognition and annotation on the word sequence.

[0026] Furthermore, the specific process of step 2.4 includes:

[0027] Step 2.4.1: Input the word sequence after word vector encoding into the BiLSTM model to obtain the semantic representation of each word in the context. Introduce an attention mechanism to weight the semantic representation, highlight the feature representation of the entity in the context, and obtain the feature sequence.

[0028] Step 2.4.2: Input the BiLSTM model and the feature sequence processed by the attention mechanism into the conditional random field model. By modeling the dependency relationship and transition constraint between labels, the recognition of various entities in the work order text is realized.

[0029] Furthermore, the specific process of step 3 includes:

[0030] Step 3.1: Design a tag system with business semantic hierarchy and construct a corresponding business hierarchical classification structure;

[0031] Step 3.2: Use the RoBERTa model to segment and embed the work order text into words, extract the key semantic features of the text, and output the context embedding vector.

[0032] Step 3.3: Based on the context embedding vector output by the RoBERTa model, construct a multi-level feature classification model with a hierarchical structure. This model consists of multiple sub-classifiers, each of which is responsible for identifying key features at the corresponding level, thereby achieving fine-grained extraction of key features step by step.

[0033] Step 3.4: Construct a deep neural network entity relationship extraction model based on the multi-head attention mechanism to extract the relationships between key elements.

[0034] Furthermore, the specific process of step 3.2 includes:

[0035] Step 3.2.1: Use the word segmenter built into the RoBERTa model to process the work order text, serialize the sentence or text into a sequence of words, and perform word segmentation to obtain the input sequence;

[0036] Step 3.2.2: Add special markers to the input sequence to identify sentence boundaries, which are used to mark the beginning and end of sentences, forming a standard RoBERTa input format and obtaining a standard input sequence;

[0037] Step 3.2.3: Feed the standard input sequence into the RoBERTa model and output the context embedding vector for each word;

[0038] The specific process of step 3.3 includes:

[0039] Step 3.3.1: Define the hierarchical structure and label system of business elements, and establish the corresponding sub-classifier structure; use the context embedding vector output by the RoBERTa model as input to train the sub-classifiers at each level, and complete the construction of the multi-level element classification model;

[0040] Step 3.3.2: Call the first-level sub-classifier to perform preliminary classification of the context embedding vector and identify the key element categories (such as business category, object category, etc.);

[0041] Step 3.3.3: Based on the primary classification results, select the corresponding secondary sub-classifier to further complete the sub-category identification and achieve fine-grained extraction of key elements;

[0042] The specific process of step 3.4 includes:

[0043] Step 3.4.1: Input the context embedding vector of the identified entities into the multi-head self-attention mechanism, calculate the mutual dependency weight between each entity and other entities in the work order text, and obtain a set of context-aware entity feature vectors.

[0044] Step 3.4.2: For the target entity pair of the relationship to be predicted, the corresponding two context-aware entity feature vectors are fused to obtain the entity pair fused feature vector for relationship recognition.

[0045] Step 3.4.3: Input the entity pair fusion feature vector into a pre-trained deep neural network relationship classifier, which outputs the probability distribution of the entity pair belonging to each predefined relationship category. Finally, select the category with the highest probability value as the final association relationship to realize the extraction of the association relationship between key elements.

[0046] Furthermore, the specific process of step 4 includes:

[0047] Step 4.1: Match the key elements in the policy documents with the key elements of the electricity marketing business; use cosine similarity to calculate the similarity between the key elements of the marketing business and the key elements of the policy documents, and extract the key elements of the policy documents with high similarity.

[0048] Step 4.2: Use the TextRank algorithm to calculate the weight of each key element extracted from the policy document. Iterate through the converged scores to obtain words with high weights, and use these words as the summary content.

[0049] Step 4.3: Generate policy basis summaries using a Seq2Seq model improved based on a multi-head attention mechanism, and combine this with an optimization mechanism to improve the quality of the summaries.

[0050] Secondly, the present invention provides a key element extraction system for electricity marketing business based on natural language processing technology, used to implement the above-mentioned key element extraction method for electricity marketing business, comprising:

[0051] Marketing Business Work Order Classification Unit: Define category labels for electricity marketing business, label the business work order text with category labels, and construct a labeled work order dataset; use the TF-IDF method to extract the feature vector of the work order text, input it into the support vector machine for training, and realize the automatic recognition of work order business type;

[0052] Entity recognition and annotation unit: Collects and preprocesses power marketing business work order text, uses Word2Vec to convert each word in the work order text into a fixed-dimensional vector, uses the BiLSTM model to obtain the context information of the words, and uses conditional random fields to annotate the word sequence output by the BiLSTM model, automatically recognizing and annotating entities of different categories.

[0053] Key Business Element Relationship Extraction Unit: Utilizing the context embedding representation of the RoBERTa model and combining it with a pre-defined business hierarchy classification structure, a multi-level element classification model is constructed to achieve fine-grained extraction of key elements in the power marketing business. Based on the attention mechanism, a deep neural network entity relationship extraction model is constructed to extract the relationships between key elements.

[0054] Abstract Quality Improvement Unit: This unit uses a similarity algorithm to calculate the similarity between key elements of electricity marketing business and key elements of policy documents, uses the TextRank algorithm to extract keywords from the key elements of policy documents, generates policy basis abstracts based on an improved Seq2Seq model, and combines optimization mechanisms to improve abstract quality.

[0055] Compared with the prior art, the present invention has the following technical effects:

[0056] Accurate Content Recognition: By employing the BiLSTM-CRF model, this invention effectively handles entity recognition tasks in electricity marketing work orders. BiLSTM provides contextual information, while the Conditional Random Field (CRF) ensures global consistency of labels, avoiding conflicts in label sequences or unreasonable annotation results. In this way, this invention can accurately identify and annotate entities (such as customer information, equipment type, fault type, etc.) in electricity marketing work orders, laying the foundation for subsequent relation extraction and policy basis generation.

[0057] Enhanced Relationships Among Key Elements: This invention enables fine-grained extraction of key business elements. RoBERTa's semantic embedding ensures an understanding of fine-grained element information, while the hierarchical classification framework extracts detailed information about business elements layer by layer. This method effectively improves the relationships between extracted business elements in complex business text scenarios such as electricity marketing.

[0058] Intelligent Policy Basis Summary Generation Technology: This invention generates policy document summaries using the TextRank algorithm, cosine similarity matching, and a Seq2Seq model with an attention mechanism, ensuring the accuracy of the policy summaries. Compared to traditional policy summary generation technologies, this invention significantly improves the matching degree between summary content and marketing needs, making policy basis summaries more accurate. Generated according to marketing business needs, it effectively supports the efficiency of power marketing work order processing. Attached Figure Description

[0059] Figure 1 This is a flowchart of a method for extracting key elements of electricity marketing business based on natural language processing technology according to the present invention;

[0060] Figure 2 This is a diagram illustrating the composition of a key element extraction system for electricity marketing business based on natural language processing technology, as described in this invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be explained and described below with reference to the accompanying drawings. However, the following embodiments are only preferred embodiments of the present invention and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments in the specific implementation details without creative effort are all within the protection scope of the present invention.

[0062] Example 1

[0063] This embodiment provides a method for extracting key elements of electricity marketing business based on natural language processing technology. (See also...) Figure 1 As shown, the steps are as follows:

[0064] Step 1: Define category labels for electricity marketing business, label the business work order text with category labels, and construct a labeled work order dataset; use the TF-IDF method to extract the feature vector of the work order text, input it into the support vector machine for training, and realize the automatic recognition of work order business type.

[0065] The specific details of step 1 are as follows:

[0066] Step 1.1: Collect historical work order text data, define category labels for electricity marketing business, manually label each work order text with category labels to form a labeled work order dataset for training; electricity marketing business can be divided into 5 major categories: business expansion application, electricity price, metering data collection, customer service, and new business.

[0067] Step 1.2: Preprocess the labeled work order texts by using the TF-IDF (Term Frequency-Inverse Document Frequency) method to calculate the term frequency features of the work order texts, and obtain the feature vector of each work order in the lexical space.

[0068] TF (Term Frequency) represents the frequency of a word in the current document, while IDF (Inverse Document Frequency) represents the rarity of a word in the entire document set.

[0069]

[0070] TF-IDF = TF(t,d)ΔIDF(t,D)

[0071] In the formula, TF(t,d) represents the frequency of word t in document d; IDF(t,D) represents the rarity of word t in the entire document set D.

[0072] Specifically, the process of step 1.2 includes:

[0073] Step 1.2.1: Preprocess each labeled work order text, including word segmentation, removal of stop words, and standardization of capitalization, to lay a foundation for subsequent vectorization.

[0074] Step 1.2.2: The TF-IDF method is used to calculate the word frequency features of the text, quantify the importance of each word, and obtain the feature vector of each work order text in the lexical space, which is used as the input for subsequent classification modeling.

[0075] Step 1.3: Using the feature vector as input and the work order category label as output, train a support vector machine to build the model; after training, classify and predict new work orders in the power marketing business to achieve automatic identification of work order business types.

[0076] Specifically, the process of step 1.3 includes:

[0077] Step 1.3.1: Using the aforementioned feature vector as input and the work order category label as output, train a support vector machine, determine the optimal classification hyperplane by maximizing the class margin, and form a text classifier;

[0078] For the work order text feature vector v i and category label y i (The category labels are business expansion application [0], electricity price [1], metering data collection [2], customer service [3], and new business [4]), and the optimization objective of the text classifier is:

[0079]

[0080] Constraint y i (ω·v i +b)≥1, i=1,2,...,N

[0081] Where: ω is the weight vector, used to determine the direction of the hyperplane; b is the bias, which determines the specific position of the hyperplane.

[0082] Cross-validation is used to divide the training set and validation set. Parameters are optimized through grid search, and the parameter combination with the highest accuracy on the validation set is selected as the final classifier parameters.

[0083] Step 1.3.2: Apply the trained text classifier to predict new work orders for the power marketing business. Input the feature vector extracted from the new work order text into the text classifier, and determine the business category to which the work order belongs through support vectors and decision boundaries to achieve automatic classification.

[0084] Step 2: Collect and preprocess the work order text of electricity marketing business. Use Word2Vec to convert each word in the work order text into a fixed-dimensional vector. Use the BiLSTM (Bidirectional Long Short-Term Memory) model to obtain the context information of the words. Use Conditional Random Field (CRF) to label the word sequence output by the BiLSTM model, and automatically identify and label different categories of entities.

[0085] Specifically, the process of step 2 includes:

[0086] Step 2.1: Obtain the original power marketing business work order text and preprocess it to remove irrelevant information, filter out irrelevant noise, special symbols, numbers, stop words and specific format information, use the jieba Chinese word segmentation tool to segment the work order text, extract its part-of-speech features, and form a word sequence with grammatical information.

[0087] Step 2.2: Define the category labels for each entity. Manually or semi-automatically label the entity categories (e.g., customer, equipment, fault, payment, etc.) in the business work order. Convert the labeled data into a training-suitable format (BIO labeling format), where each word has a label, as shown in the table below:

[0088]

[0089] Step 2.3: Use Word2Vec to map each word in the work order text to a fixed-dimensional word vector; based on the word vector, combine the aforementioned entity category labels to annotate and align each word, and more accurately quantify the words containing entity content.

[0090] More specifically, the specific process of step 2.3 includes:

[0091] Step 2.3.1, Obtain the basic word vectors: Let the input sentence be X = [x1, x2, ..., x...]. n Using Word2Vec, each word x i Mapping to a d-dimensional vector can be represented as: The word embedding matrix of the entire sentence is then:

[0092]

[0093] Where n is the sentence length, d is the dimension of word embedding, and ω i Indicator x iIt is mapped to a d-dimensional real vector.

[0094] Step 2.3.2, obtain entity label vector: for each word x i Obtain the corresponding entity category label, and vectorize the label by converting it into a k-dimensional entity label vector t using one-hot encoding. i .

[0095] Step 2.3.3, perform annotation alignment: align the word vector ω of each word. i and its corresponding entity label vector t i The concatenation and alignment are performed to form a (d+k)-dimensional enhanced feature vector W. e =concat(ω i ,t i This allows for more accurate word quantification.

[0096] Step 2.4: Use the BiLSTM model to extract the contextual semantic features of words, and input the output of the BiLSTM model into the Conditional Random Field (CRF) layer to perform entity recognition and annotation on the word sequence.

[0097] More specifically, the specific process of step 2.4 includes:

[0098] Step 2.4.1: Input the word sequence after word vector encoding into the BiLSTM model to obtain the semantic representation of each word in the context. Introduce an attention mechanism to weight the semantic representation, highlight the feature representation of the entity in the context, and obtain the feature sequence.

[0099] For the enhanced word embedding matrix W e The BiLSTM model obtains the contextual features of each word through forward LSTM and backward LSTM.

[0100] Input: The embedding matrix W of the given input words e =[w′1,w′2,...,w′ n ], where w′ i The word vector for each word.

[0101] Output: The output of BiLSTM is a feature sequence Z = (z1, z2, ..., z...). n ), where z i It is a contextual representation of each word, containing contextual information extracted from the forward and backward LSTM layers.

[0102] The forward LSTM processes the input sequence step by step from left to right. For each word w i Each forward LSTM will compute a hidden state.

[0103]

[0104] Backward LSTM processes the input sequence step by step from right to left. For each word w i Each backward LSTM will compute a hidden state.

[0105]

[0106] Ultimately, each word x i There will be two hidden states, one from the forward LSTM and one from the backward LSTM. The BiLSTM concatenates or performs a weighted average of these two hidden states to obtain the final contextual representation of the word.

[0107]

[0108] To highlight the context weights of key entities, an attention mechanism is introduced, which is applied to the feature sequence Z output by the BiLSTM. For each feature vector z in the sequence... i The final feature vector c is calculated in the following way. i .

[0109] First, calculate z i With all vectors z in the sequence j Attention score:

[0110]

[0111] Among them, W a 、U a 、v a These are learnable weight parameters.

[0112] Then, the attention score is normalized into attention weights:

[0113]

[0114] Finally, the feature vectors are weighted and summed according to the attention weights to obtain the final feature vector c. i :

[0115]

[0116] After processing by the attention mechanism, the final feature sequence C = (c1, c2, ..., c) is obtained. n Each vector in the sequence more intelligently incorporates global contextual information.

[0117] Step 2.4.2: Input the BiLSTM model and the feature sequence processed by the attention mechanism into the conditional random field model. By modeling the dependency relationship and transition constraint between labels, the recognition of various entities in the work order text is realized.

[0118] The core idea of ​​CRF is to consider the global dependency of the label sequence within the framework of the conditional probability model (for example, a "B-customer" label is more likely to be followed by an "I-customer" label than another label starting with "B-"), and to find the optimal label sequence globally to achieve accurate identification of various entities in the work order text.

[0119] Specifically, given the input sequence, which in this case is the final feature sequence C output by the attention layer, and the label sequence Y = (y1, y2, ..., y3), ... n The probability of ). The conditional probability model of CRF is:

[0120]

[0121] in:

[0122] Indicates label y i and y i-1 The transfer score between labels (i.e., the dependency relationship between labels).

[0123] Indicates label y i The corresponding feature scores,

[0124] f i It is the feature vector at the i-th position extracted by BiLSTM, which is not the original input feature;

[0125] Z(C) is a normalization factor, summed over all possible label sequences Y′ to ensure that the sum of conditional probabilities is 1, where Y′=[y′1,y′2,…,y′]. n ] represents all possible tag sequences.

[0126]

[0127] In the statement "electricity meter malfunctions", CRF integrates the features of each word and the constraints between labels, and finally labels "electricity meter" as B-device and "malfunction" as B-fault type, outputting the most reasonable label sequence.

[0128] Step 3: Utilize the contextual embedding representation of the RoBERTa model and combine it with the preset business hierarchy classification structure to construct a multi-level element classification model, thereby achieving fine-grained extraction of key elements in the power marketing business. Based on the attention mechanism, construct a deep neural network entity relationship extraction model to extract the correlation relationships of key elements.

[0129] Specifically, the process of step 3 includes:

[0130] Step 3.1: Design a tagging system with business semantic hierarchy and construct a corresponding business hierarchical classification structure. For example, equipment type tags can be further subdivided into electricity meters, distribution cabinets, etc., to ensure that the model can identify business elements at different granularities.

[0131] Step 3.2: Use the RoBERTa model to segment and embed the work order text into words, extract the key semantic features of the text, and output the context embedding vector.

[0132] Specifically, the process of step 3.2 includes:

[0133] Step 3.2.1: Use the word segmenter built into the RoBERTa model to process the work order text, serialize the sentence or text into a sequence of words, and perform word segmentation to obtain the input sequence;

[0134] The RoBERTa model's built-in word segmenter breaks down sentences or text sequences into sub-word units contained in the model's vocabulary, resulting in a word sequence X = {x1, x2, ..., x...}. n}

[0135] Step 3.2.2: Add special markers to the input sequence to identify sentence boundaries, which are used to mark the beginning and end of sentences, forming a standard RoBERTa input format and obtaining a standard input sequence;

[0136] Add before and after each input sequence <s> and< / s> Mark up to obtain the following embedding format:

[0137] <s> Customers reported power outages due to unpaid bills and requested power restoration.< / s>

[0138] <s>The output vector corresponding to the marker can then be used as the aggregate semantic representation of the entire text.

[0139] < / s> Markers used to identify the end of a sequence help distinguish between multiple input segments.

[0140] Step 3.2.3: Feed the standard input sequence into the RoBERTa model and output the context embedding vector for each word.

[0141] RoBERTa encodes the input sequence using a Transformer encoder. Each layer captures the contextual dependencies between words through a multi-head self-attention mechanism, and finally outputs the hidden state of the last layer as a context embedding vector.

[0142] Step 3.3: Based on the context embedding vector output by the RoBERTa model, construct a multi-level feature classification model with a hierarchical structure. This model consists of multiple sub-classifiers, each of which is responsible for identifying key features at its corresponding level, thereby achieving fine-grained extraction of key features step by step.

[0143] The specific process of step 3.3 includes:

[0144] Step 3.3.1: Define the hierarchical structure and label system of business elements, and establish the corresponding sub-classifier structure; use the context embedding vector output by the RoBERTa model as input to train the sub-classifiers at each level, and complete the construction of the multi-level element classification model.

[0145] A multi-level element classification framework is constructed based on business logic, clearly defining the hierarchical relationships between elements at each level. The first level represents the major business category, the second level represents the subcategories, and the third level represents specific attributes, ensuring that the tag system satisfies mutual exclusivity and completeness. For example:

[0146] Level 1: {Customer Information, Equipment Information, Fault Information, Request Information};

[0147] Level 2 (Fault Information): {Power Supply Fault, Equipment Fault, Service Fault};

[0148] Level 3 (under power supply failure): {abnormal voltage, power outage, line fault};

[0149] Each subclassifier consists of a linear layer and an activation function. A hierarchical progressive training method is adopted. First, the first-level classifier is trained with cross-entropy loss, targeting the first-level feature labels. Then, the parameters of the first-level classifier are fixed, and the second-level subclassifier is trained with the corresponding second-level feature labels as the target (the same applies to the third level), ensuring that each classifier focuses on the task at its level.

[0150] Step 3.3.2: Call the first-level sub-classifier to perform preliminary classification of the context embedding vector and identify the categories of key elements;

[0151] Using RoBERTa model embedding vectors as input, a linear layer maps the embedding vectors to a first-level feature class space, and a softmax activation is applied to output the classification probability.

[0152] P level1 (c1) = softmax(W1·E + b1)

[0153] Where W1 and b1 are the parameters of the first-level classifier, E is the word embedding vector, and c1 represents the first-level feature category, such as fault information.

[0154] The category with the highest probability is selected as the primary classification result and used as the input condition for the secondary classification.

[0155] Step 3.3.3: Based on the primary classification results, select the corresponding secondary subclassifier to further complete the subclass identification and achieve fine-grained extraction of key elements.

[0156] In addition to the context embedding vector of RoBERTa, E glo Extract word embeddings E from the text fragments corresponding to the first-level categories. loc (e.g., word vectors of sentences related to "fault information") are aggregated into a local feature vector E through an attention mechanism. input =[E glo E loc ].

[0157] The linear layer of the second-level subclassifier maps to the corresponding subclass space:

[0158] P level2 (c2|c1)=softmax(W2·E input +b2)

[0159] Where W2 and b2 are the parameters of the secondary classifier, and c2 represents the secondary feature category.

[0160] If a higher granularity (such as a third-level feature) is required, repeat step 3.3.3, and use the second-level result c2 to call the corresponding third-level sub-classifier until the finest granular feature (such as "power outage caused by unpaid bills") is extracted.

[0161] Step 3.4: Construct a deep neural network entity relationship extraction model based on a multi-head attention mechanism to extract the relationships between key elements. This model includes an entity representation enhancement module and a relationship classification module. The former strengthens context-dependent features through an attention mechanism, while the latter predicts the relationship category between entity pairs based on fused feature vectors.

[0162] The specific process of step 3.4 includes:

[0163] Step 3.4.1: Input the context embedding vector of the identified entities into the multi-head self-attention mechanism, calculate the mutual dependency weight between each entity and other entities in the work order text, and obtain a set of context-aware entity feature vectors.

[0164] Output the entity context embedding vector z from the RoBERTa model i The entity sequence representation matrix Z = {z1, z2, ..., z3} is constructed by embedding all entity context vectors. n}, N is the number of entities, and d is the embedding dimension.

[0165] The entity sequence representation matrix Z is input into a multi-head self-attention mechanism, allowing each entity to attend to all entities in the sequence to capture semantic dependencies between entities. The multi-head attention mechanism maps entities in the input sequence to different subspaces, generating a query Q, key K, and value V matrix.

[0166]

[0167] Where, d k＝ d v It is the dimension of the head.

[0168] Each attention head performs the following operations:

[0169]

[0170] Multi-head output integration:

[0171]

[0172] Where h represents the number of attention heads, and this operation will process the outputs of h attention heads (each head has an output dimension of d). v ) spliced ​​together to form a dimension of h·d v matrix.

[0173] Step 3.4.2: For the target entity pair of the relationship to be predicted, the corresponding two context-aware entity feature vectors are fused to obtain the entity pair fused feature vector for relationship recognition.

[0174] Feature fusion is performed through dot product or concatenation operations:

[0175]

[0176] Among them, z i and z j Embedding of entity pairs.

[0177] Step 3.4.3: Input the entity pair fusion feature vector into a pre-trained deep neural network relationship classifier, which outputs the probability distribution of the entity pair belonging to each predefined relationship category. Finally, select the category with the highest probability value as the final association relationship to realize the extraction of the association relationship between key elements.

[0178] The formula for a deep neural network relation classifier is as follows:

[0179] P(y|F)=softmax(W c F+b c )

[0180] Where F represents the entity pair fused feature vector, W c and b crepresents the weights and biases of the classifier, y represents the predefined relation categories (such as attribution, causation, configuration, irrelevance), and softmax(·) is the activation function used to transform the output into a probability distribution of relation categories.

[0181] Output relationship probability distribution:

[0182] P(y|F)=[p1,p2,…,p y ]

[0183] Final Relationship:

[0184]

[0185] Step 4: Calculate the similarity between key elements of electricity marketing business and key elements of policy documents using a similarity algorithm, extract keywords from key elements of policy documents using the TextRank algorithm, generate policy basis summaries based on an improved Seq2Seq model, and improve the quality of the summaries by combining optimization mechanisms.

[0186] Specifically, the process of step 4 includes:

[0187] Step 4.1: Match the key elements in the policy documents with the key elements of the electricity marketing business; use cosine similarity to calculate the similarity between the key elements of the marketing business and the key elements of the policy documents, and extract the key elements of the policy documents with high similarity.

[0188] Calculate the similarity between the keyword set of business requirements and the policy text using cosine similarity:

[0189]

[0190] Where A is the business key element vector and B is the policy key element vector.

[0191] Step 4.2: Use the TextRank algorithm to calculate the weight of each key element extracted from the policy document. Iterate through the converged scores to obtain words with high weights, and use these words as the summary content.

[0192] The TextRank score update formula is as follows:

[0193]

[0194] Where S(ω) i ) represents node ω i The weight score is calculated, where d is the damping factor, typically taken as 0.85. The iteration continues until the weights of each node converge; In(ω i Let ω be the set of incoming edge nodes, pointing to ω. iThe set of all word nodes; Out(ω) j Let ω be the set of outgoing edge nodes. j Pointer to the set of all word nodes; weight(ω) j ,ω i ) represents the edge weight, indicating the word ω i With ω j The co-occurrence intensity or similarity; S(ω) j ) indicates word ω j The weighted score.

[0195] Step 4.3: Generate policy basis summaries using a Seq2Seq model improved based on a multi-head attention mechanism, and combine this with an optimization mechanism to improve the quality of the summaries.

[0196] Step 4.3.1 uses BiLSTM as the basic encoder to capture the complex semantic features of the input text. Given an input text sequence X = {x1, x2, ..., x...} n BiLSTM generates forward and backward hidden states at each time step t. and And combine them into word embedding h t :

[0197]

[0198] Step 4.3.2, based on the word embedding h generated by the encoder t An LSTM decoder with a multi-head attention mechanism is employed to dynamically focus the key features output by the encoder, generating a summary sequence Y = {y1, y2, ..., y...}. m}. In each step, word y is generated. t At time step t, the decoder calculates its hidden state s. t Then, attention is used to focus on the encoder's output:

[0199] s t =LSTM(s t-1 ,y t-1 ,c t )

[0200] Among them, c t It is a context vector, generated by a multi-head attention mechanism from the word embedding h produced by the encoder. t Weighted average yields:

[0201]

[0202] Among them, h i and h j This represents the hidden state vector at different positions in the input sequence, used to capture the feature representation of the input data.

[0203] Step 4.3.3: The Pointer Generation Network (PGN) optimizes the generation logic by introducing PGN to achieve a dual-mode collaboration of generation and copying, solving the problems of handling out-of-vocabulary words and repeated generation. Its core idea is to use a probability P... gen By controlling the generation (selecting words from the vocabulary) and copying (copying out-of-vocabulary words from the input text), the model can effectively paraphrase important words that are not in the vocabulary, while avoiding the generation of duplicate content.

[0204]

[0205] w h w s w y Let b be the weight matrix, b be the bias, and σ be the sigmoid function.

[0206] The final output word probability is a weighted sum of the generated word probability and the duplicate word probability:

[0207]

[0208] Among them, P vocab (w) is the generation probability of word w in the vocabulary, a t,i It is the attention weight of word w in the input sequence.

[0209] Step 4.3.4: Use SIMCLS contrastive learning to score the quality of the generated candidate summaries in order to select the highest quality summary.

[0210] The original text and candidate summaries are input into the pre-trained BERT model to obtain the embedding vector E of the original text. source Embedding vector E of candidate summary candidate Then, by calculating the similarity between the two embeddings, SIMCLS scores the candidate summaries. Cosine similarity can be used as the scoring function.

[0211]

[0212] The candidate summary with the highest score is selected as the final output. Assume the extracted key requirement is "power outage due to unpaid bills, requesting restoration of power." Through matching and similarity calculation, and model training, a policy basis summary is generated as: "According to electricity policy, power outages due to unpaid bills will only be restored after the outstanding fees are settled."

[0213] Example 2

[0214] This embodiment is a key element extraction system for electricity marketing business based on natural language processing technology, such as... Figure 2As shown, the method for extracting key elements of power marketing business as described in Example 1 consists of a marketing business work order classification unit, an entity recognition and labeling unit, a key element association extraction unit, and a summary quality improvement unit.

[0215] Marketing Business Work Order Classification Unit: Define category labels for electricity marketing business, label the business work order text with category labels, and construct a labeled work order dataset; use the TF-IDF method to extract the feature vector of the work order text, input it into the support vector machine for training, and realize the automatic recognition of work order business type; this unit is used to implement the function of step 1 in embodiment 1, and will not be described again here.

[0216] Entity recognition and annotation unit: Collects and preprocesses power marketing business work order text, uses Word2Vec to convert each word in the work order text into a fixed-dimensional vector, uses the BiLSTM model to obtain the context information of the words, and uses conditional random fields to annotate the word sequence output by the BiLSTM model, automatically recognizing and annotating entities of different categories; this unit is used to implement the function of step 2 in embodiment 1, and will not be described in detail here.

[0217] Business Key Element Relationship Extraction Unit: Utilizing the context embedding representation of the RoBERTa model and combining it with a preset business hierarchy classification structure, a multi-level element classification model is constructed to achieve fine-grained extraction of key elements in the power marketing business. Based on the attention mechanism, a deep neural network entity relationship extraction model is constructed to extract the relationships between key elements. This unit is used to implement the function of step 3 in Example 1, and will not be described in detail here.

[0218] Abstract quality improvement unit: Calculates the similarity between key elements of electricity marketing business and key elements of policy documents using a similarity algorithm, extracts keywords from key elements of policy documents using the TextRank algorithm, generates policy basis abstracts based on an improved Seq2Seq model, and improves abstract quality by combining optimization mechanisms; this module is used to implement the function of step 4 in Example 1, and will not be described in detail here.

[0219] The above description is merely an embodiment of the present invention and does not limit the scope of protection of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of protection of the present invention.

Claims

1. A method for extracting key elements of electricity marketing operations based on natural language processing technology, characterized in that, include: Step 1: Define category labels for electricity marketing business, label the business work order text with category labels, and build a labeled work order dataset; use the TF-IDF method to extract the feature vector of the work order text, input it into the support vector machine for training, and realize the automatic recognition of work order business type; Step 2: Collect and preprocess the work order text of electricity marketing business. Use Word2Vec to convert each word in the work order text into a fixed-dimensional vector. Use the BiLSTM model to obtain the context information of the words. Use conditional random field to label the word sequence output by the BiLSTM model to automatically identify and label different categories of entities. Step 3: Utilize the context embedding representation of the RoBERTa model and combine it with the preset business hierarchy classification structure to construct a multi-level element classification model, thereby achieving fine-grained extraction of key elements in the power marketing business. Based on the attention mechanism, construct a deep neural network entity relationship extraction model to extract the association relationships of key elements. Step 4: Calculate the similarity between key elements of electricity marketing business and key elements of policy documents using a similarity algorithm, extract keywords from key elements of policy documents using the TextRank algorithm, generate policy basis summaries based on an improved Seq2Seq model, and improve the quality of the summaries by combining optimization mechanisms.

2. The method for extracting key elements of electricity marketing operations according to claim 1, characterized in that, The specific process of step 1 includes: Step 1.1: Collect historical work order text data, define category labels for electricity marketing business, manually label each work order text with category labels to form a labeled work order dataset for training. Step 1.2: Preprocess the labeled work order texts by using the TF-IDF method to calculate the word frequency features of the work order texts, and obtain the feature vector of each work order in the vocabulary space. Step 1.3: Using the feature vector as input and the work order category label as output, train a support vector machine to build the model; after training, classify and predict new work orders in the power marketing business to achieve automatic identification of work order business types.

3. The method for extracting key elements of electricity marketing operations according to claim 2, characterized in that, The specific process of step 1.2 includes: Step 1.2.1: Preprocess each marked work order text, including word segmentation, removal of stop words, and standardization of capitalization; Step 1.2.2: Use the TF-IDF method to calculate the word frequency features of the text, quantify the importance of each term, and obtain the feature vector of each work order text in the lexical space.

4. The method for extracting key elements of electricity marketing operations according to claim 2, characterized in that, The specific process of step 1.3 includes: Step 1.3.1: Using the aforementioned feature vector as input and the work order category label as output, train a support vector machine, determine the optimal classification hyperplane by maximizing the class margin, and form a text classifier; Step 1.3.2: Apply the trained text classifier to predict new work orders for the power marketing business. Input the feature vector extracted from the new work order text into the text classifier, and determine the business category to which the work order belongs through support vectors and decision boundaries to achieve automatic classification.

5. The method for extracting key elements of electricity marketing operations according to claim 1, characterized in that, Step 2 includes the following specific steps: Step 2.1: Obtain the original power marketing business work order text and preprocess it to remove irrelevant information, filter out irrelevant noise, special symbols, numbers, stop words and specific format information, use the jieba Chinese word segmentation tool to segment the work order text, extract its part-of-speech features, and form a word sequence with grammatical information. Step 2.2: Define the category labels for each entity, manually or semi-automatically label the entity categories in the business work order, and convert the labeled data into a format suitable for training. Step 2.3: Use Word2Vec to map each word in the work order text to a fixed-dimensional word vector; based on the word vectors, combine the aforementioned entity category labels to annotate and align each word; Step 2.4: Use the BiLSTM model to extract the contextual semantic features of words, and input the output of the BiLSTM model into the conditional random field layer to perform entity recognition and annotation on the word sequence.

6. The method for extracting key elements of electricity marketing operations according to claim 5, characterized in that, The specific process of step 2.4 includes: Step 2.4.1: Input the word sequence after word vector encoding into the BiLSTM model to obtain the semantic representation of each word in the context. Introduce an attention mechanism to weight the semantic representation, highlight the feature representation of the entity in the context, and obtain the feature sequence. Step 2.4.2: Input the BiLSTM model and the feature sequence processed by the attention mechanism into the conditional random field model. By modeling the dependency relationship and transition constraint between labels, the recognition of various entities in the work order text is realized.

7. The method for extracting key elements of electricity marketing operations according to claim 1, characterized in that, Step 3 includes the following specific steps: Step 3.1: Design a tag system with business semantic hierarchy and construct a corresponding business hierarchical classification structure; Step 3.2: Use the RoBERTa model to segment and embed the work order text into words, extract the key semantic features of the text, and output the context embedding vector. Step 3.3: Based on the context embedding vector output by the RoBERTa model, construct a multi-level feature classification model with a hierarchical structure. This model consists of multiple sub-classifiers, each of which is responsible for identifying key features at the corresponding level, thereby achieving fine-grained extraction of key features step by step. Step 3.4: Construct a deep neural network entity relationship extraction model based on the multi-head attention mechanism to extract the relationships between key elements.

8. The method for extracting key elements of electricity marketing operations according to claim 7, characterized in that, The specific process of step 3.2 includes: Step 3.2.1: Use the word segmenter built into the RoBERTa model to process the work order text, serialize the sentence or text into a sequence of words, and perform word segmentation to obtain the input sequence; Step 3.2.2: Add special markers to the input sequence to identify sentence boundaries, which are used to mark the beginning and end of sentences, forming a standard RoBERTa input format and obtaining a standard input sequence; Step 3.2.3: Feed the standard input sequence into the RoBERTa model and output the context embedding vector for each word; The specific process of step 3.3 includes: Step 3.3.1: Define the hierarchical structure and label system of business elements, and establish the corresponding sub-classifier structure; use the context embedding vector output by the RoBERTa model as input to train the sub-classifiers at each level, and complete the construction of the multi-level element classification model; Step 3.3.2: Call the first-level sub-classifier to perform preliminary classification of the context embedding vector and identify the categories of key elements; Step 3.3.3: Based on the primary classification results, select the corresponding secondary sub-classifier to further complete the sub-category identification and achieve fine-grained extraction of key elements; The specific process of step 3.4 includes: Step 3.4.1: Input the context embedding vector of the identified entities into the multi-head self-attention mechanism, calculate the mutual dependency weight between each entity and other entities in the work order text, and obtain a set of context-aware entity feature vectors. Step 3.4.2: For the target entity pair of the relationship to be predicted, the corresponding two context-aware entity feature vectors are fused to obtain the entity pair fused feature vector for relationship recognition. Step 3.4.3: Input the entity pair fusion feature vector into a pre-trained deep neural network relationship classifier, which outputs the probability distribution of the entity pair belonging to each predefined relationship category. Finally, select the category with the highest probability value as the final association relationship to realize the extraction of the association relationship between key elements.

9. The method for extracting key elements of electricity marketing operations according to claim 1, characterized in that, Step 4 includes the following specific steps: Step 4.1: Match the key elements in the policy documents with the key elements of the electricity marketing business; use cosine similarity to calculate the similarity between the key elements of the marketing business and the key elements of the policy documents, and extract the key elements of the policy documents with high similarity. Step 4.2: Use the TextRank algorithm to calculate the weight of each key element extracted from the policy document. Iterate through the converged scores to obtain words with high weights, and use these words as the summary content. Step 4.3: Generate policy basis summaries using a Seq2Seq model improved based on a multi-head attention mechanism, and combine this with an optimization mechanism to improve the quality of the summaries.

10. A system for extracting key elements of electricity marketing business based on natural language processing technology, used to implement the method for extracting key elements of electricity marketing business as described in any one of claims 1-9, characterized in that, include: Marketing Business Work Order Classification Unit: Define category labels for electricity marketing business, annotate the business work order text with category labels, and construct a labeled work order dataset; The TF-IDF method is used to extract the feature vector of the work order text, which is then input into the support vector machine for training to achieve automatic recognition of work order business types. Entity recognition and annotation unit: Collects and preprocesses power marketing business work order text, uses Word2Vec to convert each word in the work order text into a fixed-dimensional vector, uses the BiLSTM model to obtain the context information of the words, and uses conditional random fields to annotate the word sequence output by the BiLSTM model, automatically recognizing and annotating entities of different categories. Key Business Element Relationship Extraction Unit: Utilizing the context embedding representation of the RoBERTa model and combining it with a pre-defined business hierarchy classification structure, a multi-level element classification model is constructed to achieve fine-grained extraction of key elements in the power marketing business. Based on the attention mechanism, a deep neural network entity relationship extraction model is constructed to extract the relationships between key elements. Abstract Quality Improvement Unit: This unit uses a similarity algorithm to calculate the similarity between key elements of electricity marketing business and key elements of policy documents, uses the TextRank algorithm to extract keywords from the key elements of policy documents, generates policy basis abstracts based on an improved Seq2Seq model, and combines optimization mechanisms to improve abstract quality.

Images