A semi-supervised pre-training method for power system dialogue generation

Abstract

The application belongs to the technical field of power grid, and relates to a kind of semi-supervised pre-training power system dialogue generation method, by collecting and standardizing power system dialogue data, and constructing the knowledge graph of related field, unsupervised pre-training is carried out to unlabeled data using BART model, text noise is added to enhance the robustness of the model;Using knowledge graph embedding technology, the knowledge graph information is integrated into the coding stage of BART model, so that the model can refer to domain knowledge when generating dialogue;Part of the data is manually labeled and supervised fine-tuning is carried out using cross-entropy loss function;Further expand the training data by generating pseudo-labels, in practical application, the model combines user input and knowledge graph to dynamically generate more logical replies. Finally, the dialogue quality is evaluated using the BLEU index, and the model strategy is optimized. This method effectively solves the problems of high dependence on labeled data, inaccurate and incoherent generated content in power system dialogue generation.

Description

Technical Field

[0001] This application belongs to the field of power grid technology, and more specifically, relates to a semi-supervised pre-trained method for generating dialogues in power systems. Background Technology

[0002] With the widespread adoption of smart devices and sensors in power systems, power companies have accumulated a vast amount of historical dialogue data. This data covers various scenarios, including user inquiries, fault reports, and equipment maintenance. However, most of this data is unlabeled, limiting the methods that can be directly used for supervised learning.

[0003] Traditional dialogue generation models such as Seq2Seq (Sequence-to-Sequence) and Transformer perform well in handling open-domain dialogues, but face the following challenges in specific domains such as power systems:

[0004] Insufficient utilization of unlabeled data: Most dialogue generation models rely on large amounts of labeled data for training, but in the power system field, acquiring labeled data is costly and limited in quantity. How to utilize large amounts of unlabeled data to improve model performance has become a key issue.

[0005] The domain terminology is complex: Power system dialogues contain a large number of technical terms and details, and the terminology may differ across different data sources, directly affecting the training performance of the model. Therefore, it is necessary to standardize these terms so that the model can better understand and generate domain-specific dialogues.

[0006] Knowledge graphs present significant challenges in application: knowledge graphs in the power system field contain rich information on equipment, operating parameters, and fault records. Effectively embedding this structured knowledge into dialogue generation models to enhance their generative capabilities is an important research direction.

[0007] Therefore, how to effectively utilize unlabeled data and domain knowledge graphs in power system dialogue generation, reduce dependence on labeled data, and improve the accuracy and coherence of dialogue generation are technical problems that need to be solved. Summary of the Invention

[0008] This invention provides a semi-supervised pre-training method for generating dialogues in power systems. The aim is to address the technical problem of how to effectively utilize unlabeled data and domain knowledge graphs in power system dialogue generation, reduce dependence on labeled data, and improve the accuracy and coherence of dialogue generation.

[0009] A semi-supervised pre-trained method for generating dialogue in a power system includes the following steps:

[0010] Step 1: Collect dialogue data from historical data sources of the power system and construct a knowledge graph in the power system domain; clean the collected dialogue data, remove irrelevant information, and standardize the terminology in the power system domain to obtain standardized power system dialogue data;

[0011] Step 2: The BART model is used for unsupervised pre-training on unlabeled and standardized power system dialogue data, and some words in the text are randomly masked. During pre-training, text noise is introduced, and the BART model is required to reconstruct the original text based on the noisy text to obtain the text dialogue sequence.

[0012] Step 3: Use knowledge graph embedding technology to convert the nodes and edges in the knowledge graph into vector representations. In the encoding stage of the BART model, the knowledge graph embedding is used as an additional input, along with the text embedding, and input into the encoder.

[0013] Step 4: Randomly select a portion of the power system dialogue data for manual annotation. The annotation content includes dialogue intent, slot information, and knowledge graph association. Use the annotated data to supervise and fine-tune the pre-trained BART model. Optimize the model using the cross-entropy loss function. Use the fine-tuned BART model to generate pseudo-labels for the unlabeled data. Use the generated pseudo-labels as a supplement to the training data to train the model again.

[0014] Step 5: When a user inputs a question related to the power system, the BART model encodes the input text into a contextual embedding, and combines it with relevant node embeddings from the knowledge graph as input for processing. The encoder selectively focuses on nodes related to the question using knowledge graph information. During the decoding process, the BART model dynamically selects whether to use specific information from the knowledge graph based on the generated partial content, and uses the reasoning ability of the knowledge graph to generate a more logical response.

[0015] Step 6: Use the BLEU metric to evaluate the quality of the generated dialogue, check the accuracy and coherence of the content generated by the BART model, and adjust the model strategy and optimize the knowledge graph fusion strategy based on the evaluation results.

[0016] This invention employs a semi-supervised pre-training method that effectively utilizes unlabeled data and domain knowledge graphs, significantly reducing reliance on labeled data while improving the accuracy and coherence of dialogue generation. First, power system dialogue data is collected and standardized, and a domain-specific knowledge graph is constructed. Then, the BART model is used for unsupervised pre-training on the unlabeled data, with textual noise added to enhance the model's robustness. Next, knowledge graph embedding technology is used to integrate knowledge graph information into the encoding stage of the BART model, enabling the model to reference domain knowledge when generating dialogues. The model's performance is improved by manually labeling a portion of the data and using a cross-entropy loss function for supervised fine-tuning. Subsequently, pseudo-labels are generated to further expand the training data, enhancing the model's generalization ability. In practical applications, the model combines user input and the knowledge graph to dynamically generate more logically coherent responses. Finally, the BLEU metric is used to evaluate dialogue quality and optimize the model strategy. This method effectively solves the problems of high reliance on labeled data, inaccurate and incoherent generated content in power system dialogue generation.

[0017] Preferably, step 2 includes the following steps:

[0018] Data format conversion: Convert the dialogue data into an input format suitable for training the BART model. Each dialogue data consists of question-answer pairs, where the input sequence is the historical context or question part of the dialogue, and the target sequence is the corresponding answer part.

[0019] Masked language model: Randomly select 15%-20% of the words in each input text to mask them, and replace the masked words with symbols.

[0020]

[0021] In the formula: X represents the input sequence; p mask Indicates the probability of masking words; This represents the input sequence after masking.

[0022] Text noise introduction: The sentence order in the input text is randomly shuffled, and words or sentences in the input text are randomly deleted.

[0023] X noisy =Shuffle(X)+Drop(X, p) drop );

[0024] In the formula: X noisy This represents the input sequence after noise has been introduced; Shuffle(X) represents randomly shuffling sentences or words in the input sequence; Drop(X, p) represents... drop ) indicates that according to probability p dropRemove certain words or sentences from the input sequence, with a probability set to 10%-20%;

[0025] Reconstruction task: The training objective of the BART model is to train based on the masked input sequence. and the input sequence X with added noise noisy This allows the BART model to reconstruct the original input sequence X:

[0026]

[0027] In the formula: P(x|X) noisy () represents the probability that the BART model can recover the original text from noisy text; T represents the text length; x t Let x1, x2, ..., xt be the word of the t-th word. t-1 This refers to words that have already been generated.

[0028] Loss function: During pre-training, the goal of the BART model is to minimize the loss function of the reconstruction task. The cross-entropy loss function is used to calculate the difference between each word in the BART model and the target word.

[0029]

[0030] In the formula: P(x) represents the total loss value. t |x1, x2, ..., x t-1 ) represents the probability of the word generated by the BART model at time step t;

[0031] Optimizer: During pre-training, the Adam optimizer is used to train the BART model, adjusting the model's parameters to minimize the loss function.

[0032]

[0033] In the formula: θ t Represents the parameters of the current BART model; θ t+1 The parameters represent the updated BART model parameters; α represents the learning rate; m t The first momentum represents the gradient; v t represents the second momentum of the gradient; ∈ represents a small quantity to avoid division by zero;

[0034] Convergence criterion: The pre-training process ends when the BART model's performance on the validation set stabilizes and the error value is within the set range.

[0035] Preferably, step 3 includes the following steps:

[0036] Knowledge Graph Embedding: TransE knowledge graph embedding technology is used to convert nodes and edges in the knowledge graph into low-dimensional vector representations. By mapping entities and relations to the same vector space and minimizing the loss function, the embedding vectors are trained to ensure that the relations between nodes remain consistent.

[0037] h+r≈t;

[0038] In the formula: h represents the embedding vector of the head entity; r represents the embedding vector of the relation; t represents the embedding vector of the tail entity;

[0039] Loss function:

[0040] L=∑ (h,r,t)∈τ [γ+d(h+r,t)-d(h′,r′,t′)] + ;

[0041] In the formula: τ represents the set of positive sample triples; (h, r, t) represents a positive triple, representing the head entity, relation, and tail entity; d(h+r, t) represents the distance between the head entity, relation, and tail entity, which is the Euclidean distance; (h′, r′, t′) represents the negative sample; γ represents the hyperparameter, which controls the distance between positive and negative samples;

[0042] Text embedding generation: The input dialogue sample is converted into an embedding vector using pre-trained word vectors; where the input text dialogue sequence X′=(x1, x2, ..., x... T ), where each x i Representing a word in the text, the text embedding is generated as follows:

[0043] E text = [e(x1), e(x2), ..., e(x)] T )];

[0044] In the formula: E text The embedding matrix represents the text; e(x) i ) represents the word embedding obtained through pre-trained word vectors;

[0045] Combining Knowledge Graph Embedding and Text Embedding: This approach combines node embedding from the knowledge graph with text embedding. In the encoder's input, for each entity node relevant to the current dialogue context, the corresponding embedding vector is obtained. This embedding vector is then concatenated with the text embedding.

[0046] E joint =[E text E KG ];

[0047] In the formula: E KG = [h1, h2, ..., h n] indicates the embedding of knowledge graph entity nodes related to the current dialogue; E joint This represents a joint representation of text embedding and knowledge graph embedding, combined through vector concatenation operations;

[0048] Dynamically select relevant nodes: In each dialogue, the relationship structure of the knowledge graph is used to select relevant nodes based on the current dialogue content;

[0049] Encoder Input Processing: During the encoding phase of the BART model, the encoder jointly embeds the representation E. joint The input is fed into the encoder, which processes the input sequence, captures the relationships between words in the text through a self-attention mechanism, and enhances the understanding of the power system domain by utilizing embedded information from the knowledge graph.

[0050] H enc =BART_Encoder(E joint );

[0051] Where: H enc This indicates that the encoder generates the hidden state representation; BART_Encoder indicates the encoder module of the BART model.

[0052] Preferably, step 3 further includes the BART model learning the relevance weights of each knowledge graph node to the current dialogue, and calculating the attention weights between each node embedding and the input text embedding through a self-attention mechanism, assigning higher weights to more relevant nodes:

[0053]

[0054] In the formula: α i Let represent the attention weight of the i-th knowledge graph node; q is the hidden state of the input text; k i Let represent the embedding vector of the i-th knowledge graph node; n represents the number of nodes in the knowledge graph.

[0055] Encoder State Update: Based on the attention weights, the encoder performs a weighted update of the hidden state at each time step.

[0056]

[0057] Where: H updated Represents the hidden state after weighted updating by the knowledge graph; h i Let represent the hidden state at time i.

[0058] Preferably, step 4 includes the following steps:

[0059] Manually annotated power system dialogue data: Randomly select a portion of power system dialogue data for manual annotation, including dialogue intent, slot information, and knowledge graph associations;

[0060] Supervised fine-tuning: The pre-trained BART model from step 2 is fine-tuned using labeled data. During fine-tuning, the cross-entropy loss function is used to optimize the model. The cross-entropy loss function measures the difference between the dialogue sequence predicted by the model and the labeled dialogue sequence.

[0061]

[0062] In the formula: L CE Represents the cross-entropy loss value; y i This represents the actual category or vocabulary of the labeled i-th dialogue; This represents the probability of the category or word in the i-th dialogue predicted by the model; m represents the total number of words in the dialogue.

[0063] Pseudo-label generation: The fine-tuned BART model is used to predict and generate pseudo-labels for unlabeled dialogue data; the generated pseudo-labels and the real labeled data are used to train the BART model a second time.

[0064] Preferably, step 5 includes the following steps:

[0065] Input processing: When a user inputs a question related to the power system, the input text is first encoded using the encoder of a pre-trained BART model to generate a contextual embedding representation;

[0066] Knowledge graph embedding acquisition: Using knowledge graph embedding technology, nodes and edges in the knowledge graph are converted into vector representations;

[0067] Combining contextual and knowledge graph embedding: By matching keywords or terms with the user's input text, relevant node embeddings are selected from the knowledge graph; the contextual embeddings are fused with the selected relevant node embeddings to form a comprehensive embedding, wherein the fusion strategy adopts an attention mechanism for fusion;

[0068] Decoding process: During the response generation process, the BART model's decoder dynamically selects whether information from the knowledge graph is needed based on the generated partial content;

[0069] Generate logical responses: The decoder selectively focuses on the embedding of nodes that are relevant to the question through an attention mechanism, generating more logical responses.

[0070] Preferably, step 6 includes the following steps:

[0071] Model-generated dialogue: Use the trained BART model to generate responses for each user input in the test dataset;

[0072] Calculating the BLEU score: This measure the quality of generated text by comparing its similarity to that of reference text.

[0073]

[0074] In the formula: N represents the longest length of the n-gram under consideration, with a value of 4; w n The weight of the nth n-gram is denoted as . p n BP represents the precision of the nth n-gram; BP represents the length penalty factor.

[0075] in:

[0076]

[0077] In the formula: c represents the length of the generated text, r represents the length of the reference text; e represents the base of the natural logarithm; Count clip (n-gram) represents the count of the occurrences of an n-gram in the generated text after truncating the count of the same n-gram in the reference text; Count ref (n-gram) represents the count of n-grams in the reference text; C represents the set of candidate texts;

[0078] Analyzing BLEU scores: After calculating the BLEU score for each generated dialogue, the average BLEU score of the entire test data is calculated and used as the overall evaluation index of dialogue quality. This includes the following steps:

[0079] BLEU score for a single conversation: BLEU score is calculated for each user input and generated response;

[0080] Average BLEU Score: Calculate the average BLEU score for all conversations.

[0081]

[0082] In the formula: M represents the size of the test dataset; BLEU i This represents the BLEU score of the i-th dialogue;

[0083] Strategy adjustment and optimization: Based on the BLEU score results, analyze the accuracy and consistency of the model's generated responses, determine the areas that need optimization, and optimize the corresponding strategies based on these areas.

[0084] The beneficial effects of this invention include:

[0085] This invention employs a semi-supervised pre-training method that effectively utilizes unlabeled data and domain knowledge graphs, significantly reducing reliance on labeled data while improving the accuracy and coherence of dialogue generation. First, power system dialogue data is collected and standardized, and a domain-specific knowledge graph is constructed. Then, the BART model is used for unsupervised pre-training on the unlabeled data, with textual noise added to enhance the model's robustness. Next, knowledge graph embedding technology is used to integrate knowledge graph information into the encoding stage of the BART model, enabling the model to reference domain knowledge when generating dialogues. The model's performance is improved by manually labeling a portion of the data and using a cross-entropy loss function for supervised fine-tuning. Subsequently, pseudo-labels are generated to further expand the training data, enhancing the model's generalization ability. In practical applications, the model combines user input and the knowledge graph to dynamically generate more logically coherent responses. Finally, the BLEU metric is used to evaluate dialogue quality and optimize the model strategy. This method effectively solves the problems of high reliance on labeled data, inaccurate and incoherent generated content in power system dialogue generation. Attached Figure Description

[0086] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0087] Figure 1 This is an overall step diagram provided for an embodiment of the present invention. Detailed Implementation

[0088] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0089] See Figure 1 As shown, the preferred embodiment of the present invention will be further described;

[0090] A semi-supervised pre-trained method for generating dialogue in a power system includes the following steps:

[0091] Step 1: Collect dialogue data from historical data sources of the power system and construct a knowledge graph in the power system domain; clean the collected dialogue data, remove irrelevant information, and standardize the terminology in the power system domain to obtain standardized power system dialogue data;

[0092] Step 2: The BART model is used for unsupervised pre-training on unlabeled and standardized power system dialogue data, and some words in the text are randomly masked. During pre-training, text noise is introduced, and the BART model is required to reconstruct the original text based on the noisy text to obtain the text dialogue sequence.

[0093] Step 3: Use knowledge graph embedding technology to convert the nodes and edges in the knowledge graph into vector representations. In the encoding stage of the BART model, the knowledge graph embedding is used as an additional input, along with the text embedding, and input into the encoder.

[0094] Step 4: Randomly select a portion of the power system dialogue data for manual annotation. The annotation content includes dialogue intent, slot information, and knowledge graph association. Use the annotated data to supervise and fine-tune the pre-trained BART model. Optimize the model using the cross-entropy loss function. Use the fine-tuned BART model to generate pseudo-labels for the unlabeled data. Use the generated pseudo-labels as a supplement to the training data to train the model again.

[0095] Step 5: When a user inputs a question related to the power system, the BART model encodes the input text into a contextual embedding, and combines it with relevant node embeddings from the knowledge graph as input for processing. The encoder selectively focuses on nodes related to the question using knowledge graph information. During the decoding process, the BART model dynamically selects whether to use specific information from the knowledge graph based on the generated partial content, and uses the reasoning ability of the knowledge graph to generate a more logical response.

[0096] Step 6: Use the BLEU metric to evaluate the quality of the generated dialogue, check the accuracy and coherence of the content generated by the BART model, and adjust the model strategy and optimize the knowledge graph fusion strategy based on the evaluation results.

[0097] This invention employs a semi-supervised pre-training method that effectively utilizes unlabeled data and domain knowledge graphs, significantly reducing reliance on labeled data while improving the accuracy and coherence of dialogue generation. First, power system dialogue data is collected and standardized, and a domain-specific knowledge graph is constructed. Then, the BART model is used for unsupervised pre-training on the unlabeled data, with textual noise added to enhance the model's robustness. Next, knowledge graph embedding technology is used to integrate knowledge graph information into the encoding stage of the BART model, enabling the model to reference domain knowledge when generating dialogues. The model's performance is improved by manually labeling a portion of the data and using a cross-entropy loss function for supervised fine-tuning. Subsequently, pseudo-labels are generated to further expand the training data, enhancing the model's generalization ability. In practical applications, the model combines user input and the knowledge graph to dynamically generate more logically coherent responses. Finally, the BLEU metric is used to evaluate dialogue quality and optimize the model strategy. This method effectively solves the problems of high reliance on labeled data, inaccurate and incoherent generated content in power system dialogue generation.

[0098] As one possible implementation of this embodiment, step 2 includes the following steps:

[0099] Data format conversion: Convert the dialogue data into an input format suitable for training the BART model. Each dialogue data consists of question-answer pairs, where the input sequence is the historical context or question part of the dialogue, and the target sequence is the corresponding answer part.

[0100] Masked language model: Randomly select 15%-20% of the words in each input text to mask them, and replace the masked words with symbols.

[0101]

[0102] In the formula: X represents the input sequence; omask represents the probability of masking words; This represents the input sequence after masking.

[0103] Text noise introduction: The sentence order in the input text is randomly shuffled, and words or sentences in the input text are randomly deleted.

[0104] X noisy =Shuffle(X)+Drop(X, o drop );

[0105] In the formula: X noisy This represents the input sequence after noise has been introduced; Shuffle(X) represents randomly shuffling sentences or words in the input sequence; Drop(X, p) represents... drop ) indicates that according to probability p dropRemove certain words or sentences from the input sequence, with a probability set to 10%-20%;

[0106] Reconstruction task: The training objective of the BART model is to train based on the masked input sequence. and the input sequence X with added noise noisy This allows the BART model to reconstruct the original input sequence X:

[0107]

[0108] In the formula: P(x|X) noisy () represents the probability that the BART model can recover the original text from noisy text; T represents the text length; x t Let x1, x2, ..., xt be the word of the t-th word. t-1 This refers to words that have already been generated.

[0109] Loss function: During pre-training, the goal of the BART model is to minimize the loss function of the reconstruction task. The cross-entropy loss function is used to calculate the difference between each word in the BART model and the target word.

[0110]

[0111] In the formula: P(x) represents the total loss value. t |x1, x2, ..., x t-1 ) represents the probability of the word generated by the BART model at time step t;

[0112] Optimizer: During pre-training, the Adam optimizer is used to train the BART model, adjusting the model's parameters to minimize the loss function.

[0113]

[0114] In the formula: θ t Represents the parameters of the current BART model; θ t+1 The parameters represent the updated BART model parameters; α represents the learning rate; m t The first momentum representing the gradient; v t The second momentum of the gradient is represented; ∈ indicates a small quantity to avoid division by zero.

[0115] Convergence criterion: The pre-training process ends when the BART model's performance on the validation set stabilizes and the error value is within the set range.

[0116] As one possible implementation of this embodiment, step 3 includes the following steps:

[0117] Knowledge Graph Embedding: TransE knowledge graph embedding technology is used to convert nodes and edges in the knowledge graph into low-dimensional vector representations. By mapping entities and relations to the same vector space and minimizing the loss function, the embedding vectors are trained to ensure that the relations between nodes remain consistent.

[0118] h+r≈t;

[0119] In the formula: h represents the embedding vector of the head entity; r represents the embedding vector of the relation; t represents the embedding vector of the tail entity;

[0120] Loss function:

[0121] L=∑(h, r, t)∈τ[γ+d(h+r, t)-d(h′, r′, t′)]+;

[0122] In the formula: τ represents the set of positive sample triples; (h, r, t) represents a positive triple, representing the head entity, relation, and tail entity; d(h+r, t) represents the distance between the head entity, relation, and tail entity, which is the Euclidean distance; (h′, r′, t′) represents the negative sample; γ represents the hyperparameter, which controls the distance between positive and negative samples;

[0123] Text embedding generation: The input dialogue sample is converted into an embedding vector using pre-trained word vectors; where the input text dialogue sequence X′=(x1, x2, ..., x... T ), where each x i Representing a word in the text, the text embedding is generated as follows:

[0124] E text = [e(x1), e(x2), ..., e(x)] T )];

[0125] In the formula: E text The embedding matrix represents the text; e(x) i ) represents the word embedding obtained through pre-trained word vectors;

[0126] Combining Knowledge Graph Embedding and Text Embedding: This approach combines node embedding from the knowledge graph with text embedding. In the encoder's input, for each entity node relevant to the current dialogue context, the corresponding embedding vector is obtained. This embedding vector is then concatenated with the text embedding.

[0127] E joint =[E text E KG ];

[0128] In the formula: E KG = [h1, h2, ..., h n ] indicates the embedding of knowledge graph entity nodes related to the current dialogue; Ejoint Represents the joint representation of text embedding and knowledge graph embedding, combined through vector concatenation operation;

[0129] Dynamically select relevant nodes: In each conversation, utilize the relationship structure of the knowledge graph to select relevant nodes according to the current conversation content;

[0130] Encoder input processing: In the encoding stage of the BART model, the encoder jointly embeds the representation E joint is input into the encoder, and the encoder processes the input sequence, captures the relationships between text words through the self-attention mechanism, and at the same time utilizes the embedding information of the knowledge graph to strengthen the understanding of the power system field:

[0131] H enc = BART_Encoder(E joint );

[0132] In the formula: H enc represents the hidden state representation generated by the encoder; BART_Encoder represents the encoder module of the BART model.

[0133] As a possible implementation of this embodiment, step 3 further includes the BART model learning the relevance weights of each knowledge graph node to the current conversation, calculating the attention weights between each node embedding and the input text embedding through the self-attention mechanism, and assigning higher weights to more relevant nodes:

[0134]

[0135] In the formula: α i represents the attention weight of the i-th knowledge graph node; q is the hidden state of the input text; vi represents the embedding vector of the i-th knowledge graph node; n represents the number of nodes in the knowledge graph;

[0136] Encoder state update: According to the attention weights, the encoder performs weighted updates on the hidden state at each moment:

[0137]

[0138] In the formula: H uodated represents the hidden state after weighted update by the knowledge graph; h i represents the hidden state at the i-th moment.

[0139] In this embodiment, by calculating relevance weights, the model can better identify and utilize knowledge graph nodes related to the current dialogue, thereby generating answers that are more context-aware. The self-attention mechanism enables the BART model to automatically extract key information from complex power system knowledge graphs, improving the knowledge and professionalism of dialogue generation. By learning the relevance between knowledge graph nodes and dialogue text, the model can more flexibly apply existing knowledge in different dialogue scenarios, improving its adaptability to unseen data.

[0140] Joint embedding representation combines information from both the dialogue text and the knowledge graph, making the hidden state representation generated by the encoder richer and more multidimensional, which helps to improve the quality of dialogue generation. By introducing relevant information from knowledge graph nodes during the encoding stage, the model can better understand and utilize domain knowledge, enhancing the professionalism and accuracy of dialogue generation. Joint embedding representation can help the model better grasp the context and situational information of the dialogue, thereby generating more coherent and natural dialogue content.

[0141] When generating dialogue responses, the decoder can fully utilize the rich semantic information and domain knowledge provided by the encoder to generate more natural and high-quality dialogue content; improve the coherence and consistency of the dialogue: through the self-attention mechanism, the decoder can better grasp the contextual information, and the generated dialogue responses are more logically coherent and consistent, improving the user experience; enhance the ability to handle complex problems: by utilizing joint embedding representation and the self-attention mechanism, the decoder can more effectively handle complex dialogue scenarios and problems, providing more accurate and professional answers.

[0142] As one possible implementation of this embodiment, step 4 includes the following steps:

[0143] Manually annotated power system dialogue data: Randomly select a portion of power system dialogue data for manual annotation, including dialogue intent, slot information, and knowledge graph associations;

[0144] Supervised fine-tuning: The pre-trained BART model from step 2 is fine-tuned using labeled data. During fine-tuning, the cross-entropy loss function is used to optimize the model. The cross-entropy loss function measures the difference between the dialogue sequence predicted by the model and the labeled dialogue sequence.

[0145]

[0146] In the formula: L CE Represents the cross-entropy loss value; y i This represents the actual category or vocabulary of the labeled i-th dialogue; This represents the probability of the category or word in the i-th dialogue predicted by the model; m represents the total number of words in the dialogue.

[0147] Pseudo-label generation: The fine-tuned BART model is used to predict and generate pseudo-labels for unlabeled dialogue data; the generated pseudo-labels and the real labeled data are used to train the BART model a second time.

[0148] In this embodiment, manually annotating power system dialogue data and using this data for supervised fine-tuning significantly improves the generalization ability of the pre-trained BART model in the power system dialogue generation task, enabling it to better understand and generate dialogues that meet actual needs. The cross-entropy loss function is used to optimize the model during supervised fine-tuning, effectively reducing the difference between the model's predicted dialogue sequences and the actual annotated dialogue sequences, thereby improving the accuracy of the model's generated responses. The pseudo-label generation step uses the fine-tuned BART model to predict and generate pseudo-labels from unlabeled data, and combines this with real annotated data for secondary training, maximizing the use of large-scale unlabeled data and further improving model performance. The semi-supervised learning method reduces the need for large-scale manually annotated data, lowering the cost and time of data annotation.

[0149] As one possible implementation of this embodiment, step 5 includes the following steps:

[0150] Input processing: When a user inputs a question related to the power system, the input text is first encoded using the encoder of a pre-trained BART model to generate a contextual embedding representation;

[0151] Knowledge graph embedding acquisition: Using knowledge graph embedding technology, nodes and edges in the knowledge graph are converted into vector representations;

[0152] Combining contextual and knowledge graph embedding: By matching keywords or terms with the user's input text, relevant node embeddings are selected from the knowledge graph; the contextual embeddings are fused with the selected relevant node embeddings to form a comprehensive embedding, wherein the fusion strategy adopts an attention mechanism for fusion;

[0153] Decoding process: During the response generation process, the BART model's decoder dynamically selects whether information from the knowledge graph is needed based on the generated partial content;

[0154] Generate logical responses: The decoder selectively focuses on the embedding of nodes that are relevant to the question through an attention mechanism, generating more logical responses.

[0155] As one possible implementation of this embodiment, step 6 includes the following steps:

[0156] Model-generated dialogue: Use the trained BART model to generate responses for each user input in the test dataset;

[0157] Calculating the BLEU score: This measure the quality of generated text by comparing its similarity to that of reference text.

[0158]

[0159] In the formula: N represents the longest length of the n-gram under consideration, with a value of 4; w n The weight of the nth n-gram is denoted as . p n BP represents the precision of the nth n-gram; BP represents the length penalty factor.

[0160] in:

[0161]

[0162] In the formula: c represents the length of the generated text, r represents the length of the reference text; e represents the base of the natural logarithm; Count clip (n-gram) represents the count of the occurrences of an n-gram in the generated text after truncating the count of the same n-gram in the reference text; Count ref (n-gram) represents the count of n-grams in the reference text; C represents the set of candidate texts;

[0163] Analyzing BLEU scores: After calculating the BLEU score for each generated dialogue, the average BLEU score of the entire test data is calculated and used as the overall evaluation index of dialogue quality. This includes the following steps:

[0164] BLEU score for a single conversation: BLEU score is calculated for each user input and generated response;

[0165] Average BLEU Score: Calculate the average BLEU score for all conversations.

[0166]

[0167] In the formula: M represents the size of the test dataset; BLEU i This represents the BLEU score of the i-th dialogue;

[0168] Strategy adjustment and optimization: Based on the BLEU score results, analyze the accuracy and consistency of the model's generated responses, determine the direction that needs to be optimized, and optimize the corresponding strategy based on the optimization direction;

[0169] Statistical analysis: Calculate statistical indicators such as the mean, median, and standard deviation of BLEU scores to understand the overall distribution of scores.

[0170] Distribution plot: Draw a distribution plot of the BLEU scores (such as a histogram or box plot) to observe the central tendency, dispersion, and outliers of the scores.

[0171] Filter low-scoring dialogues: Identify dialogue instances with low BLEU scores and analyze user input, generated responses, and reference responses for these instances.

[0172] Categorization: Categorize low-scoring conversations by problem type, such as short replies, grammatical errors, irrelevant content, etc.

[0173] Quality Issue Classification: Based on the analysis of low-scoring dialogues, the main types of quality issues are summarized, such as:

[0174] Inaccurate answer: The generated response deviates significantly from the reference answer.

[0175] Syntax issue: The generated response contains syntactic errors or is not grammatically correct.

[0176] Irrelevant content: The generated response is not related to the question entered by the user.

[0177] The response is too short or too long: The length of the generated response is inappropriate.

[0178] Targeted optimization: For different problem types, determine the corresponding optimization direction, for example:

[0179] Inaccurate answer: Strengthen the integration of knowledge graphs to ensure that the generated response contains accurate knowledge points.

[0180] Syntax issues: Improve the quality of training data for language models and add grammar checking mechanisms.

[0181] Content Irrelevant: Optimize contextual understanding capabilities to ensure that responses closely relate to user input.

[0182] Answer too short or too long: Adjust the generation strategy to control the length of the generated text.

[0183] Optimize knowledge graph embedding: Improve the embedding method to more effectively integrate information from the knowledge graph into the model generation process.

[0184] Add knowledge graph nodes: Expand the coverage of the knowledge graph to ensure that more knowledge points can be used by the model.

[0185] Collect more dialogue data, especially manually labeled data, to ensure data diversity and quality.

[0186] Generate pseudo-labeled data: Use data augmentation techniques to generate more training data with reference answers.

[0187] Hyperparameter tuning: Based on the evaluation results, adjust the model's hyperparameters (such as learning rate, batch size, etc.) and retrain the model.

[0188] Experimental comparison: Conduct multiple sets of comparative experiments and select the parameter settings that perform best.

[0189] Recalculate BLEU scores: After applying the optimization strategy, recalculate the BLEU scores using the same test dataset to evaluate the optimization effect.

[0190] Comparative analysis: The results were compared with the initial evaluation to confirm whether the optimization strategy effectively improved the quality of the generated dialogue.

[0191] Iterative optimization: Based on the latest evaluation results, reanalyze the existing problems, adjust the optimization direction, and carry out the next round of optimization.

[0192] Long-term mechanism: Establish a mechanism for continuous improvement, regularly evaluate model performance, and ensure the continuous improvement of the quality of generated dialogues.

[0193] In this embodiment, by calculating the BLEU score, the quality of the model-generated dialogue can be objectively evaluated, ensuring that the generated dialogue has a high degree of similarity to the reference text in terms of content and structure. The BLEU score can quantify the model's performance, providing clear goals and directions for model improvement and optimization. By analyzing the BLEU score, weaknesses in the model's generated responses, such as issues with accuracy and coherence, can be identified, allowing for targeted optimization and improvement. Adjusting and optimizing the model generation strategy based on the BLEU score can continuously improve the quality of the model-generated dialogue, ensuring its effectiveness and reliability in practical applications.

[0194] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A semi-supervised pre-trained power system dialogue generation method, characterized in that, The method comprises the following steps: Step 1: Collecting dialogue data from the historical data source of the power system, and constructing a knowledge graph of the power system field; After cleaning the collected dialogue data and removing irrelevant information, the terms in the power system field are standardized to obtain standardized power system dialogue data; Step 2: Using the BART model to perform unsupervised pre-training on the unlabeled and standardized power system dialogue data, and randomly masking part of the words in the text; During pre-training, introduce text noise, and require the BART model to reconstruct the original text based on the noisy text to obtain the text dialogue sequence; Step 3: Using knowledge graph embedding technology to convert the nodes and edges in the knowledge graph into vector representations, and inputting the knowledge graph embedding as additional input into the encoder together with the text embedding during the encoding stage of the BART model; Step 4: Randomly extracting part of the power system dialogue data for manual labeling, the labeled content including dialogue intent, slot information and knowledge graph association, using the labeled data to supervise the pre-trained BART model, using the cross-entropy loss function to optimize the model, using the fine-tuned BART model to generate pseudo-labels for unlabeled data, and using the generated pseudo-labels as supplementary training data to train the model again; Step 5: When the user inputs a power system related question, the BART model encodes the input text into context embedding, and combines the related node embedding in the knowledge graph as input for processing, and the encoder selectively focuses on the nodes related to the question through the knowledge graph information; In the decoding process, the BART model dynamically selects whether to use specific information in the knowledge graph according to the generated part of the content, and uses the reasoning ability of the knowledge graph to generate more logical replies; Step 6: Using the BLEU index to evaluate the quality of the generated dialogue, detecting the accuracy and coherence of the BART model generated content, and adjusting the model strategy and optimizing the knowledge graph fusion strategy based on the evaluation results.

2. The semi-supervised pre-trained power system dialogue generation method according to claim 1, wherein, The step 2 comprises the following steps: Data format conversion: converting the dialogue data into an input format suitable for BART model training, each dialogue data consisting of a question-answer pair, wherein the input sequence is the historical context or question part of the dialogue, and the target sequence is the corresponding answer part; Masked language model: randomly selecting 15%-20% of the words in each input text to mask, and replacing the masked words with symbols: ； wherein: X represents the input sequence; represents the probability of a masked word; represents the input sequence after the masking process; Text noise introduction: randomly rearranging the order of sentences in the input text, and randomly deleting words or sentences in the input text: ； In the formula: represents the input sequence after noise introduction; represents randomly shuffling sentences or words of the input sequence; represents deleting part of the words or sentences in the input sequence according to the probability deleting part of the words or sentences in the input sequence, with the probability set to 10%-20%. Reconstruction task: the BART model is trained to reconstruct the original input sequence X based on the masked input sequence and the input sequence with added noise Let the BART model be able to reconstruct the original input sequence X: ； wherein: denotes the probability that the BART model recovers the original text from the noisy text; T denotes the text length; denotes the tthword; denotes the previously generated words; Loss function: During pre-training, the goal of the BART model is to minimize the loss function of the reconstruction task, and the cross-entropy loss function is used to calculate the difference between each word of the BART model and the target word: ； wherein: denotes the total loss value; denotes the probability of the vocabulary generated by the BART model at time step t; Optimizer: During pre-training, the Adam optimizer is used to train the BART model to adjust the parameters of the model to minimize the loss function: ； where: denotes the parameters of the current BART model; denotes the updated parameters of the BART model; denotes the learning rate; denotes the first order momentum of the gradient; denotes the second order momentum of the gradient; denotes a small quantity to avoid division by zero; Convergence criterion: When the BART model performs stably on the validation set and the error value is within the set range, the pre-training process is completed.

3. The semi-supervised pre-trained power system dialogue generation method according to claim 1, wherein, The step 3 comprises the following steps: Knowledge graph embedding: using TransE knowledge graph embedding technology to convert nodes and edges in the knowledge graph into low-dimensional vector representations, by mapping entities and relationships into the same vector space and minimizing the loss function to train embedding vectors, so that the relationship between nodes remains consistent: ； In the formula: denotes the embedding vector of the head entity; r denotes the embedding vector of the relation; t denotes the embedding vector of the tail entity; Loss function: ； In the formula: denotes a set of positive sample triples; denotes a positive triple, denotes a head entity, a relation, and a tail entity; denotes a distance between the head entity plus the relation and the tail entity, which is an Euclidean distance; denotes a negative sample; denotes a hyperparameter, which controls the distance between the positive and negative samples; Text embedding generation: converting an input dialog sample into an embedding vector using pre-trained word vectors; wherein the input text dialog sequence wherein each represents a word in the text, the text embedding generation being as follows: ； In the formula: represents an embedding matrix of the text; represents a word embedding obtained by a pre-trained word vector Combination of knowledge graph embedding and text embedding: combine the node embedding in the knowledge graph with the text embedding, in the input of the encoder, for each entity node related to the current dialogue context, get the corresponding embedding vector, and concatenate the embedding vector with the text embedding: ； In the formula: denotes an embedding of a knowledge graph entity node related to the current dialogue; denotes a joint representation of the text embedding and the knowledge graph embedding, combined by a vector concatenation operation; Dynamic selection of related nodes: in each dialogue, use the relationship structure of the knowledge graph to select nodes related to the current dialogue content; Encoder input processing: In the encoding stage of the BART model, the encoder joint embedding representation is input into the encoder, which processes the input sequence to capture the relationships between the text words through a self-attention mechanism while leveraging the embedding information of the knowledge graph to reinforce the understanding of the power system domain: ； In the formula: denotes that the encoder generates a hidden state representation; denotes an encoder module of the BART model.

4. The semi-supervised pre-trained power system dialogue generation method according to claim 1, characterized in that, Step 3 also includes the BART model learning the relevance weight of each knowledge graph node to the current dialogue, calculating the attention weight between each node embedding and the input text embedding through the self-attention mechanism, and giving higher weight to the more relevant nodes: ； In the formula, represents the attention weight of the i-th knowledge graph node; q is a hidden state of the input text; represents an embedding vector of the i-th knowledge graph node; n represents the number of nodes in the knowledge graph; Encoder state update: according to the attention weight, the encoder updates the hidden state at each time: ； In the formula, represents the hidden state updated by the knowledge graph weighting; represents the hidden state at the i th moment.

5. The semi-supervised pre-trained power system dialogue generation method according to claim 1, wherein, Step 4 includes the following steps: Artificially annotated power system dialogue data: randomly select part of the power system dialogue data for artificial annotation, including dialogue intent, slot information and knowledge graph association; Supervised fine-tuning: use the annotated data to fine-tune the pre-trained BART model in step 2, and use cross-entropy loss function to optimize the model during fine-tuning, which measures the difference between the predicted dialogue sequence generated by the model and the annotated dialogue sequence: ； In the formula: represents the cross-entropy loss value; represents the actual classification or vocabulary of the i-th dialogue marked; represents the probability of the i-th dialogue predicted by the model; m represents the total number of vocabularies in the dialogue; Pseudo-label generation: the fine-tuned BART model is used to predict the pseudo-labels for the unlabeled dialogue data; use the generated pseudo-labels and real annotated data to train the BART model again.

6. The semi-supervised pre-trained power system dialogue generation method according to claim 1, wherein, Step 5 includes the following steps: Input processing: when the user inputs a power system related question, first use the encoder of the pre-trained BART model to encode the input text and generate context embedding representation; Knowledge graph embedding acquisition: using knowledge graph embedding technology, convert nodes and edges in the knowledge graph into vector representations; Combine context and knowledge graph embedding: select relevant node embedding from the knowledge graph by matching keywords or terms in the user input text; fuse the context embedding and the selected node embedding to form a comprehensive embedding, where the fusion strategy uses attention mechanism for fusion; Decoding process: during the generation of the reply, the decoder of the BART model dynamically selects whether to use the information in the knowledge graph according to the generated part of the content; Generate logical reply: the decoder selectively focuses on the node embedding related to the question through the attention mechanism to generate a more logical reply.

7. The semi-supervised pre-trained power system dialogue generation method according to claim 1, wherein, Step 6 includes the following steps: Model generates dialogue: use the trained BART model to generate a reply for each user input in the test data set; Calculate BLEU index: measure the quality of the generated text by comparing the similarity between the generated text and the reference text: ； In the formula, N represents the maximum length of the n-gram under investigation and takes the value 4; represents the weight of the nth n-gram, and is ; represents the precision of the nth n-gram; represents the length penalty factor; Wherein: ； ； where c represents the length of the generated text, r represents the length of the reference text; e represents the base number of the natural logarithm; represents the count of the number of occurrences of n-grams in the generated text truncated by the number of occurrences of the same n-grams in the reference text; represents the count of n-grams in the reference text; C represents the set of candidate texts; BLEU Score Analysis: After calculating the BLEU score for each generated dialogue, the average BLEU score of the entire test data is calculated as the overall evaluation indicator of dialogue quality, including the following steps: Single Dialogue BLEU Score: Calculate the BLEU score for each user input and generated reply; Average BLEU Score: Calculate the average BLEU score of all dialogues: ； wherein: M represents the size of the test dataset; represents the BLEU score of the i-th dialogue; Strategy Adjustment and Optimization: Based on the BLEU score results, analyze the accuracy and coherence of the model's generated replies, determine the direction of optimization needed, and optimize the corresponding strategies based on the optimization direction.

Images