Semi-supervised pre-training electric power system dialogue generation method

Abstract

The invention belongs to the technical field of power grids, and relates to a semi-supervised pre-training power system dialogue generation method, which comprises the following steps: collecting and standardizing power system dialogue data, constructing a knowledge graph of related fields, adopting a BART model to carry out unsupervised pre-training on unlabeled data, and adding text noise to enhance the robustness of the model; integrating knowledge graph information into a coding stage of the BART model by utilizing a knowledge graph embedding technology, so that the model can refer to domain knowledge during dialogue generation; partial data is manually labeled, and a cross entropy loss function is used for monitoring and fine tuning; training data are further expanded by generating pseudo labels, and in practical application, the model combines user input and a knowledge graph to dynamically generate replies with higher logicality. And finally, evaluating dialogue quality by using a BLEU index, and optimizing a model strategy. According to the method, the problems of high dependence on annotation data and inaccurate and incoherent generated content in dialogue generation of the power system are effectively solved.

Description

Technical Field

[0001] The present application belongs to the field of power grid technology, and more specifically, to a semi-supervised pre-training method for generating power system dialogues. Background Art

[0002] With the popularization of smart devices and sensors in power systems, power companies have accumulated a large amount of historical conversation data. These data cover a variety of scenarios such as user inquiries, fault reports, and equipment maintenance. However, most of these data are unlabeled, and there are limited methods for direct supervised learning.

[0003] Traditional dialogue generation models such as Seq2Seq (Sequence-to-Sequence) and Transformer perform well in processing open-domain dialogues, but face the following challenges in specific fields such as power systems:

[0004] Insufficient use of unlabeled data: Most dialogue generation models rely on a large amount of labeled data for training. However, in the field of power systems, the acquisition cost of labeled data is high and the quantity is limited. How to use a large amount of unlabeled data to improve model performance becomes a key issue.

[0005] Complex domain terminology: Power system dialogues contain a lot of professional terms and technical details. The terms from different data sources may be inconsistent, which directly affects the training effect of the model. Therefore, these terms need to be standardized so that the model can better understand and generate domain-related dialogues.

[0006] The application of knowledge graphs is difficult: the knowledge graphs in the field of power systems contain a wealth of equipment information, operating parameters, fault records, etc. How to effectively embed this structured knowledge into the dialogue generation model and improve the generation capability of the model is an important research direction.

[0007] Therefore, how to effectively utilize unlabeled data and domain knowledge graphs in power system dialogue generation, reduce dependence on labeled data, and improve the accuracy and coherence of dialogue generation is a technical problem that needs to be solved. Summary of the invention

[0008] The present invention provides a semi-supervised pre-training method for power system dialogue generation, which aims to solve the technical problem of how to effectively utilize unlabeled data and domain knowledge graphs in power system dialogue generation, reduce dependence on labeled data, and improve the accuracy and coherence of dialogue generation.

[0009] A semi-supervised pre-training method for generating power system dialogues comprises the following steps:

[0010] Step 1: Collect conversation data from the historical data sources of the power system and construct a knowledge graph in the field of power systems; clean the collected conversation data, remove irrelevant information, and standardize the terminology in the field of power systems to obtain standardized power system conversation data;

[0011] Step 2: Use the BART model to perform unsupervised pre-training on unlabeled and standardized power system dialogue data, and randomly mask some words in the text; during pre-training, introduce text noise and require the BART model to reconstruct the original text based on the noisy text to obtain a text dialogue sequence;

[0012] Step 3: Use knowledge graph embedding technology to convert nodes and edges in the knowledge graph into vector representations. In the encoding phase of the BART model, the knowledge graph embedding is used as an additional input and fed into the encoder together with the text embedding.

[0013] Step 4: Randomly extract some power system dialogue data for manual labeling. The labeling content includes dialogue intent, slot information, and knowledge graph association. Use the labeled data to supervise and fine-tune the pre-trained BART model. Use the cross entropy loss function to optimize the model. Use the fine-tuned BART model to generate pseudo labels for unlabeled data. Use the generated pseudo labels as a supplement to the training data and train the model again.

[0014] Step 5: When the user inputs a question related to the power system, the BART model encodes the input text into a context embedding and combines it with the relevant node embedding in the knowledge graph for processing as input. The encoder selectively focuses on nodes related to the question through the knowledge graph information. During the decoding process, the BART model dynamically selects whether to use specific information in the knowledge graph based on the generated content, and uses the reasoning ability of the knowledge graph to generate a more logical response.

[0015] Step 6: Use the BLEU indicator to evaluate the quality of the generated dialogue, detect the accuracy and coherence of the content generated by the BART model, adjust the model strategy based on the evaluation results, and optimize the knowledge graph fusion strategy.

[0016] The present invention uses a semi-supervised pre-training method to effectively utilize unlabeled data and domain knowledge graphs, significantly reducing the dependence on labeled data, while improving the accuracy and coherence of dialogue generation. First, the power system dialogue data is collected and standardized, and the knowledge graph of the relevant field is constructed. Then, the BART model is used to perform unsupervised pre-training on the unlabeled data, and text noise is added to enhance the robustness of the model. Next, the knowledge graph embedding technology is used to integrate the knowledge graph information into the encoding stage of the BART model, so that the model can refer to the domain knowledge when generating dialogues. The performance of the model is improved by manually annotating some data and using the cross entropy loss function for supervised fine-tuning. After that, the training data is further expanded by generating pseudo-labels to enhance the generalization ability of the model. In practical applications, the model combines user input and knowledge graphs to dynamically generate more logical replies. Finally, the BLEU indicator is used to evaluate the dialogue quality and optimize the model strategy. This method effectively solves the problems of high dependence on labeled data, inaccurate and incoherent generated content in power system dialogue generation.

[0017] Preferably, step 2 comprises the following steps:

[0018] Data format conversion: Convert the conversation data into an input format suitable for BART model training. Each conversation data consists of a question-answer pair, where the input sequence is the historical context or question part of the conversation, and the target sequence is the corresponding answer part;

[0019] Masked language model: 15%-20% of the words are randomly selected from each input text for masking, and the masked words are replaced with symbols:

[0020]

[0021] Where: X represents the input sequence; p mask represents the probability of masked words; represents the input sequence after masking;

[0022] Text noise introduction: randomly disrupt the order of sentences in the input text and randomly delete words or sentences in the input text:

[0023] X noisy =Shuffle(X)+Drop(X,p drop );

[0024] Where: X noisy represents the input sequence after the noise is introduced; Shuffle(X) represents the random shuffling of sentences or words in the input sequence; Drop(X, p drop ) means that according to the probability p dropDelete some words or sentences in the input sequence, where the probability is set to 10%-20%;

[0025] Reconstruction task: The training objective of the BART model is based on the masked input sequence and the input sequence X with noise added noisy Let the BART model be able to reconstruct the original input sequence X:

[0026]

[0027] Where: P(x|X noisy ) represents the probability of the BART model recovering the original text from the noisy text; T represents the length of the text; x t represents the tth word; x 1 , x 2 , ..., x t-1 Indicates the previously generated vocabulary;

[0028] Loss function: During pre-training, the goal of the BART model is to minimize the loss function of the reconstruction task. The cross entropy loss function is used to calculate the difference between each word of the BART model and the target word:

[0029]

[0030] Where: Represents the total loss value; P(x t |x 1 , x 2 , ..., x t-1 ) represents the probability of the vocabulary generated by the BART model at time step t;

[0031] Optimizer: During the pre-training process, the Adam optimizer is used to train the BART model and adjust the model parameters to minimize the loss function:

[0032]

[0033] Where: θ t Represents the parameters of the current BART model; θ t+1 represents the updated parameters of the BART model; α represents the learning rate; m t represents the first-order momentum of the gradient; v t represents the second-order momentum of the gradient; ∈ represents a small amount to avoid division by zero;

[0034] Convergence criteria: When the BART model's performance on the validation set reaches stability and the error value is within the set range, the pre-training process ends.

[0035] Preferably, step 3 comprises the following steps:

[0036] Knowledge graph embedding: Use TransE knowledge graph embedding technology to convert nodes and edges in the knowledge graph into low-dimensional vector representations. By mapping entities and relationships into the same vector space and minimizing the loss function to train the embedding vector, the relationships between nodes remain consistent:

[0037] h+r≈t;

[0038] Where: h represents the embedding vector of the head entity; r represents the embedding vector of the relationship; t represents the embedding vector of the tail entity;

[0039] Loss function:

[0040] L=∑ (h,r,t)∈τ [γ+d(h+r,t)-d(h′,r′,t′)] + ;

[0041] Where: τ represents the set of positive sample triplets; (h, r, t) represents a positive triplet, representing the head entity, the relationship and the tail entity; d(h+r, t) represents the distance between the head entity valence relationship and the tail entity, which is the Euclidean distance; (h′, r′, t′) represents the negative sample; γ represents the hyperparameter, which controls the distance between positive and negative samples;

[0042] Text embedding generation: Use pre-trained word vectors to convert the input dialogue sample into an embedding vector; where the input text dialogue sequence X′ = (x 1 , x 2 , ..., x T ), where each x i Representing a word in the text, the text embedding is generated as follows:

[0043] E text =[e(x 1 ), e(x 2 ),…,e(x T )];

[0044] Where: E text Represents the embedding matrix of the text; e(x i ) represents the word embedding obtained by pre-training word vectors;

[0045] Combination of knowledge graph embedding and text embedding: Combine the node embedding in the knowledge graph with the text embedding. In the encoder input, for each entity node related to the current conversation context, obtain the corresponding embedding vector and concatenate the embedding vector with the text embedding:

[0046] E joint =[E text ; E KG ];

[0047] Where: E KG =[h 1 ,h 2 , ..., h n ] represents the embedding of the knowledge graph entity node related to the current conversation; E joint Represents the joint representation of text embedding and knowledge graph embedding, combined through vector concatenation operation;

[0048] Dynamically select relevant nodes: In each conversation, use the relational structure of the knowledge graph to select relevant nodes based on the current conversation content;

[0049] Encoder input processing: In the encoding stage of the BART model, the encoder jointly embeds the representation E joint The input is fed into the encoder, which processes the input sequence, captures the relationship between text words through the self-attention mechanism, and uses the embedded information of the knowledge graph to enhance the understanding of the power system field:

[0050] H enc =BART_Encoder(E joint );

[0051] Where: H enc Indicates that the encoder generates hidden state representation; BART_Encoder represents the encoder module of the BART model.

[0052] Preferably, step 3 further includes the BART model learning the relevance weight of each knowledge graph node with the current conversation, calculating the attention weight between each node embedding and the input text embedding through the self-attention mechanism, and giving higher weights to more relevant nodes:

[0053]

[0054] Where: α i represents the attention weight of the i-th knowledge graph node; q is the hidden state of the input text; k i represents the embedding vector of the i-th knowledge graph node; n represents the number of nodes in the knowledge graph;

[0055] Encoder state update: According to the attention weight, the encoder performs a weighted update on the hidden state at each moment:

[0056]

[0057] Where: H updated represents the hidden state after weighted update of the knowledge graph; h i represents the hidden state at the ith moment.

[0058] Preferably, step 4 comprises the following steps:

[0059] Manually label power system conversation data: Randomly select some power system conversation data for manual labeling, including conversation intent, slot information, and knowledge graph association;

[0060] Supervised fine-tuning: Use the labeled data to supervise the fine-tuning of the pre-trained BART model in step 2. During the fine-tuning process, the cross-entropy loss function is used to optimize the model. The cross-entropy loss function measures the difference between the dialogue sequence predicted by the model and the labeled dialogue sequence:

[0061]

[0062] Where: L CE Represents the cross entropy loss value; y i represents the actual category or vocabulary of the ith dialogue annotated; represents the probability of the category or word of the i-th dialogue predicted by the model; m represents the total number of words in the dialogue;

[0063] Pseudo-label generation: The fine-tuned BART model is used to predict and generate pseudo-labels for unlabeled conversation data. The BART model is trained again using the generated pseudo-labels and real annotated data.

[0064] Preferably, the step 5 comprises the following steps:

[0065] Input processing: When a user enters a question related to the power system, the input text is first encoded using the encoder of the pre-trained BART model to generate a contextual embedding representation;

[0066] Knowledge graph embedding acquisition: Use knowledge graph embedding technology to convert nodes and edges in the knowledge graph into vector representations;

[0067] Combining context and knowledge graph embedding: Select relevant node embeddings from the knowledge graph by matching keywords or terms with the user input text; fuse the context embedding with the selected relevant node embedding to form a comprehensive embedding, where the fusion strategy uses the attention mechanism for fusion;

[0068] Decoding process: In the process of generating a response, the decoder of the BART model dynamically chooses whether the information in the knowledge graph is needed based on the part of the content that has been generated;

[0069] Generate logical replies: The decoder selectively focuses on the node embeddings related to the question through the attention mechanism, generating more logical replies.

[0070] Preferably, step 6 comprises the following steps:

[0071] Model-generated dialogue: Use the trained BART model to generate responses for each user input in the test dataset;

[0072] Calculate the BLEU metric: measure the quality of the generated text by comparing the similarity between the generated text and the reference text:

[0073]

[0074] Where: N represents the maximum length of the n-gram under consideration, which is 4; w n Represents the weight of the n-th n-gram, which is p n represents the accuracy of the nth n-gram; BP represents the length penalty factor;

[0075] in:

[0076]

[0077] Where: c represents the length of the generated text, r represents the length of the reference text; e represents the base of the natural logarithm; Count clip (n-gram) represents the count of the number of occurrences of the n-gram in the generated text truncated by the number of occurrences of the same n-gram in the reference text; Count ref (n-gram) represents the count of n-grams in the reference text; C represents the candidate text set;

[0078] Analyze BLEU score results: After calculating the BLEU score of each generated dialogue, the average BLEU score of the entire set of test data is calculated as the overall evaluation indicator of dialogue quality, including the following steps:

[0079] BLEU score for a single conversation: Calculate the BLEU score for each user input and generated response;

[0080] Average BLEU score: Calculate the average BLEU score of all dialogues:

[0081]

[0082] Where: M represents the size of the test data set; BLEU i represents the BLEU score of the i-th dialogue;

[0083] Strategy adjustment and optimization: Based on the BLEU score results, analyze the accuracy and coherence of the responses generated by the model, determine the direction that needs to be optimized, and optimize the corresponding strategy based on the optimization direction.

[0084] The beneficial effects of the present invention include:

[0085] The present invention uses a semi-supervised pre-training method to effectively utilize unlabeled data and domain knowledge graphs, significantly reducing the dependence on labeled data, while improving the accuracy and coherence of dialogue generation. First, the power system dialogue data is collected and standardized, and the knowledge graph of the relevant field is constructed. Then, the BART model is used to perform unsupervised pre-training on the unlabeled data, and text noise is added to enhance the robustness of the model. Next, the knowledge graph embedding technology is used to integrate the knowledge graph information into the encoding stage of the BART model, so that the model can refer to the domain knowledge when generating dialogues. The performance of the model is improved by manually annotating some data and using the cross entropy loss function for supervised fine-tuning. After that, the training data is further expanded by generating pseudo-labels to enhance the generalization ability of the model. In practical applications, the model combines user input and knowledge graphs to dynamically generate more logical replies. Finally, the BLEU indicator is used to evaluate the dialogue quality and optimize the model strategy. This method effectively solves the problems of high dependence on labeled data, inaccurate and incoherent generated content in power system dialogue generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0087] Figure 1 An overall step block diagram provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0088] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0089] See also Figure 1 As shown, the best embodiment of the present invention is further described;

[0090] A semi-supervised pre-training method for generating power system dialogues comprises the following steps:

[0091] Step 1: Collect conversation data from the historical data sources of the power system and construct a knowledge graph in the field of power systems; clean the collected conversation data, remove irrelevant information, and standardize the terminology in the field of power systems to obtain standardized power system conversation data;

[0092] Step 2: Use the BART model to perform unsupervised pre-training on unlabeled and standardized power system dialogue data, and randomly mask some words in the text; during pre-training, introduce text noise and require the BART model to reconstruct the original text based on the noisy text to obtain a text dialogue sequence;

[0093] Step 3: Use knowledge graph embedding technology to convert nodes and edges in the knowledge graph into vector representations. In the encoding phase of the BART model, the knowledge graph embedding is used as an additional input and fed into the encoder together with the text embedding.

[0094] Step 4: Randomly extract some power system dialogue data for manual labeling. The labeling content includes dialogue intent, slot information, and knowledge graph association. Use the labeled data to supervise and fine-tune the pre-trained BART model. Use the cross entropy loss function to optimize the model. Use the fine-tuned BART model to generate pseudo labels for unlabeled data. Use the generated pseudo labels as a supplement to the training data and train the model again.

[0095] Step 5: When the user inputs a question related to the power system, the BART model encodes the input text into a context embedding and combines it with the relevant node embedding in the knowledge graph for processing as input. The encoder selectively focuses on nodes related to the question through the knowledge graph information. During the decoding process, the BART model dynamically selects whether to use specific information in the knowledge graph based on the generated content, and uses the reasoning ability of the knowledge graph to generate a more logical response.

[0096] Step 6: Use the BLEU indicator to evaluate the quality of the generated dialogue, detect the accuracy and coherence of the content generated by the BART model, adjust the model strategy based on the evaluation results, and optimize the knowledge graph fusion strategy.

[0097] The present invention uses a semi-supervised pre-training method to effectively utilize unlabeled data and domain knowledge graphs, significantly reducing the dependence on labeled data, while improving the accuracy and coherence of dialogue generation. First, the power system dialogue data is collected and standardized, and the knowledge graph of the relevant field is constructed. Then, the BART model is used to perform unsupervised pre-training on the unlabeled data, and text noise is added to enhance the robustness of the model. Next, the knowledge graph embedding technology is used to integrate the knowledge graph information into the encoding stage of the BART model, so that the model can refer to the domain knowledge when generating dialogues. The performance of the model is improved by manually annotating some data and using the cross entropy loss function for supervised fine-tuning. After that, the training data is further expanded by generating pseudo-labels to enhance the generalization ability of the model. In practical applications, the model combines user input and knowledge graphs to dynamically generate more logical replies. Finally, the BLEU indicator is used to evaluate the dialogue quality and optimize the model strategy. This method effectively solves the problems of high dependence on labeled data, inaccurate and incoherent generated content in power system dialogue generation.

[0098] As a possible implementation of this embodiment, step 2 includes the following steps:

[0099] Data format conversion: Convert the conversation data into an input format suitable for BART model training. Each conversation data consists of a question-answer pair, where the input sequence is the historical context or question part of the conversation, and the target sequence is the corresponding answer part;

[0100] Masked language model: 15%-20% of the words are randomly selected from each input text for masking, and the masked words are replaced with symbols:

[0101]

[0102] Where: X represents the input sequence; omask represents the probability of masking words; represents the input sequence after masking;

[0103] Text noise introduction: randomly disrupt the order of sentences in the input text and randomly delete words or sentences in the input text:

[0104] X noisy =Shuffle(X)+Drop(X, drop );

[0105] Where: X noisy represents the input sequence after the noise is introduced; Shuffle(X) represents the random shuffling of sentences or words in the input sequence; Drop(X, p drop ) means that according to the probability p dropDelete some words or sentences in the input sequence, where the probability is set to 10%-20%;

[0106] Reconstruction task: The training objective of the BART model is based on the masked input sequence and the input sequence X with noise added noisy Let the BART model be able to reconstruct the original input sequence X:

[0107]

[0108] Where: P(x|X noisy ) represents the probability of the BART model recovering the original text from the noisy text; T represents the length of the text; x t represents the tth word; x 1 , x 2 , ..., x t-1 Indicates the previously generated vocabulary;

[0109] Loss function: During pre-training, the goal of the BART model is to minimize the loss function of the reconstruction task. The cross entropy loss function is used to calculate the difference between each word of the BART model and the target word:

[0110]

[0111] Where: Represents the total loss value; P(x t |x 1 , x 2 , ..., x t-1 ) represents the probability of the vocabulary generated by the BART model at time step t;

[0112] Optimizer: During the pre-training process, the Adam optimizer is used to train the BART model and adjust the model parameters to minimize the loss function:

[0113]

[0114] Where: θ t Represents the parameters of the current BART model; θ t+1 represents the updated parameters of the BART model; α represents the learning rate; m t represents the first-order momentum of the gradient; v t represents the second-order momentum of the gradient; ∈ represents a small amount to avoid division by zero;

[0115] Convergence criteria: When the BART model's performance on the validation set reaches stability and the error value is within the set range, the pre-training process ends.

[0116] As a possible implementation of this embodiment, step 3 includes the following steps:

[0117] Knowledge graph embedding: Use TransE knowledge graph embedding technology to convert nodes and edges in the knowledge graph into low-dimensional vector representations. By mapping entities and relationships into the same vector space and minimizing the loss function to train the embedding vector, the relationships between nodes remain consistent:

[0118] h+r≈t;

[0119] Where: h represents the embedding vector of the head entity; r represents the embedding vector of the relationship; t represents the embedding vector of the tail entity;

[0120] Loss function:

[0121] L=∑(h, r, t)∈τ[γ+d(h+r, t)-d(h′, r′, t′)]+;

[0122] Where: τ represents the set of positive sample triplets; (h, r, t) represents a positive triplet, representing the head entity, the relationship and the tail entity; d(h+r, t) represents the distance between the head entity valence relationship and the tail entity, which is the Euclidean distance; (h′, r′, t′) represents the negative sample; γ represents the hyperparameter, which controls the distance between positive and negative samples;

[0123] Text embedding generation: Use pre-trained word vectors to convert the input dialogue sample into an embedding vector; where the input text dialogue sequence X′ = (x 1 , x 2 , ..., x T ), where each x i Representing a word in the text, the text embedding is generated as follows:

[0124] E text =[e(x 1 ), e(x 2 ),…,e(x T )];

[0125] Where: E text Represents the embedding matrix of the text; e(x i ) represents the word embedding obtained by pre-training word vectors;

[0126] Combination of knowledge graph embedding and text embedding: Combine the node embedding in the knowledge graph with the text embedding. In the encoder input, for each entity node related to the current conversation context, obtain the corresponding embedding vector and concatenate the embedding vector with the text embedding:

[0127] E joint =[E text ; EKG ];

[0128] Where: E KG =[h 1 ,h 2 , ..., h n ] represents the embedding of the knowledge graph entity node related to the current conversation; E joint Represents the joint representation of text embedding and knowledge graph embedding, combined through vector concatenation operation;

[0129] Dynamically select relevant nodes: In each conversation, use the relational structure of the knowledge graph to select relevant nodes based on the current conversation content;

[0130] Encoder input processing: In the encoding stage of the BART model, the encoder jointly embeds the representation E joint The input is fed into the encoder, which processes the input sequence, captures the relationship between text words through the self-attention mechanism, and uses the embedded information of the knowledge graph to enhance the understanding of the power system field:

[0131] H enc =BART_Encoder(E joint );

[0132] Where: H enc Indicates that the encoder generates hidden state representation; BART_Encoder represents the encoder module of the BART model.

[0133] As a possible implementation of this embodiment, step 3 further includes the BART model learning the relevance weight of each knowledge graph node with the current conversation, calculating the attention weight between each node embedding and the input text embedding through the self-attention mechanism, and giving more relevant nodes a higher weight:

[0134]

[0135] Where: α i represents the attention weight of the i-th knowledge graph node; q is the hidden state of the input text; li represents the embedding vector of the i-th knowledge graph node; n represents the number of nodes in the knowledge graph;

[0136] Encoder state update: According to the attention weight, the encoder performs a weighted update on the hidden state at each moment:

[0137]

[0138] Where: H uodated represents the hidden state after weighted update of the knowledge graph; h i represents the hidden state at the ith moment.

[0139] In this embodiment, by calculating the relevance weights, the model can better identify and utilize the knowledge graph nodes related to the current conversation, thereby generating answers that are more in line with the context; the self-attention mechanism enables the BART model to automatically extract key information from the complex power system knowledge graph, thereby improving the knowledge and professionalism of the generated conversation; by learning the correlation between the knowledge graph nodes and the conversation text, the model can more flexibly apply existing knowledge in different conversation scenarios and improve its adaptability to unseen data.

[0140] The joint embedding representation combines the dual information of the conversation text and the knowledge graph, making the hidden state representation generated by the encoder richer and more multidimensional, which helps to improve the quality of conversation generation; by introducing relevant information of the knowledge graph nodes in the encoding stage, the model can better understand and utilize domain knowledge, and enhance the professionalism and accuracy of conversation generation; the joint embedding representation can help the model better grasp the context and contextual information of the conversation, thereby generating more coherent and natural conversation content.

[0141] When generating dialogue responses, the decoder can make full use of the rich semantic information and domain knowledge provided by the encoder to generate more natural and high-quality dialogue content; improve the coherence and consistency of the dialogue: through the self-attention mechanism, the decoder can better grasp the contextual information, and the generated dialogue responses are more logically coherent and consistent, improving the user experience; enhance the ability to handle complex problems: using joint embedding representation and self-attention mechanism, the decoder can more effectively handle complex dialogue scenarios and problems, and provide more accurate and professional answers.

[0142] As a possible implementation of this embodiment, step 4 includes the following steps:

[0143] Manually label power system conversation data: Randomly select some power system conversation data for manual labeling, including conversation intent, slot information, and knowledge graph association;

[0144] Supervised fine-tuning: Use the labeled data to supervise the fine-tuning of the pre-trained BART model in step 2. During the fine-tuning process, the cross-entropy loss function is used to optimize the model. The cross-entropy loss function measures the difference between the dialogue sequence predicted by the model and the labeled dialogue sequence:

[0145]

[0146] Where: L CE Represents the cross entropy loss value; y i represents the actual category or vocabulary of the ith dialogue annotated; represents the probability of the category or word of the i-th dialogue predicted by the model; m represents the total number of words in the dialogue;

[0147] Pseudo-label generation: The fine-tuned BART model is used to predict and generate pseudo-labels for unlabeled conversation data. The BART model is trained again using the generated pseudo-labels and real annotated data.

[0148] In this embodiment, by manually annotating the power system dialogue data and using this data for supervised fine-tuning, the generalization ability of the pre-trained BART model in the power system dialogue generation task can be significantly improved, so that it can better understand and generate dialogues that meet actual needs; in the supervised fine-tuning process, the cross entropy loss function is used to optimize the model, which can effectively reduce the difference between the dialogue sequence predicted and generated by the model and the actual annotated dialogue sequence, thereby improving the accuracy of the model-generated responses; the pseudo-label generation step uses the fine-tuned BART model to predict the unlabeled data to generate pseudo-labels, and combines the real annotated data for secondary training, which can maximize the use of large-scale unlabeled data and further improve the model performance; through the semi-supervised learning method, the demand for large-scale manually annotated data is reduced, and the cost and time of data annotation are reduced.

[0149] As a possible implementation of this embodiment, step 5 includes the following steps:

[0150] Input processing: When a user enters a question related to the power system, the input text is first encoded using the encoder of the pre-trained BART model to generate a contextual embedding representation;

[0151] Knowledge graph embedding acquisition: Use knowledge graph embedding technology to convert nodes and edges in the knowledge graph into vector representations;

[0152] Combining context and knowledge graph embedding: Select relevant node embeddings from the knowledge graph by matching keywords or terms with the user input text; fuse the context embedding with the selected relevant node embedding to form a comprehensive embedding, where the fusion strategy uses the attention mechanism for fusion;

[0153] Decoding process: In the process of generating a response, the decoder of the BART model dynamically chooses whether the information in the knowledge graph is needed based on the part of the content that has been generated;

[0154] Generate logical replies: The decoder selectively focuses on the node embeddings related to the question through the attention mechanism, generating more logical replies.

[0155] As a possible implementation of this embodiment, step 6 includes the following steps:

[0156] Model-generated dialogue: Use the trained BART model to generate responses for each user input in the test dataset;

[0157] Calculate the BLEU metric: measure the quality of the generated text by comparing the similarity between the generated text and the reference text:

[0158]

[0159] Where: N represents the maximum length of the n-gram under consideration, which is 4; w n Represents the weight of the n-th n-gram, which is p n represents the accuracy of the nth n-gram; BP represents the length penalty factor;

[0160] in:

[0161]

[0162] Where: c represents the length of the generated text, r represents the length of the reference text; e represents the base of the natural logarithm; Count clip (n-gram) represents the count of the number of occurrences of the n-gram in the generated text truncated by the number of occurrences of the same n-gram in the reference text; Count ref (n-gram) represents the count of n-grams in the reference text; C represents the candidate text set;

[0163] Analyze BLEU score results: After calculating the BLEU score of each generated dialogue, the average BLEU score of the entire set of test data is calculated as the overall evaluation indicator of dialogue quality, including the following steps:

[0164] BLEU score for a single conversation: Calculate the BLEU score for each user input and generated response;

[0165] Average BLEU score: Calculate the average BLEU score of all dialogues:

[0166]

[0167] Where: M represents the size of the test data set; BLEU i represents the BLEU score of the i-th dialogue;

[0168] Strategy adjustment and optimization: Based on the BLEU score results, analyze the accuracy and coherence of the responses generated by the model, determine the direction that needs to be optimized, and optimize the corresponding strategy based on the optimization direction;

[0169] Statistical analysis: Calculate the mean, median, standard deviation and other statistical indicators of the BLEU score to understand the overall distribution of the score.

[0170] Distribution graph: Draw a distribution graph of the BLEU score (such as a histogram or box plot) to observe the central tendency, dispersion, and outliers of the score.

[0171] Filter low-scoring dialogues: Find dialogue instances with low BLEU scores, analyze the user input, generate responses, and provide reference responses for these instances.

[0172] Classify issues: Classify low-scoring conversations into problem categories, such as short replies, grammatical errors, irrelevant content, etc.

[0173] Quality problem classification: Based on the analysis results of low-scoring conversations, the main types of quality problems are summarized, such as:

[0174] Inaccurate answer: The generated response deviates greatly from the reference answer.

[0175] Grammatical issues: The generated response contains grammatical errors or is not fluent.

[0176] Irrelevant content: The generated response is not relevant to the question entered by the user.

[0177] Answer too short or too long: The generated response is not of the appropriate length.

[0178] Targeted optimization: Determine the corresponding optimization direction for different problem types, for example:

[0179] Inaccurate answers: Strengthen the integration of knowledge graphs to ensure that the generated responses contain accurate knowledge points.

[0180] Grammatical issues: Improve the quality of training data for language models and add grammar checking mechanisms.

[0181] Irrelevant content: Optimize context understanding and ensure that responses are closely related to user input.

[0182] The answer is too short or too long: adjust the generation strategy to control the length of the generated text.

[0183] Optimize knowledge graph embedding: Improve the embedding method so that the information in the knowledge graph can be more effectively integrated into the model generation process.

[0184] Add knowledge graph nodes: Expand the coverage of the knowledge graph to ensure that more knowledge points can be used by the model.

[0185] Collect more conversation data: especially manually annotated data, to ensure data diversity and quality.

[0186] Generate pseudo-labeled data: Generate more training data with reference answers through data augmentation technology.

[0187] Hyperparameter tuning: Based on the evaluation results, adjust the model’s hyperparameters (such as learning rate, batch size, etc.) and retrain the model.

[0188] Experimental comparison: Conduct multiple sets of comparative experiments and select the parameter settings with the best performance.

[0189] Recalculate BLEU score: After the optimization strategy is applied, recalculate the BLEU score using the same test dataset to evaluate the optimization effect.

[0190] Comparative analysis: Compare with the initial evaluation results to confirm whether the optimization strategy has effectively improved the quality of generated dialogues.

[0191] Iteration: Based on the latest evaluation results, analyze existing problems again, adjust the optimization direction, and carry out the next round of optimization.

[0192] Long-term mechanism: Establish a continuous improvement mechanism to regularly evaluate model performance and ensure continuous improvement in the quality of generated dialogues.

[0193] In this embodiment, by calculating the BLEU indicator, the quality of the model-generated dialogue can be objectively evaluated to ensure that the generated dialogue has a high degree of similarity with the reference text in terms of content and structure; the BLEU scoring result can quantify the model performance and provide clear goals and directions for model improvement and optimization; by analyzing the BLEU score, the weak links in the model-generated responses, such as accuracy and coherence issues, can be identified, and targeted optimization and improvement can be performed; according to the BLEU scoring results, the model generation strategy can be adjusted and optimized to continuously improve the quality of the model-generated dialogue and ensure its effectiveness and reliability in practical applications.

[0194] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A semi-supervised pre-training method for generating power system dialogues, characterized in that: The following steps are involved: Step 1: Collect conversation data from historical data sources of the power system and build a knowledge graph in the power system domain; Clean the collected conversation data, remove irrelevant information, standardize the terminology in the power system field, and obtain standardized power system conversation data; Step 2: Use the BART model to perform unsupervised pre-training on unlabeled and standardized power system dialogue data, and randomly mask some words in the text; during pre-training, introduce text noise and require the BART model to reconstruct the original text based on the noisy text to obtain a text dialogue sequence; Step 3: Use knowledge graph embedding technology to convert nodes and edges in the knowledge graph into vector representations. In the encoding phase of the BART model, the knowledge graph embedding is used as an additional input and fed into the encoder together with the text embedding. Step 4: Randomly extract some power system dialogue data for manual labeling. The labeling content includes dialogue intent, slot information, and knowledge graph association. Use the labeled data to supervise and fine-tune the pre-trained BART model. Use the cross entropy loss function to optimize the model. Use the fine-tuned BART model to generate pseudo labels for unlabeled data. Use the generated pseudo labels as a supplement to the training data and train the model again. Step 5: When the user inputs a question related to the power system, the BART model encodes the input text into a context embedding and combines it with the relevant node embeddings in the knowledge graph for processing as input. The encoder selectively focuses on nodes related to the question through the knowledge graph information. During the decoding process, the BART model dynamically chooses whether to use specific information in the knowledge graph based on the generated content, and uses the reasoning ability of the knowledge graph to generate a more logical response; Step 6: Use the BLEU indicator to evaluate the quality of the generated dialogue, detect the accuracy and coherence of the content generated by the BART model, adjust the model strategy based on the evaluation results, and optimize the knowledge graph fusion strategy.

2. A semi-supervised pre-training power system dialogue generation method according to claim 1, characterized in that: The step 2 comprises the following steps: Data format conversion: Convert the conversation data into an input format suitable for BART model training. Each conversation data consists of a question-answer pair, where the input sequence is the historical context or question part of the conversation, and the target sequence is the corresponding answer part; Masked language model: 15%-20% of the words are randomly selected from each input text for masking, and the masked words are replaced with symbols: Where: X represents the input sequence; p maxk represents the probability of masked words; represents the input sequence after masking; Text noise introduction: randomly disrupt the order of sentences in the input text and randomly delete words or sentences in the input text: X noisy =Shuffle(X)+Drop(X,p drop ); Where: X noisy represents the input sequence after the noise is introduced; Shuffle(X) represents the random shuffling of sentences or words in the input sequence; Drop(X, p drop ) means that according to the probability p drop Delete some words or sentences in the input sequence, where the probability is set to 10%-20%; Reconstruction task: The training objective of the BART model is based on the masked input sequence and the input sequence X with noise added noisy Let the BART model be able to reconstruct the original input sequence X: Where: P(X|X noisy ) represents the probability of the BART model recovering the original text from the noisy text; T represents the length of the text; x t represents the tth word; x1, x2, ..., x t-1 Indicates the previously generated vocabulary; Loss function: During pre-training, the goal of the BART model is to minimize the loss function of the reconstruction task. The cross entropy loss function is used to calculate the difference between each word of the BART model and the target word: Where: Represents the total loss value; P(x t |x1,x2,...,x t-1 ) represents the probability of the vocabulary generated by the BART model at time step t; Optimizer: During the pre-training process, the Adam optimizer is used to train the BART model and adjust the model parameters to minimize the loss function: Where: θ t Represents the parameters of the current BART model; θ t+1 represents the updated parameters of the BART model; α represents the learning rate; m t represents the first-order momentum of the gradient; v t represents the second-order momentum of the gradient; ∈ represents a small amount to avoid division by zero; Convergence criteria: When the BART model's performance on the validation set reaches stability and the error value is within the set range, the pre-training process ends.

3. A semi-supervised pre-training power system dialogue generation method according to claim 1, characterized in that: The step 3 comprises the following steps: Knowledge graph embedding: Use TransE knowledge graph embedding technology to convert nodes and edges in the knowledge graph into low-dimensional vector representations. By mapping entities and relationships into the same vector space and minimizing the loss function to train the embedding vector, the relationships between nodes remain consistent: h+r≈t; Where: h represents the embedding vector of the head entity; r represents the embedding vector of the relationship; t represents the embedding vector of the tail entity; Loss function: L=∑ (h,r,t)∈τ [γ+d(h+r,t)-d(h′,r′,t′)] + ; Where: τ represents the set of positive sample triplets; (h, r, t) represents a positive triplet, representing the head entity, the relationship and the tail entity; d(h+r, t) represents the distance between the head entity valence relationship and the tail entity, which is the Euclidean distance; (h′, r′, t′) represents the negative sample; γ represents the hyperparameter, which controls the distance between positive and negative samples; Text embedding generation: Use pre-trained word vectors to convert the input dialogue sample into an embedding vector; where the input text dialogue sequence X′ = (x1, x2, ..., x T ), where each x i Representing a word in the text, the text embedding is generated as follows: AND text =[e(x1),e(x2),…,e(x T )]; Where: E text Represents the embedding matrix of the text; e(x i ) represents the word embedding obtained by pre-training word vectors; Combination of knowledge graph embedding and text embedding: Combine the node embedding in the knowledge graph with the text embedding. In the encoder input, for each entity node related to the current conversation context, obtain the corresponding embedding vector and concatenate the embedding vector with the text embedding: AND joint =[And text ;AND KG ]; Where: E KG =[h1, h2, ..., h n ] represents the embedding of the knowledge graph entity node related to the current conversation; E joint Represents the joint representation of text embedding and knowledge graph embedding, combined through vector concatenation operation; Dynamically select relevant nodes: In each conversation, use the relational structure of the knowledge graph to select relevant nodes based on the current conversation content; Encoder input processing: In the encoding stage of the BART model, the encoder jointly embeds the representation E joint The input is fed into the encoder, which processes the input sequence, captures the relationship between text words through the self-attention mechanism, and uses the embedded information of the knowledge graph to enhance the understanding of the power system field: H enc =BART_Encoder(E joint ); Where: H enc Indicates that the encoder generates hidden state representation; BART_Encoder represents the encoder module of the BART model.

4. The method for generating a semi-supervised pre-trained power system dialogue according to claim 1, characterized in that: Step 3 also includes the BART model learning the relevance weight of each knowledge graph node with the current conversation, and calculating the attention weight between each node embedding and the input text embedding through the self-attention mechanism, giving more relevant nodes higher weights: Where: α i Represents the attention weight of the i-th knowledge graph node; q is the hidden state of the input text; k i represents the embedding vector of the i-th knowledge graph node; n represents the number of nodes in the knowledge graph; Encoder state update: According to the attention weight, the encoder performs a weighted update on the hidden state at each moment: Where: H updated represents the hidden state after weighted update of the knowledge graph; h i represents the hidden state at the ith moment.

5. The method for generating a semi-supervised pre-trained power system dialogue according to claim 1, characterized in that: The step 4 comprises the following steps: Manually label power system conversation data: Randomly select some power system conversation data for manual labeling, including conversation intent, slot information, and knowledge graph association; Supervised fine-tuning: Use the labeled data to supervise the fine-tuning of the pre-trained BART model in step 2. During the fine-tuning process, the cross-entropy loss function is used to optimize the model. The cross-entropy loss function measures the difference between the dialogue sequence predicted by the model and the labeled dialogue sequence: Where: L CE Represents the cross entropy loss value; y i represents the actual category or vocabulary of the ith dialogue annotated; represents the probability of the category or word of the i-th dialogue predicted by the model; m represents the total number of words in the dialogue; Pseudo-label generation: The fine-tuned BART model is used to predict and generate pseudo-labels for unlabeled conversation data. The BART model is trained again using the generated pseudo-labels and real annotated data.

6. A semi-supervised pre-training power system dialogue generation method according to claim 1, characterized in that: The step 5 comprises the following steps: Input processing: When a user enters a question related to the power system, the input text is first encoded using the encoder of the pre-trained BART model to generate a contextual embedding representation; Knowledge graph embedding acquisition: Use knowledge graph embedding technology to convert nodes and edges in the knowledge graph into vector representations; Combining context and knowledge graph embedding: Select relevant node embeddings from the knowledge graph by matching keywords or terms with the user input text; fuse the context embedding with the selected relevant node embedding to form a comprehensive embedding, where the fusion strategy uses the attention mechanism for fusion; Decoding process: In the process of generating a response, the decoder of the BART model dynamically chooses whether the information in the knowledge graph is needed based on the part of the content that has been generated; Generate logical replies: The decoder selectively focuses on the node embeddings related to the question through the attention mechanism, generating more logical replies.

7. A semi-supervised pre-training power system dialogue generation method according to claim 1, characterized in that: The step 6 comprises the following steps: Model-generated dialogue: Use the trained BART model to generate responses for each user input in the test dataset; Calculate the BLEU metric: measure the quality of the generated text by comparing the similarity between the generated text and the reference text: Where: N represents the maximum length of the n-gram under consideration, which is 4; w n Represents the weight of the n-th n-gram, which is p n represents the accuracy of the nth n-gram; BP represents the length penalty factor; in: Where: c represents the length of the generated text, r represents the length of the reference text; e represents the base of the natural logarithm; Count clip (n-gram) represents the count of the number of occurrences of the n-gram in the generated text truncated by the number of occurrences of the same n-gram in the reference text; Count ref (n-gram) represents the count of n-grams in the reference text; C represents the candidate text set; Analyze BLEU score results: After calculating the BLEU score of each generated dialogue, the average BLEU score of the entire set of test data is calculated as the overall evaluation indicator of dialogue quality, including the following steps: BLEU score for a single conversation: Calculate the BLEU score for each user input and generated response; Average BLEU score: Calculate the average BLEU score of all dialogues: Where: M represents the size of the test data set; BLEU i represents the BLEU score of the i-th dialogue; Strategy adjustment and optimization: Based on the BLEU score results, analyze the accuracy and coherence of the responses generated by the model, determine the direction that needs to be optimized, and optimize the corresponding strategy based on the optimization direction.

Images