Power system language understanding pre-training method based on BERT and reinforcement learning

Abstract

The invention belongs to the technical field of electric power, and relates to an electric power system language understanding pre-training method based on BERT and reinforcement learning, which provides a professional corpus basis for a model by collecting and preprocessing data in the field of electric power; power proprietary vocabularies are added into the BERT model and fine tuning is carried out, so that the understanding ability of the model to domain terms is improved; a power system knowledge graph is constructed and embedded into the model, and domain knowledge representation of the model is enhanced; by defining a task environment and applying reinforcement learning, optimization of the model on a specific power task is realized; according to the method, limited field resources are fully utilized, the advantages of the pre-training model, the knowledge graph and reinforcement learning are combined, the language understanding ability of the BERT model in the professional field of the power system is effectively improved, and a new thought is provided for solving the problem of natural language processing in the professional field.

Description

Technical Field

[0001] The present application belongs to the field of electric power technology, and more specifically, to a language understanding pre-training method for electric power systems based on BERT and reinforcement learning. Background Art

[0002] In recent years, with the rapid development of natural language processing technology, pre-trained language models represented by BERT have achieved remarkable results in general-purpose text understanding tasks. These models have learned rich language knowledge and representation capabilities through pre-training on large-scale general text corpora, and can be applied to a variety of downstream tasks.

[0003] However, when these pre-trained models are directly applied to professional fields such as power systems, they often face the following challenges:

[0004] Insufficient domain vocabulary coverage: The vocabulary of general pre-trained models is difficult to cover the professional terms and concepts of power systems, resulting in insufficient understanding of key information.

[0005] Lack of domain knowledge: The model lacks understanding of the professional knowledge of the power system and it is difficult to accurately grasp the professional meaning and logical relationship contained in the text.

[0006] Insufficient task adaptability: Specific language understanding tasks in power systems (such as fault diagnosis, equipment status analysis, etc.) are significantly different from general tasks, and directly applying pre-trained models has poor results.

[0007] Scarce training data: Compared with general fields, labeled data for power systems is often scarce, making it difficult to support large-scale supervised learning.

[0008] To solve these problems, researchers have tried various methods to improve the performance of pre-trained models in professional fields, such as domain adaptive pre-training, knowledge injection, etc. However, these methods often require a large amount of domain data or knowledge resources and are difficult to optimize for specific tasks.

[0009] Therefore, how to effectively improve the language understanding ability of pre-trained models such as BERT in the professional field of power systems under limited domain resource conditions has become a key technical problem that needs to be solved urgently. Summary of the invention

[0010] The present invention provides a pre-training method for language understanding of power systems based on BERT and reinforcement learning, which aims to solve the technical problem of how to effectively improve the language understanding ability of pre-training models such as BERT in the professional field of power systems under limited domain resource conditions.

[0011] The power system language understanding pre-training method based on BERT and reinforcement learning includes the following steps:

[0012] Step 1: Collect power system documents, domain dialogue data, and other relevant power field text data, and pre-process the collected data to obtain power system field data;

[0013] Step 2: Use the pre-trained BERT-base model as the base model, add the special terms and terminology in the power system field to the vocabulary of the BERT-base model, and train the model on the collected power system field data to obtain the fine-tuned BERT-base model;

[0014] Step 3: Build a knowledge graph of the power system and embed the relationships and attributes in the knowledge graph into the fine-tuned BERT-base model;

[0015] Step 4: Define the environment of the power system language processing task, including states, actions, and rewards, and construct a reward function based on the task goal;

[0016] Step 5: Based on the defined environment of the power system language processing task, reinforcement learning is applied on the power task for optimization.

[0017] The present invention effectively solves the problem of improving the language comprehension ability of the BERT model in the professional field of power systems under the condition of limited domain resources through a multi-step strategy. First, by collecting and preprocessing data in the power field, a professional corpus basis is provided for the model. Secondly, the power-specific vocabulary is added to the BERT model and fine-tuned to improve the model's understanding of domain terminology. Furthermore, a power system knowledge graph is constructed and embedded in the model to enhance the model's domain knowledge representation. Finally, by defining the task environment and applying reinforcement learning, the model is optimized for specific power tasks. This method makes full use of limited domain resources, combines the advantages of pre-trained models, knowledge graphs, and reinforcement learning, effectively improves the language comprehension ability of the BERT model in the professional field of power systems, and provides a new idea for solving natural language processing problems in professional fields.

[0018] Preferably, step 2 comprises the following steps:

[0019] Collecting special terms and terminology in the power system field: Calculate word frequency and TF-IDF value based on power system field data, identify high-frequency and important terms, and supplement and verify the term list by combining existing power system field dictionaries and opinions of field experts;

[0020] Constructing domain vocabulary: Merge the identified power domain terms with the initial vocabulary of the BERT-base model, and initialize the word embedding of new words using the average word embedding method;

[0021] Vocabulary expansion: Initialize the embedding vector for the newly added words using the average embedding method based on similar words;

[0022] Word segmentation: Use the word segmenter of the BERT-base model to convert the text into a word sequence, annotate the data according to the task type, and generate corresponding labels;

[0023] Data segmentation: divide the data into training set, validation set and test set according to the predetermined ratio;

[0024] Set training parameters: set learning rate parameters, batch size and training rounds;

[0025] Start training: Load the pre-trained BERT-base model based on the set training parameters, perform fine-tuning training on power system domain data, and use the Adam optimizer to update parameters. After each training round, evaluate the model performance on the validation set and adjust the hyperparameters.

[0026] Save model: Save the fine-tuned BERT-base model weights in a file and save the model's configuration file.

[0027] Preferably, in step 2, the BERT-base model is trained using a cross entropy loss function:

[0028]

[0029] Where: Loss cross-entropy Represents the value of the cross entropy loss function; y i represents the actual label of the i-th sample; represents the probability distribution of the model prediction of the i-th sample; N represents the number of samples.

[0030] Preferably, step 3 comprises the following steps:

[0031] Data collection and collation: Collect structured and unstructured data related to the power system, and pre-process the collected data;

[0032] Named Entity Recognition: Use the fine-tuned BERT-base model to perform named entity recognition and extract key entities in the power system;

[0033] Relation extraction: Based on the fine-tuned BERT-base model, the relationship between entities is automatically extracted from the text;

[0034] Graph construction: The identified entities are used as nodes and relationships as edges to construct a knowledge graph. Each node contains the attribute information of the entity, and each edge contains the attribute information of the relationship.

[0035] Embedding training: prepare training data, including triplets of nodes and their relationships in the knowledge graph;

[0036] Model training: Use the TransE model to perform embedding training on the knowledge graph. By optimizing the objective function, the vector representations of adjacent nodes and relationships are close in the embedding space.

[0037] Generate vector representation: After training, generate a corresponding low-dimensional vector representation for each entity and relationship as the embedding vector of the knowledge graph;

[0038] Model architecture design: Build a fusion model, concatenate the output of the BERT-base model with the knowledge graph embedding vector, and input it into a fully connected layer. Then perform nonlinear transformation through the activation function, and finally input the prediction result.

[0039] Training process: Train the fusion model on power system domain data and fine-tune the parameters of the BERT-base model and the fully connected layer;

[0040] Optimization objective: Based on the constructed loss function, optimize the language understanding task of the power system.

[0041] Preferably, the optimization objective function adopts a distance-based loss function:

[0042] L TransE =∑ (h,ry)∈S ∑ (h′,r′y′)∈S′ [γ+||h+rt||2-||h′+r′-t′||2] + ;

[0043] Where: S represents the set of all correct triplets; S′ represents the set of negative samples; γ represents the hyperparameter; [x] + represents the larger value between x and 0; h, r, and t are the vector representations of entity h, relation r, and entity t, respectively;

[0044] The loss function constructed in the optimization objective is as follows:

[0045] L total =αL TransE +βLoss cross-entropy ;

[0046] Where: α and β represent weight parameters respectively; Loss cross-entropy Represents the value of the cross entropy loss function.

[0047] Preferably, step 4 comprises the following steps:

[0048] State: represents the current information and environment in the power system language processing task, including the text or dialogue fragment currently being processed, the entities and their attributes extracted from the text fragment, the current context information, the embedding vectors of related entities and relations in the knowledge graph, and the current progress of the task. The state is represented as follows:

[0049]

[0050] Where: text t Represents the text segment currently being processed; entities t Represents the entity set extracted from the current text segment; context t Represents the current context information; Represents the knowledge graph embedding vector; Indicates the current progress information of the task;

[0051] Actions represent the actions taken in the current state, including extracting entities, identifying relationships, updating knowledge graphs, and generating answers, as shown below:

[0052] a t =(Extract Entities ,Identify Relationships ,Update KnowledgeGraph ,Generate Response );

[0053] Where: Extract Entities Indicates entity extraction, extracting key entities from the current text segment; Identify Relationships Represents identification relationship, identifying relationship between extracted entities; Update KnowledgeGraph Indicates updating the knowledge graph and adding the newly extracted entities and relationships to the knowledge graph; Generate Response Generate an answer, which generates a response to the user's query based on the current state;

[0054] Reward: The reward function is constructed as follows:

[0055]

[0056] Where: R represents the total reward; T represents the total number of steps to complete the task; γ represents the discount factor;

[0057] r t =α·accuracy t +β·relevance t -λ·steps t ;

[0058] Where: α, β, λ represent weight parameters; accuracy t Indicates the accuracy score of the current step; relevance t Indicates the relevance score of the current generated answer; steps t Indicates the current number of steps. The more steps, the lower the reward.

[0059] Preferably, the PPO algorithm is used for training in step 5, and the specific steps are as follows:

[0060] Initialization: Initialize the parameters θ and φ of the policy network and the value network. The policy network is used to predict the probability distribution of actions under a given state, and the value network is used to predict the value function under a given state.

[0061] Simulation interaction: In the defined power system language processing task environment, the interaction process between the model and the environment is simulated; at each time step t, the model changes from the current state s t Start by selecting an action a through the policy network t , and then the environment returns the next state s according to the action t+1 and instant rewards t ;

[0062] Collect data: record the state, action, reward and next state during the interaction to form an interaction trajectory y:(s0,a0,r0,s1,a1,r1,…,s T );

[0063] Compute advantage function: Use the value network to estimate the advantage function:

[0064] A(s t ,a t )=Q(s t ,a t )-V(s t )≈r t +γV(s t+1 )-V(s t );

[0065] Where: V(s t ) represents the state value predicted by the value network, and γ represents the discount factor;

[0066] Calculate the objective function: Limit the step size of the policy update by clipping the objective function:

[0067]

[0068] Where: π θ Indicates the current strategy; represents the old strategy; ∈ represents the pruning range parameter;

[0069] Update the policy and value network: Use the gradient ascent method to maximize the pruned objective function L CLIP (θ), update the policy network parameters θ;

[0070] Update the value network parameters φ by minimizing the mean squared error loss:

[0071]

[0072] Where: represents the target value calculated by the advantage function;

[0073] Iterative update: In each training iteration, the steps of collecting data to update the policy and value networks are repeated, new interaction data is collected, new advantage functions and objective functions are calculated, and the parameters of the policy and value networks are updated.

[0074] The beneficial effects of the present invention include:

[0075] The present invention effectively solves the problem of improving the language comprehension ability of the BERT model in the professional field of power systems under the condition of limited domain resources through a multi-step strategy. First, by collecting and preprocessing data in the power field, a professional corpus basis is provided for the model. Secondly, the power-specific vocabulary is added to the BERT model and fine-tuned to improve the model's understanding of domain terminology. Furthermore, a power system knowledge graph is constructed and embedded in the model to enhance the model's domain knowledge representation. Finally, by defining the task environment and applying reinforcement learning, the model is optimized for specific power tasks. This method makes full use of limited domain resources, combines the advantages of pre-trained models, knowledge graphs, and reinforcement learning, effectively improves the language comprehension ability of the BERT model in the professional field of power systems, and provides a new idea for solving natural language processing problems in professional fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0077] Figure 1 An overall step block diagram provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0078] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0079] See also Figure 1 As shown, the best embodiment of the present invention is further described;

[0080] The power system language understanding pre-training method based on BERT and reinforcement learning includes the following steps:

[0081] Step 1: Collect power system documents, domain dialogue data, and other relevant power field text data, and pre-process the collected data to obtain power system field data;

[0082] Step 2: Use the pre-trained BERT-base model as the base model, add the special terms and terminology in the power system field to the vocabulary of the BERT-base model, and train the model on the collected power system field data to obtain the fine-tuned BERT-base model;

[0083] Step 3: Build a knowledge graph of the power system and embed the relationships and attributes in the knowledge graph into the fine-tuned BERT-base model;

[0084] Step 4: Define the environment of the power system language processing task, including states, actions, and rewards, and construct a reward function based on the task goal;

[0085] Step 5: Based on the defined environment of the power system language processing task, reinforcement learning is applied on the power task for optimization.

[0086] The present invention effectively solves the problem of improving the language comprehension ability of the BERT model in the professional field of power systems under the condition of limited domain resources through a multi-step strategy. First, by collecting and preprocessing data in the power field, a professional corpus basis is provided for the model. Secondly, the power-specific vocabulary is added to the BERT model and fine-tuned to improve the model's understanding of domain terminology. Furthermore, a power system knowledge graph is constructed and embedded in the model to enhance the model's domain knowledge representation. Finally, by defining the task environment and applying reinforcement learning, the model is optimized for specific power tasks. This method makes full use of limited domain resources, combines the advantages of pre-trained models, knowledge graphs, and reinforcement learning, effectively improves the language comprehension ability of the BERT model in the professional field of power systems, and provides a new idea for solving natural language processing problems in professional fields.

[0087] As a possible implementation of this embodiment, step 2 includes the following steps:

[0088] Collecting special terms and terminology in the power system field: Calculate word frequency and TF-IDF value based on power system field data, identify high-frequency and important terms, and supplement and verify the term list by combining existing power system field dictionaries and opinions of field experts;

[0089] Constructing domain vocabulary: Merge the identified power domain terms with the initial vocabulary of the BERT-base model, and initialize the word embedding of new words using the average word embedding method;

[0090] Vocabulary expansion: Initialize the embedding vector for the newly added words using the average embedding method based on similar words;

[0091] Word segmentation: Use the word segmenter of the BERT-base model to convert the text into a word sequence, annotate the data according to the task type, and generate corresponding labels;

[0092] Data segmentation: divide the data into training set, validation set and test set according to the predetermined ratio;

[0093] Set training parameters: set learning rate parameters, batch size and training rounds;

[0094] Start training: Load the pre-trained BERT-base model based on the set training parameters, perform fine-tuning training on power system domain data, and use the Adam optimizer to update parameters. After each training round, evaluate the model performance on the validation set and adjust the hyperparameters.

[0095] Save model: Save the fine-tuned BERT-base model weights in a file and save the model's configuration file.

[0096] In this embodiment, by collecting and identifying special terms and terminology in the field of power systems and adding them to the vocabulary of the BERT-base model, the model's ability to understand professional terminology in the power field is significantly improved; new words are initialized using average word embedding to ensure that the vector representation of the newly added words remains consistent with the original words, which helps the model better understand the semantics of new words.

[0097] As a possible implementation of this embodiment, in step 2, the BERT-base model is trained using a cross entropy loss function:

[0098]

[0099] Where: Loss cross-entropy Represents the value of the cross entropy loss function; y irepresents the actual label of the i-th sample; represents the probability distribution of the model prediction of the i-th sample; N represents the number of samples.

[0100] As a possible implementation of this embodiment, step 3 includes the following steps:

[0101] Data collection and collation: Collect structured and unstructured data related to the power system, and pre-process the collected data;

[0102] Named Entity Recognition: Use the fine-tuned BERT-base model to perform named entity recognition and extract key entities in the power system;

[0103] Relation extraction: Based on the fine-tuned BERT-base model, the relationship between entities is automatically extracted from the text;

[0104] Graph construction: The identified entities are used as nodes and relationships as edges to construct a knowledge graph. Each node contains the attribute information of the entity, and each edge contains the attribute information of the relationship.

[0105] Embedding training: prepare training data, including triplets of nodes and their relationships in the knowledge graph;

[0106] Model training: Use the TransE model to perform embedding training on the knowledge graph. By optimizing the objective function, the vector representations of adjacent nodes and relationships are close in the embedding space.

[0107] Generate vector representation: After training, generate a corresponding low-dimensional vector representation for each entity and relationship as the embedding vector of the knowledge graph;

[0108] Model architecture design: Build a fusion model, concatenate the output of the BERT-base model with the knowledge graph embedding vector, and input it into a fully connected layer. Then perform nonlinear transformation through the activation function, and finally input the prediction result.

[0109] Training process: Train the fusion model on power system domain data and fine-tune the parameters of the BERT-base model and the fully connected layer;

[0110] Optimization objective: Based on the constructed loss function, optimize the language understanding task of the power system.

[0111] In this embodiment, by constructing a power system knowledge graph, the domain knowledge is represented in a structured manner, providing rich background information for the model, and using a fine-tuned BERT-base model for named entity recognition and relationship extraction, thereby improving the accuracy and efficiency of knowledge graph construction; the TransE model is used for knowledge graph embedding training to generate a low-dimensional vector representation, which effectively captures the semantic information between entities and relationships, and merges the knowledge graph embedding vector with the output of the BERT-base model, significantly enhancing the model's ability to understand the power system domain knowledge.

[0112] As a possible implementation of this embodiment, the optimization objective function adopts a distance-based loss function:

[0113] L TransE =∑ (h,ry)∈S ∑ (h′,r′y′)∈S′ [γ+||h+rt||2-||h′+r′-t′||2] + ;

[0114] Where: S represents the set of all correct triplets; S′ represents the set of negative samples; γ represents the hyperparameter; [x] + represents the larger value between x and 0; h, r, and t are the vector representations of entity h, relation r, and entity t, respectively;

[0115] The loss function constructed in the optimization objective is as follows:

[0116] L total =αL TransE +βLoss cross-entropy ;

[0117] Where: α and β represent weight parameters respectively; Loss cross-entropy Represents the value of the cross entropy loss function.

[0118] In this embodiment, a distance-based loss function (TransE) is used to optimize the knowledge graph embedding, so that related entities and relationships maintain approximate relationships in the embedding space, thereby improving the quality of knowledge representation. The overall loss function is constructed by combining TransE loss and cross entropy loss, thereby achieving effective fusion of knowledge graph information and text information, and comprehensively improving the performance of the model in the power system language understanding task. By adjusting the weight parameters α and β, the importance of knowledge graph information and text information in the model learning process can be flexibly controlled to adapt to different types of power system language understanding tasks.

[0119] As a possible implementation of this embodiment, step 4 includes the following steps:

[0120] State: represents the current information and environment in the power system language processing task, including the text or dialogue fragment currently being processed, the entities and their attributes extracted from the text fragment, the current context information, the embedding vectors of related entities and relations in the knowledge graph, and the current progress of the task. The state is represented as follows:

[0121]

[0122] Where: text t Represents the text segment currently being processed; entities t Represents the entity set extracted from the current text segment; context t Represents the current context information; Represents the knowledge graph embedding vector; Indicates the current progress information of the task;

[0123] Actions represent the actions taken in the current state, including extracting entities, identifying relationships, updating knowledge graphs, and generating answers, as shown below:

[0124] a t =(Extract Entities ,Identify Relttioanships ,Update KnowledgeGraph ,Generate Response );

[0125] Where: Extract Entities Indicates entity extraction, extracting key entities from the current text segment; Identify Relationships Represents identification relationship, identifying relationship between extracted entities; Update KnowledgeGraph Indicates updating the knowledge graph and adding the newly extracted entities and relationships to the knowledge graph; Generate Response Generate an answer, which generates a response to the user's query based on the current state;

[0126] Reward: The reward function is constructed as follows:

[0127]

[0128] Where: R represents the total reward; T represents the total number of steps to complete the task; γ represents the discount factor;

[0129] r t =α·accuracy t +β·relevance t -λ·steps t ;

[0130] Where: α, β, λ represent weight parameters; accuracy t Indicates the accuracy score of the current step; relevance t Indicates the relevance score of the current generated answer; steps t Indicates the current number of steps. The more steps, the lower the reward.

[0131] In this embodiment, the reward function design takes into account multiple factors such as accuracy, relevance and number of steps. This multi-dimensional reward mechanism can evaluate the performance of the model more comprehensively. By adjusting the weight parameters α, β and λ, different goals can be flexibly balanced according to specific task requirements, thereby guiding the model to make optimal decisions in different scenarios. By incorporating the knowledge graph embedding vector into the state representation, the model can use domain knowledge to understand and deal with new and unseen situations. This learning method that combines domain knowledge helps to improve the generalization ability of the model, enabling it to better cope with various complex situations in the power system. The action space covers the complete process from entity extraction to answer generation, enabling the model to achieve end-to-end task processing. This integrated processing method can reduce information loss in the intermediate links and improve the overall task execution efficiency.

[0132] As a possible implementation of this embodiment, the PPO algorithm is used for training in step 5, and the specific steps are as follows:

[0133] Initialization: Initialize the parameters θ and φ of the policy network and the value network. The policy network is used to predict the probability distribution of actions under a given state, and the value network is used to predict the value function under a given state.

[0134] Simulation interaction: In the defined power system language processing task environment, the interaction process between the model and the environment is simulated; at each time step t, the model changes from the current state s t Start by selecting an action a through the policy network t , and then the environment returns the next state s according to the action t+1 and instant rewards t ;

[0135] Collect data: record the state, action, reward and next state during the interaction to form an interaction trajectory y:(s0,a0,r0,s1,a1,r1,…,s T );

[0136] Compute advantage function: Use the value network to estimate the advantage function:

[0137] A(s t ,a t )=Q(s t ,a t )-V(st )≈r t +γV(s t+1 )-V(s t );

[0138] Where: V(s t ) represents the state value predicted by the value network, and γ represents the discount factor;

[0139] Calculate the objective function: Limit the step size of the policy update by clipping the objective function:

[0140]

[0141] Where: π θ Indicates the current strategy; represents the old strategy; ∈ represents the pruning range parameter;

[0142] Update the policy and value network: Use the gradient ascent method to maximize the pruned objective function L CLIP (θ), update the policy network parameters θ;

[0143] Update the value network parameters φ by minimizing the mean squared error loss:

[0144]

[0145] Where: represents the target value calculated by the advantage function;

[0146] Iterative update: In each training iteration, the steps of collecting data to update the policy and value networks are repeated, new interaction data is collected, new advantage functions and objective functions are calculated, and the parameters of the policy and value networks are updated.

[0147] In this embodiment, the PPO algorithm is used for training. By clipping the objective function to limit the step of policy update, performance collapse caused by excessive policy updates can be effectively prevented. At the same time, the advantage function is estimated using the value network, which can reduce variance and improve the stability and efficiency of training.

[0148] By clearly defining states, actions, and rewards, the model's decision-making process becomes more transparent and explainable. This is particularly important for critical infrastructure areas such as power systems, as it helps operators understand and verify the basis for the model's decisions; the adoption of the PPO algorithm enables the model to use training data more effectively. By using the same batch of data multiple times for multiple rounds of strategy optimization, better learning results can be achieved with limited training resources, which is of great significance for the data scarcity problem that may exist in the power sector.

[0149] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A power system language understanding pre-training method based on BERT and reinforcement learning, characterized in that: The following steps are involved: Step 1: Collect power system documents, domain dialogue data, and other relevant power field text data, and pre-process the collected data to obtain power system field data; Step 2: Use the pre-trained BERT-base model as the base model, add the special terms and terminology in the power system field to the vocabulary of the BERT-base model, and train the model on the collected power system field data to obtain the fine-tuned BERT-base model; Step 3: Build a knowledge graph of the power system and embed the relationships and attributes in the knowledge graph into the fine-tuned BERT-base model; Step 4: Define the environment of the power system language processing task, including states, actions, and rewards, and construct a reward function based on the task goal; Step 5: Based on the defined environment of power system language processing tasks, reinforcement learning is applied on the power task for optimization.

2. The power system language understanding pre-training method based on BERT and reinforcement learning according to claim 1, characterized in that: The step 2 comprises the following steps: Collecting special terms and terminology in the power system field: Calculate word frequency and TF-IDF value based on power system field data, identify high-frequency and important terms, and supplement and verify the term list by combining existing power system field dictionaries and opinions of field experts; Constructing domain vocabulary: Merge the identified power domain terms with the initial vocabulary of the BERT-base model, and initialize the word embedding of new words using the average word embedding method; Vocabulary expansion: Initialize the embedding vector for the newly added words using the average embedding method based on similar words; Word segmentation: Use the word segmenter of the BERT-base model to convert the text into a word sequence, annotate the data according to the task type, and generate corresponding labels; Data segmentation: divide the data into training set, validation set and test set according to the predetermined ratio; Set training parameters: set learning rate parameters, batch size and training rounds; Start training: Load the pre-trained BERT-base model based on the set training parameters, perform fine-tuning training on power system domain data, and use the Adam optimizer to update parameters. After each training round, evaluate the model performance on the validation set and adjust the hyperparameters. Save model: Save the fine-tuned BERT-base model weights in a file and save the model's configuration file.

3. The power system language understanding pre-training method based on BERT and reinforcement learning according to claim 2, characterized in that: In step 2, the BERT-base model is trained using a cross entropy loss function: Where: Loss cross-entropy Represents the value of the cross entropy loss function; y i represents the actual label of the i-th sample; represents the probability distribution of the model prediction of the i-th sample; N represents the number of samples.

4. The power system language understanding pre-training method based on BERT and reinforcement learning according to claim 1, characterized in that: The step 3 comprises the following steps: Data collection and collation: Collect structured and unstructured data related to the power system and pre-process the collected data; Named Entity Recognition: Use the fine-tuned BERT-base model to perform named entity recognition and extract key entities in the power system; Relation extraction: Based on the fine-tuned BERT-base model, the relationship between entities is automatically extracted from the text; Graph construction: The identified entities are used as nodes and relationships as edges to construct a knowledge graph. Each node contains the attribute information of the entity, and each edge contains the attribute information of the relationship. Embedding training: prepare training data, including triplets of nodes and their relationships in the knowledge graph; Model training: Use the TransE model to perform embedding training on the knowledge graph. By optimizing the objective function, the vector representations of adjacent nodes and relationships are close in the embedding space. Generate vector representation: After training, generate a corresponding low-dimensional vector representation for each entity and relationship as the embedding vector of the knowledge graph; Model architecture design: Build a fusion model, concatenate the output of the BERT-base model with the knowledge graph embedding vector, and input it into a fully connected layer. Then perform nonlinear transformation through the activation function, and finally input the prediction result. Training process: Train the fusion model on power system domain data and fine-tune the parameters of the BERT-base model and the fully connected layer; Optimization objective: Based on the constructed loss function, optimize the language understanding task of the power system.

5. The power system language understanding pre-training method based on BERT and reinforcement learning according to claim 4, characterized in that: The optimization objective function adopts a distance-based loss function: L TransE =∑ (h,ry)∈s ∑ (h′,r′y′)∈S′ [γ+||h+r-t||2-||h′+r′-t′||2] + ; Where: S represents the set of all correct triplets; S′ represents the set of negative samples; γ represents the hyperparameter; [x] + represents the larger value between x and 0; h, r, and t are the vector representations of entity h, relation r, and entity t, respectively; The loss function constructed in the optimization objective is as follows: L total =αL transE +βLoss cross-entropy ; Where: α and β represent weight parameters respectively; Loss cross-entropy Represents the value of the cross entropy loss function.

6. The power system language understanding pre-training method based on BERT and reinforcement learning according to claim 1, characterized in that: The step 4 comprises the following steps: State: represents the current information and environment in the power system language processing task, including the text or dialogue fragment currently being processed, the entities and their attributes extracted from the text fragment, the current context information, the embedding vectors of related entities and relations in the knowledge graph, and the current progress of the task. The state is represented as follows: Where: text t Represents the text segment currently being processed; entities t Represents the entity set extracted from the current text segment; context t Represents the current context information; Represents the knowledge graph embedding vector; Indicates the current progress information of the task; Actions represent the actions taken in the current state, including extracting entities, identifying relationships, updating knowledge graphs, and generating answers, as shown below: a t =(Extract Entities ,Identify Relationships ,Update Knowtedge Graph ,Generate Response ); Where: Extract Entities Indicates entity extraction, extracting key entities from the current text segment; Identify Relationships Represents identification relationship, identifying relationship between extracted entities; Update Knowtedge Graph Indicates updating the knowledge graph and adding the newly extracted entities and relationships to the knowledge graph; Generate Response Generate an answer, which generates a response to the user's query based on the current state; Reward: The reward function is constructed as follows: Where: R represents the total reward; T represents the total number of steps to complete the task; γ represents the discount factor; r t =α·accuracy t +β·relevance t -λ·steps t ; Where: α, β, λ represent weight parameters; accuracy t Represents the accuracy score of the current step; relevance t Indicates the relevance score of the current generated answer; steps t Indicates the current number of steps. The more steps, the lower the reward.

7. The power system language understanding pre-training method based on BERT and reinforcement learning according to claim 1, characterized in that: In step 5, the PPO algorithm is used for training, and the specific steps are as follows: Initialization: Initialize the parameters θ and φ of the policy network and the value network. The policy network is used to predict the probability distribution of actions under a given state, and the value network is used to predict the value function under a given state. Simulation interaction: In the defined power system language processing task environment, the interaction process between the model and the environment is simulated; at each time step t, the model changes from the current state s t Start by selecting an action a through the policy network t , and then the environment returns the next state s according to the action t+1 and instant rewards t ; Collect data: record the state, action, reward and next state during the interaction to form an interaction trajectory y:(s0,a0,r0,s1,a1,r1,…,s T ); Compute advantage function: Use the value network to estimate the advantage function: A(s t ,a t )=Q(s t ,a t )-V(s t )≈r t +γV(s t+1 )-V(s t ); Where: V(s t ) represents the state value predicted by the value network, and γ represents the discount factor; Calculate the objective function: Limit the step size of the policy update by clipping the objective function: Where: π θ Indicates the current strategy; represents the old strategy; ∈ represents the pruning range parameter; Update the policy and value network: Use the gradient ascent method to maximize the pruned objective function L CLIP (θ), update the policy network parameters θ; Update the value network parameters φ by minimizing the mean squared error loss: Where: represents the target value calculated by the advantage function; Iterative update: In each training iteration, the steps of collecting data to update the policy and value networks are repeated, new interaction data is collected, new advantage functions and objective functions are calculated, and the parameters of the policy and value networks are updated.

Images