Relation extraction method in power grid dispatching field combining semantic dependency and part-of-speech embedding

Abstract

This paper proposes a method for extracting relationships in the field of power grid dispatching that combines semantic dependency and part-of-speech embedding. The method includes the following steps: S1, collecting power grid dispatching data, then performing entity and relationship annotation to obtain a data set in the field of power grid dispatching; S2, inputting the data set into the RoBERTa‑CE model for model training, and learning entity information and relationship information separately; S3, performing entity-relationship multi-task learning, first concatenating the entity context information to make entity predictions, and using cross-entropy loss to calculate the entity loss value, then concatenating the relationship context to make relationship predictions, and using cross-entropy loss to calculate the relationship loss value; adding the entity loss value and the relationship loss value, and jointly participating in the back propagation of the model, so that the overall loss value is optimized towards the minimum value until all tasks converge; S4, predicting the relationship through the fully connected layer. This method can deeply mine power grid dispatching data and quickly extract the relationship between entities in the field of power grid dispatching.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a relationship extraction method in the field of power grid scheduling that combines semantic dependency and part-of-speech embedding. Background Art

[0002] The concept of relation extraction (RE) was proposed at the MUC conference in 1988. It is one of the basic tasks of information extraction. Its purpose is to identify target relations in text entities. It is an important technical link in building knowledge graphs.

[0003] Relation extraction corresponds to the edges between entities in the graph network. We embed semantic dependency information and part-of-speech information into RoBERTa to allow the model to learn the dependencies between different words.

[0004] Relationship extraction presents the problem of multi-relationship mapping. Constructing a knowledge graph typically requires extracting unstructured information, specifically entity relationship extraction. Due to the complex expertise inherent in power system empirical knowledge, multi-relationship mapping is common. Identifying these multiple relationships is a critical issue that needs to be addressed.

[0005] By performing named entity recognition on unstructured data in a specific domain and then extracting relationships between the identified entities, a simple knowledge graph can be constructed. In the field of deep learning, named entity recognition and relationship extraction are important tasks in natural language processing. Currently, researchers are using joint extraction and pipeline models to solve these tasks.

[0006] Joint extraction can take into account both entities and relationships. Yang Yanyun converted the joint extraction of entity relationships in traditional Chinese medicine into a sequence labeling task, concatenating word vectors and character vectors in parallel as bidirectional LSTM-CRF inputs to ultimately extract relationships. Jiang Bo proposed a BERT+BiLSTM+CRF method to improve the performance of deep learning methods in causal relationship extraction tasks. Zhang et al. proposed an attention mechanism model that incorporates position awareness into the long short-term memory (LSTM) neural network, and the model achieved good results.

[0007] The other approach is the pipeline model, where entity recognition and relation extraction are treated as two independent tasks and handled separately. Pipeline methods are widely used because they do not require manual feature construction. Named entity recognition and relation extraction are performed separately. Recent research on pipeline models has improved error propagation, achieving recognition results superior to joint extraction models. Zhong divides English vocabulary into root words and uses span annotation to concatenate the enumerated candidate entities with sentences as a training example, effectively improving the accuracy of relation extraction in downstream tasks. Ye proposed a neighborhood-oriented bundling strategy that bundles spans with the same starting token into a single training example whenever possible to better distinguish entity boundaries. This strategic bundling allows for relation extraction. These two pipeline models based on span representation can achieve state-of-the-art performance by fine-tuning BERT. Furthermore, pipeline models can be tuned separately for the two tasks and are highly interpretable, allowing for separate analysis and explanation of the entity and relation extraction processes. This article focuses on methods related to relation extraction.

[0008] The field of power grid dispatching involves complex terminology. General-purpose relationship extraction methods cannot accurately capture deep semantic representations within specialized domains, making it difficult to model relationships between entities. Furthermore, regional smart grid systems record a vast amount of unstructured behavioral data. From this data, textual data such as accident investigation information, audit risk statistics, on-site inspection information, and device operation status are selected to construct a power corpus. Currently, systems have low utilization rates for this data, supporting only simple queries on the text without deep mining, and thus failing to fully exploit the underlying behavioral knowledge. Current manual mining methods are inefficient and costly. Summary of the Invention

[0009] The present invention aims to at least solve the technical problems existing in the prior art, and in particular innovatively proposes a relationship extraction method in the field of power grid scheduling that combines semantic dependency and part-of-speech embedding.

[0010] In order to achieve the above-mentioned object of the present invention, the present invention provides a method for extracting relations in the field of power grid dispatching by combining semantic dependency and part-of-speech embedding, comprising the following steps:

[0011] S1: Collect power grid dispatch data and remove sentences with unclear meaning, incomplete structure, and repeated semantics. Then, perform entity and relationship annotation to obtain a data set in the field of power grid dispatch.

[0012] S2, input the dataset into the RoBERTa-CE model for model training, learning entity information and relationship information separately. The RoBERTa-CE model includes an embedding layer and a 12-layer Transformer encoder;

[0013] S3 performs entity-relationship multi-task learning. First, the entity context information is concatenated to make entity predictions, and the entity loss value is calculated using cross-entropy loss. Then, the relationship context is concatenated to make relationship predictions, and the relationship loss value is calculated using cross-entropy loss. The entity loss value and the relationship loss value are added together and jointly participate in the back propagation of the model, so that the overall loss value is optimized towards the minimum value until all tasks converge.

[0014] S4, through the fully connected layer, predicts the relationship.

[0015] Furthermore, inputting the data set into the RoBERTa-CE model for model training includes the following steps:

[0016] S2-1, the data passes through the embedding layer to learn context features and obtain word embedding code;

[0017] S2-2, then passes through the Transformer encoder to model the word embedding encoded information and obtain the output value of the hidden layer.

[0018] Furthermore, S2-1 includes:

[0019] Input the data into the embedding layer to obtain the word embedding X containing position information, paragraph information, word encoding information, semantic dependency information and part of speech embedding embedding :

[0020] X embedding =X word +X segment +X positional +X semantics +X phrase #(1)

[0021] X word 、X segment 、X positional 、X semantics 、X phrase Represent word embedding information, segment embedding information, position embedding information, semantic dependency embedding information and part-of-speech embedding information respectively;

[0022] The semantic dependency information and part-of-speech embedding are obtained through the semantic dependency vocabulary and the part-of-speech tagging vocabulary respectively. They are obtained by converting the text into index values ​​in the table after semantic dependency tagging and part-of-speech tagging.

[0023] Furthermore, in S2-2, based on the Transformer encoder architecture, the word embedding information is modeled through a deep model structure, including:

[0024] First, embed the word X embeddingWeight learning through attention mechanism:

[0025]

[0026] where Q, K, V = X embedding #(2)

[0027] Attention() represents the attention mechanism;

[0028] Q, K, and V represent query, key, and value respectively;

[0029] d k Indicates the dimension size of K;

[0030] where Q, K, and V represent query, key, and value, and all three vectors are X embedding ;

[0031] Word embedding X embedding Perform multi-head attention learning to learn multi-channel information:

[0032] MultiHead(X embedding , X embedding , X embedding )=Concat(head1,...,head h )W O

[0033] where head i =Attention(X embedding , X embedding , X embedding )#(3)

[0034] MultiHead(X embedding , X embedding , X embedding ) means taking three identical inputs as attention and learning different weight information, MultiHead() is a multi-head attention mechanism;

[0035] Concat(head1, ..., head h ) means adding the attention of heads 1 to h;

[0036] W o Represents a weight parameter;

[0037] where head i represents the value of the i-th attention head;

[0038] Attention(X embedding , X embedding , Xembedding ) represents the calculation of attention;

[0039] In order to prevent gradient explosion and gradient disappearance caused by deep models, word embedding X embedding The vector is residually connected with the multi-head attention:

[0040] sy=X embedding +MultiHead(X embedding , X embedding , X embedding )#(4)

[0041] sy represents the embedding layer X embedding The value after the residual connection of multi-head attention;

[0042] Then perform layer normalization on sy, calculate the mean and variance on each sample, normalize the hidden layer in the neural network to the standard normal distribution, and accelerate convergence:

[0043]

[0044]

[0045]

[0046] ∈ is a very small value;

[0047] LN(sy) represents layer normalization;

[0048] μ L represents the mean;

[0049] represents the standard deviation;

[0050] The scaling parameters α and β are learnable parameters;

[0051] m is the number of neurons;

[0052] Then, the normalized output of the layer is passed through the Feed Forward neural network:

[0053] FFN(x)=max(0,xW1+b1)W2+b2#(6)

[0054] Where W1 and W2 represent two weight parameters respectively;

[0055] b1 and b2 represent two bias parameters respectively;

[0056] x represents the normalized output of the layer LN(sy);

[0057] Finally, after residual connection and layer normalization:

[0058] H=FFN(x)+x#(7)

[0059] The output value H of the 12-layer Transformer encoder can be obtained. The model undergoes multiple rounds of training and gradient optimization through the AdamW optimizer.

[0060] Furthermore, S4 includes:

[0061] In order to strengthen the boundary features of candidate spans and connect them more closely with text information, the representations of the start and end positions of the span sets of the object entity and the subject entity are spliced ​​together. The formula for the object entity is as follows:

[0062] h_o start =H 12 [:,O start ]#(8)

[0063] h_o end =H 12 [:,O end ]#(9)

[0064] obj=Concat(h_o start , h_o end )#(10)

[0065] H 12 Represents the output of the last layer of RoBERTa;

[0066] H 12 [:,O start ] means only H 12 The value of the starting position;

[0067] H 12 [:,O end ] means only H 12 The value of the end position;

[0068] O start Indicates the starting position of the object entity;

[0069] O end Indicates the end position of the object entity;

[0070] obj represents the context representation of the object entity;

[0071] Concat means Concat splicing;

[0072] h_o start , h_o end Represent the representation of the starting position of the span set and the representation of the ending position of the span set respectively;

[0073] Formulas (8) and (9) obtain the trained features of the span start and end positions respectively. Formula (10) concatenates these two features so that the model can include relevant information of the object entity and the connection of the relationship while extracting the relationship.

[0074] The formula for the object entity is as follows:

[0075] h_s start =H 12 [:,S start ]#(11)

[0076] h_s end =H 12 [:,S end ]#(12)

[0077] sub=Concat(h_s start , h_S end )#(13)

[0078] Where [:, S start ] represents a slice of the main entity;

[0079] S start Indicates the starting position of the main entity;

[0080] S end Indicates the end position of the main entity;

[0081] h_s start , h_s end Respectively represent the contextual deep semantic representation of the subject entity;

[0082] sub represents the contextual deep semantic representation of the subject entity;

[0083] The context information of the subject entity and the object entity are passed through the fully connected layer to obtain the predicted scores, and then the two are added together to obtain the predicted probabilities of various relationships. Finally, after passing through the softmax layer, the final predicted relationship type is obtained:

[0084] E p =sigmoid(W a *obj+b1)#(14)

[0085] r p =sigmoid(W b *sub+b2)#(15)

[0086] R p =E p +r p #(16)

[0087] R type =softmax(R p )#(17)

[0088] Among them E p The value of the contextual deep semantic representation of the subject entity after passing through the fully connected layer;

[0089] r p The contextual deep semantic representation of the object entity after passing through the fully connected layer;

[0090] R p Indicates E p and R p to integrate;

[0091] R typ e represents the normalized information, i.e. the predicted relationship type;

[0092] W a 、W b Represent weight parameters respectively;

[0093] b1 and b2 represent bias parameters respectively;

[0094] The object entity passes through the fully connected layer and then the softmax layer to obtain the predicted entity type:

[0095] E type =softmax(E p )#(18)

[0096] E type The entity type representing the prediction.

[0097] Furthermore, the cross entropy loss formulas for entities and relations are:

[0098] E loss =-∑(E real *log(E type )+(1-E real )*log(1-E type ))#(19)

[0099] R loss =-∑(R real *log(R type )+(1-R real )*log(1-R type ))#(20)

[0100] E loss 、R loss Represents the loss value of the predicted value and the true value of the entity and relationship respectively;

[0101] E real Represents the entity's true value;

[0102] E type Represents an entity type value;

[0103] R real Represents the true value of the relationship;

[0104] R type Represents a relationship type value.

[0105] In summary, due to the adoption of the above technical solution, the present invention can address the multi-relationship problem between entities in the field of power grid dispatching. Based on the annotation method of span representation, the subject entity and the object entity are used as a group of training instances to strengthen the connection between the two entities. The embedding layer of the RoBERTa pre-training model itself contains word embedding, position embedding and paragraph embedding information, and then introduces semantic dependency and part-of-speech embedding, which can analyze and connect words in sentences, strengthen the modeling of entities that may be related, and add part-of-speech tagging embedding to enable the model to learn more deep semantic information. After obtaining the embedding layer, the RoBERTa model is used to perform multi-task learning on entities and relationships, and a parameter hard sharing mechanism is used to fuse multi-task information. Finally, the fully connected layer is used to obtain the predicted entity relationship. This method is verified on a self-constructed data set in the field of power grid dispatching, and can significantly improve the performance of the model.

[0106] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0108] Figure 1 This is a diagram illustrating an example of the present invention.

[0109] Figure 2 It is a schematic flow diagram of the present invention.

[0110] Figure 3 This is an example diagram of the training samples of the present invention.

[0111] Figure 4 It is the semantic dependency graph of the present invention.

[0112] Figure 5 It is a schematic diagram of the Embedding layer fusion process of the model of the present invention. DETAILED DESCRIPTION

[0113] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0114] 1. Dataset construction:

[0115] This paper uses deep learning methods for analysis and modeling. To this end, we constructed a named entity recognition dataset for power grid dispatching using a power corpus for deep learning model training. The corpus used in this paper contains a large amount of unstructured data. We screened and removed sentences with ambiguous meaning, incomplete structure, and repetitive semantics, ultimately extracting 2,316 high-quality data items for training and testing. Based on the characteristics of this corpus, entity types were categorized into nine categories, including power plant, voltage level, and transmission equipment; and relationship types were categorized into five categories, including time, location, subordination, equality, and cause. This approach aims to extract information from unstructured data, facilitating the subsequent construction of knowledge graphs for applications in specific areas such as faults, maintenance, and equipment lifecycles. The dataset contains 1,971 training data items, 17,433 entities, 9,354 relationships, and over 140,000 Chinese and English characters. The training, validation, and test sets are divided in a 7:2:1 ratio.

[0116] Table 1 Relationship dataset

[0117] Type name Number of Relations time 493 reson 414 local 3317 Subordinate 2461 same 2669

[0118] 2. Methods and Model Training

[0119] This paper proposes a relationship extraction method based on span representation. The overall flow chart is as follows Figure 2 This article focuses on relation extraction. The model models entity information and relation information separately, then adds the two loss values ​​(hard parameter sharing). These two values ​​participate in the model's backpropagation, optimizing the overall loss value toward its minimum. The RoBERTa-CE model consists of an embedding layer and a Transformer encoder layer. The former converts various textual information into high-dimensional vector representations, while the latter is the training model architecture. The output of the embedding layer is input into the Transformer encoder for training.

[0120] In addition to the general segment embedding information, position embedding information, and word embedding information, the present invention also adds part-of-speech embedding information and semantic dependency embedding information to the embedding layer, so that the model can learn dependency relationships, which is conducive to strengthening the connection between entity relationships.

[0121] The methods for obtaining semantic dependencies and part-of-speech embeddings are similar. The part-of-speech vocabulary and part-of-speech embeddings are generated as follows: Use the jieba tool to perform part-of-speech tagging on the text. There are a total of 55 parts of speech. RoBERTa embedding is in the form of int or floating-point vectors. We make the 55 parts of speech into a vocabulary, namely part-of-speech words. Each part of speech corresponds to an index value, for example {"v":1,"n":2...} (verb v corresponds to index 1, noun n corresponds to index 2). A sentence s = {s1,s2,s3,...sn} is not segmented and is encoded in units of characters. The part-of-speech tagging obtains w = {w1,w2,...,wn}, which is then mapped to the index value of the part-of-speech vocabulary and embedded in the embedding layer of RoBERTa to achieve part-of-speech embedding.

[0122] By integrating semantic dependency information and part-of-speech information with the RoBERTa pre-trained model, a deeper semantic representation is achieved. Using the RoBERTa-CE model, entity and relation information are learned separately. A multi-task parameter hard sharing mechanism is employed. By training multiple tasks simultaneously, the model can account for the influence of different tasks. This influence is reflected in the shared parameters until all tasks converge. This balance of entity and relation information is fully leveraged, leveraging RoBERTa's deep semantic representations. Finally, fully connected layers are used to predict relations.

[0123] Annotation example diagram Figure 1 This paper adopts a span-based annotation method, using the Label-Studio annotation platform and a visual interface to annotate entities in sentences. Span annotation combines the entity's start position, end position, and entity type into a set.

[0124] Relationships are annotated as a five-tuple of span pairs and relations, namely (s1, e1, s2, e2, relation_type). The variables in the set are, in order, entity 1 start position, entity 1 end position, entity 2 start position, entity 2 end position, and relation type. One entity is selected from the annotated data as the subject entity, and the remaining entities are enumerated as object entities. Both the subject and object entities contain entity type information. The resulting set of subject and object entities, along with the textual information, serves as a training set. Different subjects are divided into different groups for parallel training, which strengthens the connections between entities. By enumerating subjects and objects, multiple relations can be identified.

[0125] Examples of training samples are Figure 3 As shown in the figure, a subject entity is selected and the rest are used as object entities to generate a training data.

[0126] 2.1 Pre-trained Language Model

[0127] The RoBERTa-wwm-ext Chinese pre-trained model was used. This model was pre-trained unsupervised on large-scale Chinese text data, learning rich prior knowledge and achieving excellent performance in many natural language processing tasks. RoBERTa is a variant of BERT, with the following changes:

[0128] 1. The dynamic masking strategy may have different mask positions for each round of training samples. Randomly select word units for masking. For the training sample "110kV Kunlun Station", the first round of training will replace the training sample with a special word unit "110kV Kunlun <mask>", the second round of training will replace the training samples with" <mask>For the 10kV Kunlun Station, the mask positions may change again in the third and fourth rounds. The dynamic strategy increases the randomness of the model's input data, ultimately improving the model's learning ability.

[0129] 2. RoBERTa uses full sentence input across documents and cancels the next sentence prediction.

[0130] Use larger training batches and pre-training data to improve the generalization ability of the model.

[0131] Compared to English, the most notable characteristic of Chinese is its fuzzy word boundaries, lacking delimiters to mark them. In English, delimiters mark word boundaries, allowing each word to have a complete meaning. Named entity recognition in the Chinese context requires word segmentation of the text. Errors in word segmentation by the word segmenter can affect the accuracy of named entity recognition. In the power dispatching field, there are no authoritative dictionaries for word segmentation, and general-purpose word segmenters have significant errors when applied to the power grid. Therefore, a word segmenter that uses character-based segmentation is used for encoding.

[0132] 3. The encoding layer converts the input text sequence into a series of high-dimensional vector representations. These vectors contain the word encoding information, paragraph information, and position information of the input text. They can model the long-distance dependencies of the input sequence and better represent the deep semantic information of the text.

[0133] 4. Input the generated training examples into the RoBERTa Embedding layer. Fuse word embedding, position embedding and paragraph embedding, introduce semantic dependency and part-of-speech embedding, and use semantic embedding to make the embedding layer obtain semantic dependency representation, and use part-of-speech embedding to make the embedding layer obtain part-of-speech representation. Figure 5 As shown in the figure, the subscripts of semantic dependency embedding and part-of-speech embedding are the index values ​​mapped to the vocabulary after processing. This allows the model to learn the connections between different entity representations, thereby improving the model's performance. The detailed process is as follows:

[0134] (1) Construct a semantic dependency word list and use the Language Technology Platform (LTP) developed by the Social Computing and Information Retrieval Research Center of Harbin Institute of Technology to perform semantic dependency word segmentation on sentences and map them to the index of the semantic dependency word list.

[0135] Table 2: Construction of semantic dependency mapping index word list (partial)

[0136] Relationship Type Tags Description Example Index agent relationship Agt Agent I gave her a bouquet of flowers (I <-- gave) 1 Describing the Character Feat Description He is fat (grow-->fat) 2 Time Role Time Time There was a Li Bai in the Tang Dynasty (Tang Dynasty <-- there was) 3 Consular relations Poss Possessor He has a good book to read (he <-- has) 4 Punctuation marks mPunc Punctuation Marker ，。！ 5 Customer Relations Cont Content He heard the sound of firecrackers (hear-->the sound of firecrackers) 6 root node Root Root Core nodes of the entire sentence 7

[0137] Semantic dependency information can be constructed as a graph. For example, the sentence "June 30, 110kV Kunlun Station Guangkun Line" is decomposed into semantic dependency information through LTP: ['TIME','TIME','TIME','TIME','TIME','mPUNC','FEAT','FEAT','FEAT','FEAT','FEAT','FEAT','FEAT','FEAT','Root','Root','Root']. Different relationship types are mapped to indexes to obtain semantic dependency embedding: [3,3,3,3,3,5,2,2,2,2,2,2,2,2,7]. Let the model learn more about semantic dependency information, such as Figure 4 As shown, the relationship between entities can be modeled more effectively.

[0138] (2) Construct a part-of-speech tagging vocabulary, use jieba word segmentation to tag sentences with parts of speech, map them to the index values ​​of the part-of-speech tagging vocabulary, and inject entities into the dictionary of jieba word segmentation to avoid errors caused by part-of-speech tagging in the power field.

[0139] Table 3 Construction of part-of-speech mapping index vocabulary (part)

[0140] Tags Part of Speech Index NN noun 1 JJ adjective 2 DT determiner 3 IN preposition 4 MD modal (modal verb) 5 RP particle 6 NN noun 7

[0141] Semantic dependency analysis is not affected by syntactic structure. It directly connects language units with direct semantic connections through dependency arcs and labels them with corresponding semantic relationships. This is also an important difference between semantic dependency analysis and syntactic dependency analysis.

[0142] Based on the two tables above, the text is annotated with semantic dependencies and part-of-speech tags, converted into index values ​​in the table, and the two vectors are embedded in the RoBERTa encoding layer.

[0143] Input the training data into the embedding layer of RoBERTa-CE to obtain the word embedding X containing position information, paragraph information, word encoding information, semantic dependency information and part-of-speech embedding embedding :

[0144] X embedding =X word +X segment +X positional +X semantics +X phrase #(1)

[0145] After training with RoBERTa, the hidden layer output is obtained. Entity-relationship multi-task learning first concatenates the entity's context information, makes entity predictions, and calculates the entity loss using cross-entropy. Then, the relationship context is concatenated, and the relationship predictions are made, using cross-entropy to calculate the relationship loss. The context refers to the entity's previous and next offset positions in the text. Concatenation is vector concatenation: two 2*3 vectors concatenated along the 0th dimension become 4*3. Multi-task learning of entity and relationship information is performed using a hard parameter sharing mechanism. The two loss values ​​are added together and contribute to the gradient optimization of the overall model.

[0146] 3. Multi-task learning based on the RoBERTa pre-trained model and combined with entity type information

[0147] After the training data is encoded through word embeddings in the RoBERTa embedding layer, certain contextual features are learned. Based on the Transformer encoder architecture, RoBERTa-CE models the information encoded by word embeddings through a deep model structure.

[0148] First, embed the word X embedding Weight learning through attention mechanism:

[0149]

[0150] where Q, K, V = X embedding #(2)

[0151] Attention() represents the attention mechanism;

[0152] Q, K, and V represent query, key, and value respectively;

[0153] d k Indicates the dimension size of K;

[0154] where Q, K, and V represent query, key, and value, and all three vectors are X embedding ;

[0155] In the RoBERTa architecture, multi-head attention learning is required for word embedding to learn multi-channel information:

[0156] MultiHead(X embedding , X embedding , X embedding )=Concat(head1,...,head h )W O

[0157] where head i =Attention(X embedding , X embedding , X embedding )#(3)

[0158] MultiHead(X embedding , X embedding , X embedding ) means taking three identical inputs as attention and learning different weight information, MultiHead() is a multi-head attention mechanism;

[0159] Concat(head1, ..., head h ) means adding the attention of heads 1 to h;

[0160] W O Represents a weight parameter;

[0161] where head i represents the value of the i-th attention head;

[0162] Attention(X embedding , X embedding , X embedding ) represents the calculation of attention;

[0163] In order to prevent gradient explosion and gradient disappearance caused by deep models, X embedding The vector is residually connected with the multi-head attention:

[0164] sy=X embedding +MultiHead(X embedding , X embedding , X embedding )#(4)

[0165] sy represents the embedding layer X embedding The value after the residual connection of multi-head attention;

[0166] Perform layer normalization on sy, calculate the mean and variance on each sample, normalize the hidden layer in the neural network to the standard normal distribution, and accelerate convergence:

[0167]

[0168]

[0169]

[0170] LN(sy) represents layer normalization;

[0171] μ L Represents the mean. It is the mean of the input feature vector in a specific dimension and is used to calculate the normalized offset.

[0172] It represents the standard deviation of the input feature vector in a specific dimension and is used to calculate the normalization scaling.

[0173] The scaling parameters α and β are learnable parameters. ∈ is a small value to prevent the equation from being divided by zero, and m is the number of neurons.

[0174] Then, the normalized output of the layer is passed through the Feed Forward neural network:

[0175] FFN(x)=max(0,xW1+b1)W2+b2#(6)

[0176] Where W1 and W2 represent two weight parameters respectively;

[0177] b1 and b2 represent two bias parameters respectively;

[0178] The above formula includes two linear transformations, where x represents the output of layer normalization LN(sy).

[0179] Finally, after residual connection and layer normalization:

[0180] H=FFN(x)+x#(7)

[0181] The output value H of the 12-layer encoder can be obtained. The model undergoes multiple rounds of training and gradient optimization through the AdamW optimizer.

[0182] The span-based data annotation format is different from the traditional sequence annotation format. In order to strengthen the boundary characteristics of the candidate span and connect it more closely with the text information, the representations of the start and end positions of the span set are spliced ​​together. The corresponding formula is as follows:

[0183] h_o start =H 12 [:,O start ]#(8)

[0184] h_o end =H 12 [:,O end ]#(9)

[0185] obj=Concat(h_o start , h_o end )#(10)

[0186] H 12 Represents the output of the last layer of RoBERTa, H 12 It is the output of "{sentence + entity set}" after passing through RoBERTa. For example, if the entity is in the form of {[10, 12], [14, 15]}, the position of the entity in the sentence is 10-12.

[0187] H 12 [:,O start ] means only H 12 The value of the starting position; example: H12[:, [10, 14]].

[0188] H 12 [:,O end ] means only H 12 The value of the ending position; example: H12[:,[12,15]].

[0189] O start Indicates the starting position of the object entity;

[0190] O end Indicates the end position of the object entity;

[0191] obj represents the context representation of the object entity;

[0192] Concat means Concat splicing;

[0193] h_o start , h_o end Represent the representation of the starting position of the span set and the representation of the ending position of the span set respectively;

[0194] Formulas (8) and (9) obtain the trained features of the span start and end positions respectively. Formula (10) concatenates these two features so that the model can include relevant information of the object entity and the connection of the relationship while extracting the relationship.

[0195] The main entity uses a similar method:

[0196] h_s start =H 12 [:,S start ]#(11)

[0197] h_s end =H 12 [:,S end ]#(12)

[0198] sub=Concat(h_s start , h_s end )#(13)

[0199] Where [:, S start ] represents a slice of the main entity;

[0200] S start Indicates the starting position of the main entity;

[0201] S end Indicates the end position of the main entity;

[0202] h_s start , h_s end Respectively represent the contextual deep semantic representation of the subject entity;

[0203] sub represents the contextual deep semantic representation of the subject entity;

[0204] The context information of the subject entity and the object entity are respectively passed through the fully connected layer to obtain the predicted scores. The two are added together to obtain the predicted probabilities of various relationships. After passing through the softmax layer, the final predicted relationship type is obtained:

[0205] E p =sigmoid(W a *obj+b1)#(14)

[0206] r p =sigmoid(W b *sub+b2)#(15)

[0207] R p =E p +r p #(16)

[0208] R typ e=softmax(R p )#(17)

[0209] Among them E p The value of the contextual deep semantic representation of the subject entity after passing through the fully connected layer;

[0210] r p The contextual deep semantic representation of the object entity after passing through the fully connected layer;

[0211] R p Indicates E p and R p to integrate;

[0212] R typ e represents the normalized information;

[0213] W a 、W b Represent weight parameters respectively;

[0214] b1 and b2 represent bias parameters respectively;

[0215] The object entity passes through the fully connected layer and then the softmax layer to obtain the predicted entity type.

[0216] E typ e=softmax(E p )#(18)

[0217] The cross-entropy loss function is a commonly used loss function for measuring the discrepancy between model output and actual labels in classification problems. It is widely used in various models, including classification models in machine learning and neural network models in deep learning. In classification problems, the cross-entropy loss function measures the discrepancy between the probability distribution of the model output and the actual label distribution. During training, the model continuously adjusts parameters using a gradient descent algorithm to minimize the overall loss function. By minimizing the cross-entropy loss function, the model can more accurately predict the class label for each example in classification problems, thereby improving model performance.

[0218] A cross entropy loss is applied between the predicted entity type and the true value, and a cross entropy loss is applied between the predicted relationship and the true value. These two loss values ​​are then added together (hard parameter sharing) and used together to optimize the model. This allows the model to take both entities and relationships into account, reducing error propagation in pipeline models.

[0219] E loss =-∑(E real *log(E type )+(1-E real )*log(1-E type ))#(19)

[0220] R loss =-∑(R real *log(R type )+(1-R real )*log(1-R type ))#(20)

[0221] loss = E loss +R loss #(twenty one)

[0222] E loss 、R loss Represents the loss value of the predicted value and the true value of the entity and relationship respectively;

[0223] E real Represents the true value of the entity, in the form of [4, 10];

[0224] E type Represents the entity type value, such as the "station" entity class;

[0225] R real Represents the true value of the relationship, such as [4, 10, 15, 16];

[0226] R type Indicates the relationship type value, such as the "local" relationship;

[0227] 4. Experimental Verification

[0228] The experimental environment used the PyTorch framework, CUDA version 11.1, Ubuntu, and an NVIDIA RTX3090 (24GB) graphics card. The length of candidate entities, K, was set to 12. A learning rate warm-up strategy was used, linearly increasing the learning rate from 0 to 2e-5 and then decreasing it to 0. This prevented model instability caused by excessively high learning rates during the initial training phase and accelerated model convergence. The model was evaluated every 2500 training steps, and the models with the highest accuracy were saved.

[0229] Table 3 Parameter settings

[0230] parameter value Learn rate 2e-5 Batch size 10 Epoch 10 1stm_embedding_size 1024 Hidden size 768 Bert model RoBERTa-wwm-ext Embedding size 512 optimizer AdamW

[0231] This experiment uses precision, recall, and F1 value standards to measure the performance of the model. The specific evaluation formula is as follows:

[0232] precision=correct num / predict num #(twenty two)

[0233] recall=correct num / golden num #(twenty three)

[0234]

[0235] Where precision represents the accuracy rate;

[0236] recall represents the recall rate;

[0237] correct num Indicates the number of correct predictions;

[0238] predict num Indicates the total number of predictions;

[0239] golden num Indicates the number of annotation relations;

[0240] The F1 value is the harmonic mean of precision and recall, which can balance the impact of precision and recall and more comprehensively reflect the performance of the model.

[0241] To verify the effectiveness of the proposed model for extracting relationships from power grid business data, a comparative experiment was conducted. The power grid business data entity relationship extraction model was compared with three other models (BiLSM-CRF, BERT-CRF, and BERT-BiLSM-CRF) to verify that the model achieved the best recognition results for power grid business data relationship extraction.

[0242] The experimental results are shown in Table 2. The model achieves the best recognition results compared to other comparison models. On the power grid service dataset, the accuracy is 83.80%, the recall is 80.59%, and the F1 value is 82.92%.

[0243] Table 2 Performance comparison of different relation extraction models

[0244] Model Accuracy Recall F1 value BiLSM-CRF 62.08％ 64.58％ 63.30％ BERT-CRF 87.15％ 83.51％ 85.29％ BERT-BiLSM-CRF 87.95％ 84.51％ 86.19％ RoBERTa-CE 88.87％ 84.91％ 86.84％ RoBERTa-CE* 89.55％ 85.91％ 87.92％

[0245] The ones marked with * in the table are RoBERTa with semantic dependency and part-of-speech embedding strategies.

[0246] (1) Since the outputs of the BiLSTM model are independent of each other and cannot consider the relationship information between adjacent entities, the relationship recognition performance is relatively poor. On the power grid business data, the F1 value of the model is only 63.30%.

[0247] (2) The BERT-CRF model uses the BERT pre-trained language model as the embedding layer to fully capture the contextual representation of characters, thereby better obtaining deep semantic information; on the power grid dataset, the model’s F1 value is 85.29%.

[0248] (3) The BERT-BiLSTM-CRF model also uses the BERT pre-trained language model to obtain the contextual semantics of power grid business data, while utilizing a recurrent neural network to capture richer meaning and a fully connected neural network as a classifier for analysis and classification. Due to insufficient model training data, the pre-trained language model cannot learn more knowledge, resulting in relatively poor performance in relation recognition. On the power grid dataset, the model achieved an F1 score of 86.19%.

[0249] (4) For the RoBERTa-CE model that embeds semantic dependencies and parts of speech, since there are a large number of proper nouns in the power grid business data, in order to prevent the model from predicting complete entities in advance during training, a masking strategy is adopted for complete words. The RoBERTa pre-trained language model with dynamic MASK is used to obtain the contextual semantics of the power grid business data, embed semantic dependencies and parts of speech, and effectively combine the information of the subject and object to improve the connectivity between the subject and object and strengthen the connection between related entities. Therefore, compared with the above models, its recognition performance has been significantly improved. On the power grid dataset, the F1 value of the model is 87.92%. Compared with the original embedded RoBERTa pre-trained model, the F1 value has increased by 1.08%. Compared with the BiLSM-CRF model, it has been significantly improved, achieving the best recognition effect.

[0250] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.< / mask> < / mask>

Claims

1. A relation extraction method for power grid dispatching domain combining semantic dependency and part-of-speech embedding, characterized by: The following steps are involved: S1: Collect power grid dispatch data and remove sentences with unclear meaning, incomplete structure, and repeated semantics. Then, perform entity and relationship annotation to obtain a data set in the field of power grid dispatch. S2, input the dataset into the RoBERTa-CE model for model training, learning entity information and relationship information separately. The RoBERTa-CE model includes an embedding layer and a 12-layer Transformer encoder; Inputting the data set into the RoBERTa-CE model for model training includes the following steps: S2-1, the data is passed through the embedding layer to learn context features and obtain word embedding code; S2-1 includes: Input the data into the embedding layer to obtain the word embedding X containing position information, paragraph information, word encoding information, semantic dependency information and part of speech embedding embedding : X embedding =X word +X segment +X positional +X semantics +X phrase #(1) X word 、X segment 、X positional 、X semantics 、X phrase Represent word embedding information, segment embedding information, position embedding information, semantic dependency embedding information and part-of-speech embedding information respectively; The semantic dependency information and part-of-speech embedding are obtained through the semantic dependency vocabulary and part-of-speech tagging vocabulary respectively. They are obtained by converting the text into index values ​​in the table after semantic dependency tagging and part-of-speech tagging. S2-2, then passes through the Transformer encoder to model the word embedding encoded information and obtain the output value of the hidden layer; S3 performs entity-relationship multi-task learning. First, the entity context information is concatenated to make entity predictions, and the entity loss value is calculated using cross-entropy loss. Then, the relationship context is concatenated to make relationship predictions, and the relationship loss value is calculated using cross-entropy loss. The entity loss value and the relationship loss value are added together and jointly participate in the back propagation of the model, so that the overall loss value is optimized towards the minimum value until all tasks converge. S4, through the fully connected layer, predicts the relationship; S4 includes: The representations of the start and end positions of the span sets of the object entity and the subject entity are concatenated. The formula of the object entity is as follows: h_o start =H 12 [:,O start ]#(8) h_o end =H 12 [:,O end ]#(9) obj=Concat(h_o start ,h_o end )#(10) H 12 Represents the output of the last layer of RoBERTa; H 12 [:,O start ] means only H 12 The value of the starting position; H 12 [:,O end ] means only H 12 The value of the end position; O start Indicates the starting position of the object entity; O end Indicates the end position of the object entity; obj represents the context representation of the object entity; Concat means Concat splicing; h_o start , h_o end Represent the representation of the starting position of the span set and the representation of the ending position of the span set respectively; The formula for the principal entity is as follows: h_s start =H 12 [:,S start ]#(11) h_s end =H 12 [:,S end ]#(12) sub=Concat(h_s start ,h_s end )#(13) Where [:, S start ] represents a slice of the main entity; S start Indicates the starting position of the main entity; S end Indicates the end position of the main entity; h_s start , h_s end Respectively represent the contextual deep semantic representation of the subject entity; sub represents the contextual deep semantic representation of the subject entity; The context information of the subject entity and the object entity are passed through the fully connected layer to obtain the predicted scores, and then the two are added together to obtain the predicted probabilities of various relationships. Finally, after passing through the softmax layer, the final predicted relationship type is obtained: E p =sigmoid(W a *obj+b1)#(14) r p =sigmoid(W b *sub+b2)#(15) R p =E p +r p #(16) R type =softmax(R p )#(17) Among them E p The value of the contextual deep semantic representation of the subject entity after passing through the fully connected layer; r p The contextual deep semantic representation of the object entity after passing through the fully connected layer; R p Indicates E p and R p to integrate; R type Indicates the normalized information, i.e. the predicted relationship type; W a 、W b Represent weight parameters respectively; b1 and b2 represent bias parameters respectively; The object entity passes through the fully connected layer and then the softmax layer to obtain the predicted entity type: E type =softmax(E p )#(18) E type Represents the entity type of the prediction.

2. The method for extracting relations in the field of power grid dispatching by combining semantic dependency and part-of-speech embedding according to claim 1 is characterized in that: In S2-2, based on the Transformer encoder architecture, the information encoded by word embedding is modeled through a deep model structure, including: First, embed the word X embedding Weight learning through attention mechanism: where Q、K、V=X embedding #(2) Attention() represents the attention mechanism; Q, K, and V represent query, key, and value respectively; d k Indicates the dimension size of K; where Q, K, and V represent query, key, and value, and all three vectors are X embedding ; Word embedding X embedding Perform multi-head attention learning to learn multi-channel information: MultiHead(X embedding ,X embedding ,X embedding )=Concat(head1,...,head h )W O where head i =Attention(X embedding ,X embedding ,X embedding )#(3) MultiHead(X embedding , X embedding , X embedding ) means taking three identical inputs as attention and learning different weight information, MultiHead() is a multi-head attention mechanism; Concat(head1, ..., head h ) means adding the attention of heads 1 to h; W O Represents a weight parameter; where head i represents the value of the i-th attention head; Attention(X embedding , X embedding , X embedding ) represents the calculation of attention; Word Embedding X embedding The vector is residually connected with the multi-head attention: sy=X embedding +MultiHead(X embedding ,X embedding ,X embedding )#(4) sy represents the embedding layer X embedding The value after the residual connection of multi-head attention; Then perform layer normalization on sy, calculate the mean and variance on each sample, normalize the hidden layer in the neural network to the standard normal distribution, and accelerate convergence: ∈ It is a very small value; LN(sy) represents layer normalization; μ L represents the mean; represents the standard deviation; The scaling parameters α and β are learnable parameters; m is the number of neurons; Then, the normalized output of the layer is passed through the Feed Forward neural network: FFN(x)=max(0,xW1+b1)W2+b2#(6) Where W1 and W2 represent two weight parameters respectively; b1 and b2 represent two bias parameters respectively; x represents the normalized output of the layer LN(sy); Finally, after residual connection and layer normalization: H=FFN(x)+x#(7) The output value H of the 12-layer Transformer encoder can be obtained. The model undergoes multiple rounds of training and gradient optimization through the AdamW optimizer.

3. The method for extracting relations in the field of power grid dispatching by combining semantic dependency and part-of-speech embedding according to claim 1 is characterized in that: The cross entropy loss formulas for entities and relations are: AND loss =-∑(E real *log(E type )+(1-E real )*log(1-E type ))#(19) R loss =-∑(R real *log(R type )+(1-R real )*log(1-R type ))#(20) E loss 、R loss Represents the loss value of the predicted value and the true value of the entity and relationship respectively; E real Represents the entity's true value; R real Represents the true value of the relationship; R type Represents a relationship type value.

Images