Cascade pointer network-based electricity-carbon market transaction participation entity relationship extraction method and system

Abstract

The invention relates to an electricity-carbon market transaction participation entity relation extraction method and system based on a cascade pointer network. The method comprises the following steps: constructing an electricity-carbon market transaction field word segmentation dictionary; a PowerSpanBERT model is constructed, and a model is constructed; an improved gradient accumulation-projection gradient descent adversarial training strategy is introduced to train a PowerSpanBERT model; identifying and extracting a candidate head entity set; a multi-head attention mechanism is introduced to calculate a word embedding vector output by the PowerSpanBERT model, CPN is adopted to decode the candidate head entity set feature vector and the word embedding vector to obtain a candidate tail entity set, and the relation type between head-tail entity pairs in the candidate head entity set and the candidate tail entity set is predicted; and outputting the head-tail entity pair and the corresponding relationship type thereof, and generating an entity relationship triple according to the irrigation result. According to the invention, a more accurate and efficient entity relationship extraction solution is provided for electricity and carbon market transactions.

Description

Technical Field

[0001] The present invention belongs to the technical field of relationship extraction, and relates to a method and system for extracting the relationship between entities participating in electric carbon market transactions, and in particular to a method and system for extracting the relationship between entities participating in electric carbon market transactions based on a cascade pointer network; Background Art

[0002] In the context of the growing development of the carbon market, it is very important to study the impact of carbon market expansion on the low-carbon operation of new power systems. By studying the identification and relationship extraction technology of multiple entities and multi-link participants in the electricity-carbon market transactions in the new power system, analyzing the trend of carbon market expansion, and studying the mechanism, key influencing factors and transmission mechanism of the comprehensive impact of typical carbon emission reduction policies such as carbon market expansion on green electricity, green certificates, and national certified voluntary emission reductions, at the same time, based on the transmission mechanism of key factors, studying the relationship map of the integration and evolution of the electricity-carbon market in all links of the new power system, it is conducive to promoting the low-carbon and stable operation of the new power system.

[0003] Knowledge extraction is mainly oriented towards open linked data. It extracts available knowledge units through automated technology. Knowledge units mainly include three knowledge elements: entity (concept extension), relationship and attribute. Based on this, a series of high-quality factual expressions are formed to lay the foundation for the construction of the upper model layer. The goal of relationship extraction is to solve the problem of semantic links between entities. Early relationship extraction mainly identified entity relationships by manually constructing semantic rules and templates. Subsequently, the relationship model between entities gradually replaced the manually predefined grammar and rules.

[0004] At present, the interactive relationship between market participants in the electricity carbon trading market is complicated. By studying the identification and relationship extraction technology of multiple subjects and multi-link participating entities in the electricity carbon market transaction in the new power system, analyzing the trend of carbon market expansion, and studying the mechanism, key influencing factors and transmission mechanism of typical carbon reduction policies such as carbon market expansion on the comprehensive impact of green electricity, green certificates, and national certified voluntary emission reductions. However, traditional relationship extraction methods are difficult to adapt to the rapid changes in the market.

[0005] The traditional pipeline extraction method divides named entity recognition and relationship extraction into two independent tasks and processes them step by step. It has advantages in flexibility and independence, but lacks effective interaction between the two tasks, and the implicit relationship between entities is difficult to be fully mined, resulting in unsatisfactory relationship extraction results. The joint extraction model based on sequence-to-sequence joint decoding processes entity recognition and relationship extraction tasks simultaneously through a unified framework, thereby improving the accuracy of relationship extraction. However, it is difficult to extract long sequence data and faces the problem of data redundancy. They cannot effectively deal with data redundancy and relationship overlap. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a method for extracting entity relationships of participants in electricity carbon market transactions based on a cascade pointer network. The PowerSpanBERT-GA-PGD-CPN combined model is used to jointly decode the entity relationships of participants in electricity carbon market transactions, extract entity relationship triplets, and provide a more accurate and efficient entity relationship extraction solution for electricity carbon market transactions.

[0007] The present invention solves the practical problem by adopting the following technical solutions:

[0008] A method for extracting entity relationships of electric carbon market transaction participants based on cascade pointer network includes the following steps:

[0009] Construct a word segmentation dictionary for the electric carbon market transaction field;

[0010] Based on the constructed word segmentation dictionary in the electric carbon market transaction field, the PowerSpanBERT model is constructed;

[0011] Based on the obtained PowerSpanBERT model, an improved GA-PGD adversarial training strategy is introduced to perform adversarial training of the PowerSpanBERT model;

[0012] Using the trained PowerSpanBERT model, the input text is converted into a word embedding vector including context information, and the word embedding vector is decoded using a cascade pointer network to identify and extract a set of candidate header entities;

[0013] Extract the feature vector of the candidate head entity set, introduce the multi-head attention mechanism to calculate the word embedding vector output by the PowerSpanBERT model, use CPN to decode the feature vector and word embedding vector of the candidate head entity set to obtain the candidate tail entity set, and predict the relationship type between the head-tail entity pairs in the candidate head entity set and the candidate tail entity set through the linear layer and sigmoid activation function;

[0014] The extraction module builds the PowerSpanBERT-GA-PGD-CPN model, outputs the head-tail entity pairs and their corresponding relationship types, generates entity relationship triples based on the irrigation results, and completes the joint extraction of entity relationships involved in electricity carbon market transactions.

[0015] Furthermore, the specific steps of constructing a word segmentation dictionary in the electric carbon market transaction field and determining the word segmentation vocabulary in the electric carbon market transaction field include:

[0016] First, calculate the average information content of the word, and the calculation formula is as follows:

[0017] The average information content of word ν is estimated on a large corpus using the following formula:

[0018]

[0019] In the formula, p ν is the probability of word ν appearing, -logp ν It is the amount of information contained in the word ν.

[0020] Based on the calculation result of the average information content of the characters, the point mutual information between two adjacent characters is calculated;

[0021]

[0022] In the formula, ν1 and ν2 represent two adjacent characters respectively, PMI(ν1, ν2) represents the point mutual information of characters ν1 and ν2, the probability of characters ν1 and ν2 appearing separately in the text dataset is represented by P(ν1) and P(ν2), and P(ν1, ν2) is the probability that ν1 and ν2 are adjacent in this corpus.

[0023] According to the calculation formula (1) of the average information content of word ν, the left entropy and right entropy of word ξ are calculated respectively, and the formula for the product of the left and right entropies is deduced as shown in formula (3):

[0024] H(ξ)=H(ξ left )·H(ξ right ) (3)

[0025] The minimum threshold of point mutual information PMImin of word segmentation is set to 0.5, and the minimum threshold of left and right entropy product H(ξ)min is set to 2, so as to generate a word segmentation dictionary in the field of electric carbon market transactions.

[0026] Moreover, the specific steps of constructing the PowerSpanBERT model based on the constructed electric carbon market transaction field word segmentation dictionary include:

[0027] Pre-train the SpanBERT module using text data issued in existing electricity carbon market transactions;

[0028] Based on the pre-trained SpanBERT module, the PowerSpanBERT model dedicated to the field is trained in combination with the word segmentation dictionary for the electric carbon market transaction domain constructed in step 1.

[0029] Moreover, based on the obtained PowerSpanBERT model, an improved GA-PGD adversarial training strategy is introduced to perform adversarial training of the PowerSpanBERT model:

[0030] Based on the obtained PowerSpanBERT model, a gradient accumulation strategy is proposed based on PGD adversarial training. In the process of multiple forward propagation and back propagation, the model parameters are not updated immediately, but the gradients are accumulated, as shown in formula (6):

[0031]

[0032] After K iterations are completed, the average value of the accumulated gradient is calculated and the model parameters are updated using the average gradient, as shown in formula (7):

[0033]

[0034] The improved GA-PGD adversarial training strategy is shown in formula (8):

[0035]

[0036] Moreover, the trained PowerSpanBERT model is used to convert the input text into a word embedding vector including context information, and the word embedding vector is decoded using a cascade pointer network. The specific method for identifying and extracting a set of candidate header entities is as follows:

[0037] The PowerSpanBERT model is trained using the GA-PGD adversarial training strategy to convert the input text data into word embedding vector representations that the model can process;

[0038] The generated word embedding vector is input into the cascade pointer network, which scores the position of each word through the attention mechanism and assigns the vector label 1 or label 0 using the binary classification mark start and end positions to identify and extract the possible head entity set in the text.

[0039] Moreover, the feature vector of the candidate head entity set is extracted, and the multi-head attention mechanism is introduced to calculate the word embedding vector output by the PowerSpanBERT model. The CPN is used to decode the feature vector and word embedding vector of the candidate head entity set to obtain the candidate tail entity set. The specific method of predicting the relationship type between the head-tail entity pairs in the candidate head entity set and the candidate tail entity set through the linear layer and the sigmoid activation function is as follows:

[0040] The features of the candidate head entities are extracted, and the CPN is used to decode the vector that combines the output of the multi-head attention mechanism and the features of the candidate head entities to obtain the candidate tail entity set; the word embedding vector output by the PowerSpanBERT model is passed to the relationship and tail entity joint recognition layer, and the multi-head attention mechanism output is used to obtain a vector that integrates the information of each attention head;

[0041] Based on the obtained candidate head entity set, the linear layer and sigmoid activation function are used to determine the possible relationship type, and the relationship type with the highest probability between the head entity and the tail entity is identified, which can be determined as the relationship type between the head-tail entity pair.

[0042] Furthermore, after the above steps, the following steps are included:

[0043] The PowerSpanBERT-GA-PGD-CPN model outputs head-tail entity pairs and their corresponding relationship types, generates entity-relationship triples based on the irrigation results, and evaluates the performance of the model through precision, recall, and F1 score indicators.

[0044] Moreover, the PowerSpanBERT-GA-PGD-CPN model is constructed, the head-tail entity pairs and their corresponding relationship types are output, entity relationship triples are generated according to the irrigation results, and the specific method for evaluating the performance of the model through the accuracy, recall rate, and F1 score indicators is as follows:

[0045] The output results of the PowerSpanBERT-GA-PGD-CPN model are expressed as triples, and the generated triples are manually screened and optimized to remove incorrect matches and redundant information to ensure the accuracy and reliability of the triples;

[0046] The performance of the model is evaluated through accuracy, recall rate, and F1 score indicators, and the model is adjusted and optimized based on the evaluation results, including adjusting the network structure and hyperparameters.

[0047] A system for extracting entity relationships of electric carbon market transactions based on cascade pointer networks, including:

[0048] The word segmentation dictionary construction module builds a word segmentation dictionary in the field of electric carbon market transactions and determines the word segmentation vocabulary in the field of electric carbon market transactions;

[0049] The PowerSpanBERT model building module builds the PowerSpanBERT model based on the constructed word segmentation dictionary in the electric carbon market transaction field;

[0050] An improved GA-PGD adversarial training strategy introduction module introduces an improved GA-PGD adversarial training strategy based on the obtained PowerSpanBERT model to perform adversarial training on the PowerSpanBERT model.

[0051] The candidate head entity set extraction module uses the trained PowerSpanBERT model to convert the input text into a word embedding vector including context information, uses a cascade pointer network to decode the word embedding vector, and identifies and extracts the candidate head entity set;

[0052] The candidate tail entity set and relationship extraction module extracts the feature vector of the candidate head entity set, and introduces a multi-head attention mechanism to calculate the word embedding vector output by the PowerSpanBERT model. CPN is used to decode the feature vector and word embedding vector of the candidate head entity set to obtain the candidate tail entity set. The linear layer and sigmoid activation function are used to predict the relationship type between the head-tail entity pairs in the candidate head entity set and the candidate tail entity set.

[0053] Moreover, the word segmentation dictionary construction module also includes:

[0054] The average information content calculation module of a word first calculates the average information content of a word, and the calculation formula is as follows:

[0055] The average information content of word ν is estimated on a large corpus using the following formula:

[0056]

[0057] In the formula, p ν is the probability of word ν appearing, -logp ν It is the amount of information contained in the word ν.

[0058] A point mutual information calculation module between two adjacent characters, which calculates the point mutual information between two adjacent characters based on the average information amount calculation result of the characters;

[0059]

[0060] In the formula, ν1 and ν2 represent two adjacent characters respectively, PMI(ν1, ν2) represents the point mutual information of characters ν1 and ν2, the probability of characters ν1 and ν2 appearing separately in the text dataset is represented by P(ν1) and P(ν2), and P(ν1, ν2) is the probability that ν1 and ν2 are adjacent in this corpus.

[0061] The left entropy and right entropy calculation module of the word calculates the left entropy and right entropy of the word ξ according to the average information amount calculation formula (1) of the character ν, and derives the left and right entropy product formula as shown in formula (3):

[0062] H(ξ)=H(ξ left )·H(ξ right ) (3)

[0063] The word segmentation dictionary generation module in the field of electric carbon market transactions sets the minimum threshold of point mutual information PMImin of word segmentation to 0.5, and sets the minimum threshold of left and right entropy product H(ξ)min to 2, thereby generating a word segmentation dictionary in the field of electric carbon market transactions.

[0064] Advantages and beneficial effects of the present invention:

[0065] 1. The present invention proposes a method for extracting entity relationships of participants in electricity carbon market transactions based on a cascade pointer network. First, a professional dictionary in the field of electricity carbon market transactions is constructed to improve the accuracy of text segmentation, and at the same time, it is more conducive to more accurate extraction of information units during knowledge extraction. Secondly, the PowerSpanBERT model in the exclusive field is trained in combination with the text and professional dictionary in the field of electricity carbon market transactions, and a gradient accumulation adversarial training method is introduced in the pre-training process to improve the accuracy of the model in extracting electricity knowledge and the ability to predict unfamiliar information. Then, a multi-head attention mechanism is introduced in the joint recognition layer of relations and tail entities to enhance the feature representation of the output vector of the encoding layer, realize the joint extraction of entities and relations, and extract multiple relation triplets at the same time, effectively deal with the problem of relationship overlap in the extraction process, and significantly improve the accuracy of knowledge extraction.

[0066] 2. The present invention proposes a method for extracting entity relationships of participants in electricity carbon market transactions based on a cascade pointer network. Aiming at the difficulty that entities are densely distributed and relationships overlap in Chinese texts in the field of electricity carbon trading, the knowledge extraction task is converted into a pointer network labeling problem. The input electricity text sequence is labeled at the same position for entities and relationships, and triples are directly modeled for joint extraction of entities and relationships. This solves the problem of redundant entity and overlapping relationship extraction, avoids error propagation, and significantly improves the accuracy and efficiency of entity relationship extraction and labeling in the field of electricity carbon trading. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 The PowerSpanBERT-GA-PGD-CPN model structure diagram of the present invention;

[0068] Figure 2 Schematic diagram of the SpanBERT model masking strategy of the present invention;

[0069] Figure 3 Schematic diagram of the PowerSpanBERT layer architecture and input representation of the present invention. DETAILED DESCRIPTION

[0070] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings:

[0071] A method for extracting entity relationships of electric carbon market transaction participants based on cascade pointer network includes the following steps:

[0072] Step 1: First, construct a word segmentation dictionary for the electric carbon market transaction field;

[0073] The specific steps of step 1 include:

[0074] (1) First, calculate the average information content of a word. The calculation formula is as follows:

[0075] The average information content of word ν is estimated on a large corpus using the following formula:

[0076]

[0077] In the formula, p ν is the probability of word ν appearing, -logp ν It is the amount of information contained in the word ν.

[0078] (2) Based on the calculation result of the average information amount of the characters, the point mutual information between two adjacent characters is calculated;

[0079]

[0080] In the formula, ν1 and ν2 represent two adjacent characters respectively, PMI(ν1, ν2) represents the point mutual information of characters ν1 and ν2, the probability of characters ν1 and ν2 appearing separately in the text dataset is represented by P(ν1) and P(ν2), and P(ν1, ν2) is the probability that ν1 and ν2 are adjacent in this corpus.

[0081] (3) According to the average information amount calculation formula (1) of word ν, the left entropy and right entropy of word ξ are calculated respectively, and the formula for the product of the left and right entropies is deduced as shown in formula (3):

[0082] H(ξ)=H(ξ left )·H(ξ right ) (3)

[0083] (3) Based on step (2), the minimum threshold value PMImin of the point mutual information of words during word segmentation is set to 0.5, and based on step (3), the minimum threshold value H(ξ)min of the product of left and right entropies during word segmentation is set to 2, thereby generating a word segmentation dictionary for the electric carbon market transaction field.

[0084] In this embodiment, the working principle of step 1 is:

[0085] In the step 1, the electric carbon market transaction field involves many proper nouns and industry terms. In order to help the model accurately identify and understand these proper nouns, avoid misjudging them as other irrelevant words, and improve the accuracy of entity recognition and relationship extraction, the present invention first constructs a professional dictionary in the electric carbon market transaction field and determines the professional vocabulary in the electric carbon market transaction field;

[0086] Aiming at the professional vocabulary in the field of electricity and carbon market transactions, a minimum entropy word segmentation method based on left and right entropies is proposed to construct a professional dictionary for electricity and carbon markets. The pointwise mutual information (PMI) value between two adjacent words is calculated to evaluate their co-occurrence probability. At the same time, the left and right entropy product is introduced as an auxiliary basis for word formation, and fine-grained vocabulary is aggregated to evaluate the contextual adaptability of vocabulary and complete more accurate word segmentation processing to optimize the word segmentation algorithm for electricity and carbon market transaction texts.

[0087] The average information content of word ν is estimated on a large corpus using the following formula:

[0088]

[0089] In the formula, p ν is the probability of word ν appearing, -logp ν It is the amount of information contained in the word ν.

[0090] Calculating the point mutual information PMI between two adjacent words is a metric based on mutual information (PMI), which is mainly used to evaluate the strength of the correlation between two things;

[0091] The calculation formula of PMI is as follows:

[0092]

[0093] In the formula, ν1 and ν2 represent two adjacent characters respectively, PMI(ν1, ν2) represents the point mutual information of characters ν1 and ν2, the probability of characters ν1 and ν2 appearing separately in the text dataset is represented by P(ν1) and P(ν2), and P(ν1, ν2) is the probability that ν1 and ν2 are adjacent in this corpus.

[0094] According to formula (1), the left entropy and right entropy of word ξ are calculated respectively, and the formula of the left and right entropy product is deduced as formula (3). The minimum threshold H(ξ) of the left and right entropy product when the word is set is min When the value is 2, a better vocabulary aggregation effect can be achieved.

[0095] H(ξ)=H(ξ left )·H(ξ right ) (3)

[0096] Based on the understanding of the above-mentioned Chinese word segmentation characteristics and the principles of minimum entropy, left-right entropy and mutual information, and taking into account the current lack of ready-made word segmentation dictionaries in the field of electric carbon market transactions, the present invention takes the minimum entropy Chinese word segmentation algorithm based on left-right entropy as the research basis, aiming to construct a professional dictionary to improve the accuracy of extraction.

[0097] Step 2: Based on the word segmentation dictionary of the electric carbon market transaction field constructed in step 1, build the PowerSpanBERT model;

[0098] The specific steps of step 2 include:

[0099] (1) Use text data issued in existing electricity carbon market transactions to pre-train the SpanBERT module;

[0100] (2) Based on the pre-trained SpanBERT module of step (1), the PowerSpanBERT model for the dedicated domain is trained in combination with the word segmentation dictionary for the electric carbon market transaction domain constructed in step 1.

[0101] The working principle of step 2 is:

[0102] Step 2: The professional dictionary constructed above provides specific terms and concepts in the field of electric carbon market transactions. When converting text into word embedding vectors, the PowerSpanBERT encoding layer can more accurately embed the proper nouns in the field, avoid mis-embedding or incorrect parsing, and better capture and understand the complex semantic relationships in electric carbon market transactions.

[0103] For example, "carbon emission rights trading" and "electricity market access standards", professional dictionaries can help PowerSpanBERT better identify the boundaries of these composite entities, ensuring that complete entities are accurately captured during the encoding process without causing information loss or errors due to unclear word segmentation or boundaries.

[0104] SpanBERT is an improved model based on BERT. It introduces a new training objective, SpanBoundary Objective (SBO), to better capture the internal structure and boundary information of spans (such as phrases, entities, etc.). In addition to using the original cross entropy of BERT's predicted masked words as the training objective, continuous masked words are introduced as masked boundary words as auxiliary training objectives. During training, two words are taken at the boundaries before and after the masked words, and these two words are not within the mask range. The feature vectors extracted from these two words are used to jointly predict the final word with the position encoding vector of the masked word. The model can better learn span-level dependencies and context information.

[0105] like Figure 2 As shown, x5, x6, x7, x8, and x9 are the feature vectors corresponding to the masked words. Taking the word corresponding to the prediction vector x5 as an example, the probability that the word is correctly predicted as "carbon" is:

[0106] L(carbon)=L MLM (Carbon)+L SBO (Carbon) = -logP(Carbon|x5) -logP(Carbon|x4,x10 ,p5) (4)In the formula, L MLM The calculation method of is consistent with the MLM calculation method of BERT. SBO It is calculated through a two-layer fully connected network, and p3 represents the position encoding of the word to be predicted in the mask word.

[0107] The input of the SpanBERT model is divided into three parts: word embedding, paragraph embedding, and position embedding. For a given token sequence S = [s1, s2, s3, …, s n-3 ,s n-2 ], add special tags [CLS] and [SEP] at the beginning and end of the sequence, and convert each tag in the sequence into its corresponding word vector e i , get the word embedding matrix E. Assign a position vector p to each position in the sequence i , and obtain the position embedding matrix P containing all positions. Paragraph embedding is used to distinguish two sentences and is usually used only once in a sequence, denoted as [SEP], and the paragraph embedding vector s i [SEG] Indicates the beginning of a paragraph. For each token s in the sequence i , whose input i i It is composed of word embedding, position embedding and paragraph embedding, as shown in formula (5):

[0108]

[0109] The obtained input is represented as I = [i1, i2, ..., i n-1 ,i n ] is fed to the Transformer layer of the Bert model, and each token s i Get a context-aware vector representation x i :X=[x1,x2,…,x n ].

[0110] The SpanBERT module was pre-trained using regulations, management methods, and policy documents issued in electricity and carbon market transactions, such as the "Interim Regulations on Carbon Emission Trading Management", "Carbon Emission Trading Management Methods (Trial)", "Enterprise Greenhouse Gas Emission Accounting and Reporting Guidelines (Trial)", "Emission Monitoring Plan Review and Emission Report Verification Reference Guide", "Greenhouse Gas Voluntary Emission Reduction Project Design and Implementation Guidelines", etc., and combined with the professional dictionary in the field of electricity and carbon market transactions constructed in step 1, the PowerSpanBERT model for the exclusive field was trained. The model architecture and input representation are as follows: Figure 3 shown.

[0111] Step 3: Based on the PowerSpanBERT model obtained in step 2, an improved GA-PGD adversarial training strategy is introduced to perform adversarial training of the PowerSpanBERT model;

[0112] The specific method of step 3 is:

[0113] Based on the PowerSpanBERT model obtained in step 2, a gradient accumulation strategy is proposed based on PGD adversarial training. In multiple forward propagation and backpropagation processes, the model parameters are not updated immediately, but the gradients are accumulated, as shown in formula (6):

[0114]

[0115] After K iterations are completed, the average value of the accumulated gradient is calculated and the model parameters are updated using the average gradient, as shown in formula (7):

[0116]

[0117] The improved GA-PGD adversarial training strategy is shown in formula (8):

[0118]

[0119] The working principle of step 3 is:

[0120] Step 3: During the PowerSpanBERT training process, in order to improve the robustness and generalization ability of the model, an improved adversarial training is introduced to introduce tiny perturbations (adversarial samples) into the input data to enhance the model's resistance to noise and attacks.

[0121] PGD ​​(Projected Gradient Descent) is a gradient-based adversarial sample generation method. In each iteration, the input sample x is perturbed along the gradient direction of the loss function L(x,θ) and the perturbation r is generated according to the gradient direction. Since each iteration requires forward propagation and back propagation, the computational cost increases significantly. To this end, a gradient accumulation strategy is proposed based on PGD adversarial training. In the process of multiple forward propagation and back propagation, the model parameters are not updated immediately, but these gradients are accumulated, as shown in formula (6):

[0122]

[0123] After K iterations are completed, the average value of the accumulated gradient is calculated and the model parameters are updated using the average gradient, as shown in formula (7):

[0124]

[0125] The improved GA-PGD optimization objective is shown in formula (8):

[0126]

[0127] The above-mentioned improved GA-PGD adversarial training virtually expands the original data by K times without increasing additional computing overhead, thereby improving the utilization rate of gradients in training. At the same time, after multiple iterations and accumulated gradients, the model can learn from multiple different norm constraints. Compared with the PGD algorithm with a single norm constraint, the generalization error is smaller, thereby enhancing the generalization ability of the model.

[0128] Step 4: Use the trained PowerSpanBERT model to convert the input text into a word embedding vector including context information, use a cascade pointer network to decode the word embedding vector, and identify and extract a set of candidate header entities;

[0129] The specific method of step 4 is:

[0130] (1) Use the GA-PGD adversarial training strategy in step 3 to train the PowerSpanBERT model and convert the input text data into a word embedding vector representation that the model can process;

[0131] (2) The generated word embedding vector is input into the cascade pointer network, which scores the position of each word through the attention mechanism and assigns the vector label 1 or label 0 using the binary classification mark start and end positions to identify and extract the possible head entity set in the text.

[0132] The working principle of step 4 is:

[0133] Step 4: The PowerSpanBERT model encodes the input original text, that is, the PowerSpanBERT model is used to convert the text into a word embedding vector. First, the BertTokenizer component is used to segment the words and convert the words into ids. Then, the BertModel component is used to encode the words, and each word is encoded into a 768-dimensional word vector representation.

[0134] like Figure 1 As shown, the word embedding vector H obtained in the previous step n The word vector is input into the Cascade Pointer Network (CPN), and CPN is used to decode the word vector to identify all possible head entities, and the start and end positions are marked using binary classification.

[0135] First, a linear layer is used to calculate the scores of the start and end positions of the i-th word as the head entity:

[0136]

[0137] In the formula, is the weight parameter, is the bias parameter.

[0138] Then, the output of the linear layer is passed through the sigmoid activation function to obtain the probability that each position is the start or end position of the head entity, as shown in formula (10):

[0139]

[0140] Where P i start-sub and P i end-sub They represent the probability of identifying the i-th token in the input sequence as the start position and the end position of the head entity, respectively.

[0141] The probability is converted into a binary label according to the threshold (set to 0.5), as shown in formula (11). If the probability exceeds the threshold, the corresponding label will be assigned label 1, otherwise it will be assigned label 0.

[0142]

[0143] Then, the nearest matching principle is used to pair the identified start and end to obtain a set of candidate head entities, each of which is denoted as (i, j).

[0144] Step 5: Extract the feature vector of the candidate head entity set, introduce the multi-head attention mechanism to calculate the word embedding vector output by the PowerSpanBERT model, use CPN to decode the feature vector and word embedding vector of the candidate head entity set to obtain the candidate tail entity set, and predict the relationship type between the head-tail entity pairs in the candidate head entity set and the candidate tail entity set through the linear layer and sigmoid activation function;

[0145] The specific method of step 5 is:

[0146] (1) extracting the feature vector of the candidate head entity set obtained in step 4;

[0147] (2) Based on (1), a multi-head attention mechanism is introduced to calculate the word embedding vector output by the PowerSpanBERT model in step 4. The decoding method is the same as that in step 4. CPN is used to decode the feature vector and word embedding vector of the candidate head entity set to obtain the candidate tail entity set;

[0148] (3) Based on the candidate head entity set obtained in step 4, the linear layer and sigmoid activation function are used to determine the possible relationship type, and the relationship type with the highest probability between the head entity and the tail entity in (2) is identified, which can be determined as the relationship type between the head-tail entity pair.

[0149] The working principle of step 5 is:

[0150] The head entity identified by the head entity decoding layer is passed to the relation and tail entity joint recognition layer, and the relevant text embedding vector and position information are passed at the same time. The input of the relation and tail entity joint recognition layer is the output H of the PowerSpanBERT encoding layer. n In addition, the features of the header entity identified in the header entity decoding layer are also considered.

[0151] A multi-headed attention mechanism is introduced in the relationship and tail entity joint recognition layer. The multi-headed attention mechanism is used to capture the complex dependencies in the text, enhance the feature representation of the output vector of the encoding layer, and improve the accuracy of the semantic vector describing the meaning of the entity.

[0152] For the input vector H n Perform a linear transformation to obtain its query vector Q n , key vector K n Sum value vector V n For each head, the dot product of the query vector and the key vector is calculated. In order to prevent the dot product value from being too large and causing the gradient to disappear, the dimension d of the key vector is used k Scaling the dot product attention is as follows:

[0153]

[0154] Each head calculates attention independently, then the outputs of all heads are concatenated together and linearly transformed to generate the final output:

[0155] Multi_attention(Q n , K n ,V n )=[attention1,attention2,...,attention n ]·W sub

[0156] Where W sub is the weight matrix of the linear transformation.

[0157] Compute the relevance score and scale it:

[0158]

[0159] Then the scores are normalized so that the association score of each entity pair is converted into a probability distribution, and the attention weight α of the i-th entity is obtained through the softmax function. i Then use the normalized attention weight α i Perform weighted summation with the value vector of the i-th entity to obtain the weighted global vector Sum i , as shown in formula (14).

[0160]

[0161] Where V i is the value vector of the i-th entity.

[0162] The input of the relation and tail entity joint recognition layer is the output H of the encoding layer. n In addition, the characteristics of the head entity are also considered (refers to the average of all labeled vectors contained in the jth candidate head entity):

[0163]

[0164] In the formula, and They represent the probability of identifying the i-th token in the input sequence as the start position and the end position of the tail entity, respectively.

[0165] Similarly, the probabilities of the start and end positions of the tail entity are converted into binary classification labels according to the threshold, and the start and end of the identified tail entity are paired using the nearest matching principle to obtain a set of candidate tail entities, each of which is denoted as (m, n).

[0166] For each candidate head entity (i, j), a linear layer and a sigmoid activation function are used to determine the type of relationship between the head entity and each possible tail entity. The probability of the relationship between each candidate head entity and the tail entity is shown in formula (16):

[0167]

[0168] In the formula, [Sum i ;Sum j ] is the sequence representation of the head entity. rel is the weight parameter, b rel is the bias parameter.

[0169] For each candidate head entity (i, j) and candidate tail entity (m, n), use the probability of relation classification To determine the type of relationship between them:

[0170]

[0171] Step 6. Build the PowerSpanBERT-GA-PGD-CPN model, output the recognition results of the head entity in step 4 and the tail entity and relationship in step 5, generate entity-relationship triples based on the irrigation results, and evaluate the performance of the model through the accuracy, recall, and F1 score indicators.

[0172] The specific method of step 6 is:

[0173] (1) The output results of the PowerSpanBERT-GA-PGD-CPN model are expressed as triples, and the generated triples are manually screened and optimized to remove incorrect matches and redundant information to ensure the accuracy and reliability of the triples;

[0174] (2) Evaluate the performance of the model through accuracy, recall rate, and F1 score indicators, and adjust and optimize the model based on the evaluation results, including adjusting the network structure and hyperparameters, to improve the overall performance of the model.

[0175] The working principle of step 6 is:

[0176] The accuracy, recall and F1 value are used as the evaluation criteria for entity relationship recognition results. Among them, the accuracy Precision (Prec) represents the ratio of the number of correct entity relationships recognized to the total number of entity relationships actually recognized, and its calculation formula is:

[0177]

[0178] Recall (Rec) represents the ratio of the number of correctly identified entity relationships to the total number that should be identified theoretically:

[0179]

[0180] The F1 value is used to comprehensively evaluate the accuracy and recall rate. The F1 value can be used to comprehensively evaluate the recognition effect of the model. The calculation formula of the F1 value is:

[0181]

[0182] N A N represents the total number of entities in the labeled dataset, that is, the actual standard answer given by the labeled dataset; B Represents the total number of entities identified by the model; N C It indicates the total number of correct entities recognized by the model, that is, the number of entities recognized by the model that are consistent with the entities in the actual labeled dataset.

[0183] A system for extracting entity relationships of electric carbon market transactions based on cascade pointer networks, including:

[0184] The word segmentation dictionary construction module builds a word segmentation dictionary in the field of electric carbon market transactions;

[0185] The PowerSpanBERT model building module builds the PowerSpanBERT model based on the constructed word segmentation dictionary in the electric carbon market transaction field;

[0186] An improved GA-PGD adversarial training strategy introduction module introduces an improved GA-PGD adversarial training strategy based on the obtained PowerSpanBERT model to perform PowerSpanBERT model pre-training;

[0187] The candidate header entity set extraction module uses the trained PowerSpanBERT model to convert the input text into a word embedding vector including context information, and then uses a cascade pointer network to decode the word embedding vector to identify and extract the candidate header entity set;

[0188] The candidate tail entity set and relationship extraction module extracts the feature vector of the candidate head entity set, and introduces a multi-head attention mechanism to calculate the word embedding vector output by the PowerSpanBERT model. The CPN is used to decode the feature vector and word embedding vector of the candidate head entity set to obtain the candidate tail entity set. The linear layer and sigmoid activation function are used to predict the relationship type between the head-tail entity pairs in the candidate head entity set and the candidate tail entity set.

[0189] The extraction module builds the PowerSpanBERT-GA-PGD-CPN model, outputs the head-tail entity pairs and their corresponding relationship types, generates entity relationship triples based on the irrigation results, and completes the joint extraction of entity relationships involved in electricity carbon market transactions.

[0190] The word segmentation dictionary building module also includes:

[0191] The average information content calculation module of a word first calculates the average information content of a word, and the calculation formula is as follows:

[0192] The average information content of word ν is estimated on a large corpus using the following formula:

[0193]

[0194] In the formula, p ν is the probability of word ν appearing, -logp ν It is the amount of information contained in the word ν.

[0195] A point mutual information calculation module between two adjacent characters, which calculates the point mutual information between two adjacent characters based on the average information amount calculation result of the characters;

[0196]

[0197] In the formula, ν1 and ν2 represent two adjacent characters respectively, PMI(ν1, ν2) represents the point mutual information of characters ν1 and ν2, the probability of characters ν1 and ν2 appearing separately in the text dataset is represented by P(ν1) and P(ν2), and P(ν1, ν2) is the probability that ν1 and ν2 are adjacent in this corpus.

[0198] The left entropy and right entropy calculation module of the word calculates the left entropy and right entropy of the word ξ according to the average information amount calculation formula (1) of the character ν, and derives the left and right entropy product formula as shown in formula (3):

[0199] H(ξ)=H(ξ left )·H(ξ right ) (3)

[0200] The word segmentation dictionary generation module in the field of electric carbon market transactions sets the minimum threshold of point mutual information PMImin of word segmentation to 0.5, and sets the minimum threshold of left and right entropy product H(ξ)min to 2, thereby generating a word segmentation dictionary in the field of electric carbon market transactions.

[0201] It should be emphasized that the embodiments of the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation modes. Any other implementation modes derived by those skilled in the art based on the technical solutions of the present invention also fall within the scope of protection of the present invention.

Claims

1. A method for extracting entity relationships of electric carbon market participants based on cascade pointer networks, characterized in that: The following steps are involved: Construct a word segmentation dictionary for the electric carbon market transaction field; Based on the constructed word segmentation dictionary in the electric carbon market transaction field, the PowerSpanBERT model is constructed; Based on the obtained PowerSpanBERT model, an improved gradient accumulation-projection gradient descent adversarial training strategy GA-PGD is introduced to perform adversarial training of the PowerSpanBERT model; Using the trained PowerSpanBERT model, the input text is converted into a word embedding vector including context information, and the word embedding vector is decoded using a cascade pointer network to identify and extract a set of candidate header entities; Then extract the feature vector of the candidate head entity set, introduce the multi-head attention mechanism to calculate the word embedding vector output by the PowerSpanBERT model, use the cascade pointer network CPN to decode the feature vector and word embedding vector of the candidate head entity set to obtain the candidate tail entity set, and predict the relationship type between the head-tail entity pairs in the candidate head entity set and the candidate tail entity set through the linear layer and sigmoid activation function; The PowerSpanBERT-GA-PGD-CPN model is constructed to output head-tail entity pairs and their corresponding relationship types, generate entity relationship triples based on irrigation results, and complete the joint extraction of entity relationships involved in electricity carbon market transactions.

2. According to claim 1, a method for extracting entity relationships of electric carbon market transaction participants based on cascade pointer network is characterized by: The specific steps of constructing the word segmentation dictionary in the electric carbon market transaction field include: First, calculate the average information content of the word, and the calculation formula is as follows: The average information content of word ν is estimated on a large corpus using the following formula: In the formula, p ν is the probability of word ν appearing, -logp ν It is the amount of information contained in the word ν; Based on the calculation result of the average information content of the characters, the point mutual information between two adjacent characters is calculated; In the formula, ν1 and ν2 represent two adjacent characters, PMI(ν1, ν2) represents the point mutual information of characters ν1 and ν2, the probability of characters ν1 and ν2 appearing alone in the text dataset is represented by P(ν1) and P(ν2), and P(ν1, ν2) is the probability that ν1 and ν2 are adjacent in this corpus; According to the calculation formula (1) of the average information content of word ν, the left entropy and right entropy of word ξ are calculated respectively, and the formula for the product of the left and right entropies is deduced as shown in formula (3): H(ξ)=H(ξ left )·H(ξ right ) (3) The minimum threshold of point mutual information PMImin of word segmentation is set to 0.5, and the minimum threshold of left and right entropy product H(ξ)min is set to 2, and then the word segmentation dictionary in the field of electric carbon market transaction is generated; The specific steps of constructing the PowerSpanBERT model based on the constructed electric carbon market transaction field word segmentation dictionary include: Pre-train the SpanBERT module using text data issued in existing electricity carbon market transactions; Based on the pre-trained SpanBERT module, the PowerSpanBERT model dedicated to the field is trained in combination with the word segmentation dictionary for the electric carbon market transaction domain constructed in step 1.

3. According to claim 1, a method for extracting entity relationships of electric carbon market transaction participants based on cascade pointer network, characterized in that: The specific method of introducing the improved GA-PGD adversarial training strategy based on the obtained PowerSpanBERT model and conducting adversarial training of the PowerSpanBERT model is as follows: Based on the obtained PowerSpanBERT model, a gradient accumulation strategy is proposed based on PGD adversarial training. In the process of multiple forward propagation and back propagation, the model parameters are not updated immediately, but the gradients are accumulated, as shown in formula (6): After K iterations are completed, the average value of the accumulated gradient is calculated and the model parameters are updated using the average gradient, as shown in formula (7): The improved GA-PGD adversarial training strategy is shown in formula (8):

4. According to claim 1, a method for extracting entity relationships of electric carbon market transactions based on cascade pointer networks is characterized by: The improved GA-PGD adversarial training strategy is adopted to train the PowerSpanBERT model, and the trained PowerSpanBERT model is used to convert the input text into a word embedding vector including context information, and the cascade pointer network is used to decode the word embedding vector, and the specific method of identifying and extracting the candidate head entity set is as follows: The PowerSpanBERT model is trained using the GA-PGD adversarial training strategy to convert the input text data into word embedding vector representations that the model can process; The generated word embedding vector is input into the cascade pointer network, which scores the position of each word through the attention mechanism and assigns the vector label 1 or label 0 using the binary classification mark start and end positions to identify and extract the possible head entity set in the text.

5. According to claim 1, a method for extracting entity relationships of electric carbon market transactions based on cascade pointer networks is characterized by: The feature vector of the candidate head entity set is extracted, and the multi-head attention mechanism is introduced to calculate the word embedding vector output by the PowerSpanBERT model. The CPN is used to decode the feature vector and word embedding vector of the candidate head entity set to obtain the candidate tail entity set. The specific method for predicting the relationship type between the head-tail entity pairs in the candidate head entity set and the candidate tail entity set through the linear layer and the sigmoid activation function is as follows: The features of the candidate head entities are extracted, and the CPN is used to decode the vector that combines the output of the multi-head attention mechanism and the features of the candidate head entities to obtain the candidate tail entity set; the word embedding vector output by the PowerSpanBERT model is passed to the relationship and tail entity joint recognition layer, and the multi-head attention mechanism output is used to obtain a vector that integrates the information of each attention head; Based on the obtained candidate head entity set, the linear layer and sigmoid activation function are used to determine the possible relationship type, and the relationship type with the highest probability between the head entity and the tail entity is identified, which can be determined as the relationship type between the head-tail entity pair.

6. The method for extracting entity relationships of electric carbon market transaction participants based on cascade pointer network according to claim 1 is characterized by: The above steps also include the following steps: building a PowerSpanBERT-GA-PGD-CPN model, outputting the recognition results of the head entity and the tail entity and the relationship, generating entity-relationship triples according to the irrigation results, and evaluating the performance of the model through the accuracy, recall, and F1 score indicators.

7. The method for extracting entity relationships of electric carbon market transactions based on cascade pointer networks according to claim 1 is characterized by: The PowerSpanBERT-GA-PGD-CPN model is constructed, the head-tail entity pairs and their corresponding relationship types are output, entity relationship triples are generated according to the irrigation results, and the specific method for evaluating the performance of the model through the accuracy, recall rate, and F1 score indicators is as follows: (1) The output results of the PowerSpanBERT-GA-PGD-CPN model are expressed as triples, and the generated triples are manually screened and optimized to remove incorrect matches and redundant information to ensure the accuracy and reliability of the triples; (2) Evaluate the performance of the model using accuracy, recall, and F1 score indicators, and adjust and optimize the model based on the evaluation results, including adjusting the network structure and hyperparameters.

8. A system for extracting entity relationships of electric carbon market transactions based on cascade pointer networks, characterized by: include: The word segmentation dictionary construction module builds a word segmentation dictionary in the field of electric carbon market transactions; The PowerSpanBERT model building module builds the PowerSpanBERT model based on the constructed word segmentation dictionary in the electric carbon market transaction field; An improved GA-PGD adversarial training strategy introduction module introduces an improved GA-PGD adversarial training strategy based on the obtained PowerSpanBERT model to perform PowerSpanBERT model pre-training; The candidate header entity set extraction module uses the trained PowerSpanBERT model to convert the input text into a word embedding vector including context information, and then uses a cascade pointer network to decode the word embedding vector to identify and extract the candidate header entity set; The candidate tail entity set and relationship extraction module extracts the feature vector of the candidate head entity set, and introduces a multi-head attention mechanism to calculate the word embedding vector output by the PowerSpanBERT model. The CPN is used to decode the feature vector and word embedding vector of the candidate head entity set to obtain the candidate tail entity set. The linear layer and sigmoid activation function are used to predict the relationship type between the head-tail entity pairs in the candidate head entity set and the candidate tail entity set. The extraction module builds the PowerSpanBERT-GA-PGD-CPN model, outputs the head-tail entity pairs and their corresponding relationship types, generates entity relationship triples based on the irrigation results, and completes the joint extraction of entity relationships involved in electricity carbon market transactions.

9. The system for extracting entity relationships of electric carbon market transactions based on cascade pointer networks according to claim 8 is characterized by: The word segmentation dictionary building module also includes: The average information content calculation module of a word first calculates the average information content of a word, and the calculation formula is as follows: The average information content of word ν is estimated on a large corpus using the following formula: In the formula, p ν is the probability of word ν appearing, -logp ν It is the amount of information contained in the word ν. A point mutual information calculation module between two adjacent characters, which calculates the point mutual information between two adjacent characters based on the average information amount calculation result of the characters; In the formula, ν1 and ν2 represent two adjacent characters respectively, PMI(ν1, ν2) represents the point mutual information of characters ν1 and ν2, the probability of characters ν1 and ν2 appearing separately in the text dataset is represented by P(ν1) and P(ν2), and P(ν1, ν2) is the probability that ν1 and ν2 are adjacent in this corpus. The left entropy and right entropy calculation module of the word calculates the left entropy and right entropy of the word ξ according to the average information amount calculation formula (1) of the character ν, and derives the left and right entropy product formula as shown in formula (3): H(ξ)=H(ξ left )·H(ξ right ) (3) The word segmentation dictionary generation module in the field of electric carbon market transactions sets the minimum threshold of point mutual information PMImin of word segmentation to 0.5, and sets the minimum threshold of left and right entropy product H(ξ)min to 2, thereby generating a word segmentation dictionary in the field of electric carbon market transactions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the processor executes the computer program to implement the method according to any one of claims 1 to 7.

Images