Power field named entity recognition method based on dynamically combinable multi-head attention

Abstract

The invention discloses an electric power field named entity recognition method based on dynamically combinable multi-head attention, which comprises the following steps of: constructing an LERoBERTa-DCMHA model, and inputting a collected electric power text word sequence into the model to obtain a word feature vector of the electric power text word sequence; performing power word vector training according to the dictionary combination and the power text word sequence to obtain a word feature vector of the power text word sequence; performing feature fusion on the character feature vector and the word feature vector to obtain a feature vector of the power text character and word sequence; performing bidirectional feature coding on the feature vectors of the power text word sequence to obtain an output state sequence; performing labeling probability sorting on the output state sequence to obtain a final entity recognition result; according to the method, the dynamically combinable multi-head attention and the convolutional pooling layer TextCNN are added into the pre-training language model, so that the model named entity recognition performance and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the field of electric power knowledge graph technology, and specifically to a method for named entity recognition in the electric power field based on dynamically combined multi-head attention. Background Art

[0002] With the prosperous development of the power industry, a large number of electrical equipment are accompanied by a large amount of relevant text data during the evaluation, inspection, and maintenance process. Before the emergence of revolutionary computer technologies such as cloud computing, 5G, and AI, these data were often filtered, screened, and sorted manually. Manual processing is not only time-consuming and labor-intensive, but also error-prone and structurally chaotic, resulting in high consumption and low efficiency. Named entity recognition of text data in the power field uses computers to automatically identify and extract named entity objects related to power from unstructured and semi-structured power text word sequences. These named entities lay the foundation for the subsequent construction of intelligent information of power systems and related research and evaluation.

[0003] At present, named entity recognition technology based on deep learning has made great progress, but the calculation and utilization efficiency of its model are still insufficient; at the same time, named entity recognition tasks in vertical fields have problems such as unregistered words and small number of tags. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a method for named entity recognition in the electric power field based on dynamically combined multi-head attention, which aims to solve the problems in the background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for named entity recognition in the electric power field based on dynamically combined multi-head attention, comprising the following steps: Step S1: constructing a corpus data set, wherein the corpus data set includes a plurality of electric power text word sequences; Step S2: construct a LERoBERTa-DCMHA model, input the electric power text word sequence into the LERoBERTa-DCMHA model, and obtain the word feature vector of the electric power text word sequence; The specific process of building the LERoBERTa-DCMHA model is: Replace the multi-head attention in the pre-trained language model RoBERTa with dynamic combined multi-head attention DCMHA to obtain the RoBERTa-DCMHA model; then add the convolutional pooling layer TextCNN to the RoBERTa-DCMHA model to obtain the LERoBERTa-DCMHA model; The LERoBERTa-DCMHA model is composed of N groups of TextCNN-DFM modules stacked in sequence, and the TextCNN-DFM module is composed of a dynamic combination of multi-head attention DCMHA, a residual connection and layer normalization module, a feedforward neural network module, a residual connection module, a feature fusion module and a layer normalization module, a convolutional pooling layer TextCNN and a linear layer; the processing flow of the first group of TextCNN-DFM modules is as follows: first, the input data, i.e., the electric power text word sequence, is sequentially input into the dynamic combination of multi-head attention and convolutional pooling layers, and the output of the dynamic combination of multi-head attention and the input data are input into the residual connection and layer normalization module, input the output of the residual connection and layer normalization module into the feedforward neural network module, input the output of the feedforward neural network module and the output of the residual connection and layer normalization module into the residual connection module, input the output of the convolution pooling layer after passing through the linear layer and the output of the residual connection module into the feature fusion module, input the output of the feature fusion module into the layer normalization module, use the output of the layer normalization module as the output of the first group of TextCNN-DFM modules, and use the output of the last group of TextCNN-DFM modules as the final output of the LERoBERTa-DCMHA model, i.e., the word feature vector of the electric power text word sequence; Step S3: construct a dictionary combination, and perform electric power vocabulary word vector training based on the dictionary combination and the electric power text word sequence to obtain a word feature vector of the electric power text word sequence; Step S4: performing feature fusion on the character feature vector and the word feature vector of the electric power text word sequence to obtain the feature vector of the electric power text word sequence; Step S5: using a bidirectional Lattice LSTM encoding model to perform bidirectional feature encoding on the feature vector of the electric power text word sequence to obtain an output state sequence of the electric power text word sequence; Step S6: sorting the output state sequence of the electric power text word sequence by labeling probability to obtain the final entity recognition result.

[0006] Furthermore, the multi-head attention in the pre-trained language model RoBERTa is replaced by the dynamic combined multi-head attention DCMHA. The Compose function operation is introduced into the multi-head attention in the pre-trained language model RoBERTa to obtain the dynamic combined multi-head attention DCMHA. The core of the dynamic combined multi-head attention is the Compose function. Given a query and key , the Compose function converts the attention matrix , Represents the field of real numbers of dimensional column vector space, transformed into a vector space with trainable parameters The attention matrix : ; In the formula, the trainable parameters , represents the weight matrix, , Both indicate independence Dynamic weight of Indicates independence from Dynamic gating weights of , Indicates independence from Dynamic weight of Indicates independence from Dynamic gating weights of Represents the Compose function; Will Expand: ; In the formula, express function; express Activation function; represents the root mean square normalization operation; Represents a reshape operation; Represents element-wise product.

[0007] Furthermore, the specific process of inputting the electric power text word sequence into the LERoBERTa-DCMHA model to obtain the word feature vector of the electric power text word sequence is as follows: Step S2.1: Read the electric power text word sequence; Step S2.2: Integrate the existing text vocabulary information in the electric power field into each character sequence in the electric power text word sequence to obtain a vocabulary-enhanced electric power text word sequence; Step S2.3: Use existing text data in the power field to pre-train and fine-tune the LERoBERTa-DCMHA model; Step S2.4: Input the vocabulary-enhanced electric power text word sequence into the fine-tuned LERoBERTa-DCMHA model to obtain a character input vector; Step S2.5: Perform feature training on the character input vector to obtain a character feature vector of the electric power text word sequence.

[0008] Furthermore, the feature fusion module in the LERoBERTa-DCMHA model is expressed as: ; In the formula, Represents the output vector after feature fusion; represents the input vector; Represents the weight of the feature channel of the convolutional pooling layer; Represents the channel processing operation of the convolution pooling layer; Indicates increasing the weight of the feature channel before the convolution pooling layer; It represents the feature channel processing operation after adding the convolutional pooling layer.

[0009] Furthermore, the specific process of constructing the dictionary combination is: Step S3.11: respectively obtain Chinese Wikipedia corpus and electric power field word segmentation dictionary; Step S3.12: Based on the GloVe pre-trained language model, perform Jieba word segmentation training on the Chinese Wikipedia corpus to obtain the Wikipedia word vector; Step S3.13: Perform improved word segmentation training on the electric power field word segmentation dictionary to obtain the electric power field word segmentation word vector; Step S3.14: Combine the Wikipedia word vector and the electric power field word vector into a dictionary combination.

[0010] Furthermore, according to the dictionary combination and the electric power text word sequence, the electric power vocabulary word vector training is performed to obtain the word feature vector of the electric power text word sequence as follows: Step S3.21: Read the word sequence in the electric power text word sequence; Step S3.22: Merge the word sequence in the electric power text word sequence with the vectors in the dictionary combination to obtain an updated word sequence of the electric power text word sequence; Step S3.23: Input the updated word sequence of the electric power text word sequence into the LERoBERTa-DCMHA model to obtain a word input vector, perform feature training on the word input vector, and obtain a word feature vector of the electric power text word sequence.

[0011] Furthermore, the specific process of step S5 is as follows: Step S5.1: Based on the softmax activation function and the bidirectional Lattice LSTM encoding model, the feature vector of the electric power text word sequence is encoded with a word vector to obtain a word output vector of the electric power text word sequence; Step S5.2: Based on the bidirectional Lattice LSTM encoding model, the feature vector of the electric power text word sequence is encoded with a word vector to obtain a word output vector of the electric power text word sequence; Step S5.3: performing feature vector fusion on the character output vector of the electric power text word sequence and the word output vector of the electric power text word sequence to obtain an updated feature vector of the electric power text word sequence; Step S5.4: Based on the bidirectional Lattice LSTM encoding model, the updated feature vector of the electric power text word sequence and the feature vector of the electric power text word sequence are respectively subjected to hidden layer unidirectional encoding and reverse encoding to obtain the unidirectional feature vector and reverse feature vector of the electric power text word sequence respectively; Step S5.5: perform vector concatenation on the unidirectional feature vector and the reverse feature vector of the electric power text word sequence to obtain the bidirectional feature vector of the electric power text word sequence, thereby calculating the output state sequence of the electric power text word sequence.

[0012] Furthermore, the specific process of step S5.1 is as follows: Step S5.11: Based on the softmax activation function and the tanh layer in the bidirectional Lattice LSTM encoding model, encode the feature vector of the electric power text word sequence to obtain the input gate of each index position, the forget gate of each index position, the output gate of each index position, and the candidate value of each index position; Step S5.12: Calculate the memory unit of each index position according to the input gate of each index position, the forget gate of each index position and the candidate value of each index position; Step S5.13: Calculate the word output vector of each index position, that is, the word output vector of the electric power text word sequence, according to the output gate of each index position and the memory unit of each index position.

[0013] Furthermore, the specific process of step S5.3 is as follows: Step S5.31: performing vector fusion on the character output vector of the electric power text word sequence and the word output vector to obtain the word output vector of the gate control unit; Step S5.32: according to the word output vector of the gated unit and the word output vector of each index position of the bidirectional Lattice LSTM encoding model, normalized weight calculations are performed on the word granularity information and the word granularity information, respectively, to obtain the word granularity weight of each index position and the word granularity weight of each index position; Step S5.33: According to the character granularity weight of each index position and the word granularity weight of each index position, the word output vector and the word output vector of each index position of the bidirectional Lattice LSTM encoding model are updated, and the updated feature vector of each index position is calculated, that is, the updated feature vector of the electric power text word sequence.

[0014] Furthermore, the specific process of step S6 is as follows: Step S6.1: Use the CRF model to perform entity sequence labeling on the output state sequence of the electric power text word sequence to obtain the probability of the word label in the output state sequence; Step S6.2: Based on the Viterbi algorithm, the sentence label with the maximum probability of the output state sequence is calculated as the result of entity recognition, thereby completing entity recognition.

[0015] Compared with the existing technology, the present invention has the following beneficial effects: the present invention adopts a pre-trained language model with a dynamically combined multi-head attention structure, which can capture text features at multiple levels, process complex sentence structures and ambiguous phrases, and ensure the overall consistency of label sequences, thereby significantly improving the performance of named entity recognition; by adding a convolutional pooling layer TextCNN, the extraction of local information is further enhanced, thereby improving the accuracy of the model in extracting professional vocabulary information in the power field. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The present invention is a flow chart of the method.

[0017] Figure 2 Schematic diagram of the LERoBERTa-DCMHA model structure of the present invention. DETAILED DESCRIPTION

[0018] like Figure 1 As shown, the present invention provides a technical solution: a method for named entity recognition in the electric power field based on dynamically combined multi-head attention, comprising the following steps:

[0019] Step S1: construct a corpus data set, wherein the corpus data set includes a plurality of electric power text word sequences.

[0020] The specific process of step S1 is: collecting power text data, preprocessing the power text data, obtaining power text word sequences, and forming the obtained power text word sequences into a corpus data set.

[0021] The specific process of obtaining the electric power text word sequence is as follows: performing text cleaning on the electric power text data, including removing non-text content, redundant blank characters, special symbols and repeated text lines in the electric power text data; performing part-of-speech tagging on the electric power text data after text cleaning to obtain the electric power text word sequence.

[0022] Step S2: construct a LERoBERTa-DCMHA model, input the electric power text word sequence into the LERoBERTa-DCMHA model, and obtain the word feature vector of the electric power text word sequence.

[0023] Among them, the specific process of building the LERoBERTa-DCMHA model is:

[0024] The multi-head attention in the pre-trained language model (RoBERTa) is replaced with dynamic combined multi-head attention (DCMHA) to obtain the RoBERTa-DCMHA model; then a convolutional pooling layer (TextCNN) is added to the RoBERTa-DCMHA model to enhance the extraction of local information, and the LERoBERTa-DCMHA model is obtained.

[0025] like Figure 2 As shown in the figure, the LERoBERTa-DCMHA model consists of N groups of TextCNN-DFM modules stacked in sequence, wherein the TextCNN-DFM module consists of a dynamic combination of multi-head attention (DCMHA), a residual connection and layer normalization module, a feedforward neural network module, a residual connection module, a feature fusion module and a layer normalization module, a convolutional pooling layer (TextCNN) and a linear layer; the processing flow of the first group of TextCNN-DFM modules is as follows: first, the input data (electricity text word sequence) is sequentially input into the dynamic combination of multi-head attention (DCMHA) and the convolutional pooling layer, and the output of the dynamic combination of multi-head attention and the input data are sequentially input into the convolutional pooling layer. Input the residual connection and layer normalization module, input the output of the residual connection and layer normalization module into the feedforward neural network module, input the output of the feedforward neural network module and the output of the residual connection and layer normalization module into the residual connection module, input the output of the convolutional pooling layer after passing through the linear layer and the output of the residual connection module into the feature fusion module, input the output of the feature fusion module into the layer normalization module, use the output of the layer normalization module as the output of the first group of TextCNN-DFM modules, and use the output of the last group of TextCNN-DFM modules as the final output of the LERoBERTa-DCMHA model (the word feature vector of the electric power text word sequence).

[0026] Among them, in step S2, the specific process of inputting the electric power text word sequence into the LERoBERTa-DCMHA model to obtain the word feature vector of the electric power text word sequence is:

[0027] Step S2.1: Read the electric power text word sequence.

[0028] Step S2.2: Integrate the existing text vocabulary information in the electric power field into each character sequence in the electric power text word sequence to obtain a vocabulary-enhanced electric power text word sequence.

[0029] Step S2.3: Pre-train and fine-tune the LERoBERTa-DCMHA model using existing text data in the power field.

[0030] Step S2.4: Input the vocabulary-enhanced electric power text word sequence into the fine-tuned LERoBERTa-DCMHA model to obtain a character input vector.

[0031] Step S2.5: Perform feature training on the character input vector to obtain a character feature vector of the electric power text word sequence.

[0032] Among them, in step S2, the multi-head attention in the pre-trained language model (RoBERTa) is replaced with dynamic combined multi-head attention (DCMHA). The Compose function operation is introduced into the multi-head attention in the pre-trained language model (RoBERTa) to obtain dynamic combined multi-head attention (DCMHA); the core of dynamic combined multi-head attention (DCMHA) is the Compose function. Given a query and key , the Compose function converts the attention matrix , Represents the field of real numbers of dimensional column vector space, transformed into a vector space with trainable parameters The new attention matrix : (1); In the formula, the trainable parameters , represents the weight matrix, , Both indicate independence Dynamic weight of Indicates independence from Dynamic gating weights of , Indicates independence from Dynamic weight of Indicates independence from Dynamic gating weights of Represents a Compose function.

[0033] Expand formula (1): (2); In the formula, express function; express Activation function; represents the root mean square normalization operation; Represents a reshape operation; Represents element-wise product.

[0034] The overall process of dynamic combined multi-head attention (DCMHA) can be expressed as: (3); In the formula, , , Query ,key Sum The projection matrix, is the middle dimension, is the hidden layer dimension; represents the attention score matrix; Indicates An attention score matrix; Indicates the parameters used when performing the first Compose operation; Indicates the parameters used when performing the second Compose operation; Represents the output of the attention layer; Indicates The output of the attention layer is ; represents the attention weight tensor; Indicates An attention weight tensor; is the output projection matrix; express function; Indicates transpose.

[0035] The feature fusion module in the LERoBERTa-DCMHA model is expressed as: (4); In the formula, Represents the output vector after feature fusion; represents the input vector; Represents the weight of the feature channel of the convolutional pooling layer (TextCNN); Represents the channel processing operation of the convolution pooling layer (TextCNN); Indicates increasing the weight of the feature channel before the convolution pooling layer (TextCNN); It represents the feature channel processing operation after adding the convolutional pooling layer (TextCNN). Less than , to ensure that the output vector of the modified feature channel is the feature body.

[0036] Step S3: construct a dictionary combination, and perform electric power vocabulary word vector training based on the dictionary combination and the electric power text word sequence to obtain a word feature vector of the electric power text word sequence.

[0037] In step S3, the specific process of constructing the dictionary combination is: Step S3.11: Obtain Chinese Wikipedia corpus and electric power field word segmentation dictionary respectively.

[0038] Step S3.12: Based on the GloVe pre-trained language model, perform Jieba word segmentation training on the Chinese Wikipedia corpus to obtain the Wikipedia word vector.

[0039] Step S3.13: Improve the word segmentation training of the electric power field word segmentation dictionary to obtain the word vector of the electric power field word segmentation.

[0040] Step S3.14: Combine the Wikipedia word vector and the electric power field word vector into a dictionary combination.

[0041] In step S3, the specific process of training the electric power vocabulary word vector according to the dictionary combination and the electric power text word sequence and obtaining the word feature vector of the electric power text word sequence is as follows: Step S3.21: Read the word sequence in the power text word sequence.

[0042] Step S3.22: Merge the word sequence in the electric power text word sequence with the vectors in the dictionary combination to obtain an updated word sequence of the electric power text word sequence.

[0043] Step S3.23: Input the updated word sequence of the electric power text word sequence into the LERoBERTa-DCMHA model to obtain a word input vector, perform feature training on the word input vector, and obtain a word feature vector of the electric power text word sequence.

[0044] Step S4: Fusing the character feature vector and the word feature vector of the electric power text word sequence to obtain the feature vector of the electric power text word sequence.

[0045] Step S5: Use the bidirectional Lattice LSTM encoding model to perform bidirectional feature encoding on the feature vector of the electric power text word sequence to obtain the output state sequence of the electric power text word sequence.

[0046] Step S5.1: Based on the softmax activation function and the bidirectional Lattice LSTM encoding model, the feature vector of the electric power text word sequence is encoded with a word vector to obtain the word output vector of the electric power text word sequence.

[0047] The specific process of step S5.1 is as follows: Step S5.11: Based on the softmax activation function and the tanh layer in the bidirectional Lattice LSTM encoding model, the feature vector of the electric power text word sequence is encoded to obtain the input gate of each index position, the forget gate of each index position, the output gate of each index position and the candidate value of each index position.

[0048] Step S5.12: Calculate the memory unit of each index position according to the input gate of each index position, the forget gate of each index position and the candidate value of each index position.

[0049] Step S5.13: Calculate the word output vector of each index position, that is, the word output vector of the electric power text word sequence, according to the output gate of each index position and the memory unit of each index position.

[0050] Step S5.2: Based on the bidirectional Lattice LSTM encoding model, the feature vector of the electric power text word sequence is encoded to obtain the word output vector of the electric power text word sequence.

[0051] Step S5.3: Perform feature vector fusion on the character output vector of the electric power text word sequence and the word output vector of the electric power text word sequence to obtain an updated feature vector of the electric power text word sequence.

[0052] The specific process of step S5.3 is: Step S5.31: Perform vector fusion on the character output vector of the electric power text word sequence and the word output vector to obtain the word output vector of the gate control unit.

[0053] Step S5.32: According to the word output vector of the gated unit and the word output vector of each index position of the bidirectional Lattice LSTM encoding model, normalized weight calculations are performed on the word granularity information and the word granularity information, respectively, to obtain the word granularity weight of each index position and the word granularity weight of each index position.

[0054] Step S5.33: According to the character granularity weight of each index position and the word granularity weight of each index position, the word output vector and the character output vector of each index position are updated, and the updated feature vector of each index position, that is, the updated feature vector of the electric power text word sequence, is calculated.

[0055] Step S5.4: Based on the bidirectional Lattice LSTM encoding model, the updated feature vector of the electric power text word sequence and the feature vector of the electric power text word sequence are respectively unidirectionally encoded and reversely encoded at the hidden layer to obtain the unidirectional feature vector and reverse feature vector of the electric power text word sequence respectively.

[0056] Step S5.5: perform vector concatenation on the unidirectional feature vector and the reverse feature vector of the electric power text word sequence to obtain the bidirectional feature vector of the electric power text word sequence, thereby calculating the output state sequence of the electric power text word sequence.

[0057] Step S6: sorting the output state sequence of the electric power text word sequence by labeling probability to obtain the final entity recognition result.

[0058] Step S6.1: Use the CRF model to perform entity sequence labeling on the output state sequence of the electric power text word sequence, obtain the probability of the word label in the output state sequence, and satisfy: (5); In the formula, represents an arbitrary sequence of labels for the output; Represents the weight parameter of the CRF model; Represents the bias term of the CRF model; The vector corresponding to the output state sequence to be labeled for the entity sequence; is the label sequence with the highest probability; represents the observation sequence; Indicates a given Under the conditions, Probability of occurrence.

[0059] Step S6.2: Based on the Viterbi algorithm, the sentence label with the maximum probability of the output state sequence is calculated as the result of entity recognition, thereby completing entity recognition.

[0060] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A named entity recognition method in the power field based on dynamically combined multi-head attention, characterized by: The steps include: Step S1: constructing a corpus data set, wherein the corpus data set includes a plurality of electric power text word sequences; Step S2: construct a LERoBERTa-DCMHA model, input the electric power text word sequence into the LERoBERTa-DCMHA model, and obtain the word feature vector of the electric power text word sequence; The specific process of building the LERoBERTa-DCMHA model is: Replace the multi-head attention in the pre-trained language model RoBERTa with dynamic combined multi-head attention DCMHA to obtain the RoBERTa-DCMHA model; then add the convolutional pooling layer TextCNN to the RoBERTa-DCMHA model to obtain the LERoBERTa-DCMHA model; The LERoBERTa-DCMHA model is composed of N groups of TextCNN-DFM modules stacked in sequence, and the TextCNN-DFM module is composed of a dynamic combination of multi-head attention DCMHA, a residual connection and layer normalization module, a feedforward neural network module, a residual connection module, a feature fusion module and a layer normalization module, a convolutional pooling layer TextCNN and a linear layer; the processing flow of the first group of TextCNN-DFM modules is as follows: first, the input data, i.e., the electric power text word sequence, is sequentially input into the dynamic combination of multi-head attention and convolutional pooling layers, and the output of the dynamic combination of multi-head attention and the input data are input into the residual connection and layer normalization module, input the output of the residual connection and layer normalization module into the feedforward neural network module, input the output of the feedforward neural network module and the output of the residual connection and layer normalization module into the residual connection module, input the output of the convolution pooling layer after passing through the linear layer and the output of the residual connection module into the feature fusion module, input the output of the feature fusion module into the layer normalization module, use the output of the layer normalization module as the output of the first group of TextCNN-DFM modules, and use the output of the last group of TextCNN-DFM modules as the final output of the LERoBERTa-DCMHA model, i.e., the word feature vector of the electric power text word sequence; Step S3: construct a dictionary combination, and perform electric power vocabulary word vector training based on the dictionary combination and the electric power text word sequence to obtain a word feature vector of the electric power text word sequence; Step S4: performing feature fusion on the character feature vector and the word feature vector of the electric power text word sequence to obtain the feature vector of the electric power text word sequence; Step S5: using a bidirectional Lattice LSTM encoding model to perform bidirectional feature encoding on the feature vector of the electric power text word sequence to obtain an output state sequence of the electric power text word sequence; Step S6: sorting the output state sequence of the electric power text word sequence by labeling probability to obtain the final entity recognition result.

2. The method for named entity recognition in the electric power field based on dynamically combined multi-head attention according to claim 1 is characterized in that: Replacing the multi-head attention in the pre-trained language model RoBERTa with dynamic combined multi-head attention DCMHA is to introduce the Compose function operation into the multi-head attention in the pre-trained language model RoBERTa to obtain dynamic combined multi-head attention DCMHA; the core of dynamic combined multi-head attention is the Compose function. and key , the Compose function converts the attention matrix , Represents the field of real numbers of dimensional column vector space, transformed into a vector space with trainable parameters The attention matrix : ； In the formula, the trainable parameters , represents the weight matrix, , Both indicate independence Dynamic weight of Indicates independence from Dynamic gating weights of , Indicates independence from Dynamic weight of Indicates independence from Dynamic gating weights of Represents the Compose function; Will Expand: ； In the formula, express function; express Activation function; represents the root mean square normalization operation; Represents a reshape operation; Represents element-wise product.

3. The method for named entity recognition in the electric power field based on dynamically combined multi-head attention according to claim 2 is characterized in that: The specific process of inputting the electric power text word sequence into the LERoBERTa-DCMHA model to obtain the word feature vector of the electric power text word sequence is as follows: Step S2.1: Read the electric power text word sequence; Step S2.2: Integrate the existing text vocabulary information in the electric power field into each character sequence in the electric power text word sequence to obtain a vocabulary-enhanced electric power text word sequence; Step S2.3: Use existing text data in the power field to pre-train and fine-tune the LERoBERTa-DCMHA model; Step S2.4: Input the vocabulary-enhanced electric power text word sequence into the fine-tuned LERoBERTa-DCMHA model to obtain a character input vector; Step S2.5: Perform feature training on the character input vector to obtain a character feature vector of the electric power text word sequence.

4. The method for named entity recognition in the electric power field based on dynamically combined multi-head attention according to claim 3 is characterized in that: The feature fusion module in the LERoBERTa-DCMHA model is expressed as: ； In the formula, Represents the output vector after feature fusion; represents the input vector; Represents the weight of the feature channel of the convolutional pooling layer; Represents the channel processing operation of the convolution pooling layer; Indicates increasing the weight of the feature channel before the convolution pooling layer; It represents the feature channel processing operation after adding the convolutional pooling layer.

5. The method for named entity recognition in the electric power field based on dynamically combined multi-head attention according to claim 4 is characterized in that: The specific process of constructing a dictionary combination is: Step S3.11: respectively obtain Chinese Wikipedia corpus and electric power field word segmentation dictionary; Step S3.12: Based on the GloVe pre-trained language model, perform Jieba word segmentation training on the Chinese Wikipedia corpus to obtain the Wikipedia word vector; Step S3.13: Perform improved word segmentation training on the electric power field word segmentation dictionary to obtain the electric power field word segmentation word vector; Step S3.14: Combine the Wikipedia word vector and the electric power field word vector into a dictionary combination.

6. The method for named entity recognition in the electric power field based on dynamically combined multi-head attention according to claim 5 is characterized in that: According to the dictionary combination and the electric power text word sequence, the electric power vocabulary word vector training is carried out to obtain the word feature vector of the electric power text word sequence as follows: Step S3.21: Read the word sequence in the electric power text word sequence; Step S3.22: Merge the word sequence in the electric power text word sequence with the vectors in the dictionary combination to obtain an updated word sequence of the electric power text word sequence; Step S3.23: Input the updated word sequence of the electric power text word sequence into the LERoBERTa-DCMHA model to obtain a word input vector, perform feature training on the word input vector, and obtain a word feature vector of the electric power text word sequence.

7. The method for named entity recognition in the electric power field based on dynamically combined multi-head attention according to claim 6 is characterized in that: The specific process of step S5 is: Step S5.1: Based on the softmax activation function and the bidirectional Lattice LSTM encoding model, the feature vector of the electric power text word sequence is encoded with a word vector to obtain a word output vector of the electric power text word sequence; Step S5.2: Based on the bidirectional Lattice LSTM encoding model, the feature vector of the electric power text word sequence is encoded with a word vector to obtain a word output vector of the electric power text word sequence; Step S5.3: performing feature vector fusion on the character output vector of the electric power text word sequence and the word output vector of the electric power text word sequence to obtain an updated feature vector of the electric power text word sequence; Step S5.4: Based on the bidirectional Lattice LSTM encoding model, the updated feature vector of the electric power text word sequence and the feature vector of the electric power text word sequence are respectively subjected to hidden layer unidirectional encoding and reverse encoding to obtain the unidirectional feature vector and reverse feature vector of the electric power text word sequence respectively; Step S5.5: perform vector concatenation on the unidirectional feature vector and the reverse feature vector of the electric power text word sequence to obtain the bidirectional feature vector of the electric power text word sequence, thereby calculating the output state sequence of the electric power text word sequence.

8. The method for named entity recognition in the electric power field based on dynamically combined multi-head attention according to claim 7 is characterized in that: The specific process of step S5.1 is: Step S5.11: Based on the softmax activation function and the tanh layer in the bidirectional Lattice LSTM encoding model, encode the feature vector of the electric power text word sequence to obtain the input gate of each index position, the forget gate of each index position, the output gate of each index position, and the candidate value of each index position; Step S5.12: Calculate the memory unit of each index position according to the input gate of each index position, the forget gate of each index position and the candidate value of each index position; Step S5.13: Calculate the word output vector of each index position, that is, the word output vector of the electric power text word sequence, according to the output gate of each index position and the memory unit of each index position.

9. The method for named entity recognition in the electric power field based on dynamically combined multi-head attention according to claim 8 is characterized in that: The specific process of step S5.3 is: Step S5.31: performing vector fusion on the character output vector of the electric power text word sequence and the word output vector to obtain the word output vector of the gate control unit; Step S5.32: according to the word output vector of the gated unit and the word output vector of each index position of the bidirectional Lattice LSTM encoding model, normalized weight calculations are performed on the word granularity information and the word granularity information, respectively, to obtain the word granularity weight of each index position and the word granularity weight of each index position; Step S5.33: According to the character granularity weight of each index position and the word granularity weight of each index position, the word output vector and the word output vector of each index position of the bidirectional Lattice LSTM encoding model are updated, and the updated feature vector of each index position is calculated, that is, the updated feature vector of the electric power text word sequence.

10. The method for named entity recognition in the electric power field based on dynamically combined multi-head attention according to claim 9 is characterized in that: The specific process of step S6 is: Step S6.1: Use the CRF model to perform entity sequence labeling on the output state sequence of the electric power text word sequence to obtain the probability of the word label in the output state sequence; Step S6.2: Based on the Viterbi algorithm, the sentence label with the maximum probability of the output state sequence is calculated as the result of entity recognition, thereby completing entity recognition.

Images