Label and text joint representation-based element extraction method

Abstract

The embodiment of the invention discloses an element extraction method based on label and text joint representation, and the method comprises the steps: carrying out the element definition of a standard text in the power field, and marking an element label; performing a language model shielding task on a preset model by using the marked power field standard text to obtain a pre-training model; using the pre-training model to encode a standard body text, a label and an attribute feature to obtain an encoding vector; performing fluctuation pooling convolution operation on the coding vector of the standard body text to obtain a body feature vector; fusing the text feature vector and the coding vectors of the labels and the attribute features by using a label attention mechanism to obtain a fusion vector, and inputting the fusion vector into a classifier to obtain a power field standard element type; according to the method, through tag attribute embedding and tag embedding enhanced text representation, the accuracy and robustness of standard element extraction in the power field are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular, to a method for extracting elements based on the joint representation of tags and text bodies. Background Art

[0002] With the growth of the economy and continuous technological innovation, informatization and intelligence have achieved good results in many fields. Informatization and intelligence also play an important role in the power field. With the improvement of business capabilities, the number of power field standards has been increasing continuously, and the previous business models are difficult to effectively and quickly process a large number of standards.

[0003] The standard elements in the power field refer to the classification of the types and characteristics of standards according to relevant laws and regulations. The extraction of standard elements in the power field refers to using natural language processing methods to extract features from standard texts and identify their element types, and the element types are often multiple. The extraction of standard elements in the power field can effectively and quickly identify the element information of standards, and effectively reduce labor costs. From the perspective of business optimization, the rapid identification of the types and characteristics of standards by standard elements can be applied to the business of standard research and judgment analysis. In the informatization construction, the extraction of standard elements can support system functions such as case matching and standard retrieval, and improve the usability and robustness of the informatization system.

[0004] Therefore, there is an urgent need for a method for extracting standard elements in the power field with high accuracy and robustness. Summary of the Invention

[0005] The main purpose of the present invention is to provide a method for extracting elements based on the joint representation of tags and text bodies, which has high accuracy and robustness in the extraction of standard elements in the power field.

[0006] To achieve the above object, the present application provides a method for extracting elements based on the joint representation of tags and text bodies, and the method includes:

[0007] Define elements for the standard text in the power field and mark element tags;

[0008] Perform a masked language model task on the marked standard text in the power field on a preset model to obtain a pre-trained model;

[0009] Use the pre-trained model to encode the standard text body, tags, and attribute features to obtain encoded vectors;

[0010] Perform a wave pooling convolution operation on the encoded vectors of the standard text body to obtain text body feature vectors;

[0011] For the text feature vector, as well as the encoding vectors of the label and the attribute feature, a label attention mechanism is used for fusion to obtain a fusion vector, and the fusion vector is input into a classifier to obtain the standard element type in the power field.

[0012] Optionally, the standard text in the power field includes multiple standard texts, and each standard text includes multiple words;

[0013] The element definition of the standard text in the power field and the marking of element labels include:

[0014] A corresponding label set is set for each of the standard texts;

[0015] Determine the attribute features of the standard text in the power field, and corresponding element labels are set under each of the attribute features.

[0016] Optionally, the use of the marked standard text in the power field to perform a masked language model task on a preset model to obtain a pre-trained model includes:

[0017] After performing a word masking operation on the standard text, use the preset model to predict the masked words of the standard text for learning to obtain a pre-trained model.

[0018] Optionally, the specific processing process of encoding the standard text, label, and attribute features using the pre-trained model to obtain encoding vectors includes:

[0019] H x = BERT(X)

[0020] H a = BERT(A)

[0021] H y = BERT(Y)

[0022] Where X represents the standard text in the power field, A represents the attribute features of the standard text in the power field, and Y represents the label of the standard text in the power field; H x 、H a and H y respectively represent the encoding vectors of the standard text, the label, and the attribute feature, and BERT is the pre-trained model.

[0023] Optionally, the wave pooling convolution operation on the encoding vector of the standard text to obtain a text feature vector includes:

[0024] Based on a preset wave function and convolution network parameters, perform convolution processing on the encoding vector of the standard text to obtain an extraction vector;

[0025] Perform a max pooling operation on the extracted vector to obtain the text feature vector.

[0026] Optionally, for the text feature vector, as well as the encoding vectors of the label and the attribute feature, use a label attention mechanism to fuse them to obtain a fused vector, and input the fused vector into a classifier to obtain the standard element types in the power field, including:

[0027] Add the text feature vector, as well as the encoding vectors of the label and the attribute feature, bit by bit to obtain a first feature representation;

[0028] Feed the first feature representation into a feed-forward neural network for further fusion to obtain a second feature representation;

[0029] Calculate attention scores for the second feature representation and the label sets under the corresponding attribute features respectively to obtain the fused vector;

[0030] Use the fused vector to predict the text element type and obtain the probability that the text belongs to each element type.

[0031] Optionally, the preset model uses bert-base-Chinese as the basic structure, adopts the transformer encoder part, has 12 network layers, a hidden layer dimension of 768, and supports a maximum sequence length of 512 characters.

[0032] On the other hand, the present application provides an element extraction device based on the joint representation of labels and text, including:

[0033] A data processing module for defining elements in the standard text of the power field and marking element labels;

[0034] A first training module for performing a masked language model task on the preset model using the marked standard text of the power field to obtain a pre-trained model;

[0035] A model processing module for encoding the standard text, label, and attribute features using the pre-trained model to obtain encoding vectors;

[0036] The model processing module is further configured to perform a fluctuating pooling convolution operation on the encoding vector of the standard text to obtain a text feature vector;

[0037] The model processing module is further configured to use a label attention mechanism to fuse the text feature vector, as well as the encoding vectors of the label and the attribute feature, to obtain a fused vector, and input the fused vector into a classifier to obtain the standard element types in the power field.

[0038] On the other hand, this application provides an electronic device, including a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor is caused to execute the steps as described in the first aspect and any possible implementation manner thereof.

[0039] On the other hand, this application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is caused to execute each step in the method as described in the first aspect.

[0040] This application provides a method for extracting elements based on the joint representation of tags and text. By defining elements in standard texts in the power field and marking element tags, a masked language model task is performed on the marked standard texts in the power field using a preset model to obtain a pre-trained model. The pre-trained model is used to encode the standard text, tags, and attribute features to obtain encoded vectors. A wave pooling convolution operation is performed on the encoded vectors of the standard text to obtain text feature vectors. For the text feature vectors, as well as the encoded vectors of the tags and the attribute features, a tag attention mechanism is used for fusion to obtain a fusion vector, and the fusion vector is input into a classifier to obtain the standard element types in the power field. This method enhances the representation of text through tag attribute embedding and tag embedding, improving the accuracy and robustness of standard element extraction in the power field. Description of the Drawings

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0042] Among them:

[0043] Figure 1 is a schematic flowchart of a method for extracting elements based on the joint representation of tags and text provided by an embodiment of this application;

[0044] Figure 2 is a schematic diagram of a standard element extraction model in the power field incorporating tag information provided by an embodiment of this application;

[0045] Figure 3 is a schematic structural diagram of a device for extracting elements based on the joint representation of tags and text provided by an embodiment of this application. Detailed Embodiments

[0046] To enable those skilled in the art to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of this application.

[0047] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices.

[0048] Referring to "embodiments" herein means that the specific features, structures or characteristics described in connection with the embodiments can be included in at least one embodiment of this application. The phrase appears in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0049] The normalization exponential function, or Softmax function, mentioned in the embodiments of this application is a generalization of the logistic function. It can "compress" a K-dimensional vector z containing arbitrary real numbers into another K-dimensional real vector σ(z), such that the range of each element is between (0,1), and the sum of all elements is 1. This function is mostly used in multi-classification problems.

[0050] The principle of the method in the embodiments of this application mainly lies in: first, constructing a pre-trained language model oriented to the standards in the power field; by using this pre-trained language model to encode the standard text in the power field, and adopting a fluctuation pooling strategy to screen out effective feature values from the power text feature sequence; for label information and attribute information, using a pre-trained model for encoding, and enhancing the representation of the text through label attribute embedding and label embedding; finally, using a discriminator to determine which label has the highest execution degree of the element, so as to achieve the purpose of element extraction. The method in this application has achieved good results in the task of extracting standard elements in the power field.

[0051] The embodiments of this application will be described below in conjunction with the accompanying drawings in the embodiments of this application.

[0052] Please refer to Figure 1, which is a schematic flowchart of a method for extracting elements based on the joint representation of tags and text provided by an embodiment of the present application. As Figure 1 shown, the method includes:

[0053] 101. Define elements in the standard text of the power field and mark element tags.

[0054] The method in the embodiment of the present application can be executed by an element extraction device based on the joint representation of tags and text, specifically a terminal device.

[0055] Specifically, standard texts in the power field can be collected, which may include types such as planning, scheduling, metering, operation, and infrastructure. For each standard text, standard elements, that is, element tags, are marked.

[0056] In an alternative embodiment, the above-mentioned standard texts in the power field include multiple standard texts, and each standard text includes multiple words;

[0057] The above step 101 includes:

[0058] Set a corresponding tag set for each of the above standard texts;

[0059] Determine the attribute characteristics of the above-mentioned standard texts in the power field, and set corresponding element tags under each of the above attribute characteristics.

[0060] Specifically, in the standard text of the power field, each standard text contains multiple words, and each standard text corresponds to its tag set Y; and the attribute characteristics A for power elements are summarized, and there are fixed element tags under each attribute characteristic.

[0061] 102. Perform a masked language model task on the above-marked standard texts in the power field on a preset model to obtain a pre-trained model.

[0062] The masked language model task mentioned in the embodiment of the present application refers to the training task of the word masking (Masked Language Model, MLM) method.

[0063] Word masking in natural language processing is a common pre-trained model method. Its core idea is to mask a specific percentage of words in a given sentence, and then let the model predict these masked words based on the other words in the sentence. This method not only enables the model to learn language rules but also improves its performance in downstream tasks. Compared with other natural language processing technologies, MLM has unique advantages, such as being able to capture context information and automatically learn language rules without manual annotation.

[0064] Optionally, the preset model in the embodiments of the present application uses bert-base-Chinese as the basic structure, adopts the transformer encoder part, has 12 network layers, a hidden layer dimension of 768, and supports a maximum sequence length of 512 characters.

[0065] In an alternative embodiment, step 102 includes:

[0066] After performing a word masking operation on the above standard text, use the above preset model to predict the masked words in the above standard text for learning to obtain a pre-trained model.

[0067] Specifically, the standard text in the power field can be used to perform a masked language model task on the training preset model. Perform a word masking operation on the standard text, replace the words in the standard text with special symbols, and then the model predicts the masked words. The training process can be described as:

[0068]

[0069] Among them, BERT is the pre-trained model, and m(·) is the word masking operation.

[0070] 103. Use the above pre-trained model to encode the standard text body, label, and attribute features to obtain encoded vectors.

[0071] Using the pre-trained model obtained in the above step 102, the standard text body, label, and attribute can be encoded to obtain corresponding representation vectors.

[0072] In an alternative embodiment, the specific processing process of using the above pre-trained model to encode the standard text body, label, and attribute features to obtain encoded vectors includes:

[0073] H x = BERT(X)

[0074] H a = BERT(A)

[0075] H y = BERT(Y)

[0076] Among them, X represents the above standard text in the power field, A represents the attribute features of the above standard text in the power field, and Y represents the label of the above standard text in the power field; H x 、H a and H y respectively represent the encoded vectors of the above standard text body, the above label, and the above attribute features, and BERT is the above pre-trained model.

[0077] 104. Perform a wave pooling convolution operation on the encoded vector of the above standard text body to obtain a text body feature vector.

[0078] The text sequence length of the factual description part is relatively long, which contains rich semantic information. As a simple and effective feature extractor, the convolutional network can, in combination with the pooling layer, quickly learn the effective features of the text within a certain distance. In the embodiments of the present application, a feature extraction network composed of a convolutional neural network can be used to extract features from the long text sequence.

[0079] The corresponding wave function and convolutional network parameters can be set as needed, and the embodiments of the present application do not limit this.

[0080] In an optional implementation manner, the above-mentioned performing a wave pooling convolution operation on the encoded vector of the above standard text body to obtain a text body feature vector includes:

[0081] Based on a preset wave function and convolutional network parameters, perform a convolution process on the encoded vector of the above standard text body to obtain an extraction vector;

[0082] Perform a max pooling operation on the above extraction vector to obtain the above text body feature vector.

[0083] Specifically, performing a wave convolution calculation on the standard text body representation to obtain a representation vector, and the process can be:

[0084]

[0085] C i =σ(W·H x +b)

[0086] where σ() is the wave function, and W and b are convolutional network parameters;

[0087] Perform a max pooling operation on the extracted vector:

[0088] h X =MaxPooling(C)

[0089] For each convolution with a length of h, m are set, and the finally pooled feature map obtained by the convolution of this type of convolution kernel is an m-dimensional vector.

[0090] 105. For the above text body feature vector, as well as the encoded vectors of the above labels and the above attribute features, use a label attention mechanism to perform fusion to obtain a fusion vector, and input the above fusion vector into a classifier to obtain the standard element type in the power field.

[0091] In the embodiments of the present application, for the main text, labels, and attribute vectors, a label attention mechanism is used to fuse them to obtain a fused vector, and the fused vector is input into a classifier to obtain the standard element types in the power field. Specifically, it may include three steps: feature fusion, feature transformation, and attention weighted representation.

[0092] In an alternative embodiment, for the encoding vectors of the above standard main text, the above labels, and the above attribute features, a label attention mechanism is used to fuse them to obtain a fused vector, and the above fused vector is input into a classifier to obtain the standard element types in the power field, including:

[0093] Adding the encoding vector of the above standard main text and the encoding vector of the above attribute feature bit by bit to obtain a first feature representation;

[0094] Feeding the above first feature representation into a feed-forward neural network for further fusion to obtain a second feature representation;

[0095] Calculating attention scores for the above second feature representation and the label set corresponding to the corresponding attribute feature respectively to obtain the above fused vector;

[0096] Using the above fused vector to predict the text element type to obtain the probability that the text belongs to each element type.

[0097] Optionally, the first feature representation is obtained by adding the main text representation and the attribute representation bit by bit, which may include adding the encoding vector of the standard main text and the encoding vector of the above attribute feature bit by bit, or may be adding the encoding vector of the standard main text, the encoding vector of the label, and the encoding vector of the attribute feature bit by bit.

[0098] Among them, in the feature transformation step, the feature representation can be fed into a feed-forward neural network for further fusion. In this way, sparse features are integrated, and the representation ability is improved through network parameter learning. The process can be expressed by the formula:

[0099] h = f(W 1 h f + b 1 )

[0100] Among them, h f is the first feature representation, and W 1 and b 1 are transformation parameters.

[0101] In the attention weighted representation step, the representation integrating the attribute information and the label set under the attribute are used to calculate attention scores respectively. The attention scores explicitly show the degree of attention of the text to different interaction objects. The process can be:

[0102]

[0103] h o = s i H Y

[0104] h o That is, the fusion vector can be understood as the representation of the standard text fusion feature vector.

[0105] Then, the standard text fusion feature vector representation h o can be used to predict the element type of the text:

[0106]

[0107] Finally, the probability that the text belongs to each element type is obtained.

[0108] In practical applications, a trained model can be used to identify the element types of standard texts in the power field. The model can predict and output the probability that the text belongs to each element type. The extraction of standard elements in the power field can effectively and quickly identify the element information of the standard, and can effectively reduce the labor cost. From the perspective of business optimization, the standard element recognition can quickly distinguish the types and characteristics of the standards, and can be applied to the business of standard research and judgment analysis. In the informatization construction, the extraction of standard elements can support system functions such as case matching and standard retrieval, and improve the usability and robustness of the informatization system.

[0109] The element extraction method based on the joint representation of tags and text in the embodiments of the present application proposes a pre-trained model for power field standards, which strengthens the feature extraction ability of standard texts; aiming at the problem of sparse text representation features, hierarchical max-pooling convolution operations are used to obtain text feature vectors; in the embodiments of the present application, tags and attribute information are incorporated into the standard element prediction, effectively improving the model's understanding of standard texts and the element prediction results.

[0110] Figure 2 It is a schematic diagram of a power field standard element extraction model incorporating tag information provided by an embodiment of the present application. As Figure 2 shown, a method for element extraction based on the joint representation of tags and text provided by an embodiment of the present application may specifically include the following steps:

[0111] a1. Annotation of power field standard text elements: Collect a large number of power field standard texts, including types such as planning, scheduling, metering, operation, and infrastructure. For each standard text, annotate the standard elements, and the elements can include multiple types. For example, a total of 10 types of elements can be included.

[0112] Specifically, for the power field standard text X = {w 1 , w 2 , w 3 ... wn}, where w i represents the i-th word in the text, and n is the length of the standard text. Each standard text corresponds to its label set Y = {y 1 , y 2 , y 3 ... y s}}. And summarize the attribute characteristics A = {a 1 , a 2 , a 3 ... a m}} for power elements, where each attribute has a fixed element label.

[0113] a2. Produce pre-training data: Collect multiple standard texts in the power field and process the texts using a masking mechanism, with the masked part being the training target;

[0114] a3. Train the pre-training model: For the pre-training model, this application can use bert-base-Chinese as the basic structure, adopt the transformer encoder part, with 12 network layers, a hidden layer dimension of 768, and a maximum supported sequence length of 512 characters. Use the prepared data to train for the masked language model task, with the optimizer being AdamW, the learning rate being 3e-5, the training batch size being 64, and the training target being

[0115]

[0116] where BERT is the pre-training model (parameters), and m(·) is the masking operation on words.

[0117] a4. Use the pre-training model obtained in the previous step to encode the standard text, labels, and attributes, and obtain the corresponding representation vectors. The process is as follows:

[0118] H x = BERT(X)

[0119] H a = BERT(A)

[0120] H y = BERT(Y)

[0121] where H x , H a and H y respectively represent the encoding vectors of the above standard text, labels, and attributes.

[0122] a5. Use a feature extraction network composed of a convolutional neural network to extract features from the long text sequence. Specifically, the convolutional kernel size is 256, and the number of layers is 3. The process is as follows:

[0123]

[0124] C i = σ(WH x + b)

[0125] where σ is the wave function, W and b are convolutional network parameters, and max pooling operation is performed on the extracted vector:

[0126] h X = MaxPooling(C)

[0127] The above h X is the text feature vector, which is denoted as H Figure 2 in xc . For each convolution of length h, m are set, and the finally pooled feature map obtained by the convolution of this type of convolution kernel is an m-dimensional vector.

[0128] a6. For the text, label, and attribute vectors, the label attention mechanism is used for fusion to obtain the fusion vector, and the fusion vector is input into the classifier to obtain the standard element type in the power field. Specifically, it may include:

[0129] a6.1 Feature fusion, adding the text representation, label representation, and attribute representation bit by bit:

[0130] H f = αh x + βH y + εH a

[0131] where α, β, and ε are hyperparameters.

[0132] a6.2 Feature transformation, sending the feature representation into a feedforward neural network for further fusion, integrating sparse features in this way, and improving the representation ability through network parameter learning. The process can be expressed by the formula:

[0133] h = f(W 1 H f + b 1 )

[0134] where W 1 and b 1 are transformation parameters, and f is a non-linear function.

[0135] a6.3 Attention weighted representation, calculating the attention scores for the representation integrating attribute information and the label set under this attribute respectively. The attention scores explicitly show the degree of attention of the text to different interaction objects. The process is as follows:

[0136]

[0137] h o = s i H Y

[0138] Then, the standard text fusion representation vector is used to represent h o Predict the element type of the text:

[0139]

[0140] Finally, the probability that the text belongs to each element type is obtained. During the training process, the learning rate is 3e-5, the optimizer is AdamW, and the hidden layer dimension is 768.

[0141] To more clearly illustrate the element extraction method based on the joint representation of tags and text in this application, experimental comparisons are presented below.

[0142] Specifically, to verify the effectiveness of the method proposed in this application, comparisons are made with multiple well-performing models such as Bi-LSTM-Att, FastText, and BERT. The macro-average F1 value (Macro-F1), micro-average F1 (Micro-F1), and their mean (Average-F1) are used for measurement. The macro-average F1 is obtained by calculating the P and R values for each class, while the micro-average F1 is obtained by calculating the P and R values for all samples. Therefore, when the class is imbalanced, Macro-F1 can better reflect the performance of the model on the minority class. The experimental results are shown in Table 1, which is a comparison experiment result table for standard element extraction in the power field.

[0143]

[0144] Table 1

[0145] As can be seen from Table 1, the method for extracting standard elements in the power field based on the joint representation of tags and text proposed in this application has better performance than other conventional methods. Compared with the previously best-performing Roberta model, this application has improved in all three metrics. The reason is that the joint representation of tags and text helps the model understand the meaning of the standard text, further improving the effect of element extraction.

[0146] Based on the description of the foregoing method embodiments, an element extraction device based on the joint representation of tags and text is further provided in an embodiment of this application.

[0147] Figure 3 As shown in the structure diagram of an element extraction device based on the joint representation of tags and text provided in an embodiment of this application, Figure 3 as shown, the device 300 includes:

[0148] A data processing module 310 is configured to define elements in the standard text of the power field and mark element tags.

[0149] A first training module 320 is configured to perform a masked language model task on the above-mentioned marked standard text of the power field on a preset model to obtain a pre-trained model.

[0150] A model processing module 330 is configured to encode the standard text, tags, and attribute features using the above-mentioned pre-trained model to obtain encoded vectors.

[0151] The above-mentioned model processing module 330 is further configured to perform a wave pooling convolution operation on the encoded vectors of the above-mentioned standard text to obtain text feature vectors.

[0152] The above-mentioned model processing module 330 is further configured to fuse the above-mentioned text feature vectors, as well as the encoded vectors of the above-mentioned tags and the above-mentioned attribute features, using a label attention mechanism to obtain a fused vector, and input the above-mentioned fused vector into a classifier to obtain the standard element types in the power field.

[0153] It can be understood that the relevant content related to each module in the above device has been described in detail in the foregoing method embodiments, and specific reference can be made to the content in the method embodiments; that is, an element extraction device based on joint representation of tags and text provided in the present application can execute any step in the Figure 1 or Figure 2 illustrated embodiments, which will not be elaborated here.

[0154] In an embodiment of the present application, an electronic device is further proposed. The electronic device may include a processor and a memory, and the memory stores a computer program. When the computer program is executed by the processor, it will execute any step in the Figure 1 or Figure 2 illustrated method embodiments. The electronic device may further include input / output devices, etc. In a specific implementation manner, the electronic device may be a terminal device, etc.

[0155] In an embodiment, a computer-readable storage medium is further proposed. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor is caused to execute any step in the above-mentioned method embodiments.

[0156] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0157] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0158] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the patent of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A method for extracting elements based on joint representation of labels and text, characterized in that: The method comprises: Define elements of standard texts in the power sector and mark element labels; Using the marked standard text in the electric power field to perform a masked language model task on a preset model to obtain a pre-trained model; Encoding the standard text, labels and attribute features using the pre-trained model to obtain an encoding vector; Performing a wave pooling convolution operation on the encoding vector of the standard text to obtain a text feature vector; The text feature vector, and the encoding vectors of the label and the attribute feature are fused using a label attention mechanism to obtain a fused vector, and the fused vector is input into a classifier to obtain a standard element type in the power field.

2. The method for extracting elements based on joint representation of labels and text according to claim 1, characterized in that: The standard text in the electric power field includes a plurality of standard texts, each of which includes a plurality of words; The element definition and element labeling of the standard text in the power field include: Each of the standard texts sets a corresponding tag set; The attribute characteristics of the standard text in the electric power field are determined, wherein a corresponding element label is set under each of the attribute characteristics.

3. The method for extracting elements based on joint representation of labels and text according to claim 2 is characterized in that: The method of using the marked electric power field standard text to perform a masked language model task on a preset model to obtain a pre-trained model includes: After performing a word masking operation on the standard text, the preset model is used to predict the masked words of the standard text for learning to obtain a pre-trained model.

4. The method for extracting elements based on joint representation of labels and text according to claim 3 is characterized in that: The specific process of using the pre-trained model to encode the standard text, labels and attribute features to obtain the encoding vector includes: H x =BERT(X) H a =BERT(A) H y =BERT(Y) Wherein, X represents the standard text in the electric power field, A represents the attribute feature of the standard text in the electric power field, and Y represents the label of the standard text in the electric power field; H x , H a and H y Respectively represent the encoding vectors of the standard text, the label and the attribute feature, and BERT is the pre-trained model.

5. The method for extracting elements based on joint representation of labels and text according to claim 4 is characterized in that: The step of performing a wave pooling convolution operation on the encoding vector of the standard text to obtain a text feature vector includes: Based on the preset wave function and convolution network parameters, the encoding vector of the standard text is convolved to obtain an extraction vector; A maximum pooling operation is performed on the extracted vector to obtain the text feature vector.

6. The method for extracting elements based on joint representation of labels and text according to claim 1, characterized in that: The text feature vector, the encoding vector of the label and the attribute feature are fused using the label attention mechanism to obtain a fusion vector, and the fusion vector is input into the classifier to obtain the standard element type in the power field, including: Adding the text feature vector and the encoding vectors of the label and the attribute feature in a position-by-position manner to obtain a first feature representation; Sending the first feature representation to a feedforward neural network for further fusion to obtain a second feature representation; Calculating attention scores for the second feature representation and the label set under the corresponding attribute feature respectively to obtain the fusion vector; The fusion vector is used to predict the text element type to obtain the probability that the text belongs to each element type.

7. The method for extracting elements based on joint representation of labels and text according to claim 3 is characterized in that: The preset model uses bert-base-Chinese as the basic structure, adopts the transformer encoder part, has 12 network layers, a hidden layer dimension of 768, and supports a maximum sequence length of 512 characters.

8. A feature extraction device based on joint representation of tags and text, characterized in that: include: The data processing module is used to define the elements of the standard text in the power field and mark the element labels; A first training module is used to perform a masked language model task on a preset model using the marked standard text in the electric power field to obtain a pre-trained model; A model processing module, used to encode the standard text, labels and attribute features using the pre-trained model to obtain an encoding vector; The model processing module is further used to perform a ripple pooling convolution operation on the encoding vector of the standard text to obtain a text feature vector; The model processing module is also used to use the label attention mechanism to fuse the text feature vector, the label and the encoding vector of the attribute feature to obtain a fusion vector, and input the fusion vector into the classifier to obtain the standard element type in the power field.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

Images