A knowledge extraction system for power standards

Abstract

The present invention discloses an electric power standard knowledge extraction system, which belongs to the technical field of electric power professional knowledge, and is used to establish an efficient construction of electric power standard professional knowledge map; an electric power standard field database and a model training module are introduced, the model training module obtains electric power standard field data from the electric power standard field database, and obtains a BERT pre-training model through pre-training, obtains a text embedded representation of unit text data, and a parameter modification module is used to modify the next sentence prediction task in the BERT pre-training model. The advantage of this method is that the input unit text data is longer, so that the BERT pre-training model can learn longer dependencies; at the same time, the parameter modification module changes the single-character MASK operation in the original BERT pre-training model to the MASK operation of continuous characters, so that more entity semantic information can be integrated into the BERT pre-training model, which can be more conducive to the improvement of entity recognition task performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of electric power professional knowledge, and in particular is an electric power standard knowledge extraction system. Background Art

[0002] The power standard documents contain a large amount of power professional knowledge, which currently exists in the form of unstructured data such as books. In order to effectively utilize this knowledge, knowledge graphs are a good way. How to establish an efficient construction of power standard professional knowledge graphs has become an important step.

[0003] Therefore, this paper proposes a knowledge extraction system for power standard documents. Summary of the invention

[0004] In order to solve the problems existing in the above-mentioned solutions, the present invention provides a power standard knowledge extraction system.

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] A power standard knowledge extraction system includes a feature annotation module and a model training module; the feature annotation module is used to receive unit text data, and identify the explicitness and implicitness of the unit text data through an explicit and implicit recognition unit, and perform different annotations according to the explicitness and implicitness to obtain knowledge tuples;

[0007] The model training module is used to obtain power standard field data from the power standard field database, and obtain a BERT pre-training model through pre-training, thereby obtaining a text embedded representation of the knowledge tuple;

[0008] The improved Transformer is used to encode the text embedded representation of unit text data, and the result is calculated through the Attention mechanism, and the result is input into the fragment pointer network for further calculation;

[0009] When encoding each Chinese character, the sliding window method is used for local modeling. For each Chinese character, Attention is calculated for the m adjacent Chinese characters. The calculation complexity is linearly related to the length of the text sequence n. The width w between adjacent Chinese characters is obtained. When the number of layers of the fragment pointer network is L, the field of view is L*m*w.

[0010] Furthermore, the unit text data is the result of rule segmentation performed by the rule segmentation module on the processed text file.

[0011] Furthermore, it also includes a text format recognition module and a text conversion module. The text format recognition module is used to recognize the power standard text. When the text format recognition module recognizes that the format of the power standard text is the PDF data format, the text format recognition module is connected to the text conversion module. The text conversion module converts the PDF data format into the text format by text conversion, and marks the converted text format file as a processed text file.

[0012] Furthermore, the feature annotation module includes an explicit annotation unit, an implicit annotation unit, and an explicit and implicit recognition unit;

[0013] When the feature annotation module receives the unit text data, the explicitness or implicitness of the unit text data is identified through the explicitness or implicitness identification unit.

[0014] Furthermore, the process of encoding the text embedded representation of unit text data using the improved Transformer includes the following steps:

[0015] The original text is encoded through different linear transformations, and then the results are calculated through the Attention mechanism:

[0016] head i =Attetion(QW i Q , KW i K , VW i V )

[0017] MultiHead(Q,K,V)=Concat(head 1 , ..., head h )W O

[0018] Among them, head i represents the calculation method of a head entity contained in the model, i represents the index number of the head entity, i is a positive integer, and i = 1, 2...h; where W i Q , W i K , W i V Indicates that the original sequence feature dimension is d model The sequences Q, K, and V are mapped to the three inputs of the head; MultiHead means concatenating the calculation results of all head entities and multiplying them by a W O The matrix is ​​finally used as the final output result of the current layer and then input into the fragment pointer network for further calculation.

[0019] Furthermore, when the text format recognition module recognizes that the electric power standard text is in a non-PDF text format, the text format recognition module will connect to a conversion tool that can convert the electric power standard text in the data format, and convert the electric power standard text in the non-PDF text format into a text format; the text conversion module will subject the converted text format to expert review, organize the converted text, and finally output a neat text, marking the neat text as a processed text file.

[0020] Furthermore, the knowledge tuple is a data group obtained by annotating unit text data through a feature annotation module.

[0021] Furthermore, it also includes a parameter modification module, which is used to modify the next sentence prediction task in the BERT pre-training model.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. The present invention introduces an electric power standard field database and a model training module. The model training module obtains electric power standard field data from the electric power standard field database, and obtains a BERT pre-training model through pre-training, thereby obtaining a text embedded representation of the unit text data. The present invention is provided with a parameter modification module, and the parameter modification module is used to modify the next sentence prediction task in the BERT pre-training model. The advantage of this method is that the input unit text data is longer, so that the BERT pre-training model can learn longer dependencies, so that the length of each input unit text data can reach a maximum of 512 or even longer. At the same time, the parameter modification module changes the single-character MASK operation in the original BERT pre-training model to the MASK operation of continuous characters, so that more entity semantic information can be integrated into the BERT pre-training model, which can be more conducive to the improvement of entity recognition task performance.

[0024] 2. We specifically proposed a decoding scheme for the fragment pointer network, which is connected to the Adaptive-Transformer layer above. Different from the traditional CRF decoding scheme, the decoding scheme we proposed here has better efficiency and stronger targeting than the CRF decoding scheme for sparse and embedded entities on long texts, thereby alleviating the inconsistency and missingness of manual annotation standards. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 The schematic diagram of a power standard knowledge extraction system of the present invention is shown in FIG. DETAILED DESCRIPTION

[0027] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] like Figure 1 As shown, a power standard knowledge extraction system includes a text format recognition module, the text format recognition module is used to recognize the power standard text, wherein the data format of the power standard text includes multiple types, the most basic data format of the power standard text is the PDF data format, in some special occasions, the format of the power standard text may be other types of data formats, when the text format recognition module recognizes that the format of the power standard text is the PDF data format, the text format recognition module is connected to the text conversion module, the text conversion module converts the PDF data format into the text format by text conversion, and marks the converted text format file as a processed text file;

[0029] Among them, the conversion tool for converting PDF data format is OCR;

[0030] It should be noted that when the text format recognition module recognizes that the electric power standard text is in a non-PDF text format, the text format recognition module will connect to a conversion tool that can convert the electric power standard text in the data format, and convert the electric power standard text in the non-PDF text format into a text format; the text conversion module will conduct an expert review on the converted text format, organize the converted text, and finally output a neat text, mark the neat text as a processed text file, and send the processed text file to the rule segmentation module;

[0031] It should be explained that the rule segmentation module is used to perform rule segmentation on the received processed text file, and segment the processed text file into a plurality of unit text data, wherein the unit text data includes the body + title; and then send the unit text data to the feature annotation module; the feature annotation module includes an explicit annotation unit, an implicit annotation unit and an explicit and implicit recognition unit;

[0032] When the feature annotation module receives the unit text data, the explicitness or implicitness of the unit text data is identified by the explicitness or implicitness identification unit. When the explicitness or implicitness identification unit determines that the unit text data is an explicit relationship, the following annotation method is adopted:

[0033] For example, "For liquid-immersed reactors, the reference temperature is 75°C", the final annotation is ["liquid-immersed reactor", "reference temperature", "75°C"];

[0034] The marking standards are:

[0035] "Liquid immersed reactor": start_e1, "Reference temperature": rel, "75℃": end_e1

[0036] "start_e1" represents the head entity, "rel" represents the relationship, and "end_e1" represents the tail entity;

[0037] When the explicit and implicit identification unit determines that the unit text data is an implicit relationship, the following annotation method is used:

[0038] “Inrush current level / n The maximum peak current when the reactor is excited is times the rated current. ", marked in the form of ["inrush current level", "definition", "maximum peak current and times the rated current"],["inrush current level","English","inrush current level"];

[0039] The marking standards are:

[0040] "Inrush current level": def1, "The maximum peak current when the reactor is excited is equal to times the rated current”: def2.

[0041] It should be noted that the results of explicit or implicit annotation units will have the following problems:

[0042] First, the input text,data is generally long, and the distance between knowledge tuples is also relatively long.

[0043] Second, the lengths of the head entity and the tail entity are highly diverse, and are sparsely distributed;

[0044] 3. From the examples given in the explicit relations above, it can be seen that there is overlap of head entities between different knowledge tuples.

[0045] It should be noted that the knowledge tuple is a data group after the unit text data is annotated by the feature annotation module; for example, "liquid immersed reactor": start_e1, "reference temperature": rel, "75°C": end_e1 is a knowledge tuple;

[0046] To this end, the present invention introduces an electric power standard field database and a model training module. The model training module obtains electric power standard field data from the electric power standard field database, and obtains a BERT pre-training model through pre-training, thereby obtaining a text embedded representation of the unit text data. The present invention is provided with a parameter modification module, and the parameter modification module is used to modify the next sentence prediction task in the BERT pre-training model. The advantage of this method is that the input unit text data is longer, so that the BERT pre-training model can learn longer dependencies, so that the length of each input unit text data can reach a maximum of 512 or even longer.

[0047] At the same time, the parameter modification module changes the single-character MASK operation in the original BERT pre-training model to the MASK operation of continuous characters. This can integrate more entity semantic information into the BERT pre-training model, which can be more conducive to improving the performance of entity recognition tasks.

[0048] At the same time, the present invention uses the improved Transformer to perform context encoding representation of the text: first, the original text is encoded through different linear transformations, and then the result is calculated through the Attention mechanism:

[0049]

[0050] MultiHead(Q,K,V)=Concat(head 1 ,...,head h )W O

[0051] Among them, head i represents the calculation method of a head entity contained in the model, i represents the index number of the head entity, i is a positive integer, and i = 1, 2...h; where W i Q , W i K , W iV Indicates that the original sequence feature dimension is d model The sequences Q, K, and V are mapped to the three inputs of the head; MultiHead means concatenating the calculation results of all head entities and multiplying them by a W O The matrix is ​​finally used as the final output result of the current layer and then input into the fragment pointer network for further calculation.

[0052] The Attention mechanism here is designed as follows for the long text scenario here:

[0053] For any Chinese character t in the input text sequence S, for a Chinese character r in the text segment [tS, t], the similarity is calculated as follows:

[0054]

[0055] Among them, W k , W q is the matrix of linear transformation of K and Q, p t-r For relative position encoding, the formula is as follows:

[0056]

[0057]

[0058] For each head, the present invention adds a mask function to form a distance in the interval [0,1]:

[0059]

[0060] Where R is a hyperparameter that controls the smoothness of the function, z is a value between [0, S], and z is a parameter that needs to be learned. For each head, z is different. The purpose of formula 6 is to measure the focus of each head differently, so as to perform targeted learning while reducing the amount of calculation. The final attention weight is calculated as follows:

[0061]

[0062] The current head output is calculated as follows:

[0063]

[0064] Among them, W v A matrix called V, where the final output of the current layer is output by formula 2.

[0065] At the same time, the present invention also reduces the amount of attention calculation and improves the performance of the model through the following means:

[0066] When encoding each Chinese character, unlike the original Transformer, which performs global modeling on long documents, the complexity is too high, making the model difficult to train. Therefore, the sliding window method is used here for local modeling. For each Chinese character, only the attention is calculated for the m Chinese characters near it. The calculation complexity is linearly related to the length of the text sequence n. At the same time, in order to broaden the field of view of the model, the idea of ​​dilated convolution is borrowed here. In the sliding window, there will be a gap of a certain width w (hyperparameter) between two adjacent Chinese characters. When the number of layers of Adaptive-Transformer is L, the field of view can reach L*m*w.

[0067] In actual training, the number of layers of Adaptive-Transformer is 6. The first 3 layers use smaller sliding windows to model local information; and the last 3 layers use larger sliding windows to expand the receptive field.

[0068] The above is the model design scheme for dealing with data problem 1. In order to deal with the sparsity of entities in problem 2 and the overlap of entities in problem 3, we specifically proposed a decoding scheme of the fragment pointer network. In the network layer connected to the above Adaptive-Transformer layer, different from the traditional CRF decoding scheme, the decoding scheme we proposed here has better efficiency and stronger pertinence than the CRF decoding scheme in terms of sparsity and embedded entities on long texts. The specific method is as follows:

[0069] 1. The length of the input text is n. Assuming that the number of tags of the current entity is 1, the continuous fragments in the input text can contain embedded entities, that is, the fragments of the entity can be repeated, then the number of candidate entities is If the number of entity tags is k, then it is classified from k multi-label categories: The problem of selecting several entities from entities;

[0070] Second, use two fully connected layers to encode the beginning of the entity (H s ) and end (H e ) position, and the encoding of other positions i and j is represented by H i , H j ;

[0071] 3. Cascade the above two vectors, the difference of the two vectors, the dot product of the two vectors, and finally cascade;

[0072] s i，j =Hi +H j +(H i -H j )+(H i ⊙H j )

[0073] Fourth, perform a softmax operation at the end:

[0074]

[0075] The final loss function is defined as follows:

[0076]

[0077] Among them, p a is the head and tail set of all entities of type a in the sample, Q a It is the head and tail set of all non-entities or entities of non-type a in the sample, thereby alleviating the inconsistency and missingness of manual annotation standards.

[0078] In the embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation; the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the method of this embodiment.

[0079] It is also obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention.

[0080] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0081] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical method of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A power standard knowledge extraction system, characterized in that: It includes feature annotation module, model training module, rule segmentation module, text format recognition module and text conversion module; The text format recognition module is used to recognize the power standard text. When the text format recognition module recognizes that the format of the power standard text is a PDF data format, the text format recognition module is connected to the text conversion module. The text conversion module converts the PDF data format into a text format by a text conversion method, and marks the converted text format file as a processed text file. When the text format recognition module recognizes that the electric power standard text is in a non-PDF data format, the text format recognition module will connect to a conversion tool that can convert the electric power standard text in the data format, and convert the electric power standard text in the non-PDF data format into a text format; the text conversion module will conduct an expert review on the converted text format, organize the converted text, and finally output a neat text, and mark the neat text as a processed text file; The rule segmentation module is used to perform rule segmentation on the received processed text file, and segment the processed text file into a plurality of unit text data, wherein the unit text data includes the body + title; and then send the unit text data to the feature annotation module; The feature annotation module includes an explicit annotation unit, an implicit annotation unit, and an explicit and implicit identification unit; the feature annotation module is used to receive unit text data, and identify the explicitness and implicitness of the unit text data through the explicit and implicit identification unit, and perform annotations in different ways according to the explicitness and implicitness to obtain knowledge tuples; The knowledge tuple is a data group after the unit text data is annotated by a feature annotation module; The model training module is used to obtain power standard field data from the power standard field database, and obtain a BERT pre-training model through pre-training, and input the labeled data group into the BERT pre-training model to obtain a text embedded representation of the knowledge tuple; The improved Transformer is used to encode the text embedded representation, and the result is calculated through the Attention mechanism. The result is input into the fragment pointer network to continue the calculation to obtain the extraction result; The process of encoding text embedded representation using the improved Transformer includes the following steps: The original text is encoded through different linear transformations, and then the results are calculated through the Attention mechanism: ； ； in, Indicates the calculation method of a head entity contained in the model, i represents the index number of the head entity, i is a positive integer, and i=1,2...h; , , Indicates that the original sequence feature dimension is The sequences Q, K, and V are mapped to the three inputs of the head; MultiHead means concatenating the calculation results of all head entities and multiplying them by one The matrix is ​​finally used as the final output result of the current layer, and then input into the fragment pointer network to continue calculation to obtain the extraction result; When encoding each Chinese character, the sliding window method is used for local modeling. For each Chinese character, Attention is calculated for the m adjacent Chinese characters. The calculation complexity is linearly related to the length of the text sequence n. The width w between adjacent Chinese characters is obtained. When the number of layers of the improved Transformer is L, the field of view is L*m*w. It also includes a parameter modification module, which is used to modify the next sentence prediction task in the BERT pre-training model to make the input unit text data longer, so that the BERT pre-training model can learn longer dependencies. At the same time, the parameter modification module changes the single-character MASK operation in the original BERT pre-training model to the MASK operation of continuous characters, and integrates more entity semantic information into the BERT pre-training model.

Images