A method, apparatus, and storage medium for joint extraction of text entities and relations in the cement industry based on deep learning and graph theory.

Abstract

This invention relates to a method, apparatus, and storage medium for joint extraction of text entities and relations in the cement industry based on deep learning and graph theory. The method includes: Step S1: performing word embedding processing on the input natural language text to obtain a word sequence; Step S2: extracting the feature vector of each word; Step S3: generating first weights for each word from other words; Step S4: obtaining initial weights from other words on the given word; Step S5: forming an initial adjacency matrix based on the initial weights of each word on other words; Step S6: calculating attention scores to obtain a target adjacency matrix; Step S7: obtaining feature vectors fused with adjacency node information based on the obtained target adjacency matrix; Step S8: obtaining the features of the subsequences; Step S9: inputting the features of the two subsequences into an analysis model to obtain the relationship between the two subsequences. Compared with existing technologies, this invention has advantages such as high accuracy.

Description

TECHNICAL FIELD

[0001] The application relates to the field of cement, in particular to a cement field text entity-relation joint extraction method and device based on deep learning and graph theory and a storage medium. BACKGROUND

[0002] At present, a certain number of mature entity relation extraction algorithms have been applied. For these algorithms, the following is summarized: 1. According to the annotation of the model training data, it can be divided into supervised model, semi-supervised model and unsupervised model, among which the supervised model is the best and the most in-depth research, and the semi-supervised model has also made certain progress. 2. According to the model structure, it can be divided into pipeline model and joint model, the former realizes the simple idea of connecting entity recognition and relation extraction as two sub-tasks in series, and the latter connects the two sub-tasks in parallel and uses a model to solve the two sub-problems; at present, most researchers believe that the joint learning model can better avoid error accumulation and has higher research value.

[0003] In the prior art, most of the joint learning models use the similarity between two words as the word feature vector, and then combine some word features to generate relation extraction. However, under this mode, the adaptability to some complex long texts is poor, and it can only be applied to some daily explanatory documents, and the accuracy for professional technical documents such as cement materials is low, or a large amount of training set data is required. SUMMARY

[0004] The purpose of the application is to provide a cement field text entity-relation joint extraction method and device based on deep learning and graph theory and a storage medium.

[0005] The purpose of the application can be achieved by the following technical solutions:

[0006] A cement field text entity-relation joint extraction method based on deep learning and graph theory comprises:

[0007] Step S1: performing word embedding processing on the input natural language text to obtain a word sequence;

[0008] Step S2: extracting a feature vector of each word;

[0009] Step S3: for each word, based on the distance between other words and the word, respectively generating a first weight of the other words acting on the word;

[0010] Step S4: for each word, based on the distance between the feature vector of other words and the feature vector of the word, generating a second weight of the other words acting on the word, and summing the first weight and the second weight to obtain a weight initial value of the other words acting on the word.

[0011] Step S5: forming an initial adjacency matrix based on the weight initial value of each word to other words;

[0012] Step S6: enhancing the feature vector of each word, calculating the attention score and modifying the initial adjacency matrix to obtain a target adjacency matrix;

[0013] Step S7: based on the obtained target adjacency, the weighted sum of the enhanced word feature vectors is obtained, and the feature vector fused with the adjacency node information is obtained;

[0014] Step S8: reorganizing the words to obtain a plurality of sub-sequences, and taking the feature vectors of the first and last words of the sub-sequences as the features of the sub-sequences;

[0015] Step S9: inputting the features of two sub-sequences into an analysis model to obtain the relationship between the two sub-sequences.

[0016] The process of the word embedding processing adopts BERT.

[0017] In the step S3, the greater the distance between the two words, the smaller the first weight, and the value range of the first weight is [0, 1].

[0018] In the step S4, the distance between the feature vectors of the two words is the Euclidean distance of the feature vectors.

[0019] The second weight is specifically:

[0020]

[0021] Wherein: W Similarity [i,j] is the second weight of the jth word to the ith word, X[i,p] is the feature value of the ith word in dimension p, X[j,p] is the feature value of the jth word in dimension p, d is the dimension of the word vector feature, N is the total number of words, and p is all the features of the word vector.

[0022] The step S6 specifically includes:

[0023] Step S61: linearly transforming the feature vector of each word;

[0024] Step S62: splicing the linearly transformed feature vectors of any two words to obtain the attention score of the two words;

[0025] Step S63: modifying the adjacency matrix based on the obtained attention score and the first threshold to obtain a target adjacency matrix, wherein the first threshold is obtained by training and learning;

[0026] Step S64: normalizing the target adjacency matrix.

[0027] The feature vector fused with the adjacent node information is:

[0028]

[0029] Wherein: o i is the feature vector of the i-th word fused with the adjacent node information, sigma (·) is a sigmoid activation function, alpha ij is the element of the i-th row and the j-th column in the target adjacent matrix, is the feature vector of the i-th word after linear transformation, and

[0030] The loss function of the analysis model includes a cross-entropy loss function, a symmetry loss function and a dependency loss function.

[0031] A cement field text entity-relation joint extraction device based on deep learning and graph theory, comprising a memory, a processor, and a program stored in the memory, characterized in that the processor executes the program to realize the method as described above.

[0032] A storage medium having a program stored thereon, characterized in that the program is executed to realize the method as described above.

[0033] Compared with the prior art, the present application has the following beneficial effects: it can accurately extract the relationship of complex long text, and can obtain relatively accurate relationship extraction effect under small samples. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 It is the main step flow diagram of the method of the present application. DETAILED DESCRIPTION

[0035] The present application will be described in detail below in conjunction with the drawings and specific embodiments. The present embodiment is implemented on the basis of the technical solution of the present application, and detailed implementation and specific operation process are given, but the protection scope of the present application is not limited to the following examples.

[0036] A cement field text entity-relation joint extraction method based on deep learning and graph theory, as shown in Figure 1 , comprising:

[0037] Step S1: performing word embedding processing on the input natural language text to obtain a word sequence;

[0038] Since the computer cannot directly interpret natural language data, the text needs to be mapped to a vector space before inputting the data into the model, that is, the word embedding operation is performed. In the present embodiment, a pre-trained Bert model is used for word embedding training.

[0039] Using the disclosed pre-trained model, a dictionary is generated according to existing training data, a more accurate word segmenter is trained, and the result after word segmentation is input into the bert model. The output of the last hidden layer in the output result is selected as the token of the word.

[0040] Compared with the embedding method of Word2Vec, the parameters of the embedding model can be included in the training, and the parameter model can be adjusted by using the subsequent loss function calculation result. It also allows users to choose not to train the parameters of the embedding model.

[0041] Step S2: extracting the feature vector of each word, wherein the feature vector of the word is the word vector, and the original word vector is obtained based on a traditional method.

[0042] Step S3: for each word, based on the distance between other words and the word, respectively generating a first weight of other words acting on the word;

[0043] Specifically, the original meaning of a word and its specific meaning in a sentence have a certain gap, and one or more words on the left and right of the word will directly affect the meaning of the word in most cases. This case is also referred to as "contextual meaning".

[0044] In a specific implementation, the first weight is used to reflect the strength of the inter-word relevance of the relevant positions. Generally, the greater the distance between two words, the smaller the first weight, and the value range of the first weight is [0, 1], which can be assigned by experience.

[0045] Step S4: for each word, based on the distance between the feature vectors of other words and the feature vector of the word, generating a second weight of other words acting on the word, and summing the first weight and the second weight to obtain a weight initial value of other words acting on the word, wherein the distance between the feature vectors of two words is the Euclidean distance of the feature vectors.

[0046] In the graph task, nodes with similar features are more likely to have similar edges (i.e., edges with similar attributes, weights, or directions). This part is a conventional means in the art, and therefore will not be described again.

[0047] In this embodiment, the second weight is specifically:

[0048]

[0049] W Similarity[i,j] is the second weight of the jth word to the ith word, X[i,p] is the feature value of the ith word in dimension p, X[j,p] is the feature value of the jth word in dimension p, d is the dimension of the vector feature of the word, N is the total number of words, and p is the traversal of all features of the word vector.

[0050] Step S5: forming an initial adjacency matrix based on the initial weight values of each word to other words;

[0051] Step S6: enhancing the feature vector of each word, calculating the attention score and modifying the initial adjacency matrix to obtain a target adjacency matrix, specifically including:

[0052] Step S61: performing linear transformation on the feature vector of each word;

[0053] Step S62: splicing the feature vectors of any two words after linear transformation to obtain the attention score of the two words;

[0054] Step S63: modifying the adjacency matrix based on the obtained attention score and the first threshold to obtain the target adjacency matrix, wherein the first threshold is obtained by training and learning;

[0055] During the modification process, if the value of any element in the adjacency matrix is greater than the first threshold, the element in the adjacency matrix is replaced by the corresponding attention score, otherwise the corresponding element in the adjacency matrix is modified to 0.

[0056] Wherein, the attention score corresponding to the element in the ith row and jth column of the adjacency matrix specifically refers to the attention score of the jth word to the ith word.

[0057] Step S64: normalizing the target adjacency matrix.

[0058] Step S7: based on the obtained target adjacency, performing weighted summation on the enhanced feature vector of the word to obtain a feature vector fused with adjacency node information, and the feature vector fused with adjacency node information is:

[0059]

[0060] Wherein: o i is the feature vector of the ith word fused with adjacency node information, σ(·) is a sigmoid activation function, and α ij is the element in the ith row and jth column of the target adjacency matrix, is the feature vector of the ith word after linear transformation,

[0061] Step S8: reorganizing the words to obtain a plurality of sub-sequences, and taking the feature vectors of the first and last words of the sub-sequences as the features of the sub-sequences.

[0062] Step S9: inputting the features of the two sub-sequences into an analysis model to obtain the relationship of the two sub-sequences, in the embodiment, the loss function of the analysis model includes a cross-entropy loss function, a symmetry loss function and a dependency loss function.

[0063] Wherein, the information entropy is a measure for describing the amount of information needed to eliminate the uncertainty of the real data; the cross-entropy, that is, the information entropy of a data set relative to a standard data set, describes the difference between it and the standard data. In model training, by optimizing to reduce the cross-entropy, the distribution of the predicted data can be made as close as possible to the distribution of the real data. Therefore, the cross-entropy can be used as the loss function of model training.

[0064] In the field of natural language processing, the relationship between two entities is usually abstracted as an SPO triple, where S (subject) represents the subject, O (object) represents the object, and the relationship is emitted by the subject and received by the object. Mapped to a graph structure, a directed edge must be drawn from the subject to the object. Among the label categories involved in the method, there is a relationship in which the subject and the object can be interchanged, that is, the two entities participating in the relationship are both the subject and the object. Therefore, for this type of relationship, the probability matrix obtained by the model prediction should also have symmetry.

[0065] The existence of the relationship is premised on that the two corresponding Spans are entities. In other words, for any word, the maximum probability of belonging to a certain relationship should be less than the maximum probability of belonging to a certain entity.

[0066] If the above functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. A cement field text entity-relation joint extraction method based on deep learning and graph theory, characterized by, The method comprises the following steps: Step S1: performing word embedding processing on the input natural language text to obtain a word sequence; Step S2: extracting a feature vector of each word; Step S3: for each word, generating a first weight of other words acting on the word based on the distance between the other words and the word; Step S4: for each word, generating a second weight of other words acting on the word based on the distance between the feature vectors of the other words and the feature vector of the word, and summing the first weight and the second weight to obtain a weight initial value of the other words acting on the word; Step S5: forming an initial adjacency matrix based on the weight initial values of the other words by each word; Step S6: performing enhancement processing on the feature vectors of the words, calculating attention scores, and modifying the initial adjacency matrix to obtain a target adjacency matrix; Step S7: performing weighted summation on the enhanced feature vectors of the words based on the obtained target adjacency to obtain feature vectors fused with adjacency node information; Step S8: recombining the words to obtain a plurality of sub-sequences, and taking the feature vectors of the first and last words of the sub-sequences as the features of the sub-sequences; Step S9: inputting the features of two sub-sequences into an analysis model to obtain the relationship between the two sub-sequences.

2. The cement field text entity-relation joint extraction method based on deep learning and graph theory according to claim 1, characterized in that, The word embedding processing process adopts BERT.

3. The cement field text entity-relation joint extraction method based on deep learning and graph theory according to claim 1, characterized in that, In the step S3, the greater the distance between two words, the smaller the first weight, and the value range of the first weight is [0, 1].

4. The cement field text entity-relation joint extraction method based on deep learning and graph theory according to claim 1, characterized in that, In the step S4, the distance between the feature vectors of two words is the Euclidean distance of the feature vectors.

5. The cement field text entity-relation joint extraction method based on deep learning and graph theory according to claim 4, characterized in that, The second weight is specifically: where: W similarity [i,j] is the second weight of the jth word for the ith word, X[i,p] is the feature value of the ith word in dimension p, X[j,p] is the feature value of the jth word in dimension p, d is the dimension of the vector feature of the word, N is the total number of words, and p is the traversal of all features of the word vector.

6. The cement field text entity-relation joint extraction method based on deep learning and graph theory according to claim 1, characterized in that, The step S6 specifically comprises: Step S61: performing linear transformation on the feature vectors of the words; Step S62: splicing the linearly transformed feature vectors of any two words to obtain the attention score of the two words; Step S63: modifying the adjacency matrix based on the obtained attention score and a first threshold to obtain a target adjacency matrix, wherein the first threshold is obtained by training and learning; Step S64: performing normalization processing on the target adjacency matrix.

7. The cement field text entity-relation joint extraction method based on deep learning and graph theory according to claim 1, characterized in that, The feature vector fused with adjacency node information is: wherein: o i is the feature vector of the i-th word fused with the adjacent node information, σ(·) is a sigmoid activation function, and α ij is the element in the i-th row and j-th column of the target adjacency matrix, is the feature vector of the i-th word after linear transformation.

8. The cement field text entity-relation joint extraction method based on deep learning and graph theory according to claim 1, characterized in that, The loss function of the analysis model comprises a cross-entropy loss function, a symmetry loss function, and a dependency loss function.

9. A cement field text entity-relation joint extraction device based on deep learning and graph theory, comprising a memory, a processor, and a program stored in the memory, characterized in that, The processor implements the method of any one of claims 1-8 when executing the program.

10. A storage medium having stored thereon a program, characterized by The program is executed to implement the method of any one of claims 1-8.

Images