A relation extraction method based on fine-grained prompt learning and bidirectional graph convolution

Abstract

The application discloses a relation extraction method based on fine-grained prompt learning and bidirectional graph convolution, and belongs to the technical field of natural language processing and information extraction. The application firstly carries out text preprocessing, that is, inserting entity multi-granularity markers and prompt learning templates in unstructured text; constructing sentence sequence features, generating sentence sequence feature representation; constructing sentence dependency features, generating sentence dependency feature representation; constructing the mixed embedding representation of the sentence, and outputting the relation extraction result. The application introduces the morphological and hypernym semantic information of entity types and entity words, enhances the morphological and semantic information of the prompt learning template and the context semantic expression ability, captures the sequence features and dependency features implied in the sentence, mines the multi-dimensional morphological and semantic features of the text, and improves the relation extraction performance.

Description

TECHNICAL FIELD

[0001] The present application relates to a relationship extraction method based on fine-grained prompt learning and bidirectional graph convolution, belonging to the technical field of natural language processing and information extraction. BACKGROUND

[0002] Relationship extraction is an important task in the field of information extraction and knowledge graph construction, and its goal is to identify the semantic relationship between entities in unstructured natural language text. Relationship extraction technology is widely used in semantic search, question answering systems, information recommendation and other fields, and has important application value.

[0003] According to the different data annotation methods, the existing relationship extraction methods can be divided into four categories: supervised method, semi-supervised method, unsupervised method, and remote supervision method. The supervised relationship extraction method needs to manually annotate the entity pairs and their corresponding relationship types in the text. The relationship extraction task belongs to the multi-classification problem. This method requires high quality of annotation data, and the annotation data requires a lot of manpower. The scale of the annotation data required by the supervised method is limited. The semi-supervised relationship extraction method uses annotated data as seeds, and learns features and identifies unannotated data through iterative seeds during training. Compared with the supervised relationship extraction method, the semi-supervised relationship extraction method has lower requirements for the quality of the seed data, but the selection of the seed may cause semantic drift problem. The unsupervised relationship extraction method does not need manual annotation data, and realizes the entity relationship extraction task through clustering and similarity measurement methods. The remote supervision relationship extraction method aligns the text according to the relationship facts existing in the remote knowledge base to realize the automatic annotation of unannotated data, but may produce errors. The task definition of the supervised relationship extraction method is clear and the feature learning efficiency is high. At present, more relationship extraction methods adopt the supervised method for relationship extraction.

[0004] Currently, supervised relation extraction methods are primarily based on deep learning, aiming to automatically extract semantic features from text using various neural networks to achieve relation extraction. Deep learning-based relation extraction models include those based on Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, Transformers, and Graph Convolutional Networks (GCNs). These methods leverage deep learning techniques to improve the performance and efficiency of relation extraction, significantly reducing the human cost of feature construction. CNNs are characterized by their ability to handle large-scale data. Recurrent Neural Networks (RNNs) are neural network models that process sequential data and can handle long-distance dependencies. Long Short-Term Memory (LSTM) aims to solve the long-term dependency problem in RNNs; its difference lies in that its hidden states not only contain information from the current moment but also include memory states from past moments and the selection states of control gates, allowing for control over the flow and preservation of information through gating mechanisms. Transformers use self-attention mechanisms to capture key information in text. Graph Convolutional Networks (GCNs) are neural network models that process graph-structured data and can extract node features using the relationships between nodes. BERT (Bidirectional Encoder Representation from Transformers), as a pre-trained language model, can capture token-level and sentence-level features of text. It is pre-trained using large-scale unlabeled data and achieves excellent performance in various natural language processing tasks such as entity recognition and relation extraction.

[0005] Existing relation extraction methods suffer from the following main problems: First, current methods lack in-depth textual information mining, primarily utilizing entity type information while neglecting to delve into part-of-speech, word class, and contextually implicit features. Second, relation extraction methods based on pre-trained language models or long short-term memory networks generate linear structural representations of text, characterizing the contextual sequence features. Relation extraction methods based on graph neural networks generate dependency structural representations of text, characterizing the graph structural features. Existing relation extraction methods rarely capture both contextual sequence features and graph structural features simultaneously, resulting in insufficient text feature representation and neglecting implicit features within the text. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of current relation extraction methods, such as a lack of in-depth text information mining, insufficient text feature representation, and a limited simultaneous capture of text sequence and dependency features. To address these issues, a relation extraction method based on fine-grained cue learning and bidirectional graph convolutional networks is proposed. The method is characterized by: First, the introduction of a fine-grained cue learning enhancement method. Based on the cue learning template, multi-granularity entity markers are introduced, including entity types of head and tail entities, part-of-speech tags of entity words, and semantic information of hypernyms. This fully mines the semantic information of the text, enhances the contextual semantic expression ability, and improves the text sequence feature representation. Second, the introduction of bidirectional graph convolutional networks, attention mechanisms, and dependency trees to generate sentence dependency feature representations. An adjacency matrix is ​​constructed based on the text dependency tree, and an attention mechanism is introduced to perform soft pruning on the dependency tree, thus constructing a text dependency feature representation. By fusing text sequence feature representation and dependency feature representation, the implicit sequence and dependency features of sentences are captured from both sequence structure and graph structure perspectives. This uncovers multi-dimensional lexical and semantic features of the text, improving relation extraction performance.

[0007] To achieve the above objectives, the present invention adopts the following technical solution.

[0008] The relation extraction method based on fine-grained cue learning and bidirectional graph convolution includes the following steps:

[0009] Step 1: Text preprocessing, which involves inserting entity multi-granularity markers and cue learning templates into unstructured text;

[0010] Among them, entity multi-granularity markers include entity type, entity part of speech, and entity hypernym.

[0011] Step 1: Specifically includes the following sub-steps:

[0012] Step 1.1: For unstructured text, insert entity multi-granularity tags;

[0013] Step 1.1A: Generate entity types. For unstructured text, use an entity recognition and annotation tool to assign BIO tags and obtain entity types;

[0014] The BIO tag means: B (Begin) indicates the beginning of the entity, I (Inside) indicates the middle of the entity, and O (Outside) indicates that it does not belong to any type.

[0015] Step 1.1B: Generate entity part-of-speech tags. For the entity words identified in Step 1.1A, use a part-of-speech tagging tool to obtain the part-of-speech tags of the entity words, i.e., the entity part-of-speech tags;

[0016] Step 1.1C: Generate the hypernyms of the entities. For the entity words identified in Step 1.1A, obtain the hypernyms of the entities;

[0017] Step 1.1D: Generate entity multi-granularity tokens. Integrate the entity types, entity parts of speech, and entity hypernyms generated in Steps 1.1A, 1.1B, and 1.1C into entity multi-granularity tokens;

[0018] Define the entity multi-granularity marker as:

[0019] <S u :e s :Pos s :Up s >, < / S u :e s :Pos s :Up s >,

[0020] <O b :e o :Pos o :Up o >, < / O b :e o :Pos o :Up o >,

[0021] Among them, S u Represents the head entity, e s Indicates the header entity type, Pos s Indicates the part of speech of the head entity word, Up s O is a superordinate word indicating the head entity word. b Represents the tail entity, e o Indicates the tail entity type, Pos o Indicates the part of speech of the final entity word, Up o A superordinate noun indicating the suffix of a substance word. A prefix or suffix of a substance word may contain one or more tokens. s ,e o ∈ε, where ε is a predefined set of entity types.

[0022] Insert multi-granularity entity markers before and after the beginning and end entities in a sentence.

[0023] Step 1.2: For unstructured text, introduce cue learning to bridge the gap between pre-training and fine-tuning, and construct cue learning templates and their corresponding tag word sets;

[0024] Step 1.2A: Construct a prompt learning template;

[0025] Insert entity multi-granularity markers into this template;

[0026] Step 1.2B: Constructing cue learning input. The sentence processed in Step 1.1 is merged with the cue learning template to generate a cue learning sequence input;

[0027] Step 1.2C: Construct a set of prompt learning tags;

[0028] An adaptive tag selection mechanism is introduced to map relationship tags to a set of tags.

[0029] Step 2: Construct the sequence features of the sentence and generate the sequence feature representation of the sentence;

[0030] The sentence after preprocessing in step 1, including entities, multi-granularity entity tags, and cue learning templates, is input into the pre-trained language model BERT to generate a sequence feature representation of the sentence.

[0031] Step 2: Specifically includes the following sub-steps:

[0032] Step 2.1: Segment the sentence into words;

[0033] For the sentence after preprocessing in step 1, it is divided into a token sequence and marked with [CLS] and [SEP]. [CLS] is the start marker of the sentence, used to represent the final hidden state as the entire sentence's representation vector. [SEP] is the separator between different sentences.

[0034] Step 2.2: Generate the embedded representation of the sentence;

[0035] For each token in the sentence, the pre-trained model BERT performs three embedding operations: generating token embeddings, generating segment embeddings, and generating position embeddings.

[0036] Step 2.2A: Generate token embedding;

[0037] Token embedding is a fixed word vector. For each token in a sentence, its token embedding, i.e., TokenEmbedding, is generated through the embedding layer in BERT.

[0038] Step 2.2B: Generate sentence segment embeddings, i.e., Segment embeddings;

[0039] For a sentence, the Segment Embedding is generated through the embedding layer in BERT.

[0040] Step 2.2C: Generate the position embedding;

[0041] For each token in the sentence, the pre-trained model BERT generates a position embedding for each token's location.

[0042] Step 2.2D: Merge the three embeddings generated in Step 2.2A, Step 2.2B, and Step 2.2C to generate the embedding representation of the sentence.

[0043] Step 2.3: Generate the sequence feature representation of the sentence;

[0044] The sentence generated in step 2.2 is embedded into the BERT Transformer encoder.

[0045] Step 2.3A: Based on the sentence embedding representation, calculate Q, V, and K in the multi-head self-attention mechanism, where Q represents Query, V represents Value, and K represents Key;

[0046] Step 2.3B: Generate self-attention. Perform n different linear transformations on Q, V, and K to generate n self-attention points;

[0047] Step 2.3C: Using residual connections and layer normalization in the pre-trained BERT model, the results generated by multi-head self-attention are input into the feedforward layer of BERT. The input of each multi-head self-attention layer is the output of the previous layer.

[0048] Step 2.3D: Generate the sequence feature representation of the sentence;

[0049] The output of the last layer of multi-head self-attention is used as the sequence feature representation of the sentence;

[0050] Step 3: Construct the dependency features of the sentence and generate the dependency feature representation of the sentence;

[0051] Step 3: Specifically includes the following sub-steps:

[0052] Step 3.1: For a sentence, construct the dependency tree of the sentence and its corresponding adjacency matrix, and input them into a bidirectional graph convolutional network;

[0053] Step 3.1A: Construct the dependency tree of the sentence. Use Stanford Parser to extract the dependency tree of the sentence;

[0054] Step 3.1B: Construct the adjacency matrix of the sentence dependency tree;

[0055] The dependency tree of the sentence generated in step 3.1A is a directed graph, and the nodes of the graph are V. s The token represents a sentence, and the edge E in the graph represents the edge E. sThis represents the dependency relationship between two nodes. Based on the dependency tree of the sentence, construct its adjacency matrix. If node v... i and v j If they are directly connected in the dependency tree, then edge e ij Initialize to 1, otherwise set to 0.

[0056] Step 3.2: Using the adjacency matrix generated in Step 3.1 as input, a bidirectional graph convolutional network with L layers is used to propagate information by aggregating the neighbor nodes of nodes and edges in the forward and backward propagation stages, respectively.

[0057] Step 3.2A: Forward Propagation. During the forward propagation phase, each node propagates information to its outgoing neighbor nodes;

[0058] Step 3.2B: Backpropagation. During the backpropagation phase, each node propagates information to its incoming neighbor nodes;

[0059] Step 3.2C: Bidirectional aggregation. The embedding vectors of each node's neighbors are weighted and averaged, and then concatenated with the embedding vector of the current node to achieve bidirectional aggregation;

[0060] Step 3.3: Perform soft pruning on the dependency tree of the sentence based on the attention mechanism;

[0061] Step 3.3A: Calculate the attention coefficient;

[0062] An attention mechanism is introduced to prune the sentence dependency tree. The weights of different dependency edges are calculated and numerically weighted, with the weights between nodes ranging from [0,1].

[0063] Step 3.3B: Generate the dependency feature representation of the sentence;

[0064] For the attention coefficients calculated in step 3.3A, the updated node vectors are obtained by weighted summation with the corresponding node vectors. As the graph convolutional network updates by passing information to neighboring nodes in each round, a soft pruning strategy is used to continuously reduce the impact of noisy nodes and useless edges on the graph convolutional network model. The final layer output of the bidirectional graph convolutional network is used as the dependency feature representation of the sentence.

[0065] Step 4: Construct a hybrid embedding representation of the sentence and output the relation extraction results;

[0066] Step 4: Specifically includes the following sub-steps:

[0067] Step 4.1: Generate a hybrid embedding representation of the sentence;

[0068] For a sentence, the sequence feature representation generated in step 2 and the dependency feature representation generated in step 3 are concatenated to generate a hybrid embedding representation of the sentence;

[0069] Step 4.2: Perform a linear transformation on the hybrid embedding representation of the sentence;

[0070] The sentence hybrid embedding representation generated in step 4.1 is input into the fully connected layer and linearly transformed to map the high-dimensional input vector to a low-dimensional space, thereby achieving feature combination.

[0071] Step 4.3: Output the relationship extraction results;

[0072] The vector representation generated in step 4.2 is input into the Softmax layer to obtain the predicted relationship vector between the two entities. During training, the cross-entropy loss function is used to calculate the error between the predicted result and the true label.

[0073] Beneficial effects

[0074] This invention addresses the relation extraction problem by proposing a relation extraction method based on fine-grained cue learning and bidirectional graph convolution. Compared with existing technologies, it has the following advantages:

[0075] 1. Compared with current relation extraction methods, the proposed method has portability and robustness, is not limited by the source of the corpus, is not limited by the language of the corpus, and can process Chinese and English text.

[0076] 2. This invention proposes a sentence sequence feature representation generation method based on fine-grained cue learning enhancement. First, auxiliary multi-granularity entity markers are inserted into the original corpus to capture head and tail entities and their types, as well as the parts of speech and hypernyms of the head and tail entities. Then, cue learning is introduced to bridge the gap between pre-training and fine-tuning by inserting multi-granularity entity markers into the cue learning template. Finally, the sentence embedding representation and sequence feature representation are generated. This invention's sentence sequence feature representation generation method, by introducing multi-granularity entity markers, captures entity types, lexical information of entity words, and semantic information of hypernyms; by introducing fine-grained cue learning, it enhances the lexical and semantic information of the cue learning template, enriches the text sequence feature representation, and improves the performance of relation extraction.

[0077] 3. This invention proposes a sentence dependency feature representation generation method based on a bidirectional graph convolutional network. First, a dependency tree and adjacency matrix for the sentence are constructed, and a bidirectional graph convolutional neural network is used to extract dependency features. Second, an attention mechanism is used to soft-prune the dependency tree, generating the sentence's dependency feature representation. This method, by introducing a bidirectional graph convolutional neural network, an attention mechanism, and a dependency tree to generate the sentence's dependency feature representation, captures the implicit semantic dependency features in the text, improving relation extraction performance.

[0078] 4. The relation extraction method of this invention integrates the sequence feature representation and dependency feature representation of text. The hybrid embedding representation of a sentence consists of the sequence feature representation and dependency feature representation of the sentence. Furthermore, the hybrid embedding representation of the sentence is input into a fully connected layer and a Softmax layer to obtain the relation extraction result. This method captures the implicit sequence features and dependency features of sentences, mines multi-dimensional lexical and semantic features of text, and improves the relation extraction performance.

[0079] 5. Experiments were conducted on publicly available datasets, and the results demonstrate the effectiveness and superiority of the proposed method. This method has broad application prospects in question-answering systems, information retrieval, and opinion mining. Attached Figure Description

[0080] Figure 1 This is a flowchart illustrating an embodiment of a relation extraction method based on fine-grained cue learning and bidirectional graph convolution according to the present invention. Detailed Implementation

[0081] The relation extraction system based on the method of this invention uses PyCharm as the development tool, Python as the development language, and PyTorch as the development framework. The preferred embodiment of the relation extraction method based on fine-grained cue learning enhancement and bidirectional graph convolutional networks of this invention will be described in detail below with reference to specific examples.

[0082] Example 1

[0083] This embodiment describes the process of using a relation extraction method based on fine-grained cue learning enhancement and bidirectional graph convolutional networks as described in this invention, such as... Figure 1 As shown.

[0084] This invention provides a relation extraction method based on fine-grained cue learning and bidirectional graph convolution, comprising four modules:

[0085] The first module is the text preprocessing module, which includes building multi-granularity tokens, building cue learning templates, and a set of tag words;

[0086] The second module is the sequence feature generation module, which includes word segmentation, generating sentence embedding representations, and generating sentence sequence feature representations;

[0087] The third module is the dependency feature generation module, which includes constructing the dependency tree and its adjacency matrix of the sentence, forward propagation and backward propagation, and soft pruning of the dependency tree;

[0088] The fourth module is the relation extraction module, which includes constructing a hybrid embedding representation of sentences, linear transformation, and outputting the relation extraction results.

[0089] from Figure 1As can be seen, the specific steps include the following:

[0090] Step 1: Text preprocessing, which involves inserting entity multi-granularity markers and cue learning templates into unstructured text;

[0091] Among them, entity multi-granularity markers include entity type, entity part of speech, and entity hypernym.

[0092] Step 1: Specifically includes the following sub-steps:

[0093] Step 1.1: For unstructured text, insert entity multi-granularity tags;

[0094] Step 1.1A: Generate entity types. For unstructured text, use an entity recognition and annotation tool to assign BIO tags and obtain entity types;

[0095] The BIO tag means: B (Begin) indicates the beginning of the entity, I (Inside) indicates the middle of the entity, and O (Outside) indicates that it does not belong to any type.

[0096] For sentence S, obtain the BIO tag sequence NER corresponding to each token in sentence S:

[0097] NER = (N1, N2, ..., N n (1)

[0098] Where, N i This represents the BIO tag corresponding to the i-th token, and n represents the number of tokens.

[0099] Step 1.1B: Generate entity part-of-speech tags. For the entity words identified in Step 1.1A, use a part-of-speech tagging tool to obtain the part-of-speech tags of the entity words, i.e., the entity part-of-speech tags;

[0100] For sentence S, obtain the part-of-speech (POS) sequence of the entity corresponding to each token in sentence S:

[0101] POS = (P1, P2, ..., P n (2)

[0102] Among them, P i Let represent the part-of-speech tag corresponding to the i-th token, and n represent the number of tokens.

[0103] In this embodiment, the Stanford POSTagger part-of-speech tagging tool is used to obtain the part-of-speech tags of entity words.

[0104] Step 1.1C: Generate the hypernyms of the entities. For the entity words identified in Step 1.1A, obtain the hypernyms of the entities;

[0105] The process of obtaining the superordinate terms of an entity involves two steps: First, query the thesaurus of the entity; then, query the superordinate terms of the words in the thesaurus. If a word has multiple levels of superordinate terms, the lowest level superordinate term is selected.

[0106] Specifically, for an entity word e, the thesaurus e of that entity word is queried, as shown in formula (3):

[0107] e = synsets(e) (3)

[0108] Further, query the hypernyms of words in the thesaurus e, and select the lowest-level hypernym Up;

[0109] Up=hypernym(e) (4)

[0110] Here, synsets(·) represents the function to get the set of synonyms for a word, and hypernym(·) represents the function to get the hypernym of a word.

[0111] In this embodiment, the Python package pythownet provided by the Hownet database and the Python package nltk provided by the Wordnet database are used to obtain entity hypernyms.

[0112] Step 1.1D: Generate entity multi-granularity tokens. Integrate the entity types, entity parts of speech, and entity hypernyms generated in Steps 1.1A, 1.1B, and 1.1C into entity multi-granularity tokens;

[0113] Define the entity multi-granularity marker as:

[0114] <S u :e s :Pos s :Up s >, < / S u :e s :Pos s :Up s >

[0115] <O b :e o :Pos o :Up o >, < / O b :e o :Pos o :Up o >

[0116] Among them, S u Represents the head entity, e s Indicates the header entity type, Pos sIndicates the part of speech of the head entity word, Up s O is a superordinate word indicating the head entity word. b Represents the tail entity, e o Indicates the tail entity type, Pos o Indicates the part of speech of the final entity word, Up o A superordinate noun indicating the suffix of a substance word. A prefix or suffix of a substance word may contain one or more tokens. s ,e o ∈ε, where ε is a predefined set of entity types.

[0117] First, for sentence S, entity multi-granularity markers are inserted before and after the head and tail entities in sentence S, respectively, in the following form:

[0118]

[0119] Among them, X s Represents the head entity, X s ={t s ,t s+1 ,...,t s+m-1}, t s ,t s+1 ,...,t s+m-1 Let m be the token and m be the length of the header entity. X o Represents the tail entity, X o ={t o ,t o+1 ,...,t o+l-1}, t o ,t o+1 ,...,t o+l-1 Let l be the token and l be the length of the tail entity. For sentences after inserting entity multi-granularity tags, <S b :e s :Pos s :Up s > is the head entity X s The start tag of <O b :e o :Pos o :Up o > for tail entity X o The start tag of < / S b :e s :Pos s :Up s > is the head entity X s The closing tag, < / O b :e o :Pos o :Up o> for tail entity X o The closing tag.

[0120] Step 1.2: For unstructured text, introduce cue learning to bridge the gap between pre-training and fine-tuning, and construct cue learning templates and their corresponding tag word sets;

[0121] Step 1.2A: Construct a prompt learning template;

[0122] For the following relation extraction template based on prompt learning:

[0123] X s is[MASK]X o

[0124] Insert entity multi-granularity markers into this template, and the relation extraction template is constructed as follows:

[0125] T = <S u :e s :Pos s :Up s >X s < / S u :e s :Pos s :Up s >is[MASK] (6)

[0126] <O b :e o :Pos o :Up o >X o < / O b :e o :Pos o :Up o >

[0127] Step 1.2B: Constructing cue-learning input. The sentence processed in Step 1.1 is merged with the cue-learning template to generate a cue-learning sequence input.

[0128] Specifically, the string template T containing the [MASK] field from step 1.2A is compared with the input sequence. By concatenating the parts, the following sentence is constructed:

[0129]

[0130] Here, [CLS] represents the beginning of a sentence, and [SEP] represents the sentence separator.

[0131] X promptAs a sequence input, it is fed into the pre-trained language model BERT for encoding, generating sentence sequence representations, and constructing sentence sequence features.

[0132] Step 1.2C: Construct a set of tag words for cue learning;

[0133] An adaptive tag selection mechanism is introduced to map relationship tags to a set of tags.

[0134] For a set of relation labels Y, the learning hint requires setting a corresponding set of label words M(Y), that is, constructing a mapping from set Y to set of label words M(Y): M:Y→V, where V is the dictionary of the pre-trained language model. The mapping method is to remove function words and connectors from the relation labels.

[0135] For example, for the relation label y = "per:city_of_death", it is mapped to the label set "{person,city,death}", which means removing the symbols "_" and "of" from the relation label y.

[0136] Step 2: Construct the sequence features of the sentence and generate the sequence feature representation of the sentence;

[0137] The sentence after preprocessing in step 1, including entities, multi-granularity entity tags, and cue learning templates, is input into the pre-trained language model BERT to generate a sequence feature representation of the sentence.

[0138] Step 2: Specifically includes the following sub-steps:

[0139] Step 2.1: Segment the sentence into words;

[0140] For the sentence S after preprocessing in step 1, it is divided into a token sequence and marked with [CLS] and [SEP]. [CLS] is the start marker of the sentence, used to represent the final hidden state as the entire sentence's representation vector. [SEP] is the separator between different sentences.

[0141] Step 2.2: Generate the embedded representation of the sentence;

[0142] For each token in sentence S, the pre-trained model BERT performs three embedding operations: generating token embeddings, generating segment embeddings, and generating position embeddings.

[0143] Step 2.2A: Generate token embedding;

[0144] The token embedding is a fixed word vector. For each token in the sentence, its token embedding is generated through the embedding layer in BERT, i.e., TokenEmbedding(t i The calculation method is as follows:

[0145] TokenEmbedding(t i ) = E word (t i (8)

[0146] Among them, t i E represents the i-th token in the input sequence. word (·) indicates the word vector embedding layer.

[0147] Step 2.2B: Generate sentence segment embeddings, i.e., Segment embeddings;

[0148] For sentence S, a segmentation embedding (SegmentEmbedding(s)) is generated through a BERT embedding layer. i The calculation method is as follows:

[0149] SegmentEmbedding(s i ) = E seg (s i (9)

[0150] Among them, s i E represents the i-th sentence of the input sequence. seg (s i ) represents the segmentation embedding layer in BERT.

[0151] Step 2.2C: Generate the position embedding;

[0152] For each token in sentence S, the pre-trained model BERT generates a position embedding, i.e., PositionEmbedding(i), for each token's location. The calculation method is as follows:

[0153] PositionEmbedding(i)=E pos (i) (10)

[0154] Where i represents the i-th token, E pos (·) represents the location embedding layer in BERT;

[0155] Step 2.2D: Merge the three embeddings generated in Step 2.2A, Step 2.2B, and Step 2.2C to generate the embedding representation of sentence S;

[0156] E i =TokenEmbedding(t i )+PositionEmbedding(i)+SegmentEmbedding(s i (11)

[0157] Among them, E i This represents the embedding representation of the i-th token in sentence S, with the "+" symbol indicating concatenation.

[0158] Step 2.3: Generate the sequence feature representation of the sentence;

[0159] The sentence generated in step 2.2 is embedded into the BERT Transformer encoder.

[0160] Step 2.3A: Based on the sentence embedding representation, calculate Q, V, and K in the multi-head self-attention mechanism. The calculation method is as follows:

[0161] Q = E × W Q

[0162] V = E × W V (12)

[0163] K = E × W K

[0164] Where E represents the sentence embedding, Q represents the query, V represents the value, K represents the key, and W represents the key. Q W V W K All of these are learnable weight parameters.

[0165] Step 2.3B: Generate self-attention. Perform n different linear transformations on Q, V, and K to generate n self-attention points. The calculation method is as follows:

[0166]

[0167] MHSA = concat(SA1,...,SA) n (13)

[0168] Among them, SA i Let d represent the self-attention of the i-th token. K This represents the vector dimension of each self-attention head. MHSA stands for Multi-head Self-Attention, and concat means concatenation. The multi-head self-attention layer obtains contextual information by calculating the attention weight of the token.

[0169] Step 2.3C: Using residual connections and layer normalization from the pre-trained BERT model, the results generated by multi-head self-attention are input into the feedforward layer of BERT. The input h' of each multi-head self-attention layer is the input h of the previous layer. n-1 Output:

[0170] h'=LN(h n-1 +MHSA(h n-1 (14)

[0171] Where LN represents layer normalization.

[0172] Step 2.3D: Generate the sequence feature representation of the sentence;

[0173] The output of the last multi-head self-attention layer is used as the sequence feature representation of the sentence, and the calculation method is as follows:

[0174] H p =LN(h'+FF(h')) (15)

[0175] Where FF represents the feedforward layer, H p This refers to the sequence feature representation of a sentence.

[0176] Step 3: Construct the dependency features of the sentence and generate the dependency feature representation of the sentence;

[0177] Step 3: Specifically includes the following sub-steps:

[0178] Step 3.1: For sentence S, construct the dependency tree of sentence S and its corresponding adjacency matrix, and input them into a bidirectional graph convolutional network;

[0179] Step 3.1A: Construct the dependency tree of sentence S. Use Stanford Parser to extract the dependency tree of the sentence;

[0180] Step 3.1B: Construct the adjacency matrix of the dependency tree for sentence S;

[0181] For the dependency tree of sentence S generated in step 3.1A, this dependency tree is a directed graph, and the nodes of the graph are V. s The token represents a sentence, and the edge E in the graph represents the edge E. s This represents the dependency relationship between two nodes. Based on the dependency tree of sentence S, construct its adjacency matrix, where each element e... ij The calculation method is as follows:

[0182]

[0183] That is, if node v i and v j If they are directly connected in the dependency tree, then edge e ijInitialize to 1, otherwise set to 0.

[0184] Step 3.2: Using the adjacency matrix generated in Step 3.1 as input, a bidirectional graph convolutional network with L layers is used to propagate information by aggregating the neighbor nodes of nodes and edges in the forward and backward propagation stages, respectively.

[0185] Step 3.2A: Forward Propagation. During the forward propagation phase, each node propagates information to its outgoing neighbor nodes. The calculation formula is as follows:

[0186]

[0187] in, This represents the hidden features of the token at level l. This represents the set of all tokens originating from tokenv. and Let represent the learnable outbound weight matrix, and ReLU(·) represent the activation function.

[0188] Step 3.2B: Backpropagation. During the backpropagation phase, each node propagates information to its incoming neighbor nodes. The calculation formula is as follows:

[0189]

[0190] in, This represents the set of all tokens that have reached tokenv. and Let represent the learnable incoming edge weight matrix, and ReLU(·) represent the activation function.

[0191] Step 3.2C: Bidirectional Aggregation. The embedding vectors of each node's neighbors are weighted and averaged, then concatenated with the node's own embedding vector to achieve bidirectional aggregation. The calculation formula is as follows:

[0192]

[0193] Step 3.3: Perform soft pruning on the dependency tree of the sentence based on the attention mechanism;

[0194] Step 3.3A: Calculate the attention coefficient;

[0195] An attention mechanism is introduced to prune the sentence dependency tree. This is achieved by calculating and numerically weighting the weights of different dependency edges, with the weights between nodes ranging from [0,1]. The attention coefficient α... i,j The calculation method is as follows:

[0196]

[0197] Where Q represents Query, K represents Key, V represents Value, and W represents Value. i softQ and W i softK Both are learnable parameter matrices, where d represents the dimension of the node vector. This is a normalization coefficient to avoid gradient explosion or vanishing problems caused by excessively large or small dot products. This represents the adjacency matrix after soft pruning, with attention coefficient α. i,j The value is between [0, 1].

[0198] Step 3.3B: Generate the dependency feature representation of the sentence;

[0199] The attention coefficients calculated in step 3.3A are weighted and summed with the corresponding node vectors to obtain the updated node vectors. As the graph convolutional network updates the vectors by passing information to its neighbors in each round, By employing a soft pruning strategy to continuously reduce the impact of noisy nodes and useless edges on the graph convolutional network model, the final layer output of the bidirectional graph convolutional network is used as the dependency feature representation H of the sentence. s .

[0200] Step 4: Construct a hybrid embedding representation of the sentence and output the relation extraction results;

[0201] Step 4: Specifically includes the following sub-steps:

[0202] Step 4.1: Generate a hybrid embedding representation of the sentence;

[0203] For sentence S, the sequence feature representation generated in step 2 and the dependency feature representation generated in step 3 are concatenated to generate a hybrid embedding representation of the sentence:

[0204] H final =concat(H p H s ) (twenty one)

[0205] Among them, H p H represents the sequence features of a sentence. s This represents the dependency features of the sentence.

[0206] Step 4.2: Perform a linear transformation on the hybrid embedding representation of the sentence;

[0207] The sentence hybrid embedding representation generated in step 4.1 is input into the fully connected layer and linearly transformed to map the high-dimensional input vector to a low-dimensional space, thereby achieving feature combination. The calculation method is as follows:

[0208] H final =ReLU(WH H final +b H ) (twenty two)

[0209] Among them, W H and b H These are the weight matrix and bias vector of the fully connected layer, H. final It is the output of the hidden layer.

[0210] Step 4.3: Output the relationship extraction results;

[0211] The vector representation generated in step 4.2 is input into the Softmax layer to obtain the predicted vector of the relationship between the two entities. The calculation method is as follows:

[0212] p r (r|H final = softmax(W) r H final +b r ) (twenty three)

[0213] Where, p r W represents probability. r and b r These are the weight matrix and bias vector of the Softmax layer, respectively.

[0214] During training, the cross-entropy loss function is used to calculate the error between the predicted result and the true label. The calculation method is as follows:

[0215]

[0216] Where N is the number of samples, M is the number of relation categories, and y ij p is the true label of sample i. r This represents the probability of predicting the label. The relation extraction model parameters are updated by minimizing the loss function.

[0217] To illustrate the relation extraction effect of this invention, this experiment compares three methods under the same conditions, using the same training set, validation set, and test set.

[0218] The first method employs a relation extraction approach based on contextual segment representation, using a pre-trained language model and dynamic segment graphs to construct segment representations. The second method trains independent encoders for entity recognition and relation extraction, inserting entity type markers into the text to assist relation extraction. The third method is the relation extraction method of this invention.

[0219] The evaluation metrics used are: Precision, Recall, and F1-score, which are three important indicators for evaluating the performance of classification tasks. The calculation formulas are as follows:

[0220]

[0221] In this context, TP stands for True Positive, meaning that the predicted value is positive and the actual value is also positive; FP stands for False Positive, meaning that the predicted value is positive but the actual value is negative; and FN stands for False Negative, meaning that the predicted value is negative but the actual value is positive.

[0222] The relation extraction evaluation results on the CHIP2020 dataset are as follows: the first method achieves a precision of 54.8%, a recall of 55.2%, and an F1-score of 55.0%. The second method achieves a precision of 63.1%, a recall of 58.6%, and an F1-score of 60.8%. The proposed method achieves a precision of 69.9%, a recall of 59.5%, and an F1-score of 64.3%. Experiments demonstrate the effectiveness of the proposed relation extraction method based on fine-grained cue learning and bidirectional graph convolution.

[0223] The above description is merely a preferred embodiment of the present invention, and the present invention should not be limited to the content disclosed in this embodiment and the accompanying drawings. Any equivalent or modified embodiments made without departing from the spirit of the present invention fall within the scope of protection of the present invention.

Claims

1. A relation extraction method based on fine-grained prompt learning and bidirectional graph convolution, characterized in that: Relying on fine-grained prompt learning and bidirectional graph convolutional network; the model includes four modules: The first module is a text preprocessing module, including generating multi-granularity markers, constructing prompt learning templates and label word sets; The second module is a sequence feature generation module, including word segmentation, generating sentence embedding representation, and generating sentence sequence feature representation; The third module is a dependency feature generation module, including constructing a sentence dependency tree and its adjacency matrix, forward propagation and backward propagation, and soft pruning of the dependency tree; The fourth module is a relation extraction module, including constructing a hybrid embedding representation of a sentence, linear transformation, and outputting a relation extraction result; The relation extraction method based on fine-grained prompt learning and graph convolutional network includes the following steps: Step 1: text preprocessing, that is, inserting entity multi-granularity markers and prompt learning templates in unstructured text; Step 1.1: for unstructured text, insert entity multi-granularity markers; Step 1.2: for unstructured text, introduce prompt learning to bridge the gap between pre-training and fine-tuning, construct prompt learning templates and their corresponding label word sets; Step 2: construct the sequence features of the sentence, and generate the sequence feature representation of the sentence; Step 2.1: perform word segmentation on the sentence; Step 2.2: generate the embedding representation of the sentence; Step 2.3: generate the sequence feature representation of the sentence; Step 3: construct the dependency features of the sentence, and generate the dependency feature representation of the sentence; Step 3.1: for sentence S, construct the dependency tree of sentence S and its corresponding adjacency matrix, and input it into the bidirectional graph convolutional network; Step 3.2: input the adjacency matrix generated in step 3.1 into the bidirectional graph convolutional network with L layers, and propagate information by aggregating the neighbor nodes of the nodes and edges in the forward propagation and backward propagation stages respectively; Step 3.3: perform soft pruning on the dependency tree of the sentence based on the attention mechanism; Step 4: construct the hybrid embedding representation of the sentence, and output the relation extraction result; Step 4.1: generate the hybrid embedding representation of the sentence; For sentence S, concatenate the sequence feature representation generated in step 2 and the dependency feature representation generated in step 3 to generate the hybrid embedding representation of the sentence; Step 4.2: perform linear transformation on the hybrid embedding representation of the sentence; Input the sentence hybrid embedding representation generated in step 4.1 into the fully connected layer to perform linear transformation, map the input high-dimensional vector to a low-dimensional space, and realize feature combination; Step 4.3: output the relation extraction result; Input the vector representation generated in step 4.2 into the Softmax layer to obtain the relationship prediction vector between the two entities; in the training process, use the cross-entropy loss function to calculate the error between the prediction result and the true label, and update the relation extraction model parameters by minimizing the loss function Loss.

2. The relation extraction method based on fine-grained prompt learning and bidirectional graph convolution according to claim 1, characterized in that: Step 1.1 specifically includes: Step 1.1A: generate entity types; for unstructured text, use entity recognition annotation tools to assign BIO labels to obtain entity types; Wherein, the BIO label refers to: B (Begin) represents the beginning of the entity, I (Inside) represents the middle of the entity, and O (Outside) represents not belonging to any type; For the sentence S, obtain the BIO label sequence corresponding to each Token in the sentence S NER: NER = (N1, N2,..., N n ) (1) wherein N i represents the BIO label corresponding to the i-th Token, n represents the number of tokens; Step 1.1B: Generate entity part-of-speech; For the entity words identified in step 1.1A, use a part-of-speech tagging tool to obtain the part-of-speech of the entity words, i.e. entity part-of-speech; For the sentence S, obtain the entity part-of-speech sequence corresponding to each token in the sentence S POS: POS = (P1, P2,..., P n ) (2) where P i denotes the part-of-speech tag corresponding to the i-th token, and n denotes the number of tokens. Step 1.1C: Generate the hypernym of the entity; For the entity words identified in step 1.1A, obtain the hypernym of the entity; The process of obtaining the entity hypernym is divided into two steps: first, query the synonym set of the entity word; Then, query the hypernym of the words in the synonym set; If a word has multiple levels of hypernyms, select the lowest level hypernym; In particular, for an entity term e, query a synonym set of the entity term As shown in equation (3): Further, the synonym set is queried The hypernyms of the middle word are selected, and the lowest level hypernym Up is selected; Wherein, synsets(·) represents a function of obtaining the synonym set of the word, and hypernym(·) represents a function of obtaining the hypernym of the word; Step 1.1D: Generate entity multi-granularity markers; Integrate the entity type, entity part-of-speech and entity hypernym generated in steps 1.1A, 1.1B and 1.1C into entity multi-granularity markers; The entity multi-granularity markers are defined as: <S u : e s : Pos s : Up s >, < / S u : e s : Pos s : Up s > < O b : e o : Pos o : Up o >, < / O b : e o : Pos o : Up o > wherein S u denotes a head entity, e s denotes a head entity type, Pos s denotes a part-of-speech of the head entity word, Up s denotes a hypernym of the head entity word, O b denotes a tail entity, e o denotes a tail entity type, Pos o denotes a part-of-speech of the tail entity word, Up o denotes a hypernym of the tail entity word; the head entity word or the tail entity word comprises one or more tokens; e s ,e o ∈ε,ε is a predefined set of entity types; First, for the sentence S, insert the entity multi-granularity markers in front of and behind the head entity and the tail entity in the sentence S, and the specific form is as follows: wherein X s represents a head entity, X s = {t s , t s+1 ,..., t s+m-1}, t s , t s+1 ,..., t s+m-1 are tokens, and m is the length of the head entity; X o represents a tail entity, X o = {t o , t o+1 ,..., t o+l-1}, t o , t o+1 ,..., t o+l-1 are tokens, and l is the length of the tail entity; is the sentence after inserting the multi-granularity labels of the entity, <S b :e s :Pos s :Up s > is the start tag of the head entity X s , <O b :e o :Pos o :Up o > is the start tag of the tail entity X o , < / S b :e s :Pos s :Up s > is the end tag of the head entity X s , and < / O b :e o :Pos o :Up o > is the end tag of the tail entity X o .

3. The relation extraction method based on fine-grained prompt learning and bidirectional graph convolution according to claim 1, characterized in that: Step 1.2 is specifically: Step 1.2A: Construct prompt learning template; For the following relationship extraction template based on prompt learning: X s is[MASK]X o Insert the entity multi-granularity markers in the template, and the relationship extraction template is constructed as: Step 1.2B: Construct prompt learning input; Fuse the sentence processed in step 1.1 with the prompt learning template to generate the prompt learning sequence input; In particular, the string template T in step 1.2A containing the [MASK] field is concatenated with the input sequence Patching, the sentence is then constructed as follows: Wherein, [CLS] represents the beginning marker of the sentence; [SEP] represents the separator of the sentence; X prompt As a sequence input, it is input into the pre-trained language model BERT for encoding to generate sentence sequence representation and build sentence sequence features. Step 1.2C: Construct the label word set of prompt learning; Introduce a self-adaptive label word selection mechanism to map the relationship label to the label word set; For the relationship label set Y, the prompt learning needs to set the corresponding label word set M(Y), that is, to construct the mapping from set Y to label word set M(Y): M: Y→V, where V is the dictionary of the pre-trained language model; The mapping method is to remove the virtual words and connectors in the relationship label.

4. The relation extraction method based on fine-grained prompt learning and bidirectional graph convolution according to claim 1, characterized in that: Step 2 is specifically: For the sentence processed in step 1, including entities, entity multi-granularity markers, and prompt learning templates, input into the pre-trained language model BERT to generate the sequence feature representation of the sentence; Step 2.1: Tokenize the sentence; For the sentence S processed in step 1, divide it into a token sequence and add [CLS] and [SEP] markers, wherein [CLS] is the beginning marker of the sentence, which is used to take the corresponding final hidden state as the representation vector of the entire sentence; [SEP] is the separator between different sentences; Step 2.2: Generate the embedding representation of the sentence; For each token in sentence S, the pre-trained model BERT performs three embedding operations: generating token embedding, generating segment embedding, and generating position embedding; Step 2.2A: Generating token embedding; The token embedding is a fixed word vector; for each token in the sentence, its token embedding is generated through the embedding layer in BERT, i.e., TokenEmbedding(t i The calculation method is as follows: TokenEmbedding(t i ) = E word (t i ) (8) where t i represents the i-th token of the input sequence, E word (·) denotes a word vector embedding layer; Step 2.2B: Generating segment embedding of sentence S; For a sentence S, the segment embedding of the sentence, i.e., SegmentEmbedding(s i ), is generated by a BERT embedding layer as follows: SegmentEmbedding(s i ) = BERT(s) (1) where BERT is a pre-trained BERT model. SegmentEmbedding(s i ) = E seg (s i ) (9) where s i represents the i-th sentence of the input sequence, E seg (s i ) represents the segment embedding layer in BERT; Step 2.2C: Generating position embedding of sentence S; For each token in sentence S, the pre-trained model BERT generates position embedding for the position where the token is located, i.e. PositionEmbedding(i), the calculation method is as follows: PositionEmbedding(i) = E pos (i) (10) where i denotes the ith token, E pos (·) denotes the position embedding layer in BERT; Step 2.2D: Fusing the three embeddings generated in steps 2.2A, 2.2B and 2.2C to generate the embedding representation of sentence S; E i = TokenEmbedding(t i )+ PositionEmbedding(i)+ SegmentEmbedding(s i ) (11) wherein E i denotes the embedding representation of the i-th token in sentence S, and the symbol "+" denotes concatenation; Step 2.3: Generating sequence feature representation of sentence S; Input the sentence embedding representation generated in step 2.2 into the Transformer encoder of BERT; Step 2.3A: According to the sentence embedding representation, calculate Q, V, K in the multi-head self-attention mechanism, the calculation method is as follows: where E is the embedding representation of a sentence, Q represents a query Query, V represents a value Value, K represents a key Key, W Q ,W V ,W K are all learnable weight parameters; Step 2.3B: Generating self-attention; for Q, V, K, n different linear transformations are performed to generate n self-attentions, the calculation method is as follows: wherein SA i denotes the self-attention of the i-th token, d K denotes the vector dimension of each self-attention head, MHSA denotes multi-head self-attention, and concat denotes concatenation; the multi-head self-attention layer obtains context information by calculating the attention weight of the token; Step 2.3C: using the pre-trained model BERT's residual connections and layer normalization, input the results generated by multi-head self-attention into the feed-forward layers of BERT; the input h' of each layer of multi-head self-attention is the output of the previous layer h n-1 : h' = LN(h n-1 + MHSA(h n-1 )) (14) Where LN represents layer normalization; Step 2.3D: Generating sequence feature representation of sentence S; The output of the last layer of multi-head self-attention is taken as the sequence feature representation of the sentence, and the calculation method is as follows: H p = LN(h' + FF(h')) (15) where FF denotes a feed-forward layer, H p i.e., the sequence feature representation of the sentence.

5. The relation extraction method based on fine-grained prompt learning and bidirectional graph convolution according to claim 1, characterized in that: Step 3.1 is specifically: Step 3.1A: Constructing the dependency tree of sentence S; using Stanford Parser to extract the dependency tree of the sentence; Step 3.1B: Constructing the adjacency matrix of the dependency tree of sentence S; For the dependency tree of the sentence S generated in step 3.1A, the dependency tree is a kind of directed graph, the nodes V s of the graph represent tokens of the sentence, the edges E s of the graph represent dependency relationships between two nodes; according to the dependency tree of the sentence S, its adjacency matrix is constructed, and the calculation method of the element e ij in the matrix is as follows: That is, if nodes v i and v j are directly connected in the dependency tree, then edge e ij is initialized to 1, otherwise to 0.

6. The relation extraction method based on fine-grained prompt learning and bidirectional graph convolution according to claim 1, characterized in that: Step 3.2 is specifically: The adjacency matrix generated in step 3.1 is taken as input, and a bidirectional graph convolutional network with L layers is used to propagate information by aggregating the neighbor nodes of nodes and edges in the forward propagation and backward propagation stages respectively; Step 3.2A: Forward propagation; in the forward propagation stage, each node propagates information to its neighbor nodes of outgoing edges, and the calculation formula is as follows: wherein, denotes the hidden features of the token at the l-th layer, denotes the set of all tokens that can be reached from the token v, and denotes the learnable out-edge weight matrix, and ReLU(·) denotes the activation function; Step 3.2B: Backward propagation; in the backward propagation stage, each node propagates information to its neighbor nodes of incoming edges, and the calculation formula is as follows: wherein, denotes a set of all tokens reaching tokenv, and denotes a learnable input edge weight matrix, and ReLU(·) denotes an activation function. Step 3.2C: Bidirectional aggregation; the neighbor node embedding vectors of each node are weightedly averaged and concatenated with the embedding vector of the node to realize bidirectional aggregation, and the calculation formula is as follows:

7. The relation extraction method based on fine-grained prompt learning and bidirectional graph convolution according to claim 1, characterized in that: Step 3.3 is specifically: Step 3.3A: Calculating attention coefficient; The attention mechanism is introduced to prune the sentence dependency tree; by calculating and numerically weighting the weights of different dependency edges, the weight values between nodes are between [0, 1], and the attention coefficient a i,j The calculation method is as follows: where Q represents Query, K represents Key, V represents Value, W i softQ and W i softK are all learnable parameter matrices, d represents the dimension of the node vector, is a normalization coefficient to avoid the problem of gradient explosion or disappearance caused by too large or too small dot product; represents the adjacency matrix after soft pruning, the attention coefficient α i,j takes value in [0,1]. Step 3.3B: Generating dependency feature representation of sentence S; For the attention coefficient calculated in step 3.3A, the updated node vector is obtained by weighting and summing it with the corresponding node vector; the information is updated as each round of graph convolution network passes to the neighbor node, By continuously reducing the influence of noise nodes and useless edges on the graph convolution network model through the soft pruning strategy, the final layer output of the bidirectional graph convolution network is taken as the dependency feature representation H of the sentence s .

8. The relation extraction method based on fine-grained prompt learning and bidirectional graph convolution according to claim 1, characterized in that: Step 4: Constructing hybrid embedding representation of sentence S and outputting relationship extraction result; specifically: Step 4.1: Generating hybrid embedding representation of sentence S; For sentence S, the sequence feature representation generated in step 2 and the dependency feature representation generated in step 3 are concatenated to generate the hybrid embedding representation of the sentence: H final = concat(H p ,H s ) (21) where H p is the sequence feature representation of the sentence, H s is the dependency feature representation of the sentence; Step 4.2: Linear transformation is performed on the hybrid embedding representation of the sentence; The sentence mixed embedding representation generated in step 4.1 is input into a fully connected layer, which is linearly transformed to map the high-dimensional input vector into a low-dimensional space, realizing feature combination. The calculation method is as follows: where W H and b H are the weight matrix and bias vector of the fully connected layer, respectively, is the hidden layer output; Step 4.3: Output the relationship extraction result; The vector representation generated in step 4.2 is input into the Softmax layer to obtain the relationship prediction vector between the two entities. The calculation method is as follows: where p r represents the probability, W r and b r are the weight matrix and bias vector of the Softmax layer, respectively. In the training process, the cross-entropy loss function is used to calculate the error between the prediction result and the true label. The calculation method is as follows: where N is the number of samples, M is the number of relation categories, y ij is the true label of sample i, p r is the probability of the predicted label; the relation extraction model parameters are updated by minimizing the loss function Loss.

Images