A zero-shot relation extraction method and model based on a dual contrastive learning framework and a cross-attention module
By employing a zero-shot relation extraction method based on a dual contrastive learning framework and a cross-attention module, the problem of low model performance in data-scarce environments is addressed. This method enhances the semantic information fusion capability of sentences and relation descriptions, achieving accurate relation extraction under zero-shot conditions.
Patent Information
- Application Number
- CN202410675741.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-28
- Publication Date
- 2026-11-06
- Estimated Expiration
- 2044-05-28
AI Technical Summary
Existing supervised learning and remote supervised learning methods cannot effectively address the problem of low model performance in data-scarce environments, and they ignore the correlation information between sentences and relations, resulting in the model's lack of ability to recognize different sentences and relations.
We employ a zero-shot relation extraction method based on a dual contrastive learning framework and a cross-attention module. We fuse semantic information of sentences and relation descriptions through a cross-attention network and enhance the model’s discriminative ability in the representation space through dual contrastive learning.
It improves the model's ability to identify unknown relationships, enhances the fusion of semantic information, and performs exceptionally well, especially in environments where high-quality data resources are scarce.
Smart Images
Figure CN118484537B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of natural language processing and relates to a relation extraction method based on zero-shot learning. Specifically, it relates to a zero-shot relation extraction method and model based on a dual contrastive learning framework and a cross-attention module. Background Technology
[0002] Relation extraction is a key task in natural language processing, referring to the identification and extraction of semantic relationships between two labeled entities obtained from a given sentence. Furthermore, various practical applications, such as information retrieval, knowledge graph construction, and intelligent question-answering systems, are built upon the knowledge content extracted through relation extraction. Recent research on supervised learning-based relation extraction has achieved significant success. Supervised learning-based relation extraction is modeled as a supervised learning task. The main process involves first preprocessing the input sentence through syntactic analysis or word segmentation, then mapping the corresponding words to a high-dimensional space to obtain word vectors, followed by semantic encoding using a neural network to obtain sentence vectors, and finally inputting the sentence vectors into a classifier to obtain the final classification result. However, these methods typically require large-scale annotated data, and manual annotation suffers from numerous problems such as time-consuming, labor-intensive, and costly processes. To address the problem of manual annotation, some studies have adopted remote supervision to generate labeled data for training. The main purpose is to utilize external knowledge bases to provide labels for entity pairs and relations in text. However, the annotated data generated by remote supervision is always of low quality and cannot fully cover relation instances that might be generated in the real world. The above analysis shows that neither supervised learning nor remote supervised learning can effectively solve the challenge of low model performance in data-scarce environments.
[0003] Therefore, in the field of relation extraction, to address the limitations described above, Zero-Shot Relation Extraction (ZSRE) was proposed to extract unknown relations that did not appear during the training phase. In supervised relation extraction, the sets of visible and invisible relations are generally the same during both the training and testing phases. However, ZSRE has disjoint sets of visible and invisible relations between the two phases. Therefore, the goal of ZSRE is to establish a connection between the representations of visible relations formed during the training phase and the representations of invisible relations formed during the testing phase.
[0004] In existing methods, most model approaches primarily achieve performance improvement by transforming task types, without fully utilizing the latent semantic information within sentences. Furthermore, they neglect the role of correlations between sentences and relationships, leaving the model lacking the ability to recognize relationships between different sentences and relationships. Summary of the Invention
[0005] To address the aforementioned technical problems in existing technologies, this invention proposes a zero-shot relation extraction algorithm based on a dual contrastive learning framework and a cross-attention module to improve the semantic information fusion between input instances and corresponding relations. First, the model's input is an input sentence X containing target entity pairs. i The relationship described in the sentence D j The input sentence and relation description are sent to the corresponding encoders to generate attribute vectors f(X) respectively. i ) and g(D j The next step is to generate the attribute vector f(X). i ) and g(D j The inputs are combined into a cross-attention network to highlight f(X). i ) relative to g(D j The attention target is then determined by the relation description g(D). j ) and input sentence f(X) i Dual contrastive learning is performed, which expands the distance between negative sample pairs while reducing the distance between positive sample pairs, and also jointly optimizes the relation classification loss. Finally, given a new sentence input X... h The model generates an attribute vector f(X). h To find the attribute vector g(D) that has the closest invisible relation. l This method not only improves the model's ability to distinguish different relationships, but also enhances the fusion of semantic information between instances and relationships.
[0006] To achieve the above objectives, the present invention provides a zero-shot relation extraction method and model based on a dual contrastive learning framework and a cross-attention module, the specific technical solution of which is as follows:
[0007] A zero-shot relation extraction method based on a dual contrastive learning framework and a cross-attention module is characterized by the following steps:
[0008] Step 1: Use the pre-trained language model BERT to generate a contextual representation for each identifier;
[0009] Step 2: Receive the feature map V(v1,v2,...,v) of the input sentence embedding vector through a cross-attention network. h Feature maps of ) and relation attribute vectors Where h is the dimension of the last hidden layer, representing the number of spatial locations on each feature map, and v i and They are respectively at V and V d The feature vector at position i in the vector;
[0010] Step 3: From the input sentence X i The corresponding relationship description D i Obtaining representation and When each sentence is embedded With relation attribute vector When they share the same relation labels, minimize the distance between them; when they have different relation labels, increase the embedding. Each sentence and its attribute vector The distance between them;
[0011] Step 4: The [CLS] identifier H0 obtained from the input sentence encoder is compared with the entity representation obtained through entity location information. and entity representation
[0012] Step 5: Generate relation attribute vectors using Sentence-BERT The sentence encoder generates the input sentence X. ′ Generate embedding vector V ′ For sentence embedding vector V ′ The improved embedding vector is obtained through a cross-attention network. For the improved embedding vector By finding the vectors with the closest distance As a final prediction.
[0013] Furthermore, step 1 is detailed as follows:
[0014] First, the input sentence is segmented using Word-Piece tokenization, adding two special identifiers, [CLS] and [SEP], to the input sentence. Entity pair information is obtained using specific entity tag vectors, and H0 is used to represent the special identifier [CLS]. After applying the tanh activation function to H0, it is input into a fully connected layer to obtain the identifier representation H0.
[0015] H0 ′ =W0[tanh(H0)]+b0 (1)
[0016] Next, the entity representation is obtained by averaging the hidden state vectors of the entity pairs. and For two entities e1 and e2 containing multiple identifiers, a tanh nonlinear operation is added and fed into a fully connected layer, followed by averaging the vector representations to form the entity representation vectors. and As shown below:
[0017]
[0018] in Let q and r represent two entity vectors, respectively, representing e. c c = 1, 2, the start and end positions of the identifiers of the two entities, H t Let W represent the hidden state vector in the input embedding, and W... e and b e The learnable parameters are the same for both entities;
[0019] Finally, by connecting H0, and To obtain the final input sentence vector representation V; the final vector representation v is obtained through the following formula:
[0020]
[0021] W1 and b1 are learnable parameters, and ⊕ represents a join operation.
[0022] Furthermore, step 2 is detailed as follows:
[0023] First, the correlation matrix R(r1,r2,...,r) h () is calculated by multiplying and fusing the cosine distance in the relevant layers. and The semantic relevance between them is calculated, and can be specifically represented as follows:
[0024]
[0025] Where ||v i ||2 and It is the expression of the second normal form of each vector, r i Represents the local sentence feature vector v i With all relational feature vectors The correlation between them.
[0026] Next, based on the contextual representation of the relevance matrix, a fusion layer is applied to generate a cross-attention map A, and the feature maps are weighted to obtain the reconstructed sentence representation. The fusion layer takes the correlation matrix R as its input and performs convolution operations using a w convolution kernel, where each local correlation vector r in R is a vector r. i These are merged into a single attention vector; the obtained attention scalar is normalized using the softmax function to obtain i. th Sentence attention at the location is used to form an attention mapping matrix A. i as follows:
[0027]
[0028] Where τ is the temperature hyperparameter;
[0029] The final output of the cross-attention network is a reconstructed input sentence vector feature map obtained by weighting and multiplying the input sentence embedding V and the attention mapping matrix A using cross-attention fusion. The specific formula is as follows:
[0030]
[0031] in ⊕ denotes matrix multiplication, and ⊕ denotes matrix addition.
[0032] Furthermore, step 3 is detailed as follows:
[0033] Define the loss function for dual contrastive learning. First, using relation attribute vectors... As the main research object, its corresponding input sentence is embedded. Defined as a secondary object, the loss function l is obtained. cl1 Embed the input sentence As the primary object, and the corresponding relational attribute vector The corresponding loss function l is obtained by treating it as a secondary object. cl2 Loss function l cl1 and l cl2 The definition is as follows:
[0034]
[0035]
[0036] in and Representing the i-th th The input sentence embedding and the i-th... th A vector of relational attributes, where * denotes the inner product calculation method; the parameter γ ensures the inner product is directly opposite. The largest inner product greater than any negative pair
[0037] Finally, the contrastive learning loss function L cl As shown below:
[0038] L cl =L cl1 +L cl2 (9)
[0039] Furthermore, step 4 is detailed as follows:
[0040] The activation function tanh is applied to obtain the final sentence embedding V, and V is sent to CAM to obtain the reconstructed embedding vector. Then, the reconstructed embedding vector is sent. To the softmax layer, to generate the i-th... th The n-dimensional classification probability distribution of known relation instances:
[0041]
[0042] Where X i Represents the i-th th One input instance, y i These are the visible relation labels, θ represents the model parameters, and W... ce and b ce All of these are learnable parameters;
[0043] For each input instance X i In other words, the cross-entropy loss function is used to calculate the relationship classification loss L. ce :
[0044]
[0045] The final loss function is as follows:
[0046] L jiont =L ce +αL cl (12)
[0047] Where α is the equilibrium hyperparameter.
[0048] Furthermore, in step 5, the embedding vector V ′ for:
[0049]
[0050] Improved embedding vector for:
[0051]
[0052] CAM stands for Cross-Attention Network Module;
[0053] The final prediction is:
[0054]
[0055] Where C() returns the result for an unknown input sentence X. i Predictive relationship, Indicates the lth th A vector of relational attributes for an unknown relationship, where dist represents the distance calculation method.
[0056] A zero-shot relation extraction model based on a dual contrastive learning framework and a cross-attention module includes the following modules:
[0057] Input encoder module: Encodes the input sentence by linking entity information and [CLS] identifier information in the sentence to obtain the embedded vector representation of the input sentence;
[0058] Cross-attention network module: After obtaining the context representation vector, the context representation vector and the relation description attribute vector are jointly input into the cross-attention network. The cross-attention network generates a cross-attention map for each feature and uses an attention mechanism to analyze the semantic relevance between the input sentence and the relation description.
[0059] Dual contrastive learning module: After obtaining the embedding vector through the cross-attention module, contrastive learning training is performed from two perspectives: input sentence and relation description. This minimizes the distance between input sentences and relation descriptions with the same relation label, while increasing the distance between input sentences and relation descriptions with different relation labels.
[0060] Relationship Classification Module: The relationship classification module performs classification learning among known relationships, enabling the model to learn to distinguish the differences between different relationships.
[0061] The present invention has the following beneficial effects:
[0062] 1. Overall, this paper utilizes the latent connections between sentences and relation descriptions to construct a zero-shot relation extraction model based on contrastive learning and attention networks. The proposed model not only provides greater interpretability for natural language processing but also achieves satisfactory results in real-world environments where high-quality data resources are scarce.
[0063] 2. From a technical perspective, a dual contrastive learning module was designed to enhance the ability to identify different semantic information and relational descriptions in the input sentence. Furthermore, a cross-attention network module was integrated, enabling effective interaction between the embedding vectors from the input sentence and the attribute vectors of the relational descriptions in the representation space. These improvements allow the model to accurately extract unknown relations in a zero-shot setting.
[0064] 3. From the perspective of experimental results, experiments were conducted on two well-known datasets, FewRel and Wiki ZSL, to fully demonstrate the effectiveness of the dual contrastive learning module in relation classification and the effectiveness of the cross-attention network in fusing semantic information. Attached Figure Description
[0065] Figure 1 This is a schematic diagram of a zero-shot relation extraction model based on a dual contrastive learning framework and a cross-attention module.
[0066] Figure 2 This refers to the cross-attention module in the model.
[0067] Figure 3 This refers to the dual contrastive learning module in the model.
[0068] Figure 4 This is the relationship classification module in the model. Detailed Implementation
[0069] The present invention will be further analyzed below with reference to specific embodiments.
[0070] like Figure 1 As shown, the zero-shot relation extraction method based on the dual contrastive learning framework and cross-attention module of the present invention includes the following steps:
[0071] Step 1: Generate the contextual representation of each identifier using the pre-trained language model BERT. First, the input sentence is segmented using Word-Piece tokenization. In addition, two special identifiers, [CLS] and [SEP], are added to the input sentence. Simultaneously, to obtain entity pair information from the original sentence, a specific entity tag vector is used, where all positions except the entity pair index are 0, thus allowing the extraction of entity pair information. H0 is used to represent the special identifier [CLS]. After applying the tanh activation function to H0, it is input into a fully connected layer to obtain the identifier representation H′0:
[0072] H0′=W0[tanh(H0)]+b0 (1)
[0073] Next, the entity representation is obtained by averaging the hidden state vectors of the entity pairs. and Specifically, for two entities e1 and e2 containing multiple identifiers, a tanh nonlinear operation is added to the model and fed into a fully connected layer, after which the vector representation is averaged to form the entity representation vector. and As shown below:
[0074]
[0075] in This represents two entity vectors, q and r, which represent e respectively. c (c = 1, 2) The start and end positions of the identifiers of the two entities, H t Let W represent the hidden state vector in the input embedding, and W... e and b e The learnable parameters are the same for both entities.
[0076] Finally, by connecting H′0, and To obtain the final input sentence vector representation V. The final vector representation v is obtained using the following formula:
[0077]
[0078] W1 and b1 are learnable parameters, and ⊕ represents a join operation.
[0079] Step 2: Receive the feature map V(v1,v2,...,v) of the input sentence embedding vector through a cross-attention network. h Feature maps of ) and relation attribute vectors Where h is the dimension of the last hidden layer, representing the number of spatial locations on each feature map, and v i and They are respectively at V and V d The eigenvector at position i in the matrix. The correlation matrix R(r1, r2, ..., r... h () is calculated by multiplying and fusing the cosine distance in the relevant layers. and The semantic relevance between them is calculated and can be represented as follows:
[0080]
[0081] Where ||v i ||2 and It is the expression of the second normal form of each vector, r i Represents the local sentence feature vector v i With all relational feature vectors The correlation between them.
[0082] Next, based on the contextual representation of the relevance matrix, a fusion layer is applied to generate a cross-attention map A, which aims to weight the feature maps to obtain a more discriminative reconstructed sentence representation. The fusion layer takes the correlation matrix R as its input and performs convolution operations using a w convolution kernel, where each local correlation vector r in R is a vector r. i These are merged into a single attention vector. The obtained attention scalar is normalized using the softmax function to obtain i. th Sentence attention at the location is used to form an attention mapping matrix A. i as follows:
[0083]
[0084] Where τ is the temperature hyperparameter. Lower temperatures lead to lower entropy, causing the distribution to concentrate at a few high-confidence locations.
[0085] The final output of the aforementioned cross-attention network is a reconstructed input sentence vector feature map obtained by weighted summation and multiplication of the input sentence embedding V and the attention mapping matrix A using cross-attention fusion. The specific formula is as follows:
[0086]
[0087] in ⊕ denotes matrix multiplication, and ⊕ denotes matrix addition.
[0088] Step 3: From the input sentence X i The corresponding relationship description D i Obtaining representation and Then, when each embedded sentence With relation attribute vector When they share the same relation labels, the dual contrastive learning module aims to minimize the distance between them; conversely, when they have different relation labels, the dual contrastive learning module aims to increase the embedding. Each sentence and its attribute vector The distance between them.
[0089] A loss function for dual contrastive learning is defined. First, using relation attribute vectors... As the main research object, its corresponding input sentence is embedded. Defined as a secondary object, the loss function l is obtained. cl1 Furthermore, the input sentence is embedded. As the primary object, and the corresponding relational attribute vector The corresponding loss function l is obtained by treating it as a secondary object. cl2 Loss function l cl1 and l cl2 The definition is as follows:
[0090]
[0091]
[0092] in and Representing the i-th th The input sentence embedding and the i-th... th A vector of relational attributes, where * denotes the inner product calculation method. γ ensures the inner product of the directly opposite attribute. The largest inner product greater than any negative pair Furthermore, the idea behind the loss function can be interpreted as making the correct relation vector stand out more than the closest incorrect relation vector. Additionally, γ is used to prevent collapse where each vector in the embedding vector space becomes too close to other vectors.
[0093] Finally, the contrastive learning loss function L in the model cl As shown below:
[0094] L cl =L cl1 +L cl2 (9)
[0095] Step 4: The [CLS] identifier H′0 obtained from the input sentence encoder is compared with the entity representation obtained through entity location information. and entity representation Furthermore, the activation function tanh is applied to obtain the final sentence embedding V, and V is sent to CAM to obtain the reconstructed embedding vector. Then, the model sends the reconstructed embedding vectors. To the softmax layer, to generate the i-th... th The n-dimensional classification probability distribution of known relation instances:
[0096]
[0097] Where X i Represents the i-th th One input instance, y i These are the visible relation labels, θ represents the model parameters, and W... ce and b ce These are all learnable parameters.
[0098] For each input instance X i In other words, the model uses the cross-entropy loss function to calculate the relationship classification loss L. ce :
[0099]
[0100] The final loss function is as follows:
[0101] L jiont =L ce +αL cl (12)
[0102] Where α is the equilibrium hyperparameter.
[0103] Step 5: The goal is to predict the relation corresponding to a completely unknown input sentence instance X′. Therefore, the model generates relation attribute vectors using Sentence-BERT. The sentence encoder generates an embedding vector V′ for the input sentence X′:
[0104]
[0105] For the sentence embedding vector V′, the model obtains the improved embedding vector through a cross-attention network.
[0106]
[0107] CAM stands for Cross-Attention Network Module.
[0108] For the improved embedding vector The model finds the vectors with the closest distance. As a final prediction:
[0109]
[0110] Where C() returns the result for an unknown input sentence X. i Predictive relationship, It indicates the lth th There are three relation attribute vectors with unknown relationships, and dist represents the distance calculation method. In the experiment, the inner product calculation is used as the vector distance calculation method.
[0111] like Figure 2-4 As shown, the zero-shot relation extraction model based on the dual contrastive learning framework and cross-attention module of the present invention includes:
[0112] Input encoder module: Encodes the input sentence by linking entity information and [CLS] identifier information in the sentence to obtain the embedded vector representation of the input sentence.
[0113] The cross-attention network module: After obtaining the context representation vector, this vector and the relation description attribute vector are jointly input into the cross-attention network. This network generates a cross-attention map for each feature to highlight the target object, thereby effectively addressing unseen classes. This network employs an attention mechanism to analyze the semantic relevance between the input sentence and the relation description, which helps focus attention on the target object and facilitates the subsequent relation matching process.
[0114] Dual contrastive learning module: After obtaining the embedding vectors through cross-attention, this module performs contrastive learning training from two perspectives: the input sentence and the relation description. The dual contrastive learning module aims to minimize the distance between input sentences and relation descriptions with the same relation label, while increasing the distance between input sentences and relation descriptions with different relation labels.
[0115] Relationship Classification Module: The relationship classification module performs classification learning among known relationships, enabling the model to learn how to distinguish the differences between different relationships.
[0116] Experimental procedure:
[0117] To demonstrate the feasibility of this method, experiments were conducted on two publicly available datasets. The first dataset is FewRel, a manually annotated balanced dataset consisting of 80 relation types, each with 700 instances. It was created by aligning candidate sentence sets with their relations through remote supervision, while a manual annotator filtered out incorrectly annotated sentences for each relation, resulting in the final relation extraction dataset. The second dataset is Wiki-ZSL, a subset of the Wiki-KB dataset. This dataset was filtered to exclude "none" relation labels and some relation labels that appeared less than 300 times to obtain the final data content. It was also created through remote supervision.
[0118] To verify the effectiveness of the model, P, R, and F1 scores were used as experimental metrics, and the model was compared with several deep learning methods on the FewRel and Wiki-ZSL datasets. Detailed descriptions of the compared models are as follows:
[0119] 1) ESIM: This model treats the zero-sample relation extraction task as a textual implication task, and uses bi-LSTM to encode the two input sequences, and provides the encoded results to the local inference model.
[0120] 2) CIM: This model also treats the zero-sample relation extraction task as a textual implication task, but unlike ESIM, it replaces bi-LSTM with a conditional constraint module, where the representation of the input sentence is used as a conditional constraint by the corresponding relation description.
[0121] 3) ZS-Bert: This model uses a representation-matching approach to process ZSREs. It designs effective embeddings for input sentences containing entity pairs and relation attribute vectors, then encodes them into the same semantic space, and then measures the similarity between the input embeddings and relation attribute vectors.
[0122]
[0123]
[0124] Table 1: Experimental results of FewRel and Wiki-ZSL datasets.
[0125] The results of the comparative experiments are shown in Table 1, and the analysis of the results leads to the following conclusions. On the FewRel dataset, the model achieves significant enhancements when considering different values of the number of unknown relations *m*. When *m* = 5, *m* = 10, and *m* = 15, the F1 score is improved by 8.47%, 5.53%, and 3.50% compared to ZS-BERT, respectively. On the Wiki-ZSL dataset, the model also outperforms existing baseline models when considering different numbers of unknown relations *m*. Specifically, when *m* = 5, the model achieves significant improvements in P (an increase of 3.42%), R (an increase of 5.24%), and F1 score (an increase of 4.32%). Furthermore, improvements are achieved when *m* = 10 and *m* = 15, with F1 scores increasing by 8.06% and 3.64%, respectively. Overall, these results not only demonstrate the model's superior ability to integrate semantic information between the input sentence and relation descriptions but also its effectiveness in distinguishing different categories of relations.
[0126] To analyze the impact of the cross-attention module and the dual contrastive learning module on the model's F1 score, ablation experiments were conducted on the FewRel and Wiki-ZSL datasets. Only the elimination experiments of the modules proposed in this paper were considered, and their respective effects were observed. Ours represents the complete model, Ours* represents the model without the relational classification module, Ours*-CAM represents the model without the cross-attention network, and Ours*-Dual represents the model without the dual contrastive learning module.
[0127] F1 FewRel Wiki-ZSL Ours*-RC 82.88 68.60 Ours*-CAM 86.27 73.82 Ours*-Dual 85.81 73.01 Ours 88.89 76.02
[0128] Table 2: Ablation experiment results on the FewRel and Wiki-ZSL datasets.
[0129] The ablation experiment results are shown in Table 2. Ours achieved better results than Ours*-RC, demonstrating the importance of the relation classification module to the overall model. Furthermore, the results from Ours*-CAM and Ours*-Dual show that integrating these two modules significantly improves performance on both datasets, indicating that each module plays a crucial role in the semantic relation fusion process of the model.
[0130] Therefore, this invention offers the following advantages: In this work, a novel model addresses the zero-shot relation extraction (ZSRE) task. The model aims to better fuse semantic information through dual contrastive learning and highlight relevant information by forming an effective attention mechanism between the input sentence and relation description, thereby enhancing the model's ability to discriminate semantic information. Experimental results obtained on two benchmark datasets demonstrate the effectiveness of the model framework. Furthermore, extensive experiments were conducted to validate the model's performance in various aspects, demonstrating that the model still outperforms the baseline model in the zero-shot setting.
Claims
1. A zero-shot relation extraction method based on a dual contrastive learning framework and a cross-attention module, characterized in that... Includes the following steps: Step 1: Use the pre-trained language model BERT to generate a contextual representation for each identifier; Step 2: Receive the feature map of the input sentence embedding vector through a cross-attention network. Feature maps of relation attribute vectors ,in It is the dimension of the last hidden layer, representing the number of spatial locations on each feature map, and at the same time and They are respectively in and In The feature vector at the location; Step 3: From the input sentence and corresponding relationship description Obtaining representation and When each sentence is embedded With relation attribute vector When they share the same relation labels, minimize the distance between them; when they have different relation labels, increase the embedding. Each sentence and its attribute vector The distance between them; Step 4: Obtain the [CLS] identifier from the input sentence encoder Entity representation obtained through entity location information and entity representation ; Step 5: Generate relation attribute vectors using Sentence-BERT The sentence encoder generates the input sentence. Generate embedding vectors For sentence embedding vectors The improved embedding vector is obtained through a cross-attention network. For the improved embedding vector By finding the vectors with the closest distance As a final prediction; Step 2 is described in detail below: First, the correlation matrix Multiplication and fusion are calculated using cosine distance in the relevant layers. and The semantic relevance between them is calculated, and can be specifically represented as follows: ) (4) in and It is the second normal form expression for each vector. Represents local sentence feature vectors With all relational feature vectors The correlation between them; Next, based on the contextual representation of the relevance matrix, a fusion layer is applied to generate a cross-attention map. Weighting the feature maps yields the reconstructed sentence representation. The fusion layer will integrate the relevant matrix. As its input, and using The convolution kernel performs a convolution operation, where each kernel in the convolution kernel performs a convolution operation. Local correlation vectors in They are merged into a single attention vector; The obtained attention scalar is normalized using the softmax function to obtain... Sentence attention at location, forming an attention mapping matrix. as follows: (5) Where τ is the temperature hyperparameter; The final output of the cross-attention network is obtained by embedding the input sentence using cross-attention fusion. Attention mapping matrix The reconstructed input sentence vector feature map obtained by weighted summation and multiplication The specific formula is as follows: (6) in Represents matrix multiplication. Represents matrix addition; Step 3 is described in detail below: Define the loss function for dual contrastive learning; first, using relation attribute vectors... As the main research object, its corresponding input sentence is embedded. Defined as a secondary object, the loss function is obtained. ; Embed the input sentence As the primary object, and the corresponding relational attribute vector The corresponding loss function is obtained by treating it as a secondary object. loss function and The definition is as follows: (7) (8) in and They represent the first The first input sentence embedding and the first A vector of relational attributes, Indicates the method for calculating the inner product; parameters Ensure the inner product of the opposite direction ( ) is greater than the maximum inner product of any negative pair ( ); Finally, the contrastive learning loss function As shown below: (9)。 2. The zero-shot relation extraction method based on the dual contrastive learning framework and cross-attention module as described in claim 1, characterized in that: Step 1 is described in detail as follows: First, the input sentence is segmented using Word-Piece tokenization, adding two special identifiers, [CLS] and [SEP], to the input sentence. Entity pair information is then obtained using specific entity tag vectors. Used to represent the special identifier [CLS], by... conduct After the activation function operation, the input is fed into the fully connected layer to obtain the identifier representation. : (1) Next, the entity representation is obtained by averaging the hidden state vectors of the entity pairs. and For two entities containing multiple identifiers and , Add to Nonlinear operations are then performed and fed into a fully connected layer, followed by averaging the vector representation to form the entity representation vector. and As shown below: (2) in (c=1, 2) represents two entity vectors. and They represent c=1, 2, the start and end positions of the identifiers of the two entities. Let represent the hidden state vector in the input embedding, and and The learnable parameters are the same for both entities; Finally, through connection , and To obtain the final input sentence vector representation. The final vector representation It is obtained through the following formula: (3) in and These are learnable parameters, and at the same time This represents a connection operation.
3. The zero-shot relation extraction method based on the dual contrastive learning framework and cross-attention module as described in claim 1, characterized in that: Step 4 is described in detail below: Apply activation function Get the final sentence embedding And send Obtain the reconstructed embedding vector in CAM. Then, the reconstructed embedding vector is sent. To the softmax layer, to generate the first A known instance of a relation Dimensional classification probability distribution: (10) in Represents the first One input instance, It is a visible relationship tag. Indicates model parameters, and All of these are learnable parameters; For each input instance In other words, the cross-entropy loss function is used to calculate the relationship classification loss. : (11) The final loss function is as follows: (12) in It is a balancing hyperparameter.
4. The zero-shot relation extraction method based on the dual contrastive learning framework and cross-attention module as described in claim 3, characterized in that: In step 5, the embedding vector for: (13) Improved embedding vector for: (14) in It is a cross-attention network module; The final prediction is: (15) in Returns the sentence for an unknown input. Predictive relationship, Indicates the first A vector of relational attributes for an unknown relationship. This indicates the distance calculation method.
5. A zero-shot relation extraction device for the zero-shot relation extraction method based on the dual contrastive learning framework and cross-attention module as described in claim 1, characterized in that... Includes the following modules: Input encoder module: Encodes the input sentence by linking entity information and [CLS] identifier information in the sentence to obtain the embedded vector representation of the input sentence; Cross-attention network module: After obtaining the context representation vector, the context representation vector and the relation description attribute vector are jointly input into the cross-attention network. The cross-attention network generates a cross-attention map for each feature and uses an attention mechanism to analyze the semantic relevance between the input sentence and the relation description. Dual contrastive learning module: After obtaining the embedding vector through the cross-attention module, contrastive learning training is performed from two perspectives: input sentence and relation description. This minimizes the distance between input sentences and relation descriptions with the same relation label, while increasing the distance between input sentences and relation descriptions with different relation labels. Relationship Classification Module: The relationship classification module performs classification learning among known relationships, enabling the model to learn to distinguish the differences between different relationships.