A semantic distillation method for zero-shot relation extraction

Abstract

This paper discloses a semantic distillation method for zero-shot relation extraction, comprising the following steps: S1: generating entity and context representation vectors from an input sentence; S2: generating entity and context representation vectors from an input relation description; S3: weakening irrelevant features in the context representation and strengthening relevant features through a bidirectional semantic distiller; S4: performing comparative learning using enhanced representations of different context representations in the same batch; and S5: performing fine-grained matching between the entity and context representations of the input sentence and those in the relation description through a fine-grained semantic matching module. This method can effectively cope with complex context interference and improve generalization capabilities across different semantic boundaries.

Description

Technical Field

[0001] The present invention relates to the field of semantic matching technology, and in particular to a semantic distillation method for zero-sample relationship extraction. Background Art

[0002] Relation extraction involves identifying and extracting relationships between entities in a given scenario. It is a crucial component of information extraction and a critical upstream step in many natural language processing tasks. While significant progress has been made in supervised learning, existing methods rely heavily on large amounts of labeled data, and the cost of labeling increases with the number of relationships. Furthermore, supervised extraction methods often lack generalization capabilities when dealing with relationship types not seen during training. Consequently, the Zero-Shot Relation Extraction (ZeroSRE) task has emerged.

[0003] ZeroSRE aims to enable the model to accurately identify and extract unknown relations by training on known relations. Currently, mainstream ZeroSRE methods include semantic matching, hint learning, and the use of large language models. Semantic matching has garnered significant attention in recent years. Semantic matching can be categorized into two main approaches: coarse-grained and fine-grained. Coarse-grained semantic matching combines the different feature representations of a sentence for holistic matching. For example, ZS-BERT projects the sentence and relation descriptions into the same embedding space for semantic matching. Because concatenating all features may introduce noise, this approach can lead to inaccurate matching and poor performance when handling similar relations. In contrast, fine-grained semantic matching breaks sentences into smaller semantic units for matching. This makes it more sensitive to contextual changes and demonstrates superior handling of complex and similar relations. For example, RE-Matching, which pioneered fine-grained semantic matching, has taken ZeroSRE performance to new heights. However, this approach's matching performance is highly dependent on the quality of the feature representation and still faces challenges when handling similar relations or complex semantics.

[0004] First, the principle of compositionality in Fregean semantics states that the meaning of a sentence is determined by the meaning of its components and how they are composed. This suggests that different components of a text must carry different semantic weights and make different contributions to the construction of the overall semantics. This can also be explained by the way photographers highlight visual focal points when composing their images. They typically enhance the visual presence of the subject while reducing the visual interference of the background to highlight the main object. Therefore, when processing text, it is necessary to enhance its main features and suppress irrelevant features, so that its core semantic relationships stand out and the text can be accurately understood. However, although existing methods have attempted to process irrelevant features, they are still unable to completely eliminate the interference caused by complex contexts. Summary of the Invention

[0005] The present invention aims to solve the technical problems existing in the prior art, and particularly innovatively proposes a semantic distillation method for zero-sample relationship extraction, which can effectively cope with complex context interference and improve the generalization ability of different semantic boundaries.

[0006] In order to achieve the above-mentioned object of the present invention, the present invention provides a semantic distillation method for zero-shot relation extraction, comprising the following steps:

[0007] S1: Generate entity and context representation vectors for the input sentence;

[0008] S2: Generate entity and context representation vectors from the input relation description;

[0009] S3: Weaken irrelevant features in context representation and strengthen relevant features through a bidirectional semantic distiller;

[0010] S4: Contrastive learning using enhanced representations of different context representations in the same batch;

[0011] S5: Fine-grained matching is performed between the entity and context representations of the input sentence and the entity and context representations in the relation description through the fine-grained semantic matching module.

[0012] In the above scheme: Step S1 also includes the following: using BERT as a pre-trained encoder to generate entity and context representation vectors for the input sentence;

[0013] S1-1: Assume the input sentence is ,in, Enter the content for this batch A sentence, Representative Sentences The individual words;

[0014] S1-2: Tokenize input sentences The head entity and the tail entity in ;

[0015] S1-2-1: Special tokens passed through BERT encoder 、 、 and Tokenize input sentences The head entity, head entity position, tail entity and tail entity position in ;

[0016] S1-3: Input sentence through bert encoder Each word in generates a corresponding hidden state;

[0017] The corresponding hidden state is generated for the input sentence using the following formula:

[0018] ;

[0019] in, express The hidden state of represents BERT encoder;

[0020] S1-4: Generate the embedding representation of the head entity and the embedding representation of the tail entity;

[0021] S1-4-1: According to the hidden state generated in step S1-3, find Header entities marked with special tokens Corresponding , and generate The corresponding head entity embedding representation;

[0022] The embedding representation of the head entity is generated by the following formula:

[0023] ;

[0024] in, Is the input sentence Embedded representation of head entities;

[0025] S1-4-2: According to the hidden state generated in step S1-3, find Header entities marked with special tokens Corresponding , and generate The corresponding head entity embedding representation;

[0026] The embedding representation of the tail entity is generated by the following formula:

[0027] ;

[0028] in, Is the input sentence The embedding representation of the head entity in represents the weight pool;

[0029] S1-5: Generate contextual embedding representations for the head entity and the tail entity;

[0030] According to the hidden state generated in step S1-3, find and Header entities marked with special tokens 、 The corresponding hidden state 、 , and generate the context embedding representation through the following formula:

[0031] ;

[0032] in, is the contextual embedding representation, Represents the concatenation operator.

[0033] In the above scheme, step S2 also includes the following contents:

[0034] Use Sentence-BERT as a pre-trained encoder to generate entity and context representations of the input relational description;

[0035] S2-1: Assume that the input relationship is described as ;

[0036] S2-2: Tag relationship description Head entity abbreviation in , tail entity abbreviation and relationship description ;

[0037] S2-3: Generate contextual embedding representation of relation description;

[0038] S2-3-1: Describe from the relationship through Python Extract the labeled relationship description ;

[0039] S2-3-2: Obtain context embedding representation using the following formula;

[0040] ;

[0041] in, is the contextual embedding representation of the relation description, Sentence-Bert encoder; S2-4: Generates the head entity embedding representation of the relationship description;

[0042] S2-4-1: Describe from the relationship through Python Extract head entity abbreviations from ;

[0043] S2-4-2: Get the head entity embedding representation using the following formula:

[0044] ;

[0045] in, It is the embedding representation of the head entity of the relationship description;

[0046] S2-5: Generate tail entity embedding representation of relation description;

[0047] S2-5-1: Describe from the relationship through Python Extract tail entity abbreviations from ;

[0048] S2-5-2: Obtain the tail entity embedding representation using the following formula:

[0049] ;

[0050] in, It is the head entity embedding representation of the relation description.

[0051] In the above scheme, step S3 also includes the following contents:

[0052] S3-1: Identifying relation-irrelevant features by fusing the irrelevant selection layers of the projection distiller;

[0053] S3-2: Reducing irrelevant features in the context representation through projection calculation layer;

[0054] S3-2-1: Remove irrelevant features from the context representation after the fully connected layer to obtain a new context representation ;

[0055] S3-3: Selecting the most relevant features from the context representation via Gaussian adaptive attention mechanism.

[0056] In the above scheme, step S3-1 also includes the following contents:

[0057] S3-1-1: Through the dot product attention mechanism, the hidden state of the bert encoder output in step S1-3 Perform a preliminary screening to remove irrelevant features;

[0058] The formula is as follows:

[0059] ;

[0060] in, is the query vector The query matrix composed of and are the key matrix and the value matrix respectively, is the transposed matrix of the key matrix, are irrelevant features obtained through dot product attention; is the normalized exponential function;

[0061] S3-1-2: Identify the hidden state output by the bert encoder in step S1-3 through the anti-attention mechanism Features that have not received much attention;

[0062] The formula is as follows:

[0063] ;

[0064] in, They are irrelevant features obtained through anti-attention;

[0065] S3-1-3: Calculate irrelevant features after vector projection ;

[0066] S3-1-4: Calculate the final irrelevant features using the following formula:

[0067] ;

[0068] in, represents the irrelevant features finally obtained;

[0069] S3-1-5: Calculate irrelevant features obtained by dot product attention The output of the classifier;

[0070] The formula is as follows:

[0071] ;

[0072] in, represents the output of the classifier, and denote classification weight and bias respectively; Function representing the gradient reversal layer;

[0073] S3-1-6: Calculate reverse gradient loss;

[0074] The formula is as follows:

[0075] ;

[0076] in, represents the reverse gradient loss, Indicates the number of samples in the batch; Represents the actual relationship label.

[0077] In the above scheme, step S3-3 also includes the following contents:

[0078] S3-3-1: Using dot product attention mechanism to calculate context representation Attention score;

[0079] The formula is as follows:

[0080] ;

[0081] in, represents the query matrix, represents the bond matrix, is the transposed matrix of the key matrix, Represents the normalized attention score;

[0082] S3-3-2: Determine the context representation after the fully connected layer The relative position relationship between each element in the sequence and the center position is calculated by a learnable offset Adjust the position weight of features in the attention distribution.

[0083] In the above scheme, step S3-3-2 also includes the following contents:

[0084] S3-3-2-1: By inputting sequence The mean to confirm;

[0085] Calculate the input sequence The mean ;

[0086] The formula is as follows:

[0087] ;

[0088] in, is the context representation after the fully connected layer The input sequence, Represents the input sequence length, For the input sequence The mean of is the learnable offset, Represents the input sequence The index length of

[0089] S3-3-2-2: Calculate the position of each element relative to the mean distance;

[0090] The formula is as follows:

[0091] ;

[0092] in, Represents the input sequence The position sequence of each element position in , express The position of each element in the distance;

[0093] S3-3-2-3: Calculate standard deviation ;

[0094] ;

[0095] in, Represents the input sequence The standard deviation of Represents the input sequence length, For the input sequence The mean of

[0096] S3-3-2-4: By standard deviation and distance To calculate the Gaussian distribution weight;

[0097] The formula is as follows:

[0098] ;

[0099] in, represents the Gaussian distribution weight, express The position of each element in the distance, is an exponential function;

[0100] S3-3-2-5: Calculate the normalized Gaussian distribution weight ;

[0101] The formula is as follows:

[0102] ;

[0103] S3-3-2-5: Calculate the final attention weight ;

[0104] Calculated using the following formula:

[0105] ;

[0106] in, is the final attention weight, is the normalized Gaussian distribution weight, Normalized attention scores calculated for the dot-product attention mechanism;

[0107] S3-3-2-6: Calculate the final output representation using the following formula;

[0108] The formula is as follows:

[0109] ;

[0110] in, is the final output representation, is the value matrix;

[0111] S3-3-2-7: Calculate the final context representation using the following formula;

[0112] The formula is as follows:

[0113] ;

[0114] in, is the final context representation, represents the new context representation, is the final output representation.

[0115] In the above scheme, step S4 also includes the following contents:

[0116] Amplify the semantic differences between similar contexts through data augmentation layers;

[0117] S4-1: Context representation after bidirectional distillation through random masking strategy Perform data enhancement processing to generate enhanced representation ;

[0118] S4-2: Context representation after bidirectional distillation through feature truncation strategy Perform data enhancement processing to generate enhanced representation ;

[0119] S4-3: linearly transform the enhanced representations generated in steps S4-1 and S4-2 through a fully connected layer;

[0120] S4-4: Convert the enhanced representation after linear transformation into an enhanced representation of consistent length through the normalization layer 、 ;

[0121] S4-5: Calculate the InfoNCE loss function using the following formula :

[0122] ;

[0123] ;

[0124] in, represents the temperature parameter, represents the similarity matrix between two enhanced representations, Indicates the batch size; is the random mask strategy enhancement representation after linear transformation, is the enhanced representation of the feature truncation strategy after linear transformation, is an exponential function.

[0125] In the above solution, step S5 further includes the following contents:

[0126] S5-1: Calculate the matching score using the following formula:

[0127] ;

[0128] in, represents the equilibrium parameter, Represents the cosine similarity calculation function, Indicates the final matching score;

[0129] S5-2: Calculate marginal ranking loss.

[0130] In the above solution, step S5-2 further includes the following contents:

[0131] S5-2-1: Select semi-hard negative samples and hard negative samples with the highest similarity based on similarity;

[0132] S5-2-1-1: Calculate the matching scores of all sentences in this batch of input content;

[0133] S5-2-1-2: Use the topk selection strategy to select the negative sample with the highest matching score, which is the hard negative sample;

[0134] S5-2-1-3: Loop through the remaining negative samples again and find a negative sample that satisfies the matching score lower than the hard negative sample, which is a semi-hard negative sample;

[0135] S5-2-2: Calculating the marginal ranking loss of hard negative samples ;

[0136] Calculated by the following formula:

[0137] ;

[0138] in, is the boundary parameter of the marginal ranking loss for hard negative samples, is the marginal ranking loss of hard negative samples, Represents the matching score between the input sentence and the relation description;

[0139] S5-2-3: Calculating marginal ranking loss for semi-hard negative samples ;

[0140] Calculated by the following formula:

[0141] ;

[0142] in, is the boundary parameter of the marginal ranking loss for semi-hard negative samples, is the marginal ranking loss for semi-hard negative samples;

[0143] S5-2-4: Calculate the average marginal ranking loss for all batches ;

[0144] ;

[0145] in, Indicates the batch size;

[0146] S5-2-5: Calculate total losses ;

[0147] Calculated using the following formula:

[0148] ;

[0149] in, represents the average marginal ranking loss, represents the reverse gradient loss, Indicates InfoNCE loss.

[0150] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: it can not only effectively strengthen key information and weaken irrelevant information, thereby improving the model's perception of key semantic relationships. At the same time, it can also capture and amplify subtle differences between similar relationships, thereby obtaining higher-quality contextual representations, thereby comprehensively improving the performance of the ZeroSRE task. The set bidirectional semantic distiller realizes semantic distillation by strengthening key information and weakening irrelevant information, thereby obtaining higher-quality contextual representations. It can also help the model learn more meaningful feature representations and effectively alleviate the misjudgment problem caused by similar relationship semantics. In addition, we have also introduced an efficient multi-negative sample strategy in the matching process, which can better guide the model to learn more diverse feature representations and further improve the model's discrimination and generalization capabilities. We also introduced a multi-negative sample selection and training strategy to further refine the relationship feature space, thereby improving the model's discrimination ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0151] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0152] Figure 1 It is a system diagram of the present invention.

[0153] Figure 2 It is a framework diagram of the fusion projection distiller of the present invention.

[0154] Figure 3 It is a framework diagram of the Gaussian adaptive selector of the present invention.

[0155] Figure 4 It is a bar chart of the scores of the method of the present invention and other methods under different numbers of unknown relations of the FewRel dataset.

[0156] Figure 5 It is a bar chart of the scores of the method of the present invention and other methods under different numbers of unknown relations of the Wiki-ZSL dataset.

[0157] Figure 6 It is a visualization diagram of the relationship representation of the RE-Matching method on the FewRel dataset.

[0158] Figure 7 It is a visualization diagram of the relationship representation of the method of the present invention on the FewRel dataset.

[0159] Figure 8 It is a schematic diagram of the F1 scores of the method of the present invention and other methods when the number of unknown relations is 10 on the FewRel dataset.

[0160] Figure 9It is a schematic diagram of the F1 score of the method of the present invention and other methods when the number of unknown relations on the Wiki-ZSL dataset is 10.

[0161] Figure 10 The method of the present invention is the number of different unknown relations and different boundary parameters of the FewRel dataset Schematic diagram of the F1 score in this case.

[0162] Figure 11 The method of the present invention is the number of different unknown relations and different boundary parameters of the FewRel dataset Schematic diagram of the F1 score in this case.

[0163] Figure 12 The number of different unknown relations and different boundary parameters of the method of the present invention in the Wiki-ZSL dataset Schematic diagram of the F1 score in this case.

[0164] Figure 13 The number of different unknown relations and different boundary parameters of the method of the present invention in the Wiki-ZSL dataset Schematic diagram of the F1 score in this case. DETAILED DESCRIPTION

[0165] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0166] like Figures 1 to 13 As shown in FIG, a semantic distillation method for zero-shot relation extraction includes the following steps:

[0167] S1: Generate entity and context representation vectors for the input sentence;

[0168] BERT is used as a pre-trained encoder to generate entity and context representation vectors for the input sentence;

[0169] S1-1: Assume the input sentence is ,in, Enter the content for this batch A sentence, Representative Sentences The individual words;

[0170] S1-2: Tokenize input sentences The head entity and the tail entity in ;

[0171] S1-2-1: Special tokens passed through BERT encoder 、 、 and Tokenize input sentences The head entity, head entity position, tail entity and tail entity position in ;

[0172] S1-3: Input sentence through bert encoder Each word in generates a corresponding hidden state;

[0173] The corresponding hidden state is generated for the input sentence using the following formula:

[0174] ;

[0175] in, express The hidden state of represents BERT encoder;

[0176] S1-4: Generate the embedding representation of the head entity and the embedding representation of the tail entity;

[0177] S1-4-1: According to the hidden state generated in step S1-3, find Header entities marked with special tokens Corresponding , and generate The corresponding head entity embedding representation;

[0178] The embedding representation of the head entity is generated by the following formula:

[0179] ;

[0180] in, Is the input sentence The embedding representation of the head entity in represents the weight pool;

[0181] S1-4-2: According to the hidden state generated in step S1-3, find Header entities marked with special tokens Corresponding , and generate The corresponding head entity embedding representation;

[0182] The embedding representation of the tail entity is generated by the following formula:

[0183] ;

[0184] in, Is the input sentence Embedded representation of mid-tail entities;

[0185] S1-5: Generate contextual embedding representations for the head entity and the tail entity;

[0186] According to the hidden state generated in step S1-3, find and Header entities marked with special tokens 、 The corresponding hidden state 、 , and generate the context embedding representation through the following formula:

[0187] ;

[0188] in, is the contextual embedding representation, Represents the concatenation operator.

[0189] S2: Generate entity and context representation vectors from the input relation description;

[0190] Use Sentence-BERT as a pre-trained encoder to generate entity and context representations of the input relational description;

[0191] S2-1: Assume that the input relationship is described as ;

[0192] S2-2: Description by labeling relationship Head entity abbreviation in , tail entity abbreviation and relationship description ;

[0193] S2-3: Generate contextual embedding representation of relation description;

[0194] S2-3-1: Describe from the relationship through Python Extract the labeled relationship description ;

[0195] S2-3-2: Obtain context embedding representation using the following formula;

[0196] ;

[0197] in, is the contextual embedding representation of the relation description, is the encoder of sentence-Bert;

[0198] S2-4: Generate head entity embedding representation of relation description;

[0199] S2-4-1: Describe from the relationship through Python Extract head entity abbreviations from ;

[0200] S2-4-2: Get the head entity embedding representation using the following formula:

[0201] ;

[0202] in, It is the embedding representation of the head entity of the relationship description;

[0203] S2-5: Generate tail entity embedding representation of relation description;

[0204] S2-5-1: Describe from the relationship through Python Extract tail entity abbreviations from ;

[0205] S2-5-2: Obtain the tail entity embedding representation using the following formula:

[0206] ;

[0207] in, It is the embedding representation of the head entity of the relationship description;

[0208] S3: Weaken irrelevant features in context representation and strengthen relevant features through a bidirectional semantic distiller;

[0209] Since irrelevant features in the contextual embedding representation may interfere with model learning, the model needs to be able to identify and weaken these irrelevant features to obtain higher quality and more effective representations. To achieve this goal, we design a bidirectional semantic distiller that both weakens the influence of irrelevant features and enhances the influence of relevant features.

[0210] The bidirectional semantic distiller includes a fusion projection distiller and a Gaussian adaptive selector. The framework of the fusion projection distiller is as follows: Figure 2 As shown, the framework of the Gaussian adaptive selector is as follows Figure 3 shown.

[0211] S3-1: Identifying relation-irrelevant features by fusing the irrelevant selection layers of the projection distiller;

[0212] S3-1-1: Through the dot product attention mechanism, the hidden state of the bert encoder output in step S1-3 Perform a preliminary screening of irrelevant features to capture potential irrelevant features;

[0213] The formula is as follows:

[0214] ;

[0215] in, is the query vector of the fixed input of the attention mechanism The query matrix composed of and are the key matrix and the value matrix respectively, is the transposed matrix of the key matrix, are irrelevant features obtained through dot product attention; is the normalized exponential function;

[0216] S3-1-2: Identify the hidden state output by the bert encoder in step S1-3 through the anti-attention mechanism Features that have not received much attention;

[0217] The formula is as follows:

[0218] ;

[0219] in, They are irrelevant features obtained through anti-attention;

[0220] S3-1-3: Calculate irrelevant features after vector projection ;

[0221] S3-1-4: Calculate the final irrelevant features using the following formula:

[0222] ;

[0223] in, represents the irrelevant features finally obtained;

[0224] S3-1-5: Calculate irrelevant features obtained by dot product attention The output of the classifier;

[0225] The formula is as follows:

[0226] ;

[0227] in, represents the output of the classifier, and denote classification weight and bias respectively; Function representing the gradient reversal layer;

[0228] S3-1-6: Calculate reverse gradient loss;

[0229] The formula is as follows:

[0230] ;

[0231] in, represents the reverse gradient loss, Indicates the number of samples in the batch; Indicates the real relationship label;

[0232] S3-2: Reducing irrelevant features in the context representation through projection calculation layer;

[0233] S3-2-1: Remove irrelevant features from the context representation after the fully connected layer to obtain a new context representation ;

[0234] Representing the context After conversion through the fully connected layer, the vector projection is calculated and projected onto irrelevant features. direction to find irrelevant features;

[0235] S3-3: Select the most relevant features from the context representation through Gaussian adaptive attention mechanism;

[0236] S3-3-1: Using dot product attention mechanism to calculate context representation Attention score;

[0237] The formula is as follows:

[0238] ;

[0239] in, represents the query matrix, represents the bond matrix, is the transposed matrix of the key matrix, Represents the normalized attention score;

[0240] S3-3-2: Determine the context representation after the fully connected layer The relative position relationship between each element in the sequence and the center position is calculated by a learnable offset Adjust the position weight of features in the attention distribution;

[0241] S3-3-2-1: By inputting sequence The mean to confirm;

[0242] Calculate the input sequence The mean ;

[0243] The formula is as follows: ;

[0244] in, is the context representation after the fully connected layer The input sequence, Represents the input sequence length, For the input sequence The mean of is the learnable offset, Represents the input sequence The index length of

[0245] S3-3-2-2: Calculate the position of each element relative to the mean distance;

[0246] The formula is as follows: ;

[0247] in, Represents the input sequence The position sequence of each element position in , express The position of each element in the distance;

[0248] S3-3-2-3: Calculate standard deviation ;

[0249] ;

[0250] in, Represents the input sequence The standard deviation of Represents the input sequence length, For the input sequence The mean of

[0251] S3-3-2-4: By standard deviation and distance To calculate the Gaussian distribution weight;

[0252] The formula is as follows: ;

[0253] in, represents the Gaussian distribution weight, express The position of each element in the distance, is an exponential function;

[0254] S3-3-2-5: Calculate the normalized Gaussian distribution weight ;

[0255] The formula is as follows: ;

[0256] along with and The Gaussian weights of different positions will also change accordingly, thereby dynamically adjusting the attention weight of each position. Specifically, when a position is away from the center position The closer it is, the greater the Gaussian weight is, and the more important the position is; on the contrary, when a position is far from the center position The farther away, the smaller the Gaussian weight and the lower the importance of the position.

[0257] S3-3-2-5: Calculate the final attention weight ;

[0258] Calculated using the following formula: ;

[0259] in, is the final attention weight, is the normalized Gaussian distribution weight, Normalized attention scores calculated for the dot-product attention mechanism;

[0260] S3-3-2-6: Calculate the final output representation using the following formula;

[0261] The formula is as follows: ;

[0262] in, is the final output representation, is the value matrix;

[0263] Representing the context Input the fully connected layer and input the output into the fusion projection distiller and Gaussian adaptive selector for further processing. In the fusion projection distiller, we use the fusion attention strategy and vector projection to maximize the recognition and remove the context representation. The irrelevant features in , thus obtaining the optimized context representation In the Gaussian adaptive selector, we mainly use the Gaussian adaptive attention mechanism to select The relevant features in , thus obtaining the context representation .

[0264] S3-3-2-7: Calculate the final context representation using the following formula;

[0265] The formula is as follows: ;

[0266] in, is the final context representation, represents the new context representation, is the final output representation.

[0267] S4: Contrastive learning using enhanced representations of different context representations in the same batch;

[0268] S4: Amplify the semantic differences between similar contexts through data enhancement layers;

[0269] S4-1: Context representation after bidirectional distillation through random masking strategy Perform data enhancement processing to generate enhanced representation ;

[0270] Set the mask probability and generate the context representation A random mask matrix with the same matrix shape as the matrix; the random mask matrix is ​​represented by the context The matrix of is multiplied element by element, so that the elements at the corresponding positions are set to zero to generate the enhanced representation ;

[0271] S4-2: Context representation after bidirectional distillation through feature truncation strategy Perform data enhancement processing to generate enhanced representation ;

[0272] The context representation after bidirectional distillation of the input is represented by the feature truncation strategy Some feature dimensions in the matrix are set to zero to truncate some dimensions of the input features and generate enhanced representations ;

[0273] S4-3: linearly transform the enhanced representations generated in steps S4-1 and S4-2 through a fully connected layer;

[0274] S4-4: Convert the enhanced representation after linear transformation into an enhanced representation of consistent length through the normalization layer 、 ;

[0275] S4-5: Calculate the InfoNCE loss function using the following formula :

[0276] ;

[0277] ;

[0278] in, represents the temperature parameter, represents the similarity matrix between two enhanced representations, Indicates the batch size; is the random mask strategy enhancement representation after linear transformation, is the enhanced representation of the feature truncation strategy after linear transformation, is an exponential function;

[0279] Following the SimCLR contrastive learning framework, we choose to use the InfoNCE loss function for representation optimization.

[0280] S5: Perform fine-grained matching between the entity and context representations of the input sentence and the entity and context representations in the relation description through the fine-grained semantic matching module;

[0281] In step S1, the input sentence Encoded, the corresponding representation is obtained , in step S3 the context is represented Input into the bidirectional semantic distiller for processing to obtain a new sentence representation At the same time, step S2 describes the relationship Encode and get the relationship description representation ; At this point we calculate the matching scores for the entity and context representations in steps S1 and S3 and the relationship description in step S2.

[0282] S5-1: Calculate the matching score using the following formula:

[0283] ;

[0284] in, represents the balance parameter, which is set to 0.33 in this embodiment. Represents the cosine similarity calculation function, Indicates the final matching score;

[0285] S5-2: Calculate marginal ranking loss;

[0286] In order to further refine the relational feature space and improve the matching accuracy, we introduced an efficient multi-negative sample strategy based on the original marginal ranking loss to achieve joint optimization. Specifically, we select negative samples by similarity and use the top-k method to consider the first k negative samples as hard negative samples, while negative samples with lower similarity than hard negative samples are considered semi-hard negative samples. During the matching process, the model not only focuses on the hard negative samples with the highest similarity to the positive samples, but also considers the negative samples with the highest similarity to the positive samples. ,

[0287] S5-2-1: Select semi-hard negative samples and hard negative samples with the highest similarity based on similarity;

[0288] S5-2-1-1: Calculate the matching scores of all sentences in this batch of input content;

[0289] S5-2-1-2: Use the topk selection strategy to select the negative sample with the highest matching score, which is the hard negative sample;

[0290] S5-2-1-3: Loop through the remaining negative samples again and find a negative sample that satisfies the matching score lower than the hard negative sample, which is a semi-hard negative sample;

[0291] S5-2-2: Calculating the marginal ranking loss of hard negative samples ;

[0292] Calculated by the following formula:

[0293] ;

[0294] in, is the boundary parameter of the marginal ranking loss for hard negative samples, is the marginal ranking loss of hard negative samples, Represents the matching score between the input sentence and the relation description;

[0295] S5-2-3: Calculating marginal ranking loss for semi-hard negative samples ;

[0296] Calculated by the following formula:

[0297] ;

[0298] in, is the boundary parameter of the marginal ranking loss for semi-hard negative samples, is the marginal ranking loss for semi-hard negative samples;

[0299] S5-2-4: Calculate the average marginal ranking loss for all batches ;

[0300] ;

[0301] in, Indicates the batch size;

[0302] go through By optimizing , the model can make the input sentence closer to the correct relationship description, while staying away from those relationship descriptions with the highest similarity but not correct, and relationship descriptions with slightly lower similarity.

[0303] S5-2-5: Calculate total losses ;

[0304] Calculated using the following formula: ;

[0305] in, represents the average marginal ranking loss, represents the reverse gradient loss, Indicates InfoNCE loss.

[0306] To evaluate the performance of different methods on the Zero-Shot Relation Extraction (ZeroSRE) task, we used two commonly used datasets in this field: FewRel and Wiki-ZSL. FewRel is a manually annotated dataset based on Wikipedia, specifically designed for few-shot tasks. However, it can also be applied to zero-shot tasks as long as the training and test sets do not contain the same relation types. It contains 80 relation types, each consisting of 700 sentence instances.

[0307] Wiki-ZSL is a dataset derived from Wiki-KB using distant supervision. Wiki-ZSL contains more data noise but has a richer set of relation types than FewRel. It contains 113 relation types and 93,383 sentence instances.

[0308] Random selection The class relations are used as the test set, the 5-class relations are used as the validation set, and the remaining relations are used as the training set. All experiments are performed on the datasets partitioned with 5 different random seeds, and the average results of each experiment are reported.

[0309] We use the F1 score as the primary evaluation metric. The F1 score is the harmonic mean of precision and recall, and effectively reflects the model's performance in tasks with an imbalanced number of class instances. We also report the precision and recall corresponding to the F1 score for a more comprehensive assessment of model performance.

[0310] To better validate this technical solution, we compare it with the following methods: AlignRE improves the performance of the ZeroSRE task through encoding pattern alignment and semantic alignment, reducing manual intervention in prototyping. RE-Matching achieves entity-context separation matching for the first time and effectively filters out irrelevant information in the context. SUMASK improves the performance of the ZeroSRE task by leveraging large language models (LLMs) through a recursive text summarization and question answering framework. ZRCM is a ZeroSRE method based on contrastive learning that improves model generalization by designing a negative sample generator and a multi-task learning structure. RelationPrompt generates synthetic data by prompting a language model to solve the zero-shot relation triple extraction task and designs a triple search decoding method to improve the performance of extracting multiple relation triplets from a single sentence. The NoGen method uses the same setup as RelationPrompt but does not use generated synthetic samples for training. PromptMatch is a ZeroSRE model based on state-of-the-art full encoding technology. It concatenates input pairs using BERT and deeply models their fine-grained semantic interactions. ZS-BERT is a Siamese network ZeroSRE model that uses BERT as an encoder. By combining classification loss with metric-based loss, it optimizes the representation space and improves the performance of nearest neighbor search.

[0311] We use Bert-base-uncased and Sentence-Bert as pre-trained encoders for input sentences and relation descriptions, respectively, and fine-tune them according to task requirements. We use the AdamW optimizer with a learning rate of 2e-6, epochs of 5, and temperature is 0.05 and the mask probability is 0.15.

[0312] To achieve the best results on both datasets, we optimized other parameters: the feature truncation ratios for FewRel and Wiki-ZSL were 0.2 and 0.1, respectively, and the batch sizes were 32 and 128, respectively. All experiments were conducted on an NVIDIA GeForce RTX 3090 GPU. The experimental results are shown in Table 1:

[0313] Table 1: Results (%) of SDZRE and baseline methods on FewRel and Wiki-ZSL datasets

[0314]

[0315] In Table 1, m represents the number of unknown relations. In Table 1, we compare the results of different methods on predicting different m types of unknown relations. Experimental results show that SDZRE significantly outperforms the previous best baseline model on both datasets, achieving state-of-the-art results. Compared with AlignRE, SDZRE achieves F1 increases of 4.70%, 5.25%, and 2.75% in FewRel and 6.03%, 6.80%, and 1.69% in Wiki-ZSL, respectively, for predicting three different numbers of unknown relations. Furthermore, our method significantly improves over the baseline RE-Matching. For predicting three different numbers of unknown relations, SDZRE achieves F1 increases of 5.21%, 8.73%, and 6.40% in FewRel and 9.37%, 7.06%, and 2.63% in Wiki-ZSL, respectively, demonstrating SDZRE's effectiveness in the ZeroSRE task and its superior generalization ability to predict more unknown relations.

[0316] Figure 4 、 5 The F1 score changes of some baseline models and SDZRE under different numbers of unknown relations are shown. As the value of m increases (from 5 to 15), the F1 scores of all models on both datasets gradually decrease. Analyzing the reason, we believe that as the value of m increases, it means that the model needs to recognize and distinguish more types of relations. How to effectively alleviate this problem is also a direction worthy of future research. In addition, on the FewRel dataset, the F1 scores of each model are generally higher than those on the Wiki-ZSL dataset, indicating that the Wiki-ZSL dataset (distantly supervised generation) poses a greater challenge to model learning.

[0317] To further verify the contribution of each module of SDZRE, we conducted ablation experiments, and the results are shown in Table 2.

[0318] Table 2: Ablation experiment results when m=10

[0319]

[0320] After removing the contrastive learning module (without Ctr.) from the model, model performance dropped significantly: by 1.83% on the FewRel dataset and by 3.00% on the Wiki-ZSL dataset, demonstrating that our designed contrastive learning framework effectively improves the model's ability to identify similar relationships and handle complex semantics. After removing the bidirectional semantic distiller (without Bid.), the model directly uses the original context representation for matching predictions. We can see a significant performance drop in the experimental results: by 2.37% on the FewRel dataset and by 3.99% on the Wiki-ZSL dataset, demonstrating that the bidirectional semantic distiller plays an important role in weakening the interference of irrelevant features and strengthening relevant features.

[0321] After removing the multi-negative sample strategy (without Mult.), the model only uses randomly selected negative samples in the matching process. Experimental results show that the model's performance drops by 1.75% on the FewRel dataset and 0.29% on the Wiki-ZSL dataset, demonstrating the importance of adopting a multi-sample strategy in improving the model's robustness and ability to handle complex matching tasks. After removing semi-hard negative samples (without Semi.), the model only uses hard negative samples selected by the Top-k method in the matching process. Experimental results show that the model's performance drops by 1.17% on the FewRel dataset and 0.39% on the Wiki-ZSL dataset, demonstrating the importance of using the Top-k method in selecting effective negative samples. Ablation experiments show that all of our proposed methods play an important role, highlighting the effectiveness of our overall framework in enhancing the model's ability to handle complex tasks and improve accuracy.

[0322] To demonstrate the impact of different combinations of data augmentation strategies, we experimentally observed the experimental results under the pairwise combination of Dropout, random masking, and feature truncation. The results are shown in Table 3.

[0323] Table 3: Comparison of F1 scores for different data augmentation strategy combinations (m=10).

[0324]

[0325] As can be seen, the combination of random masking and feature truncation performs best among all the combination strategies. We believe this is because this combination generates more representative and robust features in contrastive learning, thereby improving the model's performance on various datasets. In particular, when dealing with complex and noisy datasets, this combination demonstrates the potential to significantly enhance the model's generalization ability. This experiment ultimately selected the combination of random masking and feature truncation for data augmentation.

[0326] To further observe how our method learns better relation representations, we randomly selected five types of relations from the FewRel dataset as unknown relations and used t-SNE to reduce the dimension of the unknown relations. The results are shown in the figure below. Figure 6 、 7 As shown. Figure 6 It can be seen that the data points are more mixed and dense, especially in blue and yellow, indicating that similar relationships exist in these instances, and RE-Matching fails to effectively distinguish these relationships.

[0327] Ranru Figure 7 As shown in the figure, data points of the same color are relatively concentrated, while data points of different categories are more dispersed. This indicates that our SDZRE can better understand the differences between instances and distinguish the relationships seen, effectively alleviating the interference caused by similar relationships. To further observe the impact of contextual semantic distillation methods, we split the bidirectional semantic distiller into a fusion projection distiller and a Gaussian adaptive selector, while keeping other conditions unchanged, and conducted comparative experiments. The results are shown in Table 4.

[0328] Table 4: Performance of SDZRE under different contextual semantic distillation methods (FewRel, m=15)

[0329]

[0330] As can be seen, while each module can bring certain improvements individually, their synergistic effect is more significant. Analyzing the reasons for this, we believe that the combination of the Fusion Projection Distiller and the Gaussian Adaptive Selector not only achieves complementary model performance but also enhances contextual awareness and information extraction capabilities through architectural optimization, thus achieving the best experimental results.

[0331] To further analyze the efficiency of this technical solution, we compared the inference time and the corresponding F1 score when m=10 on the FewRel and Wiki-ZSL datasets. Figure 8 、 9 shown.

[0332] As can be seen, SDZRE significantly improves the F1 score on both datasets while also achieving faster inference speed. Although SDZRE incurs a certain amount of additional time overhead compared to RE-Matching, it still demonstrates superior performance when inference times are similar.

[0333] We also experimentally observe the impact of hyperparameters on model performance, especially focusing on two key margin parameters. and It is a key parameter in the marginal ranking loss function, which is used to control the minimum value of the similarity gap between positive and negative samples to ensure that the model can effectively distinguish between positive and negative samples, thereby improving the model's discriminative ability and robustness.

[0334] In the experiments, we observed the hyperparameters and To ensure the comparability of the experiments, the left figure shows the impact of the changes in The default value is 0.00, and the right image is fixed to the left image. The optimal value of . Figures 10-13 It can be seen that the performance curves on the two datasets show similar trends. As the hyperparameter increases from 0 to 0.06, the model performance begins to decline, indicating that the optimal values ​​of the two hyperparameters on different datasets are the same. In addition, even if and As the size increases, the model matching will not collapse, which shows the good robustness of our method.

[0335] This paper proposes a semantic distillation method, SDZRE, for the ZeroSRE task. This method uses a bidirectional semantic distiller and a contrastive learning strategy to form a semantic distillation framework. By strengthening relevant features and weakening irrelevant ones, it highlights important semantic relationships and amplifies subtle differences between similar relationships, effectively addressing complex contextual interference. It employs a fine-grained semantic matching method and introduces a multi-negative strategy of hard negative and semi-hard negative samples based on similarity during the matching process, further improving the model's generalization ability across different semantic boundaries. Experimental results demonstrate that SDZRE significantly surpasses the state-of-the-art methods while maintaining a relatively fast inference speed, achieving both performance and efficiency improvements.

Claims

1. A semantic distillation method for zero-shot relation extraction, characterized by: The following steps are involved: S1: Generate entity and context representation vectors for the input sentence; S2: Generate entity and context representation vectors from the input relation description; S3: Weaken irrelevant features in context representation and strengthen relevant features through a bidirectional semantic distiller; Step S3 also Includes the following: S3-1: Identifying relation-irrelevant features by fusing the irrelevant selection layers of the projection distiller; S3-1-1: Through the dot product attention mechanism, the hidden state of the bert encoder output in step S1-3 Perform a preliminary screening to remove irrelevant features; The formula is as follows: ； in, is the query vector The query matrix composed of and are the key matrix and the value matrix respectively, is the transposed matrix of the key matrix, are irrelevant features obtained through dot product attention; is the normalized exponential function; S3-1-2: Identify the hidden state output by the bert encoder in step S1-3 through the anti-attention mechanism Features that have not received much attention; The formula is as follows: ； in, They are irrelevant features obtained through anti-attention; S3-1-3: Calculate irrelevant features after vector projection ; S3-1-4: Calculate the final irrelevant features using the following formula: ； in, represents the irrelevant features finally obtained; S3-1-5: Calculate irrelevant features obtained by dot product attention The output of the classifier; The formula is as follows: ； in, represents the output of the classifier, and denote classification weight and bias respectively; Function representing the gradient reversal layer; S3-1-6: Calculate reverse gradient loss; The formula is as follows: ； in, represents the reverse gradient loss, Indicates the number of samples in the batch; Indicates the real relationship label; S3-2: Reducing irrelevant features in the context representation through projection calculation layer; S3-2-1: Remove irrelevant features from the context representation after the fully connected layer to obtain a new context representation ; S3-3: Select the most relevant features from the context representation through Gaussian adaptive attention mechanism; S4: Contrastive learning using enhanced representations of different context representations in the same batch; S5: Fine-grained matching is performed between the entity and context representations of the input sentence and the entity and context representations in the relation description through the fine-grained semantic matching module.

2. The semantic distillation method for zero-shot relation extraction according to claim 1, characterized in that: Step S1 also includes the following: generating entity and context representation vectors for the input sentence using BERT as a pre-trained encoder; S1-1: Assume the input sentence is ,in, Enter the content for this batch A sentence, Representative Sentences The individual words; S1-2: Tokenize input sentences The head entity and the tail entity in ; S1-2-1: Special tokens passed through BERT encoder 、 、 and Tokenize input sentences The head entity, head entity position, tail entity and tail entity position in ; S1-3: Input sentence through bert encoder Each word in generates a corresponding hidden state; The corresponding hidden state is generated for the input sentence using the following formula: ； in, express The hidden state of represents BERT encoder; S1-4: Generate the embedding representation of the head entity and the embedding representation of the tail entity; S1-4-1: According to the hidden state generated in step S1-3, find Header entities marked with special tokens Corresponding , and generate The corresponding head entity embedding representation; The embedding representation of the head entity is generated by the following formula: ； in, Is the input sentence The embedding representation of the head entity in represents the weight pool; S1-4-2: According to the hidden state generated in step S1-3, find Header entities marked with special tokens Corresponding , and generate The corresponding head entity embedding representation; The embedding representation of the tail entity is generated by the following formula: ； in, Is the input sentence Embedded representation of mid-tail entities; S1-5: Generate contextual embedding representations for the head entity and the tail entity; According to the hidden state generated in step S1-3, find and Header entities marked with special tokens 、 The corresponding hidden state 、 , and generate the context embedding representation through the following formula: ； in, is the contextual embedding representation, Represents the concatenation operator.

3. The semantic distillation method for zero-shot relation extraction according to claim 2, characterized in that: Step S2 also includes the following: S2-1: Assume that the input relationship is described as ; S2-2: Tag relationship description Head entity abbreviation in , tail entity abbreviation and relationship description ; S2-3: Generate contextual embedding representation of relation description; S2-3-1: Describe from the relationship through Python Extract the labeled relationship description ; S2-3-2: Obtain context embedding representation using the following formula; ； in, is the contextual embedding representation of the relation description, is the encoder of sentence-Bert; S2-4: Generate head entity embedding representation of relation description; S2-4-1: Describe from the relationship through Python Extract head entity abbreviations from ; S2-4-2: Get the head entity embedding representation using the following formula: ； in, It is the embedding representation of the head entity of the relationship description; S2-5: Generate tail entity embedding representation of relation description; S2-5-1: Describe from the relationship through Python Extract tail entity abbreviations from ; S2-5-2: Obtain the tail entity embedding representation using the following formula: ； in, It is the head entity embedding representation of the relation description.

4. The semantic distillation method for zero-shot relation extraction according to claim 3, characterized in that: Step S3-3 also includes the following: S3-3-1: Using dot product attention mechanism to calculate context representation Attention score; The formula is as follows: ； in, represents the query matrix, represents the bond matrix, is the transposed matrix of the key matrix, Represents the normalized attention score; S3-3-2: Determine the context representation after the fully connected layer The relative position relationship between each element in the sequence and the center position is calculated by a learnable offset Adjust the position weight of features in the attention distribution.

5. The semantic distillation method for zero-shot relationship extraction according to claim 4, characterized in that: Step S3-3-2 also includes the following: S3-3-2-1: By inputting sequence The mean to confirm; Calculate the input sequence The mean ; The formula is as follows: ； in, is the context representation after the fully connected layer The input sequence, Represents the input sequence length, For the input sequence The mean of is the learnable offset, Represents the input sequence The index length of S3-3-2-2: Calculate the position of each element relative to the mean distance; The formula is as follows: ； in, Represents the input sequence The position sequence of each element position in , express The position of each element in the distance; S3-3-2-3: Calculate standard deviation ; ； in, Represents the input sequence The standard deviation of Represents the input sequence length, For the input sequence The mean of S3-3-2-4: By standard deviation and distance To calculate the Gaussian distribution weight; The formula is as follows: ； in, represents the Gaussian distribution weight, express The position of each element in the distance, is an exponential function; S3-3-2-5: Calculate the normalized Gaussian distribution weight ; The formula is as follows: ； S3-3-2-5: Calculate the final attention weight ; Calculated using the following formula: ； in, is the final attention weight, is the normalized Gaussian distribution weight, Normalized attention scores calculated for the dot-product attention mechanism; S3-3-2-6: Calculate the final output representation using the following formula; The formula is as follows: ； in, is the final output representation, is the value matrix; S3-3-2-7: Calculate the final context representation using the following formula; The formula is as follows: ； in, is the final context representation, represents the new context representation, is the final output representation.

6. The semantic distillation method for zero-shot relation extraction according to claim 1, characterized in that: Step S4 also includes the following: Amplify the semantic differences between similar contexts through data augmentation layers; S4-1: Context representation after bidirectional distillation through random masking strategy Perform data enhancement processing to generate enhanced representation ; S4-2: Context representation after bidirectional distillation through feature truncation strategy Perform data enhancement processing to generate enhanced representation ; S4-3: linearly transform the enhanced representations generated in steps S4-1 and S4-2 through a fully connected layer; S4-4: Convert the enhanced representation after linear transformation into an enhanced representation of consistent length through the normalization layer 、 ; S4-5: Calculate the InfoNCE loss function using the following formula : ； ； in, represents the temperature parameter, represents the similarity matrix between two enhanced representations, Indicates the batch size; is the random mask strategy enhancement representation after linear transformation, is the enhanced representation of the feature truncation strategy after linear transformation, is an exponential function.

7. The semantic distillation method for zero-shot relation extraction according to claim 5, characterized in that: Step S5 also includes the following: S5-1: Calculate the matching score using the following formula: ； in, represents the equilibrium parameter, Represents the cosine similarity calculation function, Indicates the final matching score; S5-2: Calculate marginal ranking loss.

8. The semantic distillation method for zero-shot relation extraction according to claim 7, characterized in that: Step S5-2 also includes the following: S5-2-1: Select semi-hard negative samples and hard negative samples with the highest similarity based on similarity; S5-2-1-1: Calculate the matching scores of all sentences in this batch of input content; S5-2-1-2: Use the topk selection strategy to select the negative sample with the highest matching score, which is the hard negative sample; S5-2-1-3: Loop through the remaining negative samples again and find a negative sample that satisfies the matching score lower than the hard negative sample, which is a semi-hard negative sample; S5-2-2: Calculating the marginal ranking loss of hard negative samples ; Calculated by the following formula: ； in, is the boundary parameter of the marginal ranking loss for hard negative samples, is the marginal ranking loss of hard negative samples, Represents the matching score between the input sentence and the relation description; S5-2-3: Calculating marginal ranking loss for semi-hard negative samples ; Calculated by the following formula: ； in, is the boundary parameter of the marginal ranking loss for semi-hard negative samples, is the marginal ranking loss for semi-hard negative samples; S5-2-4: Calculate the average marginal ranking loss for all batches ; ； in, Indicates the batch size; S5-2-5: Calculate total losses ; Calculated using the following formula: ； in, represents the average marginal ranking loss, represents the reverse gradient loss, Indicates InfoNCE loss.

Images