Semantic distillation method oriented to zero sample relation extraction

Abstract

The invention discloses a zero sample relation extraction-oriented semantic distillation method. The method comprises the following steps of S1, generating entity and context representation vectors from input sentences; s2, generating entity and context representation vectors of the input relation description; s3, weakening irrelevant features in the context representation through a bidirectional semantic distiller, and strengthening relevant features; s4, performing comparative learning by utilizing enhanced representations of different context representations in the same batch; and S5, performing fine-grained matching on the entity and the context representation of the input sentence and the entity and the context representation in the relation description through a fine-grained semantic matching module. Complex context interference can be effectively dealt with, and generalization ability to different semantic boundaries is improved.

Description

Technical Field

[0001] The present invention relates to the field of semantic matching technology, and in particular to a semantic distillation method for zero-sample relationship extraction. Background Art

[0002] The relation extraction task refers to identifying and extracting the relationship between entities in a given scenario. It is an important component of information extraction and a key upstream link in many natural language processing tasks. Although relation extraction has made significant progress in supervised learning, existing methods are highly dependent on large-scale labeled data. As the number of relations increases, the labeling cost also increases. In addition, supervised extraction methods often lack generalization capabilities when dealing with relation types that have not been seen during training. For this reason, the zero-sample relation extraction (ZeroSRE) task came into being.

[0003] ZeroSRE aims to enable the model to accurately identify and extract unknown relations by training known relations. The current mainstream ZeroSRE methods include semantic matching, prompt learning, and the use of large language models. Among them, semantic matching methods have received widespread attention in recent years. Semantic matching is mainly divided into two methods: coarse-grained and fine-grained. Coarse-grained semantic matching performs overall matching by splicing different feature representations of a sentence together. For example, ZS-BERT projects sentences and relationship descriptions into the same embedding space for semantic matching. Since such methods may introduce noise by splicing all features, the matching is inaccurate, and such methods perform poorly when dealing with similar relations. In contrast, fine-grained semantic matching decomposes sentences into smaller semantic units for matching, is more sensitive to context changes, and thus exhibits stronger processing capabilities for complex and similar relations. For example, RE-Matching used fine-grained semantic matching for the first time, bringing ZeroSRE performance to a new level. However, the matching effect of this method is highly dependent on the quality of feature representation, and it still faces certain challenges when dealing with similar relations or complex semantics.

[0004] First, the principle of compositionality in Fregean semantics holds that the meaning of a sentence is determined by the meaning of its components and how they are composed, which means that different components of a text must carry different semantic weights and make different contributions to the construction of the overall semantics. This can also be explained by the way photographers highlight visual emphasis when composing. They usually enhance the visual presence of the subject while reducing the visual interference of the background to highlight the main object. Therefore, when processing text, it is necessary to enhance its main features and suppress irrelevant features to highlight its core semantic relationships in order to accurately understand the text. However, although existing methods have tried to process irrelevant features, they still cannot completely eliminate the interference caused by complex contexts. Summary of the invention

[0005] The present invention aims to solve the technical problems existing in the prior art, and particularly innovatively proposes a semantic distillation method for zero-sample relationship extraction, which can effectively cope with complex context interference and improve the generalization ability of different semantic boundaries.

[0006] In order to achieve the above-mentioned object of the present invention, the present invention provides a semantic distillation method for zero-sample relation extraction, comprising the following steps:

[0007] S1: Generate entity and context representation vectors from the input sentence;

[0008] S2: Generate entity and context representation vectors from the input relation description;

[0009] S3: Weaken irrelevant features in context representation and strengthen relevant features through a bidirectional semantic distiller;

[0010] S4: Contrastive learning using enhanced representations of different context representations in the same batch;

[0011] S5: Fine-grained matching is performed between the entity and context representation of the input sentence and the entity and context representation in the relationship description through the fine-grained semantic matching module.

[0012] In the above scheme: Step S1 also includes the following: generating entity and context representation vectors for the input sentence using BERT as a pre-trained encoder;

[0013] S1-1: Assume the input sentence is Among them, X i The i-th sentence of the input content of this batch, Represents the i-th sentence X i The nth word in ;

[0014] S1-2: Tokenize input sentence Xi The head entity and the tail entity in ;

[0015] S1-2-1: Special tokens passed through BERT encoder [E h ], [\E h ]、[E t ] and [\E t ]Mark the input sentence X i The head entity, the head entity position, the tail entity and the tail entity position in ;

[0016] S1-3: Input sentence X through bert encoder i Generate the corresponding hidden state for each word in;

[0017] The corresponding hidden state is generated for the input sentence through the following formula:

[0018]

[0019] in, express The hidden state of , BERT() represents the bert encoder;

[0020] S1-4: Generate the embedding representation of the head entity and the embedding representation of the tail entity;

[0021] S1-4-1: According to the hidden state generated in step S1-3, find the h ] Header entity marked by special token The corresponding and generate The corresponding head entity embedding representation;

[0022] The embedding representation of the head entity is generated by the following formula:

[0023]

[0024] in, is the input sentence X i Embedding representation of head entity;

[0025] S1-4-2: According to the hidden state generated in step S1-3, find the t ] Header entity marked by special token The corresponding and generate The corresponding head entity embedding representation;

[0026] The embedding representation of the tail entity is generated by the following formula:

[0027]

[0028] in, is the input sentence X i The embedding representation of the head entity in , WeightPooling(·) represents the weight pool;

[0029] S1-5: Generate contextual embedding representations for head entities and tail entities;

[0030] According to the hidden state generated in step S1-3, find the h ] and [E t ] Header entity marked by special token The corresponding hidden state And generate the context embedding representation through the following formula:

[0031]

[0032] Among them, h i is the contextual embedding representation, Represents the concatenation operator.

[0033] In the above scheme: step S2 also includes the following contents:

[0034] Use Sentence-BERT as a pre-trained encoder to generate entity and context representations of the input relational description;

[0035] S2-1: Assume that the input relationship is described as p i ;

[0036] S2-2: Tag relationship description p i Head entity abbreviations in h , tail entity abbreviation s t And the relationship description des = {w 1 ,K,w n};

[0037] S2-3: Generate contextual embedding representation of relation description;

[0038] S2-3-1: Describe p from the relation through Python i Extract the labeled relationship description des = {w 1 ,K,w n};

[0039] S2-3-2: Obtain the context embedding representation through the following formula;

[0040] d v =Sentence-BERT(des)

[0041] Among them, d v is the contextual embedding representation of the relationship description, Sentence-BERT(·) is the encoder of sentence-Bert; S2-4: Generate the head entity embedding representation of the relationship description;

[0042] S2-4-1: Describe p from the relation through Python i Extract the head entity abbreviation s h ;

[0043] S2-4-2: Get the head entity embedding representation through the following formula:

[0044] d h =Sentence-BERT(s h )

[0045] Among them, d h It is the head entity embedding representation of the relationship description;

[0046] S2-5: Generate tail entity embedding representation of relation description;

[0047] S2-5-1: Describe p from the relation through Python i Extract the tail entity abbreviation s t ;

[0048] S2-5-2: Get the tail entity embedding representation through the following formula:

[0049] d t = Sentence-BERT(s t )

[0050] Among them, d t It is the head entity embedding representation of the relation description.

[0051] In the above scheme: step S3 also includes the following contents:

[0052] S3-1: Identifying relation-irrelevant features by fusing irrelevant selection layers of the projection distiller;

[0053] S3-2: Reducing irrelevant features in context representation through projection computation layer;

[0054] S3-2-1: Remove irrelevant features from the context representation to obtain a new context representation;

[0055] The formula is as follows:

[0056]

[0057] in, Indicates the irrelevant features finally obtained The square of the length of is the context representation after the fully connected layer, represents a new context representation;

[0058] S3-3: Selecting the most relevant features from the contextual representation via Gaussian adaptive attention mechanism.

[0059] In the above scheme: step S3-1 also includes the following contents:

[0060] S3-1-1: Through the dot product attention mechanism, the hidden state of the bert encoder output in step S1-3 Perform a preliminary screening to remove irrelevant features;

[0061] The formula is as follows:

[0062] A dot =Softmax(Q·K T )·V

[0063] Among them, Q is the query matrix composed of query vector q, K and V are the key matrix and value matrix respectively, K T is the transposed matrix of the key matrix, A dot is an irrelevant feature obtained by dot product attention; Softmax(g) is a normalized exponential function;

[0064] S3-1-2: Identify the hidden state of the bert encoder output in step S1-3 through the anti-attention mechanism Features that have not received much attention in

[0065] The formula is as follows:

[0066] A inv =(1-Softmax(Q·K T ))·V

[0067] Among them, A inv They are irrelevant features obtained through anti-attention;

[0068] S3-1-3: The irrelevant features obtained after vector projection are calculated by the following formula;

[0069]

[0070] Among them, A i ' nv is an irrelevant feature obtained by vector projection calculation, |A dot 2 Yes A dot The square of the length of

[0071] S3-1-4: The irrelevant features finally obtained are calculated by the following formula:

[0072]

[0073] in, represents the irrelevant features finally obtained;

[0074] S3-1-5: Calculate the irrelevant feature A obtained by dot product attention dot The output of the classifier;

[0075] The formula is as follows:

[0076] Cp i =ε·GRL(A dot )+b

[0077] Among them, Cp i represents the output of the classifier, ε and b represent the classification weight and bias respectively; GRL(g) represents the function of the gradient reversal layer;

[0078] S3-1-6: Calculate reverse gradient loss;

[0079] The formula is as follows:

[0080]

[0081] Among them, L Rev represents the reverse gradient loss, N represents the number of samples in the batch; y i Represents the actual relationship label.

[0082] In the above scheme: step S3-3 also includes the following contents:

[0083] S3-3-1: Using dot product attention mechanism to calculate context representation Attention score;

[0084] The formula is as follows:

[0085]

[0086] Where Q represents the query matrix, K represents the key matrix, and K T is the transposed matrix of the key matrix, Represents the normalized attention score;

[0087] S3-3-2: Determine the context representation after the fully connected layer The relative position relationship between each element in and the center of the sequence is calculated by a learnable offset μ offset Adjust the position weight of the feature in the attention distribution.

[0088] In the above scheme: step S3-3-2 also includes the following contents:

[0089] S3-3-2-1: By inputting sequence k i The mean value μ is determined by

[0090] Calculate the input sequence k i The mean μ of

[0091] The formula is as follows:

[0092]

[0093] Among them, k i is the context representation after the fully connected layer The input sequence, l represents the input sequence k i length, μ is the input sequence k i The mean value, μ offset is the learnable offset, j represents the input sequence k i The index length of

[0094] S3-3-2-2: Calculate the distance of each element position relative to the mean μ;

[0095] The formula is as follows:

[0096] D i =|l i -μ|

[0097] Among them, l i Represents the input sequence k i The position sequence composed of each element position in D i Indicates k i The distance of each element position relative to the mean μ;

[0098] S3-3-2-3: Calculate the standard deviation σ;

[0099]

[0100] Where σ represents the input sequence k i The standard deviation of l represents the input sequence k i length, μ is the input sequence k i The mean of

[0101] S3-3-2-4: By standard deviation σ and distance D i To calculate the Gaussian distribution weight;

[0102] The formula is as follows:

[0103]

[0104] Among them, Gi represents the Gaussian distribution weight, D i Indicates k i The distance of each element position relative to the mean μ, exp(g) is an exponential function;

[0105] S3-3-2-5: Calculate the normalized Gaussian distribution weight

[0106] The formula is as follows:

[0107]

[0108] S3-3-2-5: Calculate the final attention weight W;

[0109] Calculate using the following formula:

[0110]

[0111] Among them, W is the final attention weight, is the normalized Gaussian distribution weight, Normalized attention scores calculated for the dot-product attention mechanism;

[0112] S3-3-2-6: Calculate the final output representation using the following formula;

[0113] The formula is as follows:

[0114]

[0115] in, is the final output representation, V is the value matrix;

[0116] S3-3-2-7: Calculate the final context representation using the following formula;

[0117] The formula is as follows:

[0118]

[0119] in, is the final context representation.

[0120] In the above scheme: step S4 also includes the following contents:

[0121] Amplify the semantic differences between similar contexts through data augmentation layers;

[0122] S4-1: Contextual representation after bidirectional distillation through random masking strategy Perform data enhancement processing to generate enhanced representation

[0123] S4-2: Context representation after bidirectional distillation through feature truncation strategy Perform data enhancement processing to generate enhanced representation

[0124] S4-3: linearly transform the enhanced representations generated by steps S4-1 and S4-2 through a fully connected layer;

[0125] S4-4: Convert the linearly transformed enhanced representation to an enhanced representation of consistent length through a normalization layer

[0126] S4-5: Calculate the InfoNCE loss function L by the following formula Ctr :

[0127]

[0128] Among them, τ represents the temperature parameter, m ij represents the similarity matrix between two enhanced representations, and N represents the batch size; is the random mask strategy enhancement representation after linear transformation, is the enhanced representation of the feature truncation strategy after linear transformation, and exp(·) is an exponential function.

[0129] In the above scheme: step S5 also includes the following contents:

[0130] S5-1: Calculate the matching score using the following formula:

[0131]

[0132] Among them, α represents the balance parameter, sim(·) represents the cosine similarity calculation function, and f(X i ,p) represents the final matching score;

[0133] S5-2: Calculate marginal ranking loss.

[0134] In the above scheme: step S5-2 also includes the following contents:

[0135] S5-2-1: Select semi-hard negative samples based on similarity And the hard negative sample with the highest similarity

[0136] S5-2-2: Calculating hard negative samples The marginal ranking loss

[0137] Calculated by the following formula:

[0138]

[0139] Among them, gam t is the boundary parameter of the marginal ranking loss for hard negative samples, is the marginal ranking loss of hard negative samples, f(·) represents the matching score between the input sentence and the relation description;

[0140] S5-2-3: Calculating semi-hard negative samples The marginal ranking loss

[0141] Calculated by the following formula:

[0142]

[0143] Among them, gam b is the boundary parameter of the marginal ranking loss for semi-hard negative samples, is the marginal ranking loss for semi-hard negative samples;

[0144] S5-2-4: Calculate the average marginal ranking loss L for all batches Mr ;

[0145]

[0146] Where N represents the batch size;

[0147] S5-2-5: Calculate the total loss L;

[0148] Calculate using the following formula:

[0149] L=L Rev +L Ctr +L Mr

[0150] Among them, L Mr Denotes the average marginal ranking loss L Mr , L Rev represents the reverse gradient loss, L Ctr Indicates InfoNCE loss;

[0151] S5-2-1-1: Calculate the matching scores of all sentences in this batch of input content;

[0152] S5-2-1-2: Select the negative sample with the highest matching score through the topk selection strategy, which is a hard negative sample

[0153] S5-2-1-3: Loop through the remaining negative samples again to find hard negative samples that meet the matching score ratio Low negative samples, semi-hard negative samples

[0154] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: it can not only effectively strengthen key information and weaken irrelevant information, thereby improving the model's perception of key semantic relationships. At the same time, it can also capture and amplify the subtle differences between similar relationships, thereby obtaining a higher-quality contextual representation, thereby comprehensively improving the performance of the ZeroSRE task. The set bidirectional semantic distiller realizes semantic distillation by strengthening key information and weakening irrelevant information, thereby obtaining a higher-quality contextual representation. It can also help the model learn more meaningful feature representations and effectively alleviate the problem of misjudgment caused by similar relationship semantics. In addition, we have also introduced an efficient multi-negative sample strategy in the matching process, which can better guide the model to learn more diverse feature representations and further improve the model's discrimination and generalization capabilities. We have also introduced a multi-negative sample selection and training strategy to further refine the relationship feature space, thereby improving the model's discrimination ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0155] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0156] Figure 1 It is a system diagram of the present invention.

[0157] Figure 2 It is a framework diagram of the fusion projection distiller of the present invention.

[0158] Figure 3 It is a framework diagram of the Gaussian adaptive selector of the present invention.

[0159] Figure 4 It is a bar chart of the scores of the method of the present invention and other methods under different numbers of unknown relations of the FewRel dataset.

[0160] Figure 5 It is a bar chart of the scores of the method of the present invention and other methods under different numbers of unknown relations in the Wiki-ZSL dataset.

[0161] Figure 6 It is a visualization of the relationship representation of the RE-Matching method on the FewRel dataset.

[0162] Figure 7 It is a visualization diagram of the relationship representation of the method of the present invention on the FewRel dataset.

[0163] Figure 8 It is a schematic diagram of the F1 scores of the method of the present invention and other methods when the number of unknown relations on the FewRel dataset is 10.

[0164] Fig. 9It is a schematic diagram of the F1 scores of the method of the present invention and other methods when the number of unknown relations on the Wiki-ZSL dataset is 10.

[0165] Fig.10 The method of the present invention is different in the number of unknown relations of the FewRel data set and different boundary parameters gam t Schematic diagram of the F1 score in this case.

[0166] Fig.11 The method of the present invention is different in the number of unknown relations of the FewRel data set and different boundary parameters gam d Schematic diagram of the F1 score in this case.

[0167] Fig.12 The method of the present invention is the number of different unknown relations in the Wiki-ZSL dataset and different boundary parameters gam t Schematic diagram of the F1 score in this case.

[0168] Fig.13 The method of the present invention is the number of different unknown relations in the Wiki-ZSL dataset and different boundary parameters gam d Schematic diagram of the F1 score in this case. DETAILED DESCRIPTION

[0169] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0170] like Figures 1 to 13 As shown, a semantic distillation method for zero-shot relation extraction includes the following steps:

[0171] S1: Generate entity and context representation vectors from the input sentence;

[0172] Use BERT as a pre-trained encoder to generate entity and context representation vectors for the input sentence;

[0173] S1-1: Assume the input sentence is Among them, X i The i-th sentence of the input content of this batch, Represents the i-th sentence X i The nth word in ;

[0174] S1-2: Tokenize input sentence X i The head entity and the tail entity in ;

[0175] S1-2-1: Special tokens passed through BERT encoder [E h ], [\E h ]、[E t ] and [\E t ]Mark the input sentence X i The head entity, the head entity position, the tail entity and the tail entity position in ;

[0176] S1-3: Input sentence X through bert encoder i Generate the corresponding hidden state for each word in;

[0177] The corresponding hidden state is generated for the input sentence through the following formula:

[0178]

[0179] in, express The hidden state of , BERT() represents the bert encoder;

[0180] S1-4: Generate the embedding representation of the head entity and the embedding representation of the tail entity;

[0181] S1-4-1: According to the hidden state generated in step S1-3, find the h ] Header entity marked by special token The corresponding and generate The corresponding head entity embedding representation;

[0182] The embedding representation of the head entity is generated by the following formula:

[0183]

[0184] in, is the input sentence X i The embedding representation of the head entity in , WeightPooling(·) represents the weight pool;

[0185] S1-4-2: According to the hidden state generated in step S1-3, find the t ] Header entity marked by special token The corresponding and generate The corresponding head entity embedding representation;

[0186] The embedding representation of the tail entity is generated by the following formula:

[0187]

[0188] in, is the input sentence X i Embedding representation of mid-tail entities;

[0189] S1-5: Generate contextual embedding representations for head entities and tail entities;

[0190] According to the hidden state generated in step S1-3, find the h ] and [E t ] Header entity marked by special token The corresponding hidden state And generate the context embedding representation through the following formula:

[0191]

[0192] Among them, h i is the contextual embedding representation, Represents the concatenation operator.

[0193] S2: Generate entity and context representation vectors from the input relation description;

[0194] Use Sentence-BERT as a pre-trained encoder to generate entity and context representations of the input relational description;

[0195] S2-1: Assume that the input relationship is described as p i ;

[0196] S2-2: Describing p by labeling relations i Head entity abbreviations in h , tail entity abbreviation s t And the relationship description des = {w 1 ,K,w n};

[0197] S2-3: Generate contextual embedding representation of relation description;

[0198] S2-3-1: Describe p from the relation through Python i Extract the labeled relationship description des = {w 1 ,K,w n};

[0199] S2-3-2: Obtain the context embedding representation through the following formula;

[0200] d v =Sentence-BERT(des)

[0201] Among them, d vis the contextual embedding representation of the relation description, and Sentence-BERT(·) is the encoder of sentence-Bert;

[0202] S2-4: Generate head entity embedding representation of relation description;

[0203] S2-4-1: Describe p from the relation through Python i Extract the head entity abbreviation s h ;

[0204] S2-4-2: Get the head entity embedding representation through the following formula:

[0205] d h =Sentence-BERT(s h )

[0206] Among them, d h It is the head entity embedding representation of the relationship description;

[0207] S2-5: Generate tail entity embedding representation of relation description;

[0208] S2-5-1: Describe p from the relation through Python i Extract the tail entity abbreviation s t ;

[0209] S2-5-2: Get the tail entity embedding representation through the following formula:

[0210] d t =Sentence-BERT(s t )

[0211] Among them, d t It is the head entity embedding representation of the relationship description;

[0212] S3: Weaken irrelevant features in context representation and strengthen relevant features through a bidirectional semantic distiller;

[0213] Since irrelevant features in the contextual embedding representation may interfere with model learning, the model needs to be able to identify and weaken these irrelevant features to obtain higher quality and more effective representations. To achieve this goal, we design a bidirectional semantic distiller that both weakens the impact of irrelevant features and enhances the impact of relevant features.

[0214] The bidirectional semantic distiller includes a fusion projection distiller and a Gaussian adaptive selector. The framework of the fusion projection distiller is as follows: Figure 2 As shown in Figure 2, the framework of the Gaussian adaptive selector is as follows: Figure 3 shown.

[0215] S3-1: Identifying relation-irrelevant features by fusing irrelevant selection layers of the projection distiller;

[0216] S3-1-1: Through the dot product attention mechanism, the hidden state of the bert encoder output in step S1-3 Perform a preliminary screening of irrelevant features to capture potential irrelevant features;

[0217] The formula is as follows:

[0218] A dot =Softmax(Q·K T )·V

[0219] Among them, Q is the query matrix composed of the query vector q of the fixed input of the attention mechanism, K and V are the key matrix and value matrix respectively, K T is the transposed matrix of the key matrix, A dot is an irrelevant feature obtained by dot product attention; Softmax(g) is a normalized exponential function;

[0220] S3-1-2: Identify the hidden state of the bert encoder output in step S1-3 through the anti-attention mechanism Features that have not received much attention in

[0221] The formula is as follows:

[0222] A inv =(1-Softmax(Q·K T ))·V

[0223] Among them, A inv They are irrelevant features obtained through anti-attention;

[0224] S3-1-3: The irrelevant features obtained after vector projection are calculated by the following formula;

[0225]

[0226] Among them, A i ' nv is an irrelevant feature obtained by vector projection calculation, |A dot 2 Yes A dot The square of the length of

[0227] S3-1-4: The irrelevant features finally obtained are calculated by the following formula:

[0228]

[0229] in, represents the irrelevant features finally obtained;

[0230] S3-1-5: Calculate the irrelevant feature A obtained by dot product attention dot The output of the classifier;

[0231] The formula is as follows:

[0232] Cp i =ε·GRL(A dot )+b

[0233] Among them, Cp i represents the output of the classifier, ε and b represent the classification weight and bias respectively; GRL(g) represents the function of the gradient reversal layer;

[0234] S3-1-6: Calculate reverse gradient loss;

[0235] The formula is as follows:

[0236]

[0237] Among them, L Rev represents the reverse gradient loss, N represents the number of samples in the batch; y i Indicates the real relationship label;

[0238] S3-2: Reducing irrelevant features in context representation through projection computation layer;

[0239] S3-2-1: Remove irrelevant features from the context representation to obtain a new context representation;

[0240] The formula is as follows:

[0241]

[0242] in, Indicates the irrelevant features finally obtained The square of the length of is the context representation after the fully connected layer, represents a new context representation;

[0243] Let the context be h i After conversion through the fully connected layer, projection to irrelevant features is performed through vector projection calculation in the direction of , in order to find irrelevant features;

[0244] S3-3: Select the most relevant features from the context representation through Gaussian adaptive attention mechanism;

[0245] S3-3-1: Using dot product attention mechanism to calculate context representation Attention score;

[0246] The formula is as follows:

[0247]

[0248] Where Q represents the query matrix, K represents the key matrix, and K T is the transposed matrix of the key matrix, Represents the normalized attention score;

[0249] S3-3-2: Determine the context representation after the fully connected layer The relative position relationship between each element in and the center of the sequence is calculated by a learnable offset μ offset Adjust the position weight of features in the attention distribution;

[0250] S3-3-2-1: By inputting sequence k i The mean value μ is determined by

[0251] Calculate the input sequence k i The mean μ of

[0252] The formula is as follows:

[0253] Among them, k i is the context representation after the fully connected layer The input sequence, l represents the input sequence k i length, μ is the input sequence k i The mean value, μ offset is the learnable offset, j represents the input sequence k i The index length of

[0254] S3-3-2-2: Calculate the distance of each element position relative to the mean μ;

[0255] The formula is as follows: i =|l i -μ|

[0256] Among them, l i Represents the input sequence k i The position sequence composed of each element position in D i Indicates k i The distance of each element position relative to the mean μ;

[0257] S3-3-2-3: Calculate the standard deviation σ;

[0258]

[0259] Where σ represents the input sequence k i The standard deviation of l represents the input sequence k i length, μ is the input sequence ki The mean of

[0260] S3-3-2-4: By standard deviation σ and distance D i To calculate the Gaussian distribution weight;

[0261] The formula is as follows:

[0262] Among them, G i represents the Gaussian distribution weight, D i Indicates k i The distance of each element position relative to the mean μ, exp(g) is an exponential function;

[0263] S3-3-2-5: Calculate the normalized Gaussian distribution weight

[0264] The formula is as follows:

[0265] As μ and σ change, the Gaussian weights of different positions will also change accordingly, thereby dynamically adjusting the attention weight of each position. Specifically, when a position is closer to the center position μ, the larger the Gaussian weight, the more important the position is; conversely, when a position is farther from the center position μ, the smaller the Gaussian weight, the less important the position is.

[0266] S3-3-2-5: Calculate the final attention weight W;

[0267] Calculate using the following formula:

[0268] Among them, W is the final attention weight, is the normalized Gaussian distribution weight, Normalized attention scores calculated for the dot-product attention mechanism;

[0269] S3-3-2-6: Calculate the final output representation using the following formula;

[0270] The formula is as follows:

[0271] in, is the final output representation, V is the value matrix;

[0272] Let the context be h i The input is a fully connected layer, and the output is fed into the fusion projection distiller and Gaussian adaptive selector for further processing. In the fusion projection distiller, we use the fusion attention strategy and vector projection to maximize the recognition and remove the context representation h i The irrelevant features in , thus obtaining the optimized context representation In the Gaussian adaptive selector, we mainly use the Gaussian adaptive attention mechanism to select h i The relevant features in , thus obtaining the context representation

[0273] S3-3-2-7: Calculate the final context representation using the following formula;

[0274] The formula is as follows:

[0275] in, is the final context representation.

[0276] S4: Contrastive learning using enhanced representations of different context representations in the same batch;

[0277] S4: Amplify the semantic differences between similar contexts through data augmentation layers;

[0278] S4-1: Contextual representation after bidirectional distillation through random masking strategy Perform data enhancement processing to generate enhanced representation

[0279] Set the mask probability and generate the context representation A random mask matrix with the same matrix shape as the matrix of The matrix of is multiplied element by element, so that the elements at the corresponding positions are set to zero to generate an enhanced representation

[0280] S4-2: Context representation after bidirectional distillation through feature truncation strategy Perform data enhancement processing to generate enhanced representation

[0281] The context representation after bidirectional distillation of the input is represented by the feature truncation strategy Some feature dimensions in the matrix are set to zero to truncate some dimensions of the input features and generate enhanced representations

[0282] S4-3: linearly transform the enhanced representations generated by steps S4-1 and S4-2 through a fully connected layer;

[0283] S4-4: Convert the linearly transformed enhanced representation to an enhanced representation of consistent length through a normalization layer

[0284] S4-5: Calculate the InfoNCE loss function L by the following formula Ctr :

[0285]

[0286] Among them, τ represents the temperature parameter, m ij represents the similarity matrix between two enhanced representations, and N represents the batch size; is the random mask strategy enhancement representation after linear transformation, is the enhanced representation of the feature truncation strategy after linear transformation, exp(·) is the exponential function;

[0287] Following the SimCLR contrastive learning framework, we choose to use the InfoNCE loss function for representation optimization.

[0288] S5: Perform fine-grained matching between the entity and context representation of the input sentence and the entity and context representation in the relationship description through the fine-grained semantic matching module;

[0289] In step S1, the input sentence X i Encoded, the corresponding representation is obtained In step S3, the context is represented by h i Input to the bidirectional semantic distiller for processing to obtain a new sentence representation At the same time, the relationship description p is encoded through step S2 to obtain the relationship description representation (d h ,d v ,d t ); At this point we calculate the matching scores for the entity and context representations in steps S1 and S3 and the relationship description in step S2.

[0290] S5-1: Calculate the matching score using the following formula:

[0291]

[0292] Wherein, α represents a balance parameter, which is set to 0.33 in this embodiment, sim(·) represents a cosine similarity calculation function, and f(X i ,p) represents the final matching score;

[0293] S5-2: Calculate marginal ranking loss;

[0294] In order to further refine the relational feature space and improve the matching accuracy, we introduced an efficient multi-negative sample strategy based on the original marginal ranking loss to achieve joint optimization. Specifically, we select negative samples by similarity, using the top-k method to consider the first k negative samples as hard negative samples, and negative samples with lower similarity than hard negative samples as semi-hard negative samples. During the matching process, the model not only focuses on the hard negative samples with the highest similarity to the positive samples, but also on the hard negative samples with the highest similarity to the positive samples.

[0295] S5-2-1: Select semi-hard negative samples based on similarity And the hard negative sample with the highest similarity

[0296] S5-2-1-1: Calculate the matching scores of all sentences in this batch of input content;

[0297] S5-2-1-2: Select the negative sample with the highest matching score through the topk selection strategy, which is a hard negative sample

[0298] S5-2-1-3: Loop through the remaining negative samples again to find hard negative samples that meet the matching score ratio Low negative samples, semi-hard negative samples

[0299] S5-2-2: Calculating hard negative samples The marginal ranking loss

[0300] Calculated by the following formula:

[0301]

[0302] Among them, gam t is the boundary parameter of the marginal ranking loss for hard negative samples, is the marginal ranking loss of hard negative samples, f(·) represents the matching score between the input sentence and the relation description;

[0303] S5-2-3: Calculating semi-hard negative samples The marginal ranking loss

[0304] Calculated by the following formula:

[0305]

[0306] Among them, gam b is the boundary parameter of the marginal ranking loss for semi-hard negative samples, is the marginal ranking loss for semi-hard negative samples;

[0307] S5-2-4: Calculate the average marginal ranking loss L for all batches Mr ;

[0308]

[0309] Where N represents the batch size;

[0310] Passing by L Mr By optimizing , the model can make the input sentence closer to the correct relationship description, while staying away from those relationship descriptions with the highest similarity but not correct, and those with slightly lower similarity.

[0311] S5-2-5: Calculate the total loss L;

[0312] Calculate by the following formula: L = L Rev +L Ctr +L Mr

[0313] Among them, L Mr Denotes the average marginal ranking loss L Mr , L Rev represents the reverse gradient loss, L Ctr Indicates InfoNCE loss.

[0314] In order to evaluate the performance of different methods on the Zero-Shot Relation Extraction (ZeroSRE) task, we use two commonly used datasets in this field: FewRel and Wiki-ZSL. FewRel is a manually annotated dataset based on Wikipedia, specially designed for few-shot tasks. However, it can also be applied to zero-shot tasks as long as the training set and the test set do not contain the same relation types. It contains 80 types of relations, each of which consists of 700 sentence instances.

[0315] Wiki-ZSL is a dataset derived from Wiki-KB and generated through remote supervision. Wiki-ZSL contains more data noise, but its relation types are richer than FewRel. It contains 113 types of relations and 93,383 sentence instances.

[0316] We randomly select m∈{5,10,15} relations as the test set, 5 relations as the validation set, and the remaining relations as the training set. We conduct all experiments on the datasets partitioned with 5 different random seeds, and report the average results of each experiment.

[0317] The F1 score is used as the main evaluation indicator. The F1 score is the harmonic mean of precision and recall, which can effectively reflect the performance of the model in tasks with an unbalanced number of category instances. In addition, we also report the precision and recall corresponding to the F1 score to more comprehensively evaluate the performance of the model.

[0318] In order to better verify this technical solution, the following technical solution is compared with the following methods: AlignRE improves the performance of ZeroSRE tasks through encoding pattern alignment and semantic alignment, reducing manual intervention in prototype construction. RE-Matching realizes the matching of entity and context separation for the first time, and effectively filters irrelevant information in the context. SUMASK improves the performance of ZeroSRE tasks through recursive text summarization and question answering framework using large language models (LLMs). ZRCM is a ZeroSRE method based on contrastive learning, which improves the generalization ability of the model by designing negative sample generators and multi-task learning structures. RelationPrompt generates synthetic data by prompting the language model to complete the zero-shot relation triple extraction task, and designs a triple search decoding method to improve the effect of extracting multiple relation triplets from a single sentence. The NoGen method has the same settings as RelationPrompt, but does not use the generated synthetic samples for training. PromptMatch is a ZeroSRE model based on the most advanced full encoding technology, which concatenates input pairs through BERT and deeply models their fine-grained semantic interactions. ZS-BERT is a Siamese network ZeroSRE model that uses BERT as an encoder. It optimizes the representation space and improves the nearest neighbor search effect by combining classification loss with metric-based loss.

[0319] We use Bert-base-uncased and Sentence-Bert as pre-trained encoders for input sentences and relation descriptions, respectively, and fine-tune them according to task requirements. We use the AdamW optimizer with a learning rate of 2e-6, epochs of 5, a temperature of τ of 0.05, and a mask probability of 0.15.

[0320] In order to achieve the best results for the model on the two datasets, we optimized other parameters: the feature truncation ratios of FewRel and Wiki-ZSL were 0.2 and 0.1 respectively, and the batch sizes were 32 and 128 respectively. All experiments were completed in the NVIDIA GeForce RTX 3090 environment. The experimental results are shown in Table 1:

[0321] Table 1: Results of SDZRE and baseline methods on FewRel and Wiki-ZSL datasets (%)

[0322]

[0323]

[0324] In Table 1, m represents the number of unknown relations. In Table 1, we compare the results of different methods in predicting different m types of unknown relations. The experimental results show that SDZRE significantly outperforms the previous best baseline model on both datasets and achieves the current SOTA results. In predicting three different numbers of unknown relations, compared with AlignRE, the F1 value of SDZRE in FewRel increases by 4.70%, 5.25%, and 2.75%, respectively, and the F1 value in Wiki-ZSL increases by 6.03%, 6.80%, and 1.69%, respectively. In addition, our method is more significantly improved than the baseline RE-Matching. In predicting three different numbers of unknown relations, the F1 value in FewRel increases by 5.21%, 8.73%, and 6.40%, respectively, and the F1 value in Wiki-ZSL increases by 9.37%, 7.06%, and 2.63%, respectively, indicating the superiority of SDZRE in the effectiveness of ZeroSRE tasks and the generalization ability of predicting more unknown relations.

[0325] Figure 4 , 5 The changes in F1 scores of some baseline models and SDZRE under different numbers of unknown relations are shown. As the value of m increases (from 5 to 15), the F1 scores of all models on the two datasets gradually decrease. Analyzing the reason, we believe that as the value of m increases, it means that the model needs to recognize and distinguish more types of relations. How to effectively alleviate this problem is also a direction worthy of future research. In addition, on the FewRel dataset, the F1 scores of each model are generally higher than those on the Wiki-ZSL dataset, indicating that the Wiki-ZSL dataset (remotely supervised generation) brings greater challenges to model learning.

[0326] To further verify the contribution of each module of SDZRE, we conducted an ablation experiment, and the results are shown in Table 2. After removing the contrastive learning module (w / o Ctr.) from the model, the model performance dropped significantly: it dropped by 1.83% on the FewRel dataset and 3.00% on the Wiki-ZSL dataset, indicating that the contrastive learning framework we designed effectively improved the model's ability to identify similar relationships and handle complex semantics. After removing the bidirectional semantic distiller (w / o Bid.), the model directly uses the original context representation for matching prediction. We can notice that the performance of the experimental results has dropped significantly: it dropped by 2.37% on the FewRel dataset and 3.99% on the Wiki-ZSL dataset, indicating that the bidirectional semantic distiller plays an important role in weakening the interference of irrelevant features and strengthening relevant features.

[0327] After removing the multi-negative sample strategy (w / o Mult.), the model only uses randomly selected negative samples in the matching process. Experimental results show that the performance of the model drops by 1.75% on the FewRel dataset and 0.29% on the Wiki-ZSL dataset, indicating the importance of adopting a multi-sample strategy in improving the robustness of the model and its ability to handle complex matching tasks. After removing the semi-hard negative samples (w / o Semi.), the model only uses hard negative samples selected by the Top-k method in the matching process. Experimental results show that the performance of the model drops by 1.17% on the FewRel dataset and 0.39% on the Wiki-ZSL dataset, indicating the importance of using the Top-k method in selecting effective negative samples. The results of the ablation experiment show that the methods we proposed all play an important role, which emphasizes the effectiveness of the overall framework we designed in enhancing the model to handle complex tasks and improve accuracy.

[0328] Table 2: Ablation experiment results when m=10

[0329]

[0330] In order to demonstrate the impact of different combinations of data augmentation strategies, we experimentally observed the experimental results of the pairwise combination of Dropout, random mask, and feature truncation. The results are shown in Table 3.

[0331] Table 3: Comparison of F1 scores of different data augmentation strategy combinations (m=10).

[0332]

[0333] It can be seen that among all the combination strategies, the combination of random mask and feature truncation performs the best. We believe that this is because the combination can generate more representative and robust features in contrastive learning, thereby improving the performance of the model on different datasets. In particular, when dealing with complex and noisy datasets, the combination shows the potential to significantly enhance the generalization ability of the model. The experiment in this paper finally selected the combination of random mask and feature truncation for data augmentation.

[0334] To further observe how our method learns better relation representations, we randomly selected five types of relations in the FewRel dataset as unknown relations and used t-SNE to reduce the dimension of the unknown relations. The results are shown in Figure 2. Figure 6 , 7 As shown. Figure 6 It can be seen that the data points are more mixed and dense, especially in blue and yellow, indicating that similar relationships exist in these instances, and RE-Matching fails to effectively distinguish these relationships.

[0335] However Figure 7As shown in the figure, the data points of the same color are relatively concentrated, while the distribution between different categories is relatively scattered, which shows that our SDZRE can better understand the differences between instances and distinguish the relationships seen, effectively alleviating the interference caused by similar relationships. To further observe the impact of the contextual semantic distillation method, we split the bidirectional semantic distiller into a fusion projection distiller and a Gaussian adaptive selector while keeping other conditions unchanged, and conducted comparative experiments. The results are shown in Table 4.

[0336] Table 4: Performance of SDZRE under different contextual semantic distillation methods (FewRel, m=15)

[0337]

[0338] It can be seen that although each module can bring certain improvements separately, their synergy is more significant. Analyzing the reasons, we believe that the combination of the fusion projection distiller and the Gaussian adaptive selector can not only achieve complementarity in model performance, but also improve the context perception and information extraction capabilities through architecture optimization, thus achieving the best experimental results.

[0339] To further analyze the efficiency of this technical solution, we compared the inference time and the corresponding F1 score when m = 10 on the FewRel and Wiki-ZSL datasets. Figure 8 , 9 shown.

[0340] It can be seen that SDZRE significantly improves the F1 score on both datasets and achieves faster inference speed. Although SDZRE incurs a certain amount of extra time overhead compared to RE-Matching, SDZRE still shows a more outstanding performance advantage when the inference time is similar.

[0341] We also experimentally observed the impact of hyperparameters on model performance, especially two key marginal parameters. t and gam b It is a key parameter in the marginal ranking loss function, which is used to control the minimum value of the similarity gap between positive and negative samples to ensure that the model can effectively distinguish positive and negative samples, thereby improving the model's discriminative ability and robustness.

[0342] In the experiment, we observed the hyperparameter gam under the condition of fixing the selection process. t and gam b To ensure the comparability of the experiments, the gam b The default value is 0.00, and the right picture is fixed to the gam in the left picture. t The optimal value of Figures 10 to 13It can be seen that the performance curves on the two data sets show similar trends. As the hyperparameter increases from 0 to 0.06, the model performance begins to decline, indicating that the optimal values ​​of the two hyperparameters on different data sets are the same. t and gam b As the size increases, the model matching will not collapse, which shows the good robustness of our method.

[0343] This paper proposes a semantic distillation method SDZRE for the ZeroSRE task. This method uses a bidirectional semantic distiller and a contrastive learning strategy to form a semantic distillation framework. It highlights important semantic relationships by strengthening relevant features and weakening irrelevant features, and amplifies the subtle differences between similar relationships, effectively dealing with complex contextual interference. A fine-grained semantic matching method is adopted, and a multi-negative sample strategy of hard negative samples and semi-hard negative samples is introduced based on similarity during the matching process, which further improves the model's generalization ability for different semantic boundaries. Experimental results show that SDZRE can achieve SOTA results that significantly exceed current methods while maintaining a relatively fast inference speed, achieving a dual improvement in performance and efficiency.

Claims

1. A semantic distillation method for zero-shot relation extraction, characterized by: The following steps are involved: S1: Generate entity and context representation vectors from the input sentence; S2: Generate entity and context representation vectors from the input relation description; S3: Weaken irrelevant features in context representation and strengthen relevant features through a bidirectional semantic distiller; S4: Contrastive learning using enhanced representations of different context representations in the same batch; S5: Fine-grained matching is performed between the entity and context representation of the input sentence and the entity and context representation in the relationship description through the fine-grained semantic matching module.

2. The semantic distillation method for zero-shot relation extraction according to claim 1, characterized in that: Step S1 also includes the following: generating entity and context representation vectors for the input sentence using BERT as a pre-trained encoder; S1-1: Assume the input sentence is Among them, X i The i-th sentence of the input content of this batch, Represents the i-th sentence X i The nth word in ; S1-2: Tokenize input sentence X i The head entity and the tail entity in ; S1-2-1: Special tokens passed through BERT encoder [E h ], [\E h ]、[E t ] and [\E t ]Mark the input sentence X i The head entity, the head entity position, the tail entity and the tail entity position in ; S1-3: Input sentence X through bert encoder i Generate the corresponding hidden state for each word in; The corresponding hidden state is generated for the input sentence through the following formula: in, express The hidden state of , BERT() represents the bert encoder; S1-4: Generate the embedding representation of the head entity and the embedding representation of the tail entity; S1-4-1: According to the hidden state generated in step S1-3, find the h ] Header entity marked by special token The corresponding and generate The corresponding head entity embedding representation; The embedding representation of the head entity is generated by the following formula: in, is the input sentence X i The embedding representation of the head entity in , WeightPooling(·) represents the weight pool; S1-4-2: According to the hidden state generated in step S1-3, find the t ] Header entity marked by special token The corresponding and generate The corresponding head entity embedding representation; The embedding representation of the tail entity is generated by the following formula: in, is the input sentence X i Embedding representation of mid-tail entities; S1-5: Generate contextual embedding representations for head entities and tail entities; According to the hidden state generated in step S1-3, find the h ] and [E t ] Header entity marked by special token The corresponding hidden state And generate the context embedding representation through the following formula: Among them, h i is the contextual embedding representation, Represents the concatenation operator.

3. The semantic distillation method for zero-shot relation extraction according to claim 1, characterized in that: Step S2 also includes the following contents: S2-1: Assume that the input relationship is described as p i ; S2-2: Tag relationship description p i Head entity abbreviations in h , tail entity abbreviation s t And the relationship description des = {w1, K, w n }; S2-3: Generate contextual embedding representation of relation description; S2-3-1: Describe p from the relation through Python i Extract the labeled relationship description des = {w1, K, w n }; S2-3-2: Obtain the context embedding representation through the following formula; d v =Sentence-BERT(des) Among them, d v is the contextual embedding representation of the relation description, and Sentence-BERT(·) is the encoder of sentence-Bert; S2-4: Generate head entity embedding representation of relation description; S2-4-1: Describe p from the relation through Python i Extract the head entity abbreviation s h ; S2-4-2: Get the head entity embedding representation through the following formula: d h =Sentence-BERT(s h ) Among them, d h It is the head entity embedding representation of the relationship description; S2-5: Generate tail entity embedding representation of relation description; S2-5-1: Describe p from the relation through Python i Extract the tail entity abbreviation s t ; S2-5-2: Get the tail entity embedding representation through the following formula: d t =Sentence-BERT(s t ) Among them, d t It is the head entity embedding representation of the relation description.

4. The semantic distillation method for zero-shot relation extraction according to claim 3, characterized in that: Step S3 also Includes the following: S3-1: Identifying relation-irrelevant features by fusing irrelevant selection layers of the projection distiller; S3-2: Reducing irrelevant features in context representation through projection computation layer; S3-2-1: Remove irrelevant features from the context representation to obtain a new context representation; The formula is as follows: in, Indicates the irrelevant features finally obtained The square of the length of is the context representation after the fully connected layer, represents a new context representation; S3-3: Selecting the most relevant features from the contextual representation via Gaussian adaptive attention mechanism.

5. The semantic distillation method for zero-sample relation extraction according to claim 4, characterized in that: Step S3-1 also includes the following: S3-1-1: Through the dot product attention mechanism, the hidden state of the bert encoder output in step S1-3 Perform a preliminary screening to remove irrelevant features; The formula is as follows: A dot =Softmax(Q·K T )·V Among them, Q is the query matrix composed of query vector q, K and V are the key matrix and value matrix respectively, K T is the transposed matrix of the key matrix, A dot is an irrelevant feature obtained by dot product attention; Softmax(g) is a normalized exponential function; S3-1-2: Identify the hidden state of the bert encoder output in step S1-3 through the anti-attention mechanism Features that have not received much attention in The formula is as follows: A inv =(1-Softmax(Q·K T ))·V Among them, A inv They are irrelevant features obtained through anti-attention; S3-1-3: The irrelevant features obtained after vector projection are calculated by the following formula; Among them, A i ' nv is an irrelevant feature obtained by vector projection calculation, |A dot | 2 Yes A dot The square of the length of S3-1-4: The irrelevant features finally obtained are calculated by the following formula: in, represents the irrelevant features finally obtained; S3-1-5: Calculate the irrelevant feature A obtained by dot product attention dot The output of the classifier; The formula is as follows: Cp i =ε·GRL(A dot )+b Among them, Cp i represents the output of the classifier, ε and b represent the classification weight and bias respectively; GRL(g) represents the function of the gradient reversal layer; S3-1-6: Calculate reverse gradient loss; The formula is as follows: Among them, L Rev represents the reverse gradient loss, N represents the number of samples in the batch; y i Represents the actual relationship label.

6. The semantic distillation method for zero-shot relation extraction according to claim 4, characterized in that: Step S3-3 also includes the following contents: S3-3-1: Using dot product attention mechanism to calculate context representation Attention score; The formula is as follows: Where Q represents the query matrix, K represents the key matrix, and K T is the transposed matrix of the key matrix, Represents the normalized attention score; S3-3-2: Determine the context representation after the fully connected layer The relative position relationship between each element in and the center of the sequence is calculated by a learnable offset μ offset Adjust the position weight of the feature in the attention distribution.

7. The semantic distillation method for zero-shot relationship extraction according to claim 6, characterized in that: Step S3-3-2 also includes the following: S3-3-2-1: By inputting sequence k i The mean value μ is determined by Calculate the input sequence k i The mean μ of The formula is as follows: Among them, k i is the context representation after the fully connected layer The input sequence, l represents the input sequence k i length, μ is the input sequence k i The mean value, μ offset is the learnable offset, j represents the input sequence k i The index length of S3-3-2-2: Calculate the distance of each element position relative to the mean μ; The formula is as follows: D i =|l i -m| Among them, l i Represents the input sequence k i The position sequence composed of each element position in D i Indicates k i The distance of each element position relative to the mean μ; S3-3-2-3: Calculate the standard deviation σ; Where σ represents the input sequence k i The standard deviation of l represents the input sequence k i length, μ is the input sequence k i The mean of S3-3-2-4: By standard deviation σ and distance D i To calculate the Gaussian distribution weight; The formula is as follows: Among them, G i represents the Gaussian distribution weight, D i Indicates k i The distance of each element position relative to the mean μ, exp(g) is an exponential function; S3-3-2-5: Calculate the normalized Gaussian distribution weight The formula is as follows: S3-3-2-5: Calculate the final attention weight W; Calculate using the following formula: Among them, W is the final attention weight, is the normalized Gaussian distribution weight, W% is the normalized attention score calculated by the dot product attention mechanism; S3-3-2-6: Calculate the final output representation using the following formula; The formula is as follows: in, is the final output representation, V is the value matrix; S3-3-2-7: Calculate the final context representation using the following formula; The formula is as follows: in, is the final context representation.

8. The semantic distillation method for zero-shot relationship extraction according to claim 3, characterized in that: Step S4 also includes the following contents: Amplify the semantic differences between similar contexts through data augmentation layers; S4-1: Contextual representation after bidirectional distillation through random masking strategy Perform data enhancement processing to generate enhanced representation S4-2: Context representation after bidirectional distillation through feature truncation strategy Perform data enhancement processing to generate enhanced representation S4-3: linearly transform the enhanced representations generated by steps S4-1 and S4-2 through a fully connected layer; S4-4: Convert the linearly transformed enhanced representation to an enhanced representation of consistent length through a normalization layer S4-5: Calculate the InfoNCE loss function L by the following formula Ctr : Among them, τ represents the temperature parameter, m ij represents the similarity matrix between two enhanced representations, and N represents the batch size; is the random mask strategy enhancement representation after linear transformation, is the enhanced representation of the feature truncation strategy after linear transformation, and exp(·) is an exponential function.

9. The semantic distillation method for zero-shot relation extraction according to claim 1, characterized in that: Step S5 also Includes the following: S5-1: Calculate the matching score using the following formula: Among them, α represents the balance parameter, sim(·) represents the cosine similarity calculation function, and f(X i ,p) represents the final matching score; S5-2: Calculate marginal ranking loss.

10. The semantic distillation method for zero-shot relationship extraction according to claim 9, characterized in that: Step S5-2 also includes the following contents: S5-2-1: Select semi-hard negative samples based on similarity And the hard negative sample with the highest similarity S5-2-2: Calculating hard negative samples The marginal ranking loss Calculated by the following formula: Among them, gam t is the boundary parameter of the marginal ranking loss for hard negative samples, is the marginal ranking loss of hard negative samples, f(·) represents the matching score between the input sentence and the relation description; S5-2-3: Calculating semi-hard negative samples The marginal ranking loss Calculated by the following formula: Among them, gam b is the boundary parameter of the marginal ranking loss for semi-hard negative samples, is the marginal ranking loss for semi-hard negative samples; S5-2-4: Calculate the average marginal ranking loss L for all batches Mr ; Where N represents the batch size; S5-2-5: Calculate the total loss L; Calculate using the following formula: L=L Rev +L Ctr +L Mr Among them, L Mr Denotes the average marginal ranking loss L Mr , L Rev represents the reverse gradient loss, L Ctr Indicates InfoNCE loss; S5-2-1-1: Calculate the matching scores of all sentences in this batch of input content; S5-2-1-2: Select the negative sample with the highest matching score through the topk selection strategy, which is a hard negative sample S5-2-1-3: Loop through the remaining negative samples again to find hard negative samples that meet the matching score ratio Low negative samples, semi-hard negative samples

Images