A relation extraction method based on neural kernel method
Through the neural nuclear method combined with neural network, complex semantic information in relation extraction tasks are automatically learned, which solves the limitations of manual design of features and distance functions in the existing technology, and achieves more efficient relationship extraction effect and transfer capabilities.
Patent Information
- Application Number
- CN202411431795.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-14
- Publication Date
- 2024-12-10
- Estimated Expiration
- 2044-10-14
AI Technical Summary
Existing relationship extraction techniques rely on manual design of features and distance functions, are difficult to capture complex semantic information, lack deep understanding, difficult to migrate, and rely on external tools for performance, which is easy to overfit.
The neural nuclear method is adopted, combined with neural network and kernel replacement, and optimized relationship extraction tasks through cluster kernels, instance kernels and description kernels, automatically learn the similarity and semantic dependence of relationship instances, reducing labor costs and migration complexity.
It improves the performance and effect of relationship extraction, effectively captures complex relationship characteristics, reduces the risk of overfitting, reduces dependence on external tools, and improves migration capabilities.
Smart Images
Figure CN118940746B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of natural language processing, and specifically relates to a relationship extraction method based on a neural kernel method. Background Art
[0002] With the rapid development of artificial intelligence and the technological explosion in the field of information extraction, various data (such as video, audio, pictures and text) are constantly increasing, resulting in a large amount of information emerging in electronic digital form. In order to meet the challenge of information explosion, automated tools are urgently needed to extract valuable information from massive data, and information extraction technology has therefore emerged. In the field of natural language processing, information extraction technology is widely used, among which relation extraction is an important component. Named entities in text refer to proper nouns such as names of people, places and organizations, while relation extraction is to extract semantic relationships between entities from text with labeled entity pairs. It is a basic task in natural language processing (NLP) and is widely used to support knowledge graph construction, machine translation, automatic question answering and other tasks. Unlike traditional sentence classification tasks (such as news or sentiment classification), which only assign a label to the entire sentence, the RE task should verify all entity pairs in a sentence. Since they overlap in sentences and have the same contextual features, a qualified RE model should be able to learn discriminative sentence representations related to specific entity pairs. Therefore, the relationship extraction technology currently developed also has important theoretical and practical significance, and it has attracted extensive attention from researchers in this field. These studies can be roughly divided into two frameworks: feature weighting and case matching.
[0003] In feature weighting methods, all features are extracted and then weighted with a function (a classifier) to build a metric space. During training, a decision hyperplane is learned to split the training data to minimize a predefined loss function.
[0004] In the case matching method, all instances are also represented as feature vectors and mapped to a metric space. However, the decision plane of case matching is not to learn the decision hyperplane, but to learn the decision hyperplane based on the neighborhood of each labeled instance in the training data.
[0005] The existing relationship extraction methods have the following technical problems:
[0006] 1. Shallow feature weighting methods and traditional shallow kernel models mainly rely on manually designed rules, feature engineering and distance functions, which cannot effectively capture complex semantic information, make it difficult to learn high-order abstract features and integrate external resources, and cannot fully utilize the advantages of deep learning and automatic learning. Their performance is usually weak. In addition, manually designed distance functions may not be sufficient to capture the similarities between complex relationship instances and are prone to overfitting problems.
[0007] 2. Traditional shallow kernel models are mainly based on classification feature representation, which makes it difficult to capture the contextual features and semantic dependencies of relation instances because these models lack a deep understanding and modeling ability of semantic structures.
[0008] 3. When traditional shallow kernel models are migrated to different fields, rule design is an empirical task with high labor costs and difficulty in migration, because the rules need to be redesigned for different fields, which increases labor costs and the complexity of migration.
[0009] 4. The performance of traditional kernel methods is highly dependent on external parsing toolkits and is susceptible to errors because traditional kernel methods rely on external tools to generate parse trees, and these external tools may have errors, affecting the performance of the model. Summary of the invention
[0010] The purpose of the present invention is to provide a relation extraction method based on a neural kernel method. The technical problem solved by the present invention is how to use the neural kernel method for relation extraction (Relation Extraction), and improve the classification accuracy by combining the neural network and the kernel replacement method. The neural kernel method adopted by the present invention can automatically learn the similarity between relation instances, avoiding the problems that may be caused by manually designing features and distance functions in existing relation extraction methods, and at the same time can make full use of deep learning and external resources to improve the performance and effect of relation extraction tasks.
[0011] The technical solution adopted by the present invention to solve the technical problem is as follows:
[0012] A neural kernel method-based relationship extraction method of the present invention mainly comprises the following steps:
[0013] Step S1: define and formalize the relation extraction task, and establish a mapping from the relation instance set to the relation label set;
[0014] Step S2: Construct a neural kernel method model to assign input data to the same type as the closest annotated reference instance.
[0015] Furthermore, in step S1, a label set T is defined to represent the semantic type of the relationship, and T0 represents a negative relationship type, that is, it means that there is no predefined relationship between the two named entities, thereby establishing a mapping from the relationship instance set to the relationship label set.
[0016] Furthermore, the neural kernel method model includes: cluster kernel K C , instance kernel K I , describe the kernel K D , core replacement and relationship type identification; the cluster core K CGenerate a cluster representation by clustering predefined relationship instances; the instance kernel K I The instance representation is generated by averaging from a predefined support set; the description kernel K D The description representation is generated by a predefined type description cluster; the kernel replacement demonstrates the effectiveness of neural kernel replacement by generating three composite kernels K1, K2 and K3; during the training process, the cluster kernel K C , instance kernel K I , describe the kernel K D is optimized and supervised by objectives related to the relation extraction task; during testing, each input data is input into the neural kernel method model, mapped into the metric space by the composite kernel, and then assigned to the same type as the closest annotated reference instance generated by kernel replacement through relation type recognition.
[0017] Furthermore, the cluster core K C Using clustering on the entire training set to initialize the cluster kernel can be formalized as ,in and Both represent encoders. and Same structure, encoder Used to generate a representation of a relation instance; the instance kernel K I Initialized by a randomly collected support set containing X relation instances, which can be formalized as , encoder Each relation instance is mapped to an abstract representation; the description kernel K D Initialization is performed using a predefined type description, which can be formalized as ,in represents the feature mapping function, which is used to learn the description representation; the composite kernel generated by the kernel replacement is formalized as K1=K I +K D +K C , K2=αK I +βK D +γK C , α, β, γ are all trainable parameters in the neural network, K3=K I ·K D ·K C , · represents the dot product between neural kernels.
[0018] Furthermore, in the neural kernel method model, the encoder and the neural kernel are both composed of randomly initialized neural networks, and the encoder and the neural kernel are used to learn instance representations related to the relationship extraction task and estimate the distance between relationship instances.
[0019] Furthermore, the encoder and the neural kernel are optimized during training by minimizing the distance between the input instance and the reference instance.
[0020] Furthermore, the training goal of the neural kernel is to learn the neural kernel to reduce the distance between relation instances of the same type and to expand the distance between relation instances of different relation types.
[0021] Furthermore, in the neural kernel method model, it is assumed that is a neural kernel (ζ∈{I,D,C,1,2,3}), is an encoder (η∈{I,D,C}); the set of reference instances is denoted as Θ = R I ∪R D ∪R C , R I represents the reference instance set, R D represents the set of description instances, R C Represents the set of relationship instances, which is used to construct the cluster kernel K C The KMeans algorithm is used to obtain the loss function from the training set. Given an input instance R, the loss function of predicting the input instance R is formalized as: ;
[0022] Represents a reference instance, which is used to determine the type of the input instance R; ) represents a kernel, τ represents the temperature factor; the second term on the right side of the formula is to penalize excessive divergence between relationship representations, which is used to improve the generalization ability of the model.
[0023] Furthermore, in the neural kernel method model, a new relation instance R is given, the new relation instance R and all reference instances The distance between ∈Θ is calculated by using a composite kernel (ζ∈{1, 2, 3}); the output is normalized to give a formal probability interpretation as: , A set of semantic types representing relations, Representation encoder; collects truth instances by identifying the index associated with the highest conditional probability ; , is a truth instance The type label of the nearest reference instance, which is the same as the input relation instance R as the identified relation type.
[0024] The beneficial effects of the present invention are:
[0025] The present invention proposes a relation extraction method based on a neural kernel method, which combines the advantages of neural networks and traditional kernel methods to achieve relation extraction tasks. Three neural kernels are introduced into the relation extraction task: cluster kernel, instance kernel and description kernel. Through the optimization of these neural kernels, the performance of deep neural networks is improved. Compared with the prior art, the present invention has the following advantages:
[0026] 1. The present invention uses the relevant objectives of the relationship extraction task to initialize the neural kernel, effectively captures the contextual features and semantic dependencies of the relationship instances and the similarities between complex relationship instances, and makes full use of the advantages of deep learning and automatic learning to learn high-order abstract features and integrate external resources, avoiding the existing shallow feature weighting methods and the problems that may be caused by manually designing features and distance functions in traditional shallow kernel models.
[0027] 2. In machine learning, a kernel is a distance function used to evaluate the similarity between two instances. The traditional kernel method is a memory-based method that assigns new inputs to inputs of the same type as the nearest annotated instance, and constructs a more distinguishing kernel for classification through kernel replacement. A limitation of the traditional kernel method is the use of manually designed distance functions, which rely heavily on prior knowledge and are difficult to apply to deep neural networks composed of a large number of randomly initialized parameters. In addition, the performance of the traditional kernel method is highly dependent on external parsing toolkits and is easily affected by errors that affect the performance of the model. Based on this, the present invention proposes a neural kernel strategy. The utility of the neural kernel is that in the training process, a neural kernel with a target related to the relationship extraction task can be initialized. The neural kernel does not learn a decision hyperplane, but establishes a decision based on the neighborhood of the annotated instance in the training set. The present invention achieves more effective decision boundary learning and improves the performance of the relationship extraction task through the replacement mechanism of the neural kernel.
[0028] 3. The neural kernel method-based relationship extraction method proposed in the present invention is different from the traditional shallow kernel model and does not require redesigning rules for different fields when migrating to different fields, which reduces labor costs and reduces migration complexity.
[0029] 4. The present invention conducted experiments on public datasets and demonstrated that the neural kernel method achieved state-of-the-art performance in the relationship extraction task, proving the effectiveness of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 This is the neural kernel method (NKM) model diagram. DETAILED DESCRIPTION
[0031] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings of this specification.
[0032] like Figure 1 As shown, the present invention provides a relationship extraction method based on a neural kernel method, which specifically includes the following steps:
[0033] Step S1: Define and formalize the relationship extraction task, and establish a mapping from a set of relationship instances to a set of relationship labels.
[0034] A neural kernel is a deep neural network that acts as a binary function to evaluate the distance between two input data. In traditional models, kernels are manually defined distance functions. Since neural kernels contain randomly initialized parameters, in order to use neural kernels, they should be initialized by objectives related to the relation extraction task. Therefore, in step S1, a label set T is defined to represent the semantic type of the relation, where T0 represents a negative relation type, that is, it indicates that there is no predefined relation between two named entities, thereby establishing a mapping from a set of relation instances to a set of relation labels. The specific description is as follows:
[0035] Let S = [t1, t2, ..., t N ] is a sequence from t1 to t N A named entity is a subsequence of sentence S, denoted by E ij = [t i , ..., t j All named entities in sentence S are represented as E = {E ij |1≤i≤j≤N}. Because a relation should be determined in a given context, a triple R= <E ij ,E lm ,S> to represent a relation instance, where sentence S is the context of relation R. Then, the relation instances of all sentences S can be represented as a set R={ <E ij ,E lm ,S>|E ij ≠E lm ,E ij ∈E,E lm ∈E}.
[0036] In the traditional feature weighted relation extraction method, a label set T = {T0, T1, ···, T K} to represent the semantic type of the relationship, where T0 is a negative relationship type, indicating that there is no predefined relationship between the two named entities. The purpose of relationship extraction is to establish a mapping from a set of relationship instances and a set of relationship labels, formalized as: f:R->T. This means that ∀R∈R, there exists T∈T, f(R)=T. The mapping f implements the classification task based on the features related to R.
[0037] Let S ε = {S1, S2, ···} is a training set, and S ρ = {S1, S2, ···} is a test set consisting of independent sentences. ε and S ρ All relation instances can be represented as R ε and R ρ Then, during training, by minimizing R ε The loss function in the loss is used to optimize the classifier . , where △ is represented as a function space. Function f ♯ (R) is regarded as a golden classifier, which always outputs the true relation type of R.
[0038] In the feature weighted model, using the training data R ε Find the function in the feature space Represents the decision boundary. After the training process, the training data is no longer needed and is discarded. ε Or predictions for new inputs are based only on the learned feature weighting parameters.
[0039] The neural kernel is a memory-based method that uses a case matching strategy. It uses the training data R ε Each relation instance is assigned a relation type by the closest match in . The process of finding a relation type for a given input R can be formalized as: . Where R* is a reference instance used to determine the type of input R. K is defined as a distance function used to evaluate and ϕ(•) represents an encoder, called a feature space map, which maps an input instance into an abstract representation. In deep neural networks, ϕ(•) is composed of multiple stacked neural layers (such as CNN, RNN, attention, etc.) to learn high-level abstract features of a sentence.
[0040] Step S2: Construct a neural kernel method (NKM) model to assign input data to the same type as the closest annotated reference instance.
[0041] The constructed neural kernel method (NKM) model mainly consists of five components: cluster kernel K C , instance kernel K I , describe the kernel K D, Kernel Substitution and Relation Types recognition. The designed neural kernels with different architectures (cluster kernel K C , instance kernel K I , describe the kernel K D ) to demonstrate the kernel replacement mechanism for relation extraction.
[0042] Among them, the cluster core K C Generate cluster representation through predefined relationship instance clusters; instance kernel K I Generate instance representations by averaging from a predefined support set; describe the kernel K D Description representations are generated through predefined type description clusters; kernel substitution demonstrates the utility of neural kernel substitution by generating three composite kernels K1, K2, and K3. During training, these kernels are optimized and supervised by the relevant objectives of the relation extraction task; during testing, each input data is input into the neural kernel method (NKM) model, mapped into the metric space through the composite kernel, and then assigned to the same type as the closest annotated reference instance generated by kernel substitution through relation type recognition.
[0043] Cluster core K C Using clustering on the entire training set (Train Set) to initialize the cluster kernel can be formalized as ,in and Both represent encoders. and Same structure, encoder Used to generate representations of relation instances; instance kernel K I Initialized by a randomly collected support set, the support set contains X relation instances, which can be formalized as , encoder Map each relation instance to an abstract representation; describe the kernel K D Initialization is done using predefined type descriptions, which can be formalized as ,in represents the feature mapping function, which is used to learn the description representation; the composite kernel generated by kernel replacement is formalized as K1=K I +K D +K C , K2=αK I +βK D +γK C, α, β, γ are all trainable parameters in the neural network, K3=K I ·K D ·K C , · represents the dot product between neural kernels.
[0044] like Figure 1 As shown in the figure, in the neural kernel method (NKM) model, the specific operation steps of the training objectives and reasoning process are as follows:
[0045] Among them, PLM stands for Pre-trained Language Models, that is, pre-trained language model; MLP stands for multi-layer perceptron;
[0046] (1) Training objective: In the NKM model, both the encoder and the kernel are composed of randomly initialized neural networks, which can learn instance representations related to the relation extraction task and estimate the distance between relation instances. Unlike traditional feature-weighted or neural-based models that only focus on learning instance representations or use predefined kernels, in the NKM model, the encoder and all kernels should be optimized during training by minimizing the distance between the input instance and the reference instance.
[0047] The training goal of the neural kernel is to learn the neural kernel so that the distance between relation instances of the same type can be reduced while the distance between relation instances of different relation types can be expanded. In the present invention, a contrastive learning loss function is introduced to achieve this goal.
[0048] set up is a neural kernel, where ζ∈{I,D,C,1,2,3}, is an encoder (η∈{I,D,C}). The set of reference instances is denoted as Θ = R I ∪R D ∪R C , R I represents the reference instance set, R D represents the set of description instances, R C Represents a set of relationship instances; given an input instance R, the loss function for predicting the input instance R is formalized as: ;
[0049] Represents a reference instance, which is used to determine the type of the input instance R; ) represents a kernel, τ represents the temperature factor, and is used to compare the hyperparameters of the learning process. The second term on the right side of the formula is to penalize the excessive divergence between the relationship representations, which is used to improve the generalization ability of the model.
[0050] (2) Reasoning: Given a new relation instance (New Input Instance) R, the new relation instance R and all reference instances The distance between ∈Θ is calculated by using a composite kernel The output is normalized to give a formal probability interpretation as: ;in, A set of semantic types representing relations, Indicates the encoder.
[0051] Collect truth instances by identifying the indices associated with the highest conditional probabilities . .in, is a truth instance The type label of the nearest reference instance. It is associated with the input relation instance R as the identified relation type.
[0052] The present invention discloses a relation extraction method based on a neural kernel method, which is used to implement a relation extraction task. The neural kernel method maps an input instance to a metric space and determines the type of the input instance by the nearest matching instance. During the training process, the present invention uses a target related to the relation extraction task to initialize the neural kernel, and then constructs a composite kernel by kernel replacement to improve the distinguishing ability of a deep neural network.
[0053] In order to demonstrate the effectiveness of the method proposed in the present invention, real data is used below to verify the effectiveness of a relationship extraction method based on a neural kernel method provided by the present invention.
[0054] The present invention utilizes three widely used public benchmark datasets to evaluate the performance of the present invention: the ACE 2005 English dataset, the SemEval 2010 dataset, and the CoNLL04 dataset. The ACE 2005 English dataset is a popular dataset that contains articles from different types, including news, magazines, interviews, etc. The ACE 2005 English dataset consists of more than 100,000 relationship instances, which are annotated with six positive relationship types and one negative relationship type. The SemEval2010 dataset consists of 10,717 relationship instances, which are formally divided into 8,000 training instances and 2,717 test instances. The CoNLL04 dataset is annotated with 1,283 training instances, 343 validation instances for verification, and 422 test instances.
[0055] The present invention uses SpanBERT as the model encoder of the present invention; Adam is used as the optimizer; and the traditional measurements of precision (P), recall (R) and F1-score (F1) are used to evaluate the performance.
[0056] On the same dataset, the present invention is compared with other models, as shown in Table 1 (the performance of the present invention is indicated in bold). At the same time, in order to show the impact of different neural kernels on the final performance, the present invention designs three relation extraction tasks on the same dataset. The first task is to use a model composed of three neural kernels for relation extraction, and the second task is to use a model with a single neural kernel deleted for relation extraction, which are respectively independently deleted instance kernel, description kernel and cluster kernel, denoted as “w / o K I ”, “w / o K D ” and “w / o K C ”. The third task is to perform relation extraction using the model with two kernels removed to show the performance of a single kernel, e.g., “w / o K C-D "is a model that removes the cluster kernel and the description kernel", which is equal to a kernel consisting of instance kernel K I After verification, the performance is shown in Table 2 (the best F value is indicated in bold).
[0057] Table 1 Comparison with related projects
[0058]
[0059] Table 2 Ablation studies on different datasets
[0060]
[0061] As shown in Table 1, the neural kernel method-based relation extraction method of the present invention achieves state-of-the-art performance on three evaluation datasets. The results show that compared with the traditional shallow kernel model that relies on classification features and manually designed distance functions, the neural kernel method-based relation extraction method of the present invention can automatically learn from the training dataset to calculate the similarity between relation instances, effectively avoid the overfitting problem caused by the manually designed distance function, and can use deep neural networks to learn higher abstract features and integrate external resources.
[0062] As shown in Table 2, removing each kernel from the composite kernel will lead to performance degradation on all evaluation datasets. The results show that each kernel has a positive impact on the final performance. When removing any two kernels from the composite kernel, the performance of all datasets further degrades compared to removing only one kernel. This phenomenon illustrates the effectiveness of kernel replacement in merging heterogeneous resources. In the neural case matching framework, it also demonstrates the advantage of kernel replacement for enhancing the discriminability of deep neural networks.
[0063] In summary, the relationship extraction method based on the neural kernel method provided by the present invention has excellent performance.
[0064] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A relation extraction method based on neural kernel method, It is characterized in that The following steps are involved: Step S1: define and formalize the relation extraction task, and establish a mapping from the relation instance set to the relation label set; Step S2: construct a neural kernel method model to assign the input text data to the same type as the closest annotated reference instance; The neural kernel method model includes: cluster kernel K C , instance kernel K I , describe the kernel K D , core replacement and relationship type identification; the cluster core K C Generate a cluster representation by clustering predefined relationship instances; the instance kernel K I The instance representation is generated by averaging from a predefined support set; the description kernel K D The description representation is generated by a predefined type description cluster; the core replacement is generated by generating three composite cores K 1 , K 2 and K 3 To demonstrate the effectiveness of neural kernel replacement; during training, the cluster kernel K C , instance kernel K I , describe the kernel K D The optimization is performed and supervised by the relevant objectives of the relation extraction task; during the test process, each input data is input into the neural kernel method model, mapped into the metric space by the composite kernel, and then assigned to the same type as the closest annotated reference instance generated by kernel replacement through relation type recognition; The cluster core K C Using clustering on the entire training set to initialize the cluster kernel can be formalized as in and Both represent encoders. and Same structure, encoder Used to generate a representation of a relation instance; the instance kernel K I Initialized by a randomly collected support set containing X relation instances, which can be formalized as Encoder Map each relationship instance to an abstract representation; the description kernel K D Initialization is performed using a predefined type description, which can be formalized as in represents the feature mapping function, which is used to learn the description representation; the composite kernel generated by the kernel replacement is formalized as K 1 =K I +K D +K C , K 2 =αK I +βK D +γK C , α, β, γ are all trainable parameters in the neural network, K 3 =K I ·K D ·K C , · represents the dot product between neural nuclei; In the neural kernel method model, let K ζ is a neural kernel, ζ∈{I,D,C,1,2,3}, is an encoder, η∈{I,D,C}; the reference instance set is denoted as Θ=R I ∪R D ∪R C , R I represents the reference instance set, R D represents the set of description instances, R C Represents a set of relation instances; given a relation instance R, the loss function for predicting the relation instance R is formalized as: R * Represents a reference instance, which is used to determine the type of the relationship instance R; represents a kernel, τ represents the temperature factor; the second term on the right side of the formula is to penalize excessive divergence between relational representations, which is used to improve the generalization ability of the model; In the neural kernel method model, given a new relation instance R, the new relation instance R and all reference instances R * The distance between ∈Θ is calculated by using a composite kernel is calculated; the output is normalized to give a formal probability interpretation as: T represents the set of semantic types of relations, Representation encoder; collects truth instances by identifying the index associated with the highest conditional probability is a truth instance The type label of the nearest reference instance, which is the same as the input relation instance R as the identified relation type.
2. According to claim 1, a relationship extraction method based on a neural kernel method, It is characterized in that In step S1, a label set T is defined to represent the semantic type of the relationship, and T0 represents a negative relationship type, that is, it means that there is no predefined relationship between the two named entities, thereby establishing a mapping from the relationship instance set to the relationship label set.
3. According to the neural kernel method of claim 1, It is characterized in that In the neural kernel method model, the encoder and the neural kernel are both composed of randomly initialized neural networks. The encoder and the neural kernel are used to learn instance representations related to the relationship extraction task and estimate the distance between relationship instances.
4. According to claim 3, a method for extracting relations based on a neural kernel method, It is characterized in that The encoder and kernel are optimized during training by minimizing the distance between input instances and reference instances.
5. According to claim 3, a method for extracting relations based on a neural kernel method, It is characterized in that The training goal of the neural kernel is to learn the neural kernel to reduce the distance between relation instances of the same type and expand the distance between relation instances of different relation types.