Entity relation joint extraction method and system based on deep learning

Abstract

The invention relates to the technical field of natural language processing, in particular to an entity relation joint extraction method and system based on deep learning. The method comprises the following steps: S1, preprocessing an input text; s2, converting the preprocessed word segmentation result into vector representation; s3, enhancing the understanding of a model on the context by adopting a BiLSTM, and introducing a self-attention mechanism to enhance the text feature modeling capability; s4, dynamically constructing a score matrix of a triple for relation prediction; and S5, outputting an identified triple result. The entity relationship joint extraction method and system based on deep learning are used for solving the problem of accurate capture of entity overlapping and semantic dependency relationship in a complex context, can automatically identify triads from structured and unstructured input texts, and output a triad identification result.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a method and system for jointly extracting entity relationships based on deep learning. Background Art

[0002] Knowledge graphs, originating from the development of the Semantic Web and artificial intelligence, were created to address the lack of semantic connections in massive amounts of data. They leverage structured knowledge to enhance machine understanding and support intelligent search, recommendations, and decision-making. Existing knowledge graph construction methods primarily focus on three core components: information extraction, knowledge fusion, and knowledge storage and dynamic updating. By combining traditional rule-based approaches with deep learning technologies, they are gradually forming a system for transforming unstructured and structured data into structured knowledge networks.

[0003] Information extraction technology extracts the "disease-symptom-drug" relationship from medical electronic medical records to assist clinical decision-making, captures the "enterprise-holding-risk event" chain association in the financial field to support risk control analysis, and analyzes e-commerce reviews to build a "product-attribute-user preference" knowledge network to optimize consumer experience; knowledge fusion technology breaks down data silos.

[0004] During the construction of knowledge graphs, information extraction often faces the semantic gap problem caused by the separation of entity recognition and relationship extraction. Traditional pipeline methods (first extracting entities and then determining relationships) are prone to error accumulation and have difficulty handling complex scenarios with nested entities. To avoid these problems, researchers have combined entity recognition and relationship extraction tasks to construct a joint entity-relationship extraction model. Existing deep learning-based joint extraction methods can be divided into two categories based on the modeling object: parameter sharing and joint decoding.

[0005] Parameter sharing-based method: The core idea of the parameter sharing-based method is to share the parameters in the model so that entity recognition and relationship extraction tasks can be trained together in the same model, thereby fully utilizing the correlation between tasks.

[0006] This approach uses a shared feature representation space and a unified network structure (such as LSTM, GRU, Transformer, etc.) to simultaneously perform entity recognition and relation extraction. For example, Gao et al. proposed an entity and relation extraction model based on global entity pairing and relation attention mechanism, which not only effectively solves the overlapping triple problem but also improves the temporal performance of the model.

[0007] Joint decoding-based methods: The joint decoding method models entity recognition and relation extraction tasks independently, but performs joint processing in the decoding stage. This method combines the decoding processes of the two tasks and generates entity labels and relation labels simultaneously through a joint decoding process. This method relies on an additional decoding mechanism to capture the dependencies between entities and relations. For example, Wang et al. proposed a single-stage token pair linking (TPLinker) joint extraction model. This model uses a "handshake annotation" scheme and uses an annotated matrix to extract all entities and their overlapping relations, achieving good results in relation extraction tasks.

[0008] When implementing the above technology, there are the following defects:

[0009] Limited understanding of context: When dealing with long-distance dependencies or complex semantic relationships, existing models may not be able to accurately understand the context, especially when faced with ambiguous or metaphorical expressions, resulting in errors in entity recognition or relationship judgment.

[0010] Problem of multiple overlapping relationships: There are often multiple relationships between entities. How to accurately identify these multiple overlapping relationships is a difficulty in the entity relationship extraction task.

[0011] Lack of effective fusion mechanism: When processing entities and relationships, existing technologies often lack effective information fusion mechanisms and cannot fully utilize the information between entities and relationships.

[0012] To this end, a deep learning-based entity relationship joint extraction method and system are designed to provide a technical solution to the above technical problems. Summary of the Invention

[0013] Based on this, it is necessary to provide a deep learning-based entity relationship joint extraction method and system to address the above technical problems, so as to solve the technical problems raised in the above background technology.

[0014] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0015] A joint entity relationship extraction method based on deep learning, the steps are as follows:

[0016] S1: Preprocess the input text;

[0017] S2: Convert the preprocessed word segmentation results into vector representation;

[0018] S3: By using BiLSTM to enhance the model's understanding of context, the self-attention mechanism is introduced to strengthen the text feature modeling capability;

[0019] S4: Dynamically construct the score matrix of triples for relationship prediction;

[0020] S5: Output the recognized triplet results.

[0021] As a preferred embodiment of the entity relationship joint extraction method based on deep learning provided by the present invention, in step S1, the processing steps are as follows:

[0022] Split the text into sentences according to punctuation or specified rules;

[0023] Based on the standard word segmenter, the [unused1] delimiter is inserted after the subword segmentation result of each word.

[0024] As a preferred embodiment of the entity relationship joint extraction method based on deep learning provided by the present invention, in step S2, the preprocessed word segmentation result is converted into a vector representation, and the steps are as follows:

[0025] Use the pre-trained language model RoBERTa to obtain fine-grained character embeddings, capturing deep semantic and syntactic features;

[0026] RoBERTa enhances the encoding capabilities of the bidirectional Transformer through an improved pre-training strategy;

[0027] For the input token ID sequence X={x1,x2,...,x n}, the output expression of RoBERTa is as follows:

[0028] h i =RoBERTa(x i ),i=1,2,...,n;

[0029] in, It is the word vector representation generated by RoBERTa, d represents the dimension of the embedding vector, and n represents the length of the sequence.

[0030] As a preferred embodiment of the entity relationship joint extraction method based on deep learning provided by the present invention, for the entire input sequence, the output of the RoBERTa model is a tensor, which is expressed as follows:

[0031] H roberta =RoBERTa(x1,x2,...,x n )=(d1,d2,...,d n );

[0032] in,

[0033] Sine and cosine positional encoding is used to provide additional position information for each word. The expression is as follows:

[0034]

[0035] Among them, pos∈[0,n-1] is the position index, i is the dimension index of the position encoding vector, and d is the dimension of the position encoding;

[0036] Each input word vector x is positionally encoded and added to the embedding vector to provide the position information of each word. The expression is as follows:

[0037]

[0038] As a preferred embodiment of the entity relationship joint extraction method based on deep learning provided by the present invention, in step S3, the BiLSTM is used to enhance the model's understanding of context and the self-attention mechanism is introduced to enhance the text feature modeling capability. The steps are as follows:

[0039] Assume that the input of BiLSTM is The forward and backward calculations of BiLSTM are:

[0040]

[0041]

[0042] The bidirectional output is spliced as:

[0043]

[0044] in, Since it is a bidirectional LSTM, the output dimension is twice that of the original RoBERTa.

[0045] By dynamically calculating the association weights between tokens in a sequence, adaptive focusing of key information and suppression of redundant information are achieved;

[0046] Assume that the input sequence is H LSTM , linearly project it into query matrix Q, key matrix K and value matrix V, expressed as follows:

[0047] Q=H LSTM W Q ; K=H LSTM W K ; V=H LSTM W V ;

[0048] in, is the learnable parameter matrix, d kThe vector dimensions representing the key matrix and query matrix, d v Representing the vector dimension of the value matrix, the self-attention process can be expressed as:

[0049]

[0050] Capture the relationship between contexts by transforming queries, keys, and values, and ultimately generate a context representation that contains multiple semantic information.

[0051] As a preferred embodiment of the entity relationship joint extraction method based on deep learning provided by the present invention, in step S4, the steps are as follows:

[0052] Represent all relationships At the same time, calculate the label pair (w i ,w j )of The significance of the final score function is expressed as follows:

[0053]

[0054] Where v is the score vector, drop(·) represents the dropout strategy used to prevent overfitting;

[0055] (w i ,r k ,w j ) is input into the softmax function to predict the corresponding label. The expression is as follows:

[0056]

[0057] The objective function is defined as:

[0058]

[0059] in, represents the correct label obtained from the markup.

[0060] As a preferred implementation of the entity relationship joint extraction method based on deep learning provided by the present invention, in step S5, the recognized triple results are converted into different formats when output.

[0061] A deep learning-based entity relationship joint extraction system, including a preprocessing module, an encoding module, a semantic understanding module, a relationship prediction module and an output module;

[0062] The preprocessing module is used to receive structured or unstructured text data and perform preprocessing;

[0063] The encoding module is used to map the preprocessed text into character vectors, and combines position encoding to enhance the model's ability to capture sequence temporal features;

[0064] The semantic understanding module is used to use the BiLSTM network to capture contextual features in a bidirectional manner to construct context-aware word vectors, introduce a self-attention layer to achieve dynamic allocation of semantic weights within the sequence, and focus on information aggregation of key semantic units;

[0065] The relationship prediction module is used to dynamically construct a feature matrix of the head and tail entity combination, and perform maximum index extraction along the label dimension in the inference phase to output the optimal relationship category prediction by mapping the multi-dimensional score tensor generated by the fully connected layer;

[0066] The output module is used to output the recognition result in a specified format.

[0067] It can be seen without a doubt that the above-mentioned technical solution of this application can definitely solve the technical problem to be solved by this application.

[0068] At the same time, through the above technical solutions, the present invention has at least the following beneficial effects:

[0069] 1. The present invention provides a deep learning-based entity relationship joint extraction method and system, which are used to solve the problem of accurately capturing entity overlap and semantic dependency in complex contexts. It can automatically identify triples from structured and unstructured input texts and output the identified triples.

[0070] 2. This invention strengthens the collaborative judgment capability of entity boundaries and relationship types by constructing a bidirectional semantic perception network and a cross-dimensional attention interaction mechanism, thereby improving the integrity and consistency of relation triple extraction.

[0071] 3. The present invention ensures that the model can perceive the position of each token in the sequence through position encoding, thereby improving the model's sequence processing capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0073] Figure 1 This is a software architecture diagram of the present invention;

[0074] Figure 2 This is the structural diagram of the joint extraction model of the present invention. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0076] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0077] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions therein may be combined with each other.

[0078] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0079] Reference Figure 1-Figure 2 ,A method for joint entity relationship extraction based on deep learning.

[0080] Step 1: Preprocess the received user input text, where the input text can be structured data (such as database records) or unstructured data (such as documents, web pages);

[0081] The processing steps are as follows:

[0082] Split text into sentences based on punctuation or specified rules, filter out irrelevant information such as special characters and emoticons, and standardize the formatting of sentences, specifically processing full-width and half-width characters, uppercase and lowercase conversion, and traditional and simplified Chinese conversion;

[0083] Based on the standard word segmenter, the [unused1] separator is inserted after the subword segmentation result of each word;

[0084] Step 2: Convert the preprocessed word segmentation results into vector representations. The steps are as follows:

[0085] Use the pre-trained language model RoBERTa to obtain fine-grained character embeddings, capturing deep semantic and syntactic features;

[0086] RoBERTa enhances the encoding capability of the bidirectional Transformer through an improved pre-training strategy (removing next sentence prediction, dynamic masking mechanism, and larger-scale training data), and performs context-sensitive encoding on the input token sequence. For the input token ID sequence X = {x1, x2, ..., x n}, the output of RoBERTa can be expressed as:

[0087] hi =RoBERTa(x i ),i=1,2,...,n;

[0088] in, It is the word vector representation generated by RoBERTa, d represents the dimension of the embedding vector, and n represents the length of the sequence.

[0089] For the entire input sequence, the output of the RoBERTa model is a tensor:

[0090] H roberta =RoBERTa(x1,x2,...,x n )=(d1,d2,...,d n );

[0091] in,

[0092] Use sine and cosine positional encoding to provide additional position information for each word:

[0093]

[0094] Among them, pos∈[0,n-1] is the position index, i is the dimension index of the position encoding vector, and d is the dimension of the position encoding.

[0095] Finally, each input word vector x is positionally encoded and added to the embedding vector, providing the position information of each word:

[0096]

[0097] Step 3: Address the issue of entity relationship joint extraction failing to effectively capture contextual dependencies; enhance the model's ability to understand context. The steps are as follows:

[0098] BiLSTM can enhance the model's understanding of context by simultaneously considering the forward and backward information of the sequence.

[0099] Assume that the input of BiLSTM is The forward and backward calculations of BiLSTM are:

[0100]

[0101] The bidirectional output is spliced as:

[0102]

[0103] in, Since it is a bidirectional LSTM, the output dimension is twice that of the original RoBERTa.

[0104] The self-attention mechanism is introduced to enhance the text feature modeling capability. By dynamically calculating the association weights (attention distribution) between tokens in the sequence, the adaptive focus on key information and the suppression of redundant information are achieved. Assume that the input sequence is H LSTM , linearly project it into query matrix Q, key matrix K and value matrix V, then:

[0105] Q=H LSTM W Q ; K=H LSTM W K ; V=H LSTM W V ;

[0106] in, is the learnable parameter matrix, d k The vector dimension of the key matrix (Key) and the query matrix (Query), d v Representing the vector dimension of the value matrix (Value), the self-attention process can be expressed as:

[0107]

[0108] This operation captures the relationship between contexts by transforming queries, keys, and values, and ultimately generates a context representation that contains multiple semantic information.

[0109] Step 4: Dynamically construct a triplet score matrix for relationship prediction; the steps are as follows:

[0110] List all possible triplets and assign high confidence labels through a classifier. A simple classification network is used with the input (e i ,r k ,e j ) to achieve this goal, where r k is a randomly initialized relation representation. The scoring function is defined as:

[0111] f r (h,t)=r T (E pair );

[0112] Among them, h and t represent head and tail respectively, E pair Represents the representation and projection of the spliced head and tail entities, which is used to mine the potential dependency between the two entities:

[0113]

[0114] in, d erepresents the dimension of entity pair representation, b is the trainable weight and bias; [;] is the concatenation operation; is the ReLU activation function.

[0115] The score function of the classifier can be seamlessly connected with the previous output; the mapping function from entity features to entity pairs can be adaptively learned through the matrix W; the connection between two entity pairs is not interchangeable, that is, [H head ;H tail ]≠[H tail ;H head ], which is essential for modeling asymmetric relationships.

[0116] Next, use all the relations to represent At the same time, calculate the label pair (w i ,w j )of where 4 is the number of classification labels. The final score function is defined as:

[0117]

[0118] Here, v is the score vector and drop(·) represents the dropout strategy used to prevent overfitting.

[0119] Finally, (w i ,r k ,w j ) is input into the softmax function to predict the corresponding label:

[0120]

[0121] The objective function is defined as:

[0122]

[0123] in, represents the correct label obtained from the markup.

[0124] Step 5: Output the recognized triples in various formats. The steps are as follows:

[0125] According to requirements, the output results are organized into JSON, XML, CSV and other formats, and a knowledge graph of the triple results is provided for user visual viewing. An API interface is also provided to support integration with other systems.

[0126] Achieved results:

[0127] 1. Fully utilize contextual semantic information to enhance semantic understanding: BiLSTM captures local sequence features (such as part of speech, phrase structure, and grammatical dependencies between adjacent entities) and models context through bidirectional loops. The self-attention mechanism models global semantic connections (such as cross-sentence reference and long-range logical relationships) and focuses on key information through dynamic weight allocation. This model captures richer contextual semantics and improves the ability to understand complex sentences.

[0128] 2. The software is efficient, easy to integrate and deploy: The software device of this solution adopts a modular design, has good scalability and maintainability, supports integration with other systems, and is easy to deploy in actual applications.

[0129] 3. Highly innovative and fills a technological gap: This paper transforms the joint extraction task into a new perspective of fine-grained ternary classification problem, proposes a new solution, is highly innovative, and improves the technical level in the field.

[0130] 4. Wide applicability and strong versatility: It can effectively operate on both structured and unstructured text data, and is suitable for multiple fields such as social media, finance, news, e-commerce, etc., meeting the needs of different scenarios.

[0131] Example 2

[0132] Based on the above embodiment 1, an implementation method is disclosed.

[0133] Use Python language and implement programming language and framework based on PyTorch deep learning framework;

[0134] Use the open source pre-trained language model RoBERTa and perform fine-tuning;

[0135] The training and optimization steps of the pre-trained model are as follows:

[0136] Use Adam optimizer and set appropriate learning rate and weight decay;

[0137] Use the cross entropy loss function as the loss function to measure the difference between the prediction and the actual value;

[0138] Use segmented learning rate and early stopping strategy as training strategy to prevent overfitting and catastrophic forgetting.

[0139] Example 3

[0140] Based on the above-mentioned embodiment 1 and embodiment 2, an application method is disclosed.

[0141] Service deployment: Encapsulate the model as RoBiRLAAPI and deploy it on the server for client calls.

[0142] Optimize performance through model compression, quantization, and other methods to increase model inference speed and reduce resource consumption;

[0143] Provide friendly error prompts when input is abnormal or unrecognizable.

[0144] Example 4

[0145] Based on the above-mentioned embodiments 1, 2 and 3, a deep learning-based entity relationship joint extraction system is disclosed, including a preprocessing module, an encoding module, a semantic understanding module, a relationship prediction module and an output module;

[0146] The preprocessing module is used to receive structured or unstructured text data and perform preprocessing;

[0147] The encoding module is used to map preprocessed text into character vectors and combine positional encoding to enhance the model's ability to capture sequence temporal features;

[0148] The semantic understanding module uses a BiLSTM network to bidirectionally capture contextual features to construct context-aware word vectors. It also introduces a self-attention layer to dynamically allocate semantic weights within a sequence, focusing on information aggregation of key semantic units.

[0149] The relationship prediction module is used to dynamically construct the feature matrix of the head and tail entity combinations, generate a multi-dimensional scoring tensor (entity pair × relationship × label) through fully connected layer mapping, perform maximum index extraction along the label dimension during the inference phase, and output the optimal relationship category prediction;

[0150] The output module is used to output the recognition results in a specified format and supports various application scenarios.

[0151] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for joint entity relationship extraction based on deep learning, characterized in that: Here are the steps: S1: Preprocess the input text; S2: Convert the preprocessed word segmentation results into vector representation; S3: By using BiLSTM to enhance the model's understanding of context, the self-attention mechanism is introduced to strengthen the text feature modeling capability; S4: Dynamically construct the score matrix of triples for relationship prediction; S5: Output the recognized triplet results.

2. The entity relationship joint extraction method based on deep learning according to claim 1 is characterized in that: In step S1, the processing steps are as follows: Split the text into sentences according to punctuation or specified rules; Based on the standard word segmenter, the [unused1] delimiter is inserted after the subword segmentation result of each word.

3. The entity relationship joint extraction method based on deep learning according to claim 1 is characterized in that: In step S2, the preprocessed word segmentation results are converted into vector representations. The steps are as follows: Use the pre-trained language model RoBERTa to obtain fine-grained character embeddings, capturing deep semantic and syntactic features; RoBERTa enhances the encoding capabilities of the bidirectional Transformer through an improved pre-training strategy; For the input token ID sequence X={x1,x2,...,x n }, the output expression of RoBERTa is as follows: h i =RoBERTa(x i ),i=1,2,...,n; in, It is the word vector representation generated by RoBERTa, d represents the dimension of the embedding vector, and n represents the length of the sequence.

4. The entity relationship joint extraction method based on deep learning according to claim 3 is characterized in that: For the entire input sequence, the output of the RoBERTa model is a tensor expressed as follows: H roberta =RoBERTa(x1,x2,...,x n )=(d1,d2,...,d n ); in, Sine and cosine positional encoding is used to provide additional position information for each word. The expression is as follows: Among them, pos∈[0,n-1] is the position index, i is the dimension index of the position encoding vector, and d is the dimension of the position encoding; Each input word vector x is positionally encoded and added to the embedding vector to provide the position information of each word. The expression is as follows:

5. The entity relationship joint extraction method based on deep learning according to claim 1 is characterized in that: In step S3, the BiLSTM model is used to enhance the model's understanding of context and the self-attention mechanism is introduced to strengthen the text feature modeling capability. The steps are as follows: Assume that the input of BiLSTM is The forward and backward calculations of BiLSTM are: The bidirectional output is spliced as: in, Since it is a bidirectional LSTM, the output dimension is twice that of the original RoBERTa. By dynamically calculating the association weights between tokens in a sequence, adaptive focusing of key information and suppression of redundant information are achieved; Assume that the input sequence is H LSTM , linearly project it into query matrix Q, key matrix K and value matrix V, expressed as follows: Q=H LSTM W Q ;K=H LSTM W K ;V=H LSTM W V ; in, is the learnable parameter matrix, d k The vector dimensions representing the key matrix and query matrix, d v Representing the vector dimension of the value matrix, the self-attention process can be expressed as: Capture the relationship between contexts by transforming queries, keys, and values, and ultimately generate a context representation that contains multiple semantic information.

6. The entity relationship joint extraction method based on deep learning according to claim 1 is characterized in that: In step S4, the steps are as follows: Represent all relationships At the same time, calculate the label pair (w i ,w j )of The significance of the final score function is expressed as follows: Where v is the score vector, drop(·) represents the dropout strategy used to prevent overfitting; (w i ,r k ,w j ) is input into the softmax function to predict the corresponding label. The expression is as follows: The objective function is defined as: in, represents the correct label obtained from the markup.

7. The entity relationship joint extraction method based on deep learning according to claim 1 is characterized in that: In step S5, the recognized triples are converted into different formats before being output.

8. A deep learning-based entity relationship joint extraction system, used in the deep learning-based entity relationship joint extraction method according to any one of claims 1 to 7, characterized in that: It includes preprocessing module, encoding module, semantic understanding module, relationship prediction module and output module; The preprocessing module is used to receive structured or unstructured text data and perform preprocessing; The encoding module is used to map the preprocessed text into character vectors, and combines position encoding to enhance the model's ability to capture sequence temporal features; The semantic understanding module is used to use the BiLSTM network to capture contextual features in a bidirectional manner to construct context-aware word vectors, introduce a self-attention layer to achieve dynamic allocation of semantic weights within the sequence, and focus on information aggregation of key semantic units; The relationship prediction module is used to dynamically construct a feature matrix of the head and tail entity combination, and perform maximum index extraction along the label dimension in the inference phase to output the optimal relationship category prediction by mapping the multi-dimensional score tensor generated by the fully connected layer; The output module is used to output the recognition result in a specified format.

Images