Scientific data set entity relationship joint extraction method based on deep learning

Abstract

The invention discloses a scientific data set entity relationship joint extraction method based on deep learning. The method comprises the following steps: firstly, obtaining metadata information of a scientific data set, and carrying out data cleaning to construct a corpus; secondly, corpus texts are coded based on a SciBERT pre-training model, coding features are generated, and a non-autoregression and autoregression mixed decoding strategy is adopted to form a potential relation set; and then performing entity identification by utilizing a global pointer network in combination with the coding features and the potential relationship set, marking head and tail entities and updating. And finally, mapping each sub-relation vector and all head and tail entities in the potential relation set into a matrix space, calculating the association probability of the sub-relation and the entity pair, if the association probability exceeds a threshold value, determining that the sub-relation is an effective triple, and outputting an entity relation triple in the scientific data set. According to the method, the potential relation existing in the sentence can be accurately detected, and accurate feature representation is provided for subsequent tasks such as entity recognition.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a method for jointly extracting entity relationships from scientific datasets based on deep learning. Background Art

[0002] With the continuous deepening of scientific research and the widespread use of data sharing, metadata information for scientific datasets has become an important means for researchers to access and utilize data. This metadata, including the dataset's name, keywords, abstract, subject classification, author, data volume, and publication date, provides researchers with a basic overview of the dataset and user guidance. Automatically extracting valuable relationship information from this metadata, such as the association between datasets and research topics, and authors and research fields, is crucial for improving data utilization efficiency, promoting interdisciplinary research, and constructing scientific knowledge graphs. However, existing technologies face numerous challenges when dealing with the task of extracting entity relationships from scientific text.

[0003] First, the linguistic structure of scientific texts is complex and dense with technical terms, making it difficult for general language models to accurately capture their deep semantics, resulting in inaccurate entity recognition and relationship classification. Second, the metadata information of scientific datasets may contain a variety of potential relationship types, such as the "researcher-dataset" attribution relationship, the "dataset-topic" thematic relationship, and the "dataset-institution" source relationship. Traditional relationship extraction methods often rely on manual feature engineering or fixed pattern matching, which are difficult to adapt to the complexity and diversity of text structures, and their efficiency and accuracy need to be improved. In addition, traditional relationship extraction usually divides entity recognition and relationship classification into two separate steps. This separate processing flow easily leads to the propagation of errors at different stages, ultimately affecting the reliability of the extraction results.

[0004] Therefore, there is an urgent need to develop a more efficient and accurate entity relationship extraction method to meet the needs of entity relationship extraction of scientific dataset metadata. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a method for joint extraction of entity relationships from scientific datasets based on deep learning, comprising the following steps:

[0006] Step S1: Obtain metadata information of scientific datasets from scientific dataset sharing websites, including name, keywords, abstract, subject classification, author, data volume, release date, etc., and then perform data cleaning on the obtained data. The data text after the cleaning operation can be used as a corpus for subsequent steps.

[0007] Step S2: Use the SciBERT pre-trained model to encode the corpus text obtained in step S1. SciBERT is a language model based on the BERT architecture that is pre-trained specifically for scientific text. First, input a sentence S from the corpus in step S1 into the SciBERT pre-trained model, where special [CLS] and [SEP] tags are added to sentence S. [CLS] is at the beginning of the sentence to capture the overall representation of the sentence. [SEP] is used to mark the end of a sentence or to separate two sentences. The SciBERT pre-trained model outputs a vector sequence H(S) = [h1,...h i ,...h n ], where h i Represents the vector corresponding to each token in sentence S, n represents the number of tokens in the sentence, and H(S) contains the context information of each token.

[0008] Step S3: Based on the encoding feature H(S) generated in step S2, a non-autoregressive and autoregressive hybrid decoding strategy is adopted to more accurately detect the potential relationships in the sentence and reduce error propagation.

[0009] Step S3-1: Using the predefined maximum number of potential relations and the encoded features H(S) generated in step S2, a non-autoregressive decoder is used to generate multiple potential relation candidates in parallel. Because the non-autoregressive decoder does not rely on previously generated tags, it can quickly generate multiple candidate relations, thereby improving efficiency. However, due to the lack of context dependence, the quality of the candidates may be unstable.

[0010] Step S3-2: The candidate relations generated by non-autoregression are input into the autoregressive decoder, and refined screening is performed through the masked self-attention layer and the cross-attention layer, and the candidates with confidence higher than the threshold are retained to form the final potential relation set.

[0011] Step S4: Entity recognition is performed using the global pointer network, combining the encoded features H(S) from step S2 and the potential relationship set obtained from step S3. The global pointer network recognizes the head and tail of the entity as a whole, marking the start and end positions of the entity.

[0012] Step S5: To capture the connections between different relation types, a Graph Attention Network (GAT) is introduced based on the latent relation vectors from step S3 and the head and tail entities from step S4. This network models each sub-relation vector to generate relation-aware vectors. By calculating relation-aware vectors and using an attention mechanism, information about adjacent sub-relation nodes is aggregated, and the head and tail entities are updated to capture the potential connections between different relation types, thereby improving the model's expressiveness and generalization capabilities.

[0013] Step S6: Map each sub-relationship and all entities into a matrix space. After step S5, all entities contain information about each sub-relationship to a certain extent. A sigmoid function is used to calculate the association probability between the sub-relationship and the entity pair. If it exceeds the threshold, it is considered a valid triplet. Finally, the entity relationship triplet in the scientific dataset is output.

[0014] The substantial effects of the present invention are:

[0015] 1. This paper encodes the text in the corpus using the SciBERT pre-trained model. This model, pre-trained specifically for scientific text, captures the unique linguistic characteristics and knowledge of scientific literature, learning the terminology, specialized vocabulary, expressions, and syntactic structures commonly used in the scientific field. Compared to other general-purpose language models, it performs better on scientific text, better extracting semantic information from the text and providing more accurate feature representations for subsequent tasks such as relationship detection and entity recognition.

[0016] 2. This invention uses a hybrid non-autoregressive and autoregressive decoding strategy to detect potential relationships within sentences. The non-autoregressive decoder rapidly generates multiple candidate relationships, improving efficiency, while the autoregressive decoder meticulously screens or modifies these candidates to ensure quality. This combination improves relationship detection efficiency while reducing error propagation, enabling more accurate detection of potential relationships within sentences.

[0017] 3. This method uses a global pointer network for entity recognition, jointly predicting the head and tail positions of an entity as a whole. This prioritizes entities related to potential relationships and reduces computational redundancy for irrelevant entities. This method can more accurately identify entities related to potential relationships, improving the efficiency and accuracy of entity recognition while reducing unnecessary computational resource consumption.

[0018] 4. This paper introduces a graph attention network to jointly model sub-relationships and explore the correlations between entities under different sub-relationships. This approach can capture the potential connections between different relationship types, enhance the model's semantic expression capabilities, and better understand the complex relationships between entities in scientific texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Flowchart of the method for joint extraction of entity relationships from scientific datasets based on deep learning of the present invention;

[0020] Figure 2 Comparison chart of experimental results of the present invention. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present application and are not intended to limit the present application. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art. Those skilled in the art will appreciate that the embodiments of the present application may be implemented as a system, device, equipment, method, or computer program product. Therefore, the present application may be specifically implemented in the following forms, namely, complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0022] This method proposes a joint extraction method of entity relationships in scientific datasets based on deep learning. The process is shown in Figure 1 As shown. First, metadata is obtained from the scientific dataset sharing website and cleaned to build a corpus; then, the SciBERT pre-trained model is used to encode the text of the corpus to obtain a deep contextual semantic representation; then, a non-autoregressive and autoregressive hybrid decoding strategy is used to generate a potential relationship set, while ensuring efficiency and taking into account the quality of candidate relationships; then, the potential relationship set and encoding features are input into the entity extraction decoding layer, and the global pointer network is used for entity recognition; at the same time, the graph attention network is used to model the sub-relationships, capture the connections between different relationship types, and update the relationship perception vector of the entity. Finally, the sub-relationships and entities are mapped to the same matrix space, the upper triangular area is retained for optimized storage, and the valid entity relationship triplets that exceed the threshold are screened out by calculating the probability score. Ultimately, the superior entity relationship extraction capability is demonstrated on the scientific dataset. The specific implementation steps of the present invention are as follows:

[0023] Step S1: Obtain metadata for scientific datasets from scientific dataset sharing websites, including name, keywords, abstract, subject classification, author, data size, and publication date. The crawled data is then cleaned, retaining only the text content and removing HTML tags, special characters, punctuation, and stop words. These characters are generally meaningless for natural language processing tasks and need to be removed to reduce text noise. The resulting data text can be used as a corpus for subsequent tasks.

[0024] Step S2: Use the SciBERT pre-trained model to encode the text in the corpus obtained in step S1. SciBERT is a language model based on the BERT architecture that is pre-trained specifically for scientific texts. It aims to capture the unique language characteristics and knowledge in scientific literature. SciBERT has learned the terms, professional vocabulary, expressions, and syntactic structures commonly used in the scientific field, so it performs better when processing scientific texts. First, a sentence S from the corpus of step S1 is input into the SciBERT pre-trained model, where special [CLS] and [SEP] tags are added to sentence S. [CLS] is at the beginning of the sentence in order to capture the overall representation of the sentence. [SEP] is used to mark the end of a sentence, or to separate two sentences. The SciBERT pre-trained model outputs a vector sequence H(S) = [h1,...h i ,...h n ], where h i Represents the vector corresponding to each token in sentence S, n represents the number of tokens in the sentence, and H(S) contains the context information of each token.

[0025] Step S3: After generating the encoding feature H(S) in step S2, a non-autoregressive and autoregressive hybrid decoding strategy is used to more accurately detect the potential relationships in the sentence and reduce error propagation.

[0026] Non-autoregressive decoders generate all target sequence tokens at once, without relying on previously generated tokens. This allows them to quickly generate multiple candidate relations, improving efficiency. However, the lack of contextual dependencies can lead to unstable candidate quality. Autoregressive decoders guarantee candidate quality through sequential generation, but their speed is limited by step-by-step decoding.

[0027] Therefore, when extracting potential relations, we first use a non-autoregressive decoder to generate multiple candidate relation sets in parallel, and then use an autoregressive decoder to fine-tune or correct the candidates.

[0028] Step S3-1: In the non-autoregressive decoder, the potential relation set n k The size of n is a prerequisite for decoding to begin. k is the predefined maximum number of potential relations, which must be greater than the maximum number of actual relations in the sentence. Therefore, the input of the non-autoregressive decoder is composed of n k The relation query embedding is composed of d represents the embedding dimension consistent with the encoder embedding.

[0029] The non-autoregressive decoder consists of L N The network consists of a typical Transformer decoder layer, each of which contains three sub-layers: a multi-head self-attention layer, a multi-head cross-attention layer, and a feed-forward network layer.

[0030] The first layer is a multi-head self-attention layer, which embeds the initialized latent relations into the input multi-head self-attention layer to model the mutual relationship between latent relations. See formula (1), (2):

[0031]

[0032] in W O ∈R d×d is the learnable parameter matrix. The first three are the query, key, and value projection matrices of the i-th attention head. The last one is the linear transformation parameter matrix. d represents the embedding dimension consistent with the encoder embedding, h represents the number of attention heads, and Q r The input of the non-autoregressive decoder is represented by n k Relational query embedding composition.

[0033] The second layer is a multi-head cross-attention layer that integrates the information of the input sentence, where the latent relations are associated with the corresponding trigger words in the sentence. See formulas (3), (4):

[0034]

[0035] In the formula represents the result calculated according to formula (7), and H is the context representation output by step S2.

[0036] The last layer is a fully connected feed-forward network for spatial transformation, see formula (5):

[0037]

[0038] in represents the output of the j-th decoder layer, represents the output of formula (4).

[0039] n k Relational queries are performed by N Non-autoregressive decoder embedding of a typical transformer decoder layer, the output embedding is denoted as H r ∈R nk×d Next, the feed-forward network decodes these embeddings into n k Relation type. Given an embedding The output of , the probability of predicting the relationship type is calculated by formula (6):

[0040]

[0041] in W represents the i-th output relation query embedding. r ∈R |R|×d and b r ∈R|R| is a trainable parameter, and R represents the size of the predefined relationship set.

[0042] In addition, layer normalization and residual connections are used after each sub-layer. See formula (7):

[0043] y=LN(x+sublayer(x)) (7)

[0044] The function sublayer(·) represents a multi-head self-attention layer, a multi-head cross-attention layer, or a fully connected feedforward network layer. LN(·) represents the layer normalization function, where the input sequence of each sublayer is represented by x and the output after layer normalization is represented by y.

[0045] Step S3-2: The candidate relation H generated by the non-autoregressive decoder r Input to the autoregressive decoder, which consists of L N The structure of the network consists of a typical Transformer decoder layer, each layer contains a masked self-attention layer, a cross attention layer, and a feedforward network layer.

[0046] The masked self-attention layer forces the model to focus only on the generated tokens to avoid information leakage. See formula (8):

[0047]

[0048] Q, K, V represent query, key, and value matrices, d k Represents the dimension of each attention head, M represents the mask matrix, the lower triangular part is 0 (allowing attention to the generated part), and the upper triangular part is negative infinity (masking future tokens). represents the scaling factor, and h is the number of attention heads.

[0049] The cross-attention layer injects semantic information from the input sentence into the target sequence generation process. Its calculation process is shown in formulas (3) and (4), which are the same as the non-autoregressive decoder. The calculation process of the feedforward network layer is shown in formula (6), which is the same as the non-autoregressive decoder. Finally, a confidence score is calculated for each candidate, and candidates with a probability higher than the threshold are retained to finally generate a potential relationship set.

[0050] Step S4: After obtaining the potential relationship set, the global pointer network is used to identify entities in combination with the encoding layer features H(S) from step S2. This network jointly predicts the head and tail positions of the entity as a whole, preferentially screening entities related to potential relationships and reducing the computational redundancy of irrelevant entities. The calculation is shown in formulas (9), (10), and (11).

[0051] q i,a =W q,a h i +b q,a(9)

[0052] k i,a =W k,a h i +b k,a (10)

[0053]

[0054] Where W and b represent trainable parameters. i,a and k i,a are the vector representations of the head token and tail token of the entity respectively. i and j represent the initial position and final position of the entity in the sequence respectively. O represents the relative position information of each entity in the sequence, E a ∈R ann Represents the score matrix of the entity, where a represents the number of types, n corresponds to the length of the sequence, and E a (i, j) represents the vector representation corresponding to the position of (i, j) in the matrix, that is, the vector representation of the corresponding entity. Since entity extraction in the entity relationship extraction task classifies entities as head entities or tail entities, we regard entity classification as a binary task a = 2 and classify them as head entities or tail entities.

[0055] Step S5: To capture the potential connections between different relationship types and enhance the semantic expression ability of the model, based on the potential relationships generated in step S3 and the entity vectors extracted in step S4, the sub-relationships are jointly modeled and the correlation of entities under the same sub-relationships is explored. The calculation formula is shown in formula (12).

[0056]

[0057] in represents the relation-aware vector of the jth entity under the i-th sub-relationship, W i Represents the weight matrix corresponding to the i-th sub-relationship, v j Represents the j-th entity vector extracted in step S4.

[0058] After obtaining different sub-relationship vectors with entity information, they are then input into GAT. GAT aggregates the information of adjacent sub-relationship nodes through the attention mechanism and updates the features of the central node. At the same time, a feedforward neural network is added to GAT, see formula (13), where to represent the different sub-relations corresponding to the j-th entity vector, where n represents the number of relations in the sentence.

[0059]

[0060] After different sub-relations pass through GAT, they are mapped to corresponding entities. The formulas are shown in (14) and (15):

[0061]

[0062]

[0063] in represents the vector representation of the kth subject under the i-th sub-relationship, represents the vector of the lth object corresponding to the i-th sub-relation, and Represents the entity features output by GAT, and Represents the trainable weights between the i-th sub-relationship mapped to the subject and the object, and Represents bias.

[0064] Step S6: Generate relation-aware entity vectors in step S5 and After that, each sub-relationship and all entities are mapped to the same matrix space. The construction of this matrix is intended to efficiently identify valid entity pairs in a sentence. Since the matrix naturally has symmetric characteristics, in order to optimize resource utilization and improve computational efficiency, the model only retains its upper triangular area for storage. By reducing redundant data storage, it effectively saves memory space and significantly reduces the time complexity of the algorithm. For each combination of sub-relationship and entity pair in the matrix, the sigmoid activation function is used to perform probability calculation to quantify the degree of association between the two and generate a corresponding score. If the score exceeds the pre-set threshold standard, it is determined that this sub-relationship and entity pair constitute a valid triple. The calculation process is shown in formula (16).

[0065]

[0066] Where P represents the probability of pairing between each sub-relationship and the entities with relation information. If it is greater than the threshold set by the experiment, the subject and object under the sub-relationship are considered to be valid triples.

[0067] Comparative experiment: Collect and organize metadata of scientific datasets used to evaluate model performance. Using the same dataset, compare the evaluation indicators of this model with other models with similar structures to demonstrate the effectiveness of this model.

[0068] The effectiveness of the method is evaluated by precision (P), recall (R), and F1 score (F1).

[0069] Precision measures how many of the samples that the model predicts as positive are actually positive. In other words, precision measures the accuracy of the model's predictions of positive examples. A higher value indicates a higher accuracy of the model in predicting positive examples. The calculation formula is Among them, TP represents true positives, which are the number of samples predicted to be positive and actually positive; FP represents false positives, which are the number of samples predicted to be positive but actually negative.

[0070] Recall measures the model's coverage of all truly positive samples, that is, how many truly positive samples the model can correctly predict. A higher value indicates that the model is more capable of capturing actual positive samples. The calculation formula is Among them, TP represents the true positive samples, which are predicted to be positive and are actually positive, and FN represents the false negative samples, which are predicted to be negative but are actually positive.

[0071] The F1 value is the harmonic mean of precision and recall, which indicates how many of the samples predicted as positive by the model are actually positive. It also takes into account the model's coverage of all samples that are actually positive. Because the F1 value combines precision and recall, it can more comprehensively evaluate the performance of the model for unbalanced datasets or tasks that require high precision and recall. Its calculation formula is Among them, P represents precision and R represents recall.

[0072] In the comparative experiment Figure 2 The model of the present invention is superior to the current common models in terms of precision (P), recall (R), and F1 score (F1).

[0073] Finally, it should be noted that the above embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above embodiments within the technical scope disclosed in the present application, or perform equivalent replacements for some of the technical features thereof. These modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for joint extraction of entity relationships from scientific datasets based on deep learning, characterized by: The following steps are involved: Step S1: Obtain metadata information of scientific datasets from scientific dataset sharing websites, and then perform data cleaning to construct a corpus; Step S2: Encode the corpus text based on the SciBERT pre-trained model to generate encoding features; Step S3: Based on the encoding features, a non-autoregressive and autoregressive hybrid decoding strategy is used to detect the potential relationships in the sentence and form a potential relationship set; Step S4: Utilize the global pointer network to combine the encoding features and the potential relationship set to perform entity recognition, mark the head and tail entities, and update them; Step S5: Map each sub-relationship vector and all head and tail entities in the latent relationship set into a matrix space; calculate the association probability between the sub-relationship and the entity pair through the sigmoid function. If it exceeds the threshold, it is determined to be a valid triple, and the entity relationship triple in the scientific dataset is output.

2. The method for joint extraction of entity relationships from scientific datasets based on deep learning according to claim 1 is characterized in that: The metadata information includes name, keywords, abstract, subject classification, author, data volume and publication date; The data cleaning includes removing HTML tags, various special characters, punctuation marks and stop words.

3. The method for joint extraction of entity relationships from scientific datasets based on deep learning according to claim 2 is characterized in that: The step S2 is specifically implemented as follows: first, a sentence S from a corpus is input to the SciBERT pre-trained model, wherein the sentence S is marked with special [CLS] and [SEP] tags; [CLS] is located at the beginning of the sentence to capture the overall representation of the sentence; [SEP] is used to mark the end of the sentence or to separate two sentences; the SciBERT pre-trained model outputs a vector sequence H(S) = [h1,...h i ,...h n ], where h i Represents the vector corresponding to each token in sentence S, n represents the number of tokens in the sentence, and H(S) contains the context information of each token.

4. The method for joint extraction of entity relationships from scientific datasets based on deep learning according to claim 3 is characterized in that: The specific implementation process of step S3 is as follows: Step S3-1: by pre-defining the maximum number of potential relations and the encoding feature H(S), a non-autoregressive decoder is used to generate multiple potential relation candidates in parallel; In the non-autoregressive decoder, the potential relationship set n k The size of n is a prerequisite for decoding to begin. k is the predefined maximum number of potential relations, which is greater than the maximum number of actual relations in the sentence; the non-autoregressive decoder is composed of L N The network is composed of three sublayers: a multi-head self-attention layer, a multi-head cross-attention layer, and a feed-forward network layer. Step S3-2: The candidate relations generated by non-autoregressive methods are input into the autoregressive decoder, and are screened through the masked self-attention layer and the cross-attention layer. The candidates with confidence levels higher than the threshold are retained to form the final set of potential relations.

5. The method for joint extraction of entity relationships from scientific datasets based on deep learning according to claim 4 is characterized in that: The global pointer network recognizes the head and tail of the entity as a whole and marks the start and end positions of the entity.

6. The method for joint extraction of entity relationships from scientific datasets based on deep learning according to claim 5 is characterized in that: The update described in step S4 is specifically implemented as follows: based on the potential relationship set and the head and tail entities, the graph attention network GAT is introduced to model different sub-relationship vectors in the potential relationship set respectively to obtain relationship perception vectors. Through relationship perception vector calculation and attention mechanism, the information of adjacent sub-relationship nodes is aggregated, the head and tail entities are updated, and the potential connections between different relationship types are captured.

7. The method for joint extraction of entity relationships from scientific datasets based on deep learning according to claim 6, characterized in that: The modeling of different sub-relationship vectors in the potential relationship set is specifically as follows: in represents the relation-aware vector of the jth entity under the i-th sub-relationship, W i Represents the weight matrix corresponding to the i-th sub-relationship, v j Represents the j-th entity vector.

Images