Prompt-based entity-level autoregressive relation triple extraction model training method, extraction method and device

Abstract

This invention provides a prompt-based entity-level autoregressive relation triple extraction model training method, extraction method, and apparatus. For complex scenarios such as single entity overlap, entity pair overlap, and segmented entities, it employs an extended BIO labeling mechanism to construct a training sample set, adding labels to multiple entity segments in a sentence according to different entity types to distinguish between the initial token, internal tokens, and non-entity tokens. An initial neural network model is constructed using a BERT encoder and classifier. In single-step autoregressive extraction, the model is input with the sentence, relation type, and extracted entity pairs. For different relation types, it predicts entity pairs and their relationships one by one through multiple rounds of autoregression. During training, label smoothing technology is used, combined with a cross-entropy loss function to update parameters and optimize extraction performance. This method can efficiently label segmented entities and predict entities based on multiple rounds of autoregression, effectively solving the problem of relation triple extraction in complex scenarios involving segmented entities.

Description

Technical Field

[0001] This invention relates to the field of relation triple extraction technology, and in particular to a prompt-based entity-level autoregressive relation triple extraction model training method, extraction method, and apparatus. Background Technology

[0002] With the continuous expansion of communication networks, network operation and maintenance (O&M) faces increasingly complex challenges. Knowledge graph technology is considered one of the key means to achieve intelligent network O&M. Relation triple extraction, as a core technology for constructing knowledge graphs, primarily extracts entities and their relationships from natural language text to form structured triple data. However, existing research mostly focuses on general domains, with relatively little research specifically targeting the network O&M field. Currently, a special phenomenon exists in network O&M corpora: the segmented entity problem, which is rare in general domains. Segmented entities refer to a complete entity composed of multiple discontinuous segments in the text. This phenomenon is particularly evident in specialized network O&M corpora, posing new challenges to relation triple extraction technology. Most existing relation triple extraction methods cannot effectively handle the segmented entity problem, limiting their application in specialized fields. Therefore, how to solve the segmented entity problem and improve the accuracy and completeness of relation triple extraction has become a pressing technical challenge in the current network O&M field. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a prompt-based entity-level autoregressive relation triple extraction model training method, extraction method, and apparatus to eliminate or improve one or more defects in the prior art and solve the problem of segmented entity recognition composed of discontinuous multi-segment sequences in relation triple extraction tasks.

[0004] One aspect of the present invention provides a prompt-based entity-level autoregressive relation triple extraction model training method, the method comprising the following steps:

[0005] Multiple samples are obtained, each sample being a sentence originating from the target topic environment, and each sentence is represented as a token sequence. Each sample contains multiple relation triples, each relation triple containing a subject, an object, and the relationship between them. The subject and the object, as entities, are composed of one or more entity segments, and each entity segment is a subsequence in the sentence containing one or more consecutive tokens. Within a sample, there are three types of overlaps between or within relation triples: single entity overlap, entity pair overlap, and segmented entities. Single entity overlap indicates that two relation triples share one entity; entity pair overlap indicates that two relation triples share the same entity but have different relationships; and segmented entities indicate that a relation triple contains an entity composed of multiple non-contiguous and non-overlapping entity segments.

[0006] A training sample set is constructed by adding a label to each token in the sentence of each sample. The label distinguishes between the entity segment start token, the entity segment internal token and other non-entity tokens.

[0007] An initial neural network model, including a BERT encoder and a classifier, is obtained. During a single-step extraction process, the initial neural network model takes the sentence, the relation type added to the end of the sentence, and the extracted entity pairs as input and outputs the next entity pair. The initial neural network model performs multiple rounds of autoregression to extract entity pairs one by one according to multiple relation types. Based on the prediction results and the labels, a cross-entropy loss function is constructed after label smoothing. The parameters of the initial neural network model are updated using the training sample set to obtain the target relation triplet extraction model.

[0008] In some embodiments, a training sample set is constructed by adding a label to each token in each sentence of each sample. The label distinguishes between entity segment start tokens, entity segment internal tokens, and other non-entity tokens, including:

[0009] The entity segment start token comprises three parts: the first part indicates that the current token belongs to the entity segment start token; the second part indicates the entity type, which includes subject and object; and the third part indicates the segment ordinal number of the entity segment represented by the current token within the corresponding entity.

[0010] The token within the entity segment consists of two parts: the first part indicates that the current token belongs to the entity segment; the second part indicates the entity type, which includes the subject and the object.

[0011] The other non-entity tokens include a part, the first part of which marks that the current token belongs to other non-entity tokens;

[0012] The first token of the entity segment is marked as the starting token of the entity segment, and all subsequent tokens in the entity segment starting from the second token are marked as internal tokens of the entity segment. All tokens in the entity segment have the same entity type as their corresponding tags.

[0013] In some embodiments, the classifier includes consecutive fully connected layers and a Softmax layer; the BERT encoder is pre-trained using textual data of the target topic environment.

[0014] In some embodiments, the method further includes: pruning the attention head and layers of the BERT encoder; and introducing a local attention mechanism or a cross-segment attention mechanism into the BERT encoder.

[0015] In some embodiments, the BERT encoder employs DistilBERT / TinyBERT.

[0016] In some embodiments, a cross-entropy loss function is constructed based on the prediction results and the labels, on the basis of label smoothing, and the expression for the loss is:

[0017]

[0018] Among them, y LS This represents the label after smoothing, where j represents the vector y. LS The index of each class in the table, j0 represents the index of the actual class, and α is a hyperparameter representing the degree of smoothness.

[0019] In some embodiments, the method further includes: dividing the training sample set into a training set, a validation set, and a test set; performing model training using the training set; evaluating and optimizing the target relation triplet extraction model using the validation set; calculating the standard accuracy, recall, and F1 score of the trained and optimized target relation triplet extraction model using the test set; and generating a performance evaluation report.

[0020] On the other hand, the present invention also provides a prompt-based entity-level autoregressive relation triple extraction method, the method comprising the following steps:

[0021] Obtain sentences to be recognized that are relevant to the target topic environment;

[0022] After tokenizing the sentence to be identified, input it into the target relation triplet extraction model in the above prompt-based entity-level autoregressive relation triplet extraction model training method, and output the entity recognition result.

[0023] On the other hand, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0024] On the other hand, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0025] The beneficial effects of the present invention are at least as follows:

[0026] The present invention describes a prompt-based entity-level autoregressive relation triplet extraction model training method, extraction method, and apparatus. In complex relation scenarios involving single entity overlap, entity pair overlap, and segmented entities, it utilizes an extended BIO labeling mechanism to add multiple labels to multiple entity segments of sentences in each sample according to various entity types, constructing a training sample set. The labels distinguish the starting token, internal tokens, and other non-entity tokens of sentences in each sample. An initial neural network model is constructed using a BERT encoder and classifier. During single-step autoregressive extraction, the model takes the sentence, relation type, and extracted entity pairs as input. For different relation types, it predicts entity pairs and their relationships one by one through multiple rounds of autoregression. Label smoothing technology is employed during training, combined with cross-entropy loss function to update parameters and optimize the model's extraction performance. This invention, by leveraging multi-segmentation, multi-relationship categories, and distinguishing intra-attribute methods, can efficiently label segmented entities. During model learning, multi-round autoregression is used to predict entities, effectively solving the problem of relation triplet extraction in complex scenarios with segmented entities.

[0027] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0028] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0029] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0030] Figure 1 This is a schematic diagram of the multi-level autoregressive extraction process of the model in the prompt-based entity-level autoregressive relation triplet extraction model training method according to an embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram of the single-step extraction process of the model in the prompt-based entity-level autoregressive relation triplet extraction model training method according to an embodiment of the present invention.

[0032] Figure 3 Example diagram of triples where ROUGE-1 < 1 but are judged as correct.

[0033] Figure 4ROUGE-1 score distribution for multiple incompletely matched predicted triples.

[0034] Figure 5(a) shows the F1 scores of PEAR and other LLM-based baselines at different numbers of triples.

[0035] Figure 5(b) shows the F1 scores of PEAR and other LLM-based baselines for different numbers of entity lingers.

[0036] Figure 5(c) shows the recall rates of PEAR and other LLM-based baselines at different entity score ranges. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0038] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0039] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0040] Among numerous enabling technologies, knowledge graphs, with their unique advantages, have become a key strategy for achieving intelligent network operations and maintenance, attracting widespread attention from the academic community. It is important to emphasize that the quality of the knowledge graph directly determines its actual effectiveness in downstream network operations and maintenance tasks. Currently, most knowledge graphs used in network operations and maintenance research are primarily derived from structured data, which to some extent limits the diversity of data sources and lacks unstructured data such as reasoning knowledge. To achieve higher levels of intelligence, mining high-quality knowledge from unstructured text is particularly crucial.

[0041] Relation triple extraction (RTE) technology aims to simultaneously identify entity pairs and their relationships from unstructured text, and is one of the core technologies for constructing knowledge graphs. The triples extracted by this technology are presented in the format of "<subject, relation, object>", forming the basic unit of a knowledge graph. With the deepening research on relation triple extraction, its development can be roughly divided into two camps: non-LLM (Large Language Model) methods and LLM-based methods.

[0042] Non-LLM methods typically encode text data using an encoder, then employ sequence labeling, table completion, or decoder generation to extract information related to the triples. With the rapid development of LLMs, LLM-based relation triple extraction methods generally add cue information before and after the text, inputting it into an LLM library, which then outputs formatted triples. Both methods have their advantages and limitations. In general-domain corpora, state-of-the-art non-LLM methods often outperform LLM-based methods. However, LLM-based methods, with their massive pre-trained knowledge reserves and flexible output content, demonstrate greater potential in handling complex and unique scenarios.

[0043] Despite the many exciting achievements in relation triple extraction, unfortunately, most of these have focused on extracting triples from traditional complex scenarios, such as single entity overlap (SEO) and entity pair overlap (EPO) scenarios.

[0044] However, a situation easily overlooked exists in Chinese network operations and maintenance corpora: a complete entity may consist of multiple discontinuous fragments in the text. For example, in the sentence "Methods for controlling cell coverage include adjusting the MS received signal level and the RACH access level," the entity "adjusting the RACH access level" requires combining two discontinuous text fragments, "adjust" and "RACH access level." In this application, this scenario is referred to as a "segmented entity." The reason segmented entities are overlooked is mainly because most research focuses on general domains, where entities can usually be represented by concise phrases such as "person's name" or "location." Furthermore, widely used datasets, such as NYT and WebNLG, rarely involve segmented entities.

[0045] In contrast, corpora in specialized fields such as network operations and maintenance (LOM) often contain longer and more complex entities, making the phenomenon of segmented entities more pronounced. Network operations and maintenance refers to a series of management activities conducted to ensure the safe and efficient operation of telecommunications network services, with core tasks encompassing equipment management, network monitoring, and fault analysis. In recent years, the rapid expansion of communication network scale has brought numerous challenges to network operations and maintenance, primarily manifested in resource-intensive repetitive tasks, a high dependence on specialized knowledge, and fragmented knowledge acquisition. Currently, most existing non-LLM methods lack effective mechanisms for handling segmented entities. Although LLM-based methods have recently shown the potential to address this problem by generating complete entity texts, continued use of early datasets limits researchers' full understanding and prioritization of this issue.

[0046] In view of this, the present invention provides a prompt-based entity-level autoregressive relation triple extraction model training method, which includes the following steps S101 to S103:

[0047] Step S101: Obtain multiple samples, each sample being a sentence originating from the target topic environment, and each sentence is represented as a token sequence; each sample contains multiple relation triples, each relation triple containing a subject, an object, and the relationship between them. The subject and object, as entities, are composed of one or more entity segments, and an entity segment is a subsequence in the sentence containing one or more consecutive tokens; within a sample, there are three cases: single entity overlap, entity pair overlap, and segmented entity; single entity overlap means that there is one entity that is the same between two relation triples, entity pair overlap means that there are entities that are the same but have different relationships between two relation triples, and segmented entity means that there is an entity composed of multiple discontinuous and non-overlapping entity segments within a relation triple.

[0048] Step S102: Add labels to each token in the sentence of each sample to build a training sample set. The labels distinguish between the entity segment start token, the entity segment internal token and other non-entity tokens.

[0049] Step S103: Obtain the initial neural network model, including the BERT encoder and classifier; during the single-step extraction process, the initial neural network model takes the sentence, the relation type added to the end of the sentence, and the extracted entity pairs as input and outputs the next entity pair. The initial neural network model performs multiple rounds of autoregression to extract the prediction results of entity pairs one by one according to multiple relation types; based on the prediction results and labels, a cross-entropy loss function is constructed on the basis of label smoothing, and the parameters of the initial neural network model are updated using the training sample set to obtain the target relation triplet extraction model.

[0050] In step S101, the samples to be acquired can be targeted at a specific language environment, so a target topic environment can be set to acquire the corresponding samples. For example, the samples can be collected from a network operation and maintenance environment. When configuring the samples, various relation triplet configurations are arranged to ensure learning quality. In this embodiment, the relation triplet types within the samples include at least three types: single entity overlap, entity pair overlap, and / or segmented entities.

[0051] The goal of the relation triple extraction task is to extract from a given sentence S = {w1, w2, ..., w3} with L tokens. L Extract all possible triples from the list and combine them into a single array. The format is as follows. Where r... i Indicates the relation type selected from a predefined set, s i and o iRepresenting the subject and object respectively, both are composed of w j The sequence ∈S is constructed, and the two have an r-meaning relationship. i The relationship.

[0052] The complex scenarios of relation triple extraction tasks mainly include the following:

[0053] Single entity overlap (SE0) means that there are two triples (s1,r1,o1) and (s2,r2,o2), where s1 or o1 is equal to s2 or o2.

[0054] Entity pairs overlap (EP0), which means there exist two triples (s1, r1, o1) and (s2, r2, o2), where the set {s1, o1} equals {s2, o2}, but r1 does not equal r2.

[0055] Segmented entity, that is, there exists a usable The entity represents the entity where n1>0, n2>0, j≠i+n1+1, and the intervals [i,i+n1] and [j,j+n2] have no overlap.

[0056] In step S102, labels are added to the samples to prepare for subsequent model training. In some embodiments, a training sample set is constructed by adding labels to each token in the sentence of each sample. The labels distinguish between entity segment start tokens, entity segment internal tokens, and other non-entity tokens, including:

[0057] The entity segment start token consists of three parts: the first part indicates that the current token belongs to the entity segment start token; the second part indicates the entity type, which includes subject and object; and the third part indicates the segment ordinal number of the entity segment represented by the current token within the corresponding entity.

[0058] The token within an entity segment consists of two parts: the first part indicates that the current token belongs to the entity segment; the second part indicates the entity type, which includes the subject and the object.

[0059] Other non-entity tokens include a part, the first part of which indicates that the current token belongs to other non-entity tokens;

[0060] In this entity segment, the first token is marked as the entity segment start token, and all subsequent tokens in the entity segment starting from the second token are marked as entity segment internal tokens. All tokens in the entity segment have the same entity type as their corresponding tags.

[0061] To address the issue of segmented entities, this application proposes an improved tagging mechanism—Segmented BIO. In this mechanism, each entity type is assigned at most (m+1) tags—"B1, B2, ..., I," where each B tag represents the first token of each entity segment. If the actual number of segments n is less than m, only "B1, B2, ..., Bn, I" will be used. During the conversion from tags to entity text, entity segments are concatenated in ascending order based on the subscript number of their "B" tags. Finally, based on the above description, the tag set for the Segmented BIO tagging mechanism is given:

[0062] T set ={SUBJ-B1,SUBJ-B2,…,SUBJ-B m SUBJ-I

[0063] OBJ-B1, OBJ-B2, ..., OBJ-B m ,OBJ-I,O}

[0064] Among them, T set The length is equal to d t .

[0065] This invention adds tags in a segmented, multi-tag format, distinguishing tokens based on in-sentence attributes.

[0066] Introducing multiple segment start markers (such as SUBJ-B1 and SUBJ-B2) clearly identifies the starting positions of different segments, providing richer structural information. This allows the model to directly identify discontinuous entity segments through labels and combine them into complete entities. In scenarios containing segmented entities (such as network operations and maintenance), the model recall rate is significantly improved; experiments show that segmented BIO contributes 7.1% to the F1 score improvement. When irrelevant content is interspersed between entity segments (such as "adjusting MS received signal level and RACH access level"), segmented BIO can clearly distinguish which tokens belong to different segments of the same entity, avoiding misjudgments. Adapting to diverse scenarios, it can flexibly support different numbers of segments (such as 2-segment and 3-segment entities), covering more complex entity structures through label design. By clearly marking entity segment boundaries, the model can still accurately extract target entities in noisy text (such as redundant descriptions and nested entities), improving its anti-interference capability.

[0067] In step S103, in order to achieve the goal of relation triple extraction and overcome the complexities of single entity overlap, entity pair overlap and / or segmented entities, this invention provides a prompt-based entity-level autoregressive relation triple extraction method, hereinafter referred to as PEAR.

[0068] Figure 1 and Figure 2 The framework of PEAR is shown, in which Figure 1This describes the extraction process of PEAR, which involves iteratively extracting entity pairs from a given sentence. Figure 2 This section explains the single-step extraction process in PEAR and the structure of the PEAR model. The single-step extraction process is designed as a sequence labeling task. Specifically, it involves mapping a sentence with prompts to a sequence and inputting it into the model, then having the model assign labels to each token in the original sentence. Figure 2 The output label sequence in the example is a typical example of segmented BIO. This mechanism can accurately label segmented entities, enabling the model to extract these entities. Furthermore, a label smoothing mechanism is introduced during the training phase to prevent overfitting.

[0069] The framework of PEAR can be represented by the following formula:

[0070] (s k+1 ,o k+1 ) = Model(S,r,(s k o k ))

[0071] k = 0, 1, 2, ..., N r

[0072]

[0073] Here, "Model" represents the single-step extraction process in PEAR. "Model" combines "sentence S", "relation type r", and "the k-th entity pair (s) associated with r". k ,o k (s) is taken as input, and then output (s) is output. k+1 ,o k+1 This extraction process will be executed iteratively, forming an "entity-level autoregressive" structure. Initially, (s0, o0) is empty, indicating the extraction of the first entity pair; when another empty entity pair is extracted... The process terminates at that point. It is important to note that upon obtaining... Previously, the total number of entity pairs N r It is unknown. Furthermore, a single-round autoregressive process can only extract entity pairs belonging to a specific relation type. To comprehensively extract all relation triples, PEAR needs to access all relation types (assuming a total of M) and execute M different autoregressive processes. Input information is fused using a prompt-based approach; specifically, two text descriptions are appended to the end of the original sentence, representing "relation type" and "previously extracted entity pairs," respectively, allowing the sentence and prompts to be input into the model in a unified format.

[0074] This invention uses BERT to encode input tokens, generating a sequence of feature vectors. The BERT encoder can be pre-trained using textual data within the target topic environment. First, the sentence S and its prompt S... p ={w L+1 ,w L+2 ,…,w L′ The tokens are then tokenized, and these tokens are mapped to a low-dimensional space to generate a word embedding sequence X. This sequence is then input into BERT, which outputs a feature vector sequence H. The process in BERT can be simply described by the following formula:

[0075] H = BERT(X; Θ1)

[0076] X = {x1, x2, ..., x} L′}

[0077] H = {h1, h2, ..., h} L′}

[0078] Where X and H are both L′×d. L′ represents the number of tokens in the sentence after adding prompts, L′>L. d represents the dimension of the BERT hidden layer. The symbol Θ1 represents all trainable parameters within BERT. In some embodiments, the attention head and layers of the BERT encoder can also be pruned; local attention mechanisms or cross-segment attention mechanisms can be introduced into the BERT encoder. To achieve lightweight design, the BERT encoder can also employ DistilBERT / TinyBERT.

[0079] Subsequently, a fully connected layer and Softmax are used to obtain sequence T, which represents the probability of the label, and can be described by the following formula:

[0080] T = {t1, t2, ..., t} L}

[0081] t i =Softmax((h i ·W+b)

[0082] in, and This represents the trainable parameters of the fully connected layer. Vector t i The probability corresponding to each label is d. t It should be noted that the range of index i is 1 to L, corresponding to the original sentence portion.

[0083] During training, the cross-entropy loss function is used to calculate the loss between the predicted and ground truth labels. Furthermore, a label smoothing mechanism is introduced to prevent overfitting, especially when training samples are limited. This aims to improve the robustness of PEAR.

[0084] In the formula for cross-entropy loss, the label y is typically represented as a one-hot vector. Label smoothing is a regularization technique that prevents the model from becoming overconfident in its predictions during training by decreasing the values ​​of 1 in y and increasing the values ​​of 0. The smoothed label y LS It can be represented as:

[0085]

[0086]

[0087] Where j represents vector y LS The index of each class in the table, j0 represents the index of the actual class, and α is a hyperparameter representing the degree of smoothness.

[0088] The loss function after label smoothing can be expressed as:

[0089]

[0090] In some embodiments, the method further includes: dividing the training sample set into a training set, a validation set, and a test set; performing model training using the training set; evaluating and optimizing the target relation triplet extraction model using the validation set; calculating the standard accuracy, recall, and F1 score of the trained and optimized target relation triplet extraction model using the test set; and generating a performance evaluation report.

[0091] On the other hand, the present invention also provides a prompt-based entity-level autoregressive relation triple extraction method, the method comprising the following steps S201 to S202:

[0092] Step S201: Obtain the sentences to be identified that are related to the target topic environment.

[0093] Step S202: Tokenize the sentence to be recognized and input it into the target relation triplet extraction model in the above prompt-based entity-level autoregressive relation triplet extraction model training method, and output the entity recognition result.

[0094] On the other hand, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0095] On the other hand, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0096] The effects of the present invention will be described below with reference to specific embodiments:

[0097] 1. Dataset Construction Process

[0098] In this embodiment, the experiment was conducted on the constructed CMIM23-NOM1-RA (China Mobile Innovator Marathon2023-Network Operation and Maintenance, task 1-Re Annotation) dataset. The original text of this dataset comes from Task 1 of the "2023 China Mobile Maker Marathon - Knowledge Graph-Based Network Operation and Maintenance Q&A" competition organized by China Mobile in 2023, and is accompanied by re-annotation based on the structure of this invention. Task 1 of this competition aims to extract relation triples from the Chinese network operation and maintenance corpus to construct a domain knowledge graph to complete subsequent tasks. The original dataset of Task 1 can be called "CMIM23-NOM1", which contains a training set and a test set. The training set contains 4354 labeled samples, plus 21206 unlabeled samples, which participants can freely use for various purposes. On the other hand, the test set includes 981 unlabeled samples to evaluate the effectiveness of the methods proposed by the participants.

[0099] However, the annotation of CMIM23-NOM1 follows the format of OIE (Open Information Extraction Task). OIE must extract all structured information, including relation types, from the corpus, rather than predefining a set of relation types as in restricted domain relation triple extraction. Therefore, the labeled training data in CMIM23-NOM1 contains a large number of different relation types, totaling 547. OIE possesses rich relation diversity, can discover new relation types, and exhibits good transferability. Undeniably, the establishment of CMIM23-NOM1 can facilitate the effective transfer of open-domain models from general domains to network operations and maintenance. However, open-domain research is inherently plagued by challenges such as annotation difficulties, limited accuracy, and the need for large amounts of labeled data. The mere existence of over 4354 annotated samples is insufficient to meet the demand for large amounts of annotation in open-domain research, thus the constructed knowledge graph may lack accuracy.

[0100] Therefore, this embodiment constructs a domain-defined relation triplet extraction dataset. A five-member annotation team from information and communication engineering or related disciplines was formed. They then re-annotated 4354 sentences already annotated in the CMIM23-NOM1 training set and 981 sentences in the test set. Each sample was independently annotated by two members, and then verified and confirmed by a third member. During this process, the relation types were adjusted multiple times, and some abnormal samples were handled. Ultimately, 13 different relation types commonly found in the network operations corpus were identified, resulting in 5337 re-annotated samples containing 19509 triples. Finally, these samples were divided into three datasets: 3337 for training, 1000 for validation, and 1000 for testing, collectively known as the "CMIM23-NOM1-RA" dataset.

[0101] The CMIM23-NOM1-RA dataset possesses several unique characteristics: the relation types exhibit a distinct domain-specific nature. Relations in the corpus are categorized into 13 types, effectively covering most common logical relations found in web operations corpora. The dataset is information-dense, with an average of 3.7 triples per sample. Approximately 17% of sentences contain more than 5 triples, with the most extreme case containing 50 triples. Notably, approximately 18% of the triples involve segmented entities, with the number of segments primarily ranging from 2 to 4. These features highlight its complexity and pose significant challenges to relation triple extraction methods. Finally, the specific relation types and additional features of the CMIM23-NOM1-RA dataset are detailed in Table 1.

[0102] Table 1. Relationship types and additional features of the CMIM23-NOM1-RA dataset.

[0103]

[0104] 2. Experimental Setup

[0105] Six non-LLM methods and three LLM-based methods were selected as baselines. The implementation of these baselines and the evaluation metrics used in the experiments are described below.

[0106] 2.1 Implementation of Non-LLM Baselines

[0107] Six non-LLM baselines were replicated: TPLinker, SPN, PRGC, OneRel, BiRTE, and UniREL. These are all state-of-the-art BERT-based methods in recent years. To adapt them to CMIM23-NOM1-RA, the following adjustments were made to this embodiment:

[0108] Pre-trained model parameters: To ensure compatibility with the Chinese corpus, the pre-trained parameters of these baselines were adjusted to "Chinese-BERT-WWM-EXT" to obtain better Chinese understanding, and they were aligned with the parameters used in PEAR.

[0109] Segmented Entity Processing: Given that these baselines cannot perform segmented entity extraction, triples containing segmented entities are excluded from the training set, and the remaining data is used to train models for them.

[0110] 2.2 Implementation of the LLM-based baseline

[0111] This embodiment also utilized three high-performance LLMs for experimentation: T5, GLM, and Qwen. To achieve this goal, the following strategies were implemented, referring to Table 2:

[0112] Pre-trained model parameters: For T5, after experimental comparison, "Randeng-T5-784M-MultiTask-Chinese" was selected as the pre-training parameter, which is referred to as "Rd-T5-784MTC" in the experiment. For GLM and Qwen, due to hardware limitations, "GLM4-9B-Chat" and "Qwen2-7B" were selected respectively, although they have fewer parameters, they have excellent performance.

[0113] Data Format: CMIM23-NOM1-RA has been adapted to a question-and-answer format to fit the LLM paradigm, and appropriate prompts have been added. See Appendix a for an example of the question-and-answer format.

[0114] Fine-tuning: Two fine-tuning strategies are employed. For T5, all parameters are fine-tuned to achieve optimal performance. For GLM and Qwen, LoRA fine-tuning is used, which enables satisfactory results with limited resources.

[0115] Data Augmentation: Data augmentation operations were performed. Specifically, each sample in the dataset was expanded into 10 question-answer pairs, providing a different example for each question. During validation, triples predicted by more than half of the expanded samples were selected as the final predictions for this embodiment. By employing this operation, performance improvements were observed in this embodiment.

[0116] In addition, implementation details for PEAR and all baselines are presented in Table 2. All LLM-based baselines and OneRe1 experiments were conducted on an NVIDIA A800 80GB GPU, while other experiments, including PEAR, were conducted on an NVIDIA V100 32GB GPU.

[0117] Table 2 shows the implementation details of PEAR and all baselines.

[0118] In the table, "*" indicates that the setting of this item is the same as that in the baseline paper, and "-" indicates that this item does not need to be set.

[0119]

[0120] 3. Evaluation Indicators

[0121] Two triplet criteria are used to determine the correctness of triples: one is more lenient, namely "ROUGE match", and the other is more stringent, namely "exact match".

[0122] The first criterion differs from previous work. In previous studies, entity correctness depended entirely on matching the first or last token, a lenient criterion. However, in this paper, the scenarios with longer entities and segmented entities are complex, making traditional criteria inapplicable. Therefore, this embodiment proposes a lenient criterion incorporating ROUGE, a widely used metric in text summarization used to measure the similarity between model-generated text and reference text; a higher value indicates greater similarity. In this experiment, "ROUGE matching" is defined: an entity is considered correct if the ROUGE-1 score between the predicted entity and the ground truth is greater than 0.6. A triple is considered correct if the relation type and both the subject and object are correct. ROUGE matching is used to tolerate entities with only a few token mismatches but whose basic meaning remains unchanged, and entities that do not hinder knowledge graph construction. Furthermore, this tolerant criterion demonstrates the effectiveness of the relation triple extraction method in scenarios closer to the real world.

[0123] The second criterion, "exact match," is more stringent because it requires that the predicted entity must be identical to the truth value to be considered correct. This criterion reflects the exact extraction capability of relation triplet extraction methods.

[0124] After determining the two criteria mentioned above, the standard accuracy (Prec.), recall (Rec.), and F1 score were reported to comprehensively evaluate the performance of the method.

[0125] 4. Experimental Results

[0126] The results of the experiments conducted using the above configuration are shown in Table 3.

[0127] Table 3 Experimental Results

[0128]

[0129] Table 3 shows the "Exact Match," "ROUGE Match," and "Segmented Entities" results for PEAR and all other baselines on the CMIM23-NOM1-RA dataset. "Exact Match" and "ROUGE Match" provide statistics based on triples with all labels and predictions. "Segmented Entities" represents the exact match results when only triples containing segmented entities are retained.

[0130] The “Exact Match” and “Coarse Match” columns in Table 3 show the overall performance of the methods. Clearly, PEAR outperforms other baselines on most metrics. Under the exact match condition, PEAR's F1 score is 0.4% higher than the best-performing baseline, UniRel, and 1.3% higher than the best LLM-based method, GLM4-9B-Chat. Under the ROUGE match condition, PEAR's F1 score is also 2.8% higher than the best baseline, Rd-T5-784MTC. PEAR's significant advantage under ROUGE matching is primarily attributed to its high recall, which greatly improves its F1 score. These results clearly demonstrate that PEAR outperforms other general-domain methods on the Chinese web operations corpus. The lower score under the “Exact Match” condition is due to the high challenge of the dataset.

[0131] However, it is worth noting that PEAR achieved an F1 score of over 70% in ROUGE matching. This score is a remarkable achievement, especially in specialized fields with limited corpus sources and small training sets. Furthermore, ROUGE matching better reflects PEAR's performance in real-world applications, with a score of 70.9% meeting the requirements for building high-quality knowledge graphs.

[0132] The “Segmented Entities” column in Table 3 shows a comparison of PEAR with baselines capable of extracting segmented entities. As a novel and challenging problem, the reported scores under the segmented entity condition are not high. This is because extracting segmented entities requires not only extracting multiple fragments from the text but also integrating the text, which poses a significant challenge to the model's text understanding capabilities. Furthermore, the limited number of training samples with segmented entities also contributes to the low scores. However, by comparison, PEAR still outperforms all baselines in terms of accuracy and F1 score.

[0133] 5. Experimental Analysis

[0134] 5.1 Reliability Analysis Based on ROUGE Matching

[0135] The analysis uses the ROUGE-1 score. The reliability of the "ROUGE match" is explained below, along with a statistically based interpretation of setting the metric threshold to 0.6. ROUGE aims to measure the similarity between two texts; a higher ROUGE score indicates greater similarity. Figure 3Two typical triplet cases are shown, which are considered correct even though they are not perfect matches. In example (1), the predicted subject contains additional information compared to the label, but this does not change the meaning of the entity, so it is considered correct. This situation of adding or missing parts is quite common. In example (2), the prediction splits the object of the label truth value into two separate parts, resulting in two output triples. In the context of building a knowledge graph, this situation does not affect the integrity of the graph, so it is also considered correct. For those incorrect triples, they often have very little overlap between the tokens and the label, leading to a significant change in the meaning of the entity. These triples tend to have lower ROUGE scores.

[0136] To gain a deeper understanding of the ROUGE-1 score distribution of correct and incorrect triples, nearly 200 incompletely matched predicted triples were further evaluated. The results are as follows: Figure 4 As shown, most correct triplets score above 0.7, while most incorrect triplets score below 0.5. Assuming these two distributions are independent, we set the threshold to 0.6 based on Naive Bayes classification. The above examples and statistics demonstrate that "ROUGE matching" is reliable and the threshold setting is reasonable.

[0137] 5.2. PEAR Advantages Analysis

[0138] Statistical analysis was performed on samples containing different numbers of triples and samples involving SEO and EPO scenarios. The results are shown in Table 4.

[0139] Table 4 shows the F1 scores of each method in samples with different numbers of triples and overlap patterns.

[0140]

[0141]

[0142] The samples were divided into six groups based on the number of triples, ranging from 1 to 5 and ≥6. A higher number of triples indicates greater sentence complexity. Results showed that PEAR achieved a lower F1 score than OneRel and UniRel for samples with 3 or fewer triples. However, PEAR achieved the highest score when the number of triples was 4 or more. Furthermore, the F1 scores of each method were analyzed in normal, SEO, and EPO scenarios. PEAR achieved the highest scores in the complex SEO and EPO scenarios, while UniRel scored highest in the normal scenario.

[0143] The superior performance of OneRel and UniRel in simple scenarios likely stems from their better optimization for accuracy. In the simpler scenarios mentioned above, the recall rates of different methods are very similar, giving OneRel and UniRel a greater advantage due to their higher accuracy.

[0144] However, PEAR demonstrates superior performance in extracting information from more complex text. We summarize two reasons. First, the combination of autoregressive and segmented BIO mechanisms makes PEAR more suitable for extracting entity information, including segmented entities, in complex scenarios. Second, in PEAR, the extraction of different relation types is completely independent, making it unaffected by the presence of multiple relations in a sentence.

[0145] Because complex text often carries more information, PEAR’s advantage in handling complex text compared to all other baselines earned it the highest F1 score in Table 3.

[0146] 5.3 Analysis of the effect of segmented entity extraction

[0147] A detailed analysis of the segmented entity extraction performance of PEAR and other LLM-based baselines was conducted. The aim was to understand the specific factors limiting their scores. This was analyzed from three perspectives: the range of triples in the sample, the range of tokens in the entity, and the range of segments within a segmented entity.

[0148] First, as can be seen from Figure 5(a), the extraction performance of all methods generally decreases with the increase of the number of triples. This indicates that the complexity of the text is one of the factors limiting the ability of this method to extract segmented entities.

[0149] Secondly, Figure 5(b) shows that the performance of the three LLM-based methods slightly decreases with the increase of the number of entity tokens, while PEAR remains unaffected. This is because the generation mechanism of LLM is affected by the output length, while PEAR does not exhibit this problem. In conclusion, the number of entity tokens also has a certain impact on the extraction of segmented entities.

[0150] Finally, in Figure 5(c), it is clear that correctly extracted triples are mainly concentrated when the maximum number of segments is 2, while the recall rate for entities divided into more than 3 segments is low. Note that since there are only 4 triples with 4 segments in total, the 25% recall rate of Qwen2-7B means that a triple containing an entity with 4 segments was correctly extracted. We believe that the increase in the number of segments drastically increases the extraction difficulty, while the number of training samples drastically decreases; these two factors together limit the current performance of segmented entity extraction.

[0151] 5.4 Ablation Study

[0152] Ablation studies were conducted to investigate the contributions of several modules to PEAR. Specifically, ablation experiments were performed on the relational information fusion method and label smoothing mechanism mentioned in Section 2. The results are listed in the last two rows of Table 3.

[0153] PEAR spe_tok_rela This means assigning custom, special tags to various relation types during the input fusion stage, rather than using natural language text to represent them. Results show that representing relations in natural language improves PEAR's performance by 1.8%, 1.5%, and 2.8% in exact matching, ROUGE matching, and segmented entities, respectively. This fully validates the importance of leveraging the semantic information of pre-trained models.

[0154] PEAR no_seg_ent This indicates that PEAR does not contain "segmented BIOs" and therefore cannot extract segmented entities. During training, we processed the dataset in the same way we trained a non-LLM baseline, i.e., we removed all triples containing segmented entities. The results show that PEAR... no_seg_ent The accuracy increased, but the recall decreased, and the overall F1 score was lower than PEAR (1.5% lower for exact matches and 7.1% lower for ROUGE matches). Furthermore, PEAR... no_seg_ent The performance was closer to the non-LLM baseline and failed to surpass some excellent baselines. These findings indicate that the "segmented BIO" mechanism significantly improved PEAR's performance, particularly its recall capability.

[0155] PEAR no_LS This indicates that no label smoothing mechanism was used in the loss function. Experimental results show that the method using the label-smoothed cross-entropy loss function is 1.7% better in exact matching than using only the regular cross-entropy loss, and 0.5% better in ROUGE matching and segmented entity cases. This suggests that the label smoothing mechanism can alleviate overfitting and has a certain optimization effect on model training.

[0156] To achieve intelligent network operation and maintenance and automate the construction of high-quality knowledge graphs in this field, this invention focuses on relation triple extraction technology for the Chinese network operation and maintenance corpus, and identifies a new complex scenario—segmented entities. To address the problems posed by segmented entities, this invention proposes a prompt-based entity-level autoregressive relation triple extraction method, called "PEAR". This method is an autoregressive framework containing an end-to-end sequence labeling model for extracting relation triples. During the labeling phase, a "segmented BI0" labeling scheme is introduced to effectively handle segmented entities. Furthermore, label smoothing is introduced during the training phase to mitigate overfitting. To further evaluate the performance of PEAR, CMIM23-NOM1-RA was constructed, the first high-quality domain-bounded relation triple extraction dataset in the network operation and maintenance field. This dataset is characterized by highly specialized relation types, high information density, and the inclusion of complex scenarios involving segmented entities, presenting significant challenges. Finally, extensive experiments and comprehensive analyses demonstrate that PEAR achieves state-of-the-art performance on the Chinese network operation and maintenance corpus.

[0157] In summary, the prompt-based entity-level autoregressive relation triplet extraction model training method, extraction method, and apparatus described in this invention address complex relation scenarios involving single entity overlap, entity pair overlap, and segmented entities. Through an extended BIO labeling mechanism, multiple labels are added to multiple entity segments of sentences in each sample according to various entity types to construct a training sample set. The labels distinguish the starting token, internal tokens, and other non-entity tokens of sentences in each sample. An initial neural network model is constructed using a BERT encoder and classifier. During single-step autoregressive extraction, the model takes the sentence, relation type, and extracted entity pairs as input. For different relation types, it predicts entity pairs and their relationships one by one through multiple rounds of autoregression. Label smoothing technology is employed during training, combined with cross-entropy loss function to update parameters and optimize the model's extraction performance. This invention, by leveraging multi-segmentation, multi-relationship categories, and distinguishing intra-attribute methods, can efficiently label segmented entities. During model learning, multi-round autoregression is used to predict entities, effectively solving the problem of relation triplet extraction in complex scenarios with segmented entities.

[0158] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0159] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0160] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0161] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A prompt-based method for training an entity-level autoregressive relation triplet extraction model, characterized in that, The method includes the following steps: Multiple samples are obtained, each sample being a sentence originating from the target topic environment, and each sentence is represented as a token sequence. Each sample contains multiple relation triples, each relation triple containing a subject, an object, and the relationship between them. The subject and the object, as entities, are composed of one or more entity segments, and each entity segment is a subsequence in the sentence containing one or more consecutive tokens. Within a sample, there are three types of overlaps between or within relation triples: single entity overlap, entity pair overlap, and segmented entities. Single entity overlap indicates that two relation triples share one entity; entity pair overlap indicates that two relation triples share the same entity but have different relationships; and segmented entities indicate that a relation triple contains an entity composed of multiple non-contiguous and non-overlapping entity segments. A training sample set is constructed by adding a label to each token in the sentence of each sample. The label distinguishes between the entity segment start token, the entity segment internal token and other non-entity tokens. An initial neural network model, including a BERT encoder and a classifier, is obtained. During a single-step extraction process, the initial neural network model takes the sentence, the relation type added to the end of the sentence, and the extracted entity pairs as input and outputs the next entity pair. The initial neural network model performs multiple rounds of autoregression to extract entity pairs one by one according to multiple relation types. Based on the prediction results and the labels, a cross-entropy loss function is constructed after label smoothing. The parameters of the initial neural network model are updated using the training sample set to obtain the target relation triplet extraction model.

2. The method for training a prompt-based entity-level autoregressive relation triplet extraction model according to claim 1, characterized in that, A training sample set is constructed by adding labels to each token in each sentence of each sample. The labels distinguish between entity segment start tokens, entity segment internal tokens, and other non-entity tokens, including: The entity segment start token comprises three parts: the first part indicates that the current token belongs to the entity segment start token; the second part indicates the entity type, which includes subject and object; and the third part indicates the segment ordinal number of the entity segment represented by the current token within the corresponding entity. The token within the entity segment consists of two parts: the first part indicates that the current token belongs to the entity segment; the second part indicates the entity type, which includes the subject and the object. The other non-entity tokens include a part, the first part of which marks that the current token belongs to other non-entity tokens; The first token of the entity segment is marked as the starting token of the entity segment, and all subsequent tokens in the entity segment starting from the second token are marked as internal tokens of the entity segment. All tokens in the entity segment have the same entity type as their corresponding tags.

3. The method for training a prompt-based entity-level autoregressive relation triplet extraction model according to claim 1, characterized in that, The classifier includes consecutive fully connected layers and a Softmax layer; the BERT encoder is pre-trained using text data of the target topic environment.

4. The method for training a prompt-based entity-level autoregressive relation triplet extraction model according to claim 3, characterized in that, The method further includes: pruning the attention head and layers of the BERT encoder; and introducing a local attention mechanism or a cross-segment attention mechanism into the BERT encoder.

5. The method for training a prompt-based entity-level autoregressive relation triplet extraction model according to claim 3, characterized in that, The BERT encoder uses DistilBERT / TinyBERT.

6. The method for training a prompt-based entity-level autoregressive relation triplet extraction model according to claim 1, characterized in that, Based on the prediction results and the labels, a cross-entropy loss function is constructed after label smoothing. The expression for the loss is: Among them, y LS This represents the label after smoothing, where j represents the vector y. LS The index of each class in the table, j0 represents the index of the actual class, and α is a hyperparameter representing the degree of smoothness.

7. The method for training a prompt-based entity-level autoregressive relation triplet extraction model according to claim 1, characterized in that, The method further includes: dividing the training sample set into a training set, a validation set, and a test set; using the training set to train the model; using the validation set to evaluate and optimize the target relation triplet extraction model; using the test set to calculate the standard accuracy, recall, and F1 score of the trained and optimized target relation triplet extraction model; and generating a performance evaluation report.

8. A prompt-based method for extracting entity-level autoregressive relation triples, characterized in that, The method includes the following steps: Obtain sentences to be recognized that are relevant to the target topic environment; The sentence to be identified is tokenized and then input into the target relation triplet extraction model in the prompt-based entity-level autoregressive relation triplet extraction model training method according to any one of claims 1 to 7, and the entity recognition result is output.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images