A generative-based multi-task multi-level knowledge extraction method and system and a storage medium

Abstract

This invention provides a generative, multi-task, multi-level knowledge extraction method, system, and storage medium. The method includes: Step 1: Designing a unified input / output format to transform heterogeneous task requirements into a mapping problem from source text to target text; Step 2: Adapting the decoding strategy of the generative model using a two-stage multi-task training method that includes task separation and task coupling. The beneficial effects of this invention are: By using a phased, progressive learning approach, this invention effectively alleviates the optimization conflict problem in multi-task learning, realizing the transformation from independent task learning to collaborative knowledge enhancement, and providing an effective multi-task training paradigm for complex knowledge extraction tasks.

Description

Technical Field

[0001] This invention relates to the field of information extraction and natural language processing technology, and in particular to a generative, multi-task, multi-level knowledge extraction method, system, and storage medium. Background Technology

[0002] In the fields of information extraction and natural language processing, knowledge extraction techniques aim to extract structured information from unstructured text. Currently, discriminative multi-task, multi-level knowledge extraction methods have become one of the mainstream solutions. This method, by sharing underlying network parameters, can simultaneously achieve multiple sub-tasks such as named entity recognition, relation extraction, and risk multi-label classification within a unified model. Experiments show that this discriminative architecture can effectively capture common features across different tasks, significantly improving extraction performance and computational efficiency compared to traditional single-task models.

[0003] However, as application scenarios increasingly demand higher extraction accuracy, flexibility, and scalability, existing discriminative methods have gradually revealed the following inherent limitations: 1. Poor task scalability and bloated model structure: Discriminative models typically require designing specific classifiers and complex labeling systems for each subtask. For example, entity recognition usually uses the BIO labeling strategy, while relation extraction requires a separate classifier design. This design leads to a redundant model structure as the number of tasks increases, and it is difficult to flexibly extend to new task types or labels; 2. Limited output format: Discriminative models are inherently limited by a predefined set of closed labels, and can only perform classification or sequence labeling operations. They cannot directly generate relation triples or output other unstructured complex text formats.

[0004] 3. High training difficulty and task interference: In multi-task learning scenarios, the training of discriminative methods typically relies on meticulously designed task-specific loss functions and weight balancing strategies. Gradient conflicts or negative transfer can easily occur between different subtasks, thus limiting the overall performance of the model.

[0005] Therefore, there is an urgent need for a new knowledge extraction method that can overcome the above-mentioned shortcomings and achieve complex knowledge extraction in a more flexible and efficient manner across multiple tasks and levels.

[0006] The information disclosed in this background section is intended only to enhance understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0007] To address the problems in the prior art, this invention provides a generative, multi-task, multi-level knowledge extraction method, system, and storage medium.

[0008] This invention provides a generative, multi-task, multi-level knowledge extraction method, comprising: Step 1: Design a unified input and output format to transform heterogeneous task requirements into a mapping problem from source text to target text; Step 2: Adapt the decoding strategy of the generative model using a two-stage multi-task training method that includes task separation and task coupling.

[0009] As a further improvement of the present invention, step one further includes: Step 1: Obtain the raw text and use a format converter to convert the obtained raw text into a linearized input sequence. The input sequence is composed of the source text and the task description. Step 2: Input the linearized input sequence into the generative model, which then converts it into a linearized output sequence. In the output sequence, the multi-labels for the risk classification task are concatenated according to the order in the input sequence. Step 3: Input the linearized output sequence into the output parser, and the output parser will parse the output sequence to obtain the final structured knowledge.

[0010] As a further improvement of the present invention, the generative model adopts an encoder-decoder architecture. This model receives a linearized input sequence, and its decoder generates a target sequence through an autoregressive method and outputs the extraction results containing multiple labels of entities, relationships and risks in a predefined output format.

[0011] As a further improvement of the present invention, in step 1, the task description includes a task type and a tag option, wherein the task type is described in natural language and the tag option is identified by a special symbol.

[0012] As a further improvement of the present invention, in step 2, the entity recognition task includes a task prefix, entity type and specific entity words. The entity words are concatenated after the corresponding entity type in the order in which they appear in the text, separated by symbols. The relation extraction task includes a task prefix, relation type and head and tail entities. Different entity pairs are separated by symbols. Entity pairs are represented in the form of head entity@tail entity.

[0013] As a further improvement of the present invention, step two further includes: The first stage involves multi-task training with task separation: the generative model processes only one task in a single input; the input sequence is composed of source text and task descriptions for a single task; and model parameters are shared among entity recognition, relation extraction, and risk multi-label classification tasks. The second stage involves multi-task training steps with task coupling: the generative model outputs a linearized sequence containing entity recognition, relation extraction, and risk multi-label tasks in one step of generation; the encoding stage simultaneously models the semantic association between entities and relations, as well as between entities and risks; and the decoding stage implicitly models the consistency constraints between entities and relations through a self-attention mechanism.

[0014] The present invention also discloses a generative multi-task multi-level knowledge extraction system, comprising: a memory, a processor, and a computer program stored in the memory, wherein the computer program is configured to implement the steps of the multi-task multi-level knowledge extraction method of the present invention when invoked by the processor.

[0015] The present invention also discloses a computer-readable storage medium storing a computer program configured to implement the steps of the multi-task, multi-level knowledge extraction method described in the present invention when invoked by a processor.

[0016] The beneficial effects of this invention are: 1. This invention adopts a generative model and a unified linearized input-output format, treating knowledge extraction tasks of different granularities, such as entity recognition, relation extraction, and multi-label classification, as a sequence-to-sequence generation problem. Compared with discriminative methods that require designing independent classification heads for each task, the model structure of this invention is simple. Adding a new task only requires defining a new task description, without modifying the model architecture. 2. This invention proposes a two-stage training strategy of task separation and task coupling through phased progressive learning: the first stage enables the model to master the basic capabilities of each task through single-task training, avoiding interference between tasks; the second stage enables the model to learn the semantic relationships between tasks through multi-task joint training, realizing the transformation from independent task learning to collaborative knowledge enhancement, effectively alleviating the optimization conflict problem in multi-task learning. 3. Experimental results show that the F1 score of this invention is significantly better than that of existing baseline models on the power safety dataset; 4. This invention is applicable to text knowledge extraction in vertical fields that contain professional terminology and complex tagging systems, including but not limited to scenarios that require multi-level risk identification, such as power safety, industrial safety, equipment operation and maintenance, and medical documents. Attached Figure Description

[0017] Figure 1 This is a flowchart of the generative multi-level knowledge extraction model of this invention; Figure 2 This is a diagram of the linearized input / output format of the present invention; Figure 3 This is an example diagram of the linearized input and output of the present invention; Figure 4This is the multi-task training strategy diagram of the generative model of this invention. Detailed Implementation

[0018] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.

[0019] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0020] like Figure 1 As shown, this invention discloses a generative multi-task, multi-level knowledge extraction method, aiming to explore how to use generative pre-trained language models to unify the modeling of three core knowledge extraction tasks: named entity recognition, relation extraction, and risk multi-label classification.

[0021] This embodiment uses text data from the field of power safety as an example to provide a detailed explanation of the method proposed in this invention. Text data in this field is characterized by dense technical terms, multiple risk levels, and imbalanced samples, which can fully verify the effectiveness of this invention.

[0022] To achieve this goal, a unified input / output format needs to be designed first to transform heterogeneous task requirements into a mapping problem from source text to target text. For example, sentences and task instructions can be concatenated as input, and entity labels, relation triples, and risk category labels can be transformed into linearized text sequences as output according to a specific template. Secondly, this invention studies a multi-task training strategy based on a generative architecture, exploring how to efficiently train on data containing multiple tasks. A two-stage multi-task training method, including task separation and task coupling, is proposed, adapting to the decoding strategy of the generative framework to ensure that the model can accurately and completely extract multi-level knowledge elements from the text. Experimental results show that the generative multi-task multi-level knowledge extraction method proposed in this invention achieves improvements on a self-built power safety dataset, demonstrating stronger generalization ability and reasoning interpretability, providing new ideas for building a more intelligent and general knowledge extraction system.

[0023] like Figure 1 As shown, this invention discloses a generative, multi-task, multi-level knowledge extraction method, including: Step 1: Design a unified input and output format to transform heterogeneous task requirements into a mapping problem from source text to target text; Step 1: Given the original text, the first step is to perform format conversion, transforming both input and output into linearized text. The generative model uses an encoder-decoder architecture. The model receives the input sequence, and the decoder generates the target sequence through autoregression, extracting multi-label results including entities, relationships, and risks according to a predefined output format. Finally, the final structured knowledge is obtained by parsing the output sequence.

[0024] 1. Input / output format To align various structured knowledge sets with the output of generative models, a linearized input-output format needs to be designed to guide the model in clearly distinguishing different task types and their corresponding extraction requirements. The specific input format is as follows: Figure 2 As shown.

[0025] Input format: The input sequence consists of source text and a task description concatenated together. The task description includes the task type and label options. The task type is described in natural language, such as "entity recognition" or "relation extraction"; the label options are identified using special symbols, where: Risk label: Enclosed in angle brackets, such as "<electric shock>".

[0026] Entity type label: Enclosed in parentheses, such as "(device name)".

[0027] Relationship type label: Enclosed in square brackets, such as "[Modifier]".

[0028] Different task types are identified by different special symbols, which enables the model to better distinguish the differences between task labels and facilitates the model's quick understanding of their different meanings.

[0029] Output format: In the output sequence, the multi-labels for the risk classification task are concatenated in the order they appear in the input sequence. The entity recognition task includes a task prefix, entity type, and specific entity words. Entity words are concatenated after the corresponding entity type in the order they appear in the text, separated by semicolons ";". The relation extraction task includes a task prefix, relation type, and head and tail entities. Different entity pairs are separated by semicolons ";", and entity pairs are represented in the form "head entity@tail entity". Figure 3 This demonstrates a concrete example of linearized input and output.

[0030] Step 2: Adapt the decoding strategy of the generative model using a two-stage multi-task training method that includes task separation and task coupling.

[0031] To enable the model to effectively learn the intrinsic relationships between different knowledge extraction tasks while avoiding task interference, this invention designs a two-stage multi-task training strategy: the first stage is multi-task training with task separation, and the second stage is multi-task training with task coupling, as follows: Figure 4 As shown.

[0032] Phase 1: Multi-task training with task separation. This phase aims to enable the model to independently master the basic capabilities of various knowledge extraction methods, avoiding optimization conflicts that may occur in the early stages of multi-task joint training. At this stage, the model processes only one task per input, with the input being a concatenation of source text and a task description. However, the training data includes data for three tasks: entity recognition, relation extraction, and risk multi-label classification. Model parameters are shared across different tasks, but only one task is performed in each forward propagation. Through alternating learning of different tasks, the model gradually establishes an independent and accurate mapping capability for each task.

[0033] The second stage: Task-coupled multi-task training. After the model has achieved a stable ability to extract independent tasks, the second stage introduces task-coupled multi-task training. This aims to encourage the model to learn the relationships between different knowledge points, achieving inter-task facilitation and true one-step joint extraction. At this stage, the model's input is as follows: Figure 2 and Figure 3 As shown, the model is required to output a linearized sequence of three tasks in one step of generation. This stage guides the model's encoding phase to simultaneously focus on the semantic associations between entities and relations, as well as between entities and risks, and implicitly models the consistency constraints of entities and relations using a self-attention mechanism during decoding, thereby effectively mitigating the problem of error accumulation in pipelined methods.

[0034] Through a two-stage training process, the model ensures the accuracy of each sub-task and achieves collaborative extraction of multi-level knowledge.

[0035] Experimental Results and Analysis To evaluate the performance of the proposed Generative-based Multi-task Multi-level Knowledge Extraction Model (GMKE) in this invention, experiments were conducted on a constructed power safety dataset. First, the performance evaluation metrics are introduced, and the experimental environment and parameter settings are explained. Then, the experimental results are analyzed, and finally, ablation experiments and sample analyses are performed.

[0036] Data sets and evaluation metrics The dataset contains 9600 training entries, 1200 validation entries, and 1200 test entries, which are converted to a linear form as described in this invention. The evaluation metrics for entity recognition and relation extraction are precision (P), recall (R), and F1 score, while the evaluation metric for risk multi-label classification is strict accuracy.

[0037] Experimental setup This invention uses Mengzi-t5-base as the generative base model, with a maximum input length of 280, a learning rate of 2e-5, and a training batch size of 32. The AdamW optimizer is selected for parameter updates during training. In multi-task training, the first and second training phases consist of 10 epochs each. All experiments were conducted on an NVIDIA Tesla V100 (32GB VRAM) GPU server.

[0038] Experimental Results and Analysis To evaluate the performance of the proposed generative multi-level knowledge extraction model, experiments were conducted on a constructed power safety dataset. Comparative experiments were also performed using models such as YAYI-UIE, Paddle Paddle UIE, and SLG Framework. The experimental results are shown in the table below: Table 1. Experimental results of the generative multi-level knowledge extraction model.

[0039] Table 1 shows the results of the model on a self-constructed power safety dataset. The optimal value for each indicator is shown in bold, and the suboptimal value is indicated by an underline. The gray text represents the experimental results of the discriminative model in this paper. The results of YAYI-UIE were obtained using few-shot context learning, with three examples randomly selected as prompts for each text. The results of Paddle PaddleUIE and SLG Framework are shown after 20 rounds of fine-tuning. However, since SLG Framework is not open source, this invention reproduces the experiments by referring to its paper.

[0040] Compared to the generative baseline model, the proposed GMKE model achieves optimal results on all three tasks, fully validating the effectiveness of the proposed method. The YAYI-UIE model performs relatively poorly, mainly because it only employs a few-shot cue learning strategy and does not fine-tune on power domain data. Although it performs well in general scenarios, its extraction capabilities are limited by the lack of domain knowledge when facing highly specialized entity and relation categories in the power safety field. PaddlePaddle UIE achieves an F1 score of 92.43% on the NER task, but its training strategy has certain limitations. It splits entity recognition into multiple tasks, extracting only one category of entities in each generation. This design fails to model the inherent relationships between different categories of entities. For example, "voltage level" and "equipment name" often co-occur in text and have a modifying relationship; split training makes it difficult for the model to learn the collaborative information between such entities. In relation extraction training, PaddlePaddleUIE adopts a pipeline extraction approach, using real head entities as cue words to predict their relation types and tail entities. This training method, which relies on real-world annotations, deviates from the inference stage, leading to an imbalance between precision and recall due to error accumulation. While the SLG framework achieves suboptimal results on NER and MLC tasks, its lack of support for relation extraction prevents the model from learning the synergistic effect between entity recognition and relation extraction, limiting its applicability in complex knowledge extraction scenarios. The proposed GMKE model achieves optimal performance on all three tasks: an F1 score of 93.76% on the NER task, a 0.68% improvement over the baseline-best SLG framework; an F1 score of 93.40% on the RE task, a 9.97% improvement over the baseline-best PaddlePaddle UIE; and a strict accuracy of 87.32% on the MLC task, a 2% improvement over the baseline-best SLG framework. Table 1 clearly demonstrates the two major advantages of the GMKE model: firstly, the effectiveness of the two-stage multi-task training strategy. The first stage, task-separated training, allows the model to master different tasks thoroughly, while the second stage, task-coupled training, forces the model to learn the intrinsic relationships between tasks through joint input, significantly improving the performance of each task. Secondly, the unified linearized input and output format design enables the model to model multiple entity types and their relationships simultaneously, avoiding information loss caused by task splitting.

[0041] Ablation Experiment Results and Analysis (1) Ablation experiment of two-stage training strategy Table 2 shows the experimental results for GMKE using different training strategies. "Task separation + task coupling" indicates that the present invention performs two-stage training, with each stage consisting of 10 rounds. "Task coupling" indicates that no staged training is performed, and 20 rounds of multi-task training with task coupling are performed directly.

[0042] Table 2. Impact of Training Strategies on Model Performance

[0043] The experimental results show that the GMKE model employing a two-stage training strategy outperforms the model directly trained on all three tasks. After removing the multi-task training with task separation, the F1 score for the NER task decreased by 0.4%, the F1 score for the RE task decreased by 0.23%, and the strict accuracy for the MLC task decreased by 1.16%, exhibiting a consistent performance decline across all three tasks, fully validating the effectiveness of the two-stage training strategy. The first stage of task separation training allows the model to focus on learning the basic capabilities of each task, avoiding interference from complex tasks in task coupling. The second stage of task coupling training enables the model, with a certain level of knowledge extraction capability, to further learn the intrinsic relationships between tasks, achieving synergistic knowledge enhancement. Direct task coupling training requires the model to simultaneously learn the complex mapping relationships of multiple tasks from scratch; entity boundary recognition, relational semantic modeling, and risk classification are mutually constrained, making it difficult to reach their respective optimal states. Notably, the MLC task exhibits the highest sensitivity to the training strategy, indicating that the risk classification task highly relies on the contextual information provided by entity recognition and relation extraction. The basic extraction capabilities learned in the first stage provide crucial support for the joint learning in the second stage, indirectly verifying the rationality of the GMKE multi-task collaborative design.

[0044] In summary, the two-stage training strategy is one of the key design features that enables the GMKE model to achieve excellent performance. By adopting a phased, progressive learning approach, it effectively alleviates the optimization conflict problem in multi-task learning, realizing the transformation from independent task learning to collaborative knowledge enhancement, and providing an effective multi-task training paradigm for complex knowledge extraction tasks.

[0045] Sample Analysis To visually compare the performance differences between the generative multilevel knowledge extraction model GMKE and the discriminative multilevel knowledge extraction model DMKE in power safety knowledge extraction tasks, this invention selects test samples for comparative analysis, as shown in Table 3.

[0046]

[0047] The results show that the text model of this invention outperforms the baseline model in entity recognition (F1 score of 93.76%), relation extraction (F1 score of 93.40%), and risk multi-label classification (Accuracy of 87.32%), and outperforms the discriminative model in relation extraction, while maintaining a comparable level to the discriminative model in entity recognition and risk classification. Ablation experiments further demonstrate the positive effect of the two-stage training strategy on performance improvement, showcasing the superiority of the model proposed in this invention.

[0048] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A generative, multi-task, multi-level knowledge extraction method, characterized in that, include: Step 1: Design a unified input and output format to transform heterogeneous task requirements into a mapping problem from source text to target text; Step 2: Adapt the decoding strategy of the generative model using a two-stage multi-task training method that includes task separation and task coupling.

2. The multi-task, multi-level knowledge extraction method according to claim 1, characterized in that, Step one also includes: Step 1: Obtain the raw text and use a format converter to convert the obtained raw text into a linearized input sequence. The input sequence is composed of the source text and the task description. Step 2: Input the linearized input sequence into the generative model, which then converts it into a linearized output sequence. In the output sequence, the multi-labels for the risk classification task are concatenated according to the order in the input sequence. Step 3: Input the linearized output sequence into the output parser, and the output parser will parse the output sequence to obtain the final structured knowledge.

3. The multi-task, multi-level knowledge extraction method according to claim 2, characterized in that, The generative model uses an encoder-decoder architecture. This model receives a linearized input sequence, and its decoder generates a target sequence through an autoregressive approach. It then outputs the extraction results, which include multiple labels for entities, relationships, and risks, according to a predefined output format.

4. The multi-task, multi-level knowledge extraction method according to claim 2, characterized in that, In step 1, the task description includes a task type and tag options. The task type is described in natural language, and the tag options are identified by special symbols.

5. The multi-task, multi-level knowledge extraction method according to claim 3, characterized in that, In step 2, the entity recognition task includes a task prefix, entity type, and specific entity words. The entity words are concatenated after the corresponding entity type in the order in which they appear in the text, separated by symbols. The relation extraction task includes a task prefix, relation type, and head and tail entities. Different entity pairs are separated by symbols. Entity pairs are represented in the form of head entity@tail entity.

6. The multi-task, multi-level knowledge extraction method according to claim 1, characterized in that, Step two also includes: The first stage involves multi-task training with task separation: the generative model processes only one task in a single input; the input sequence is composed of source text and task descriptions for a single task; and model parameters are shared among entity recognition, relation extraction, and risk multi-label classification tasks. The second stage involves multi-task training steps with task coupling: the generative model outputs a linearized sequence containing entity recognition, relation extraction, and risk multi-label tasks in one step of generation; the encoding stage simultaneously models the semantic association between entities and relations, as well as between entities and risks; and the decoding stage implicitly models the consistency constraints between entities and relations through a self-attention mechanism.

7. A generative, multi-task, multi-level knowledge extraction system, characterized in that, include: A memory, a processor, and a computer program stored on the memory, the computer program being configured to implement the steps of the multi-task, multi-level knowledge extraction method according to any one of claims 1-6 when invoked by the processor.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program configured to implement the steps of the multi-task, multi-level knowledge extraction method according to any one of claims 1-6 when invoked by a processor.

Images