A general information extraction method based on contrastive supervision and cross-stage distillation

Abstract

The present invention provides a general information extraction method based on contrastive supervision and cross-stage distillation, comprising selecting a pre-trained model as a base model and initializing it; initializing a first low-rank decomposition matrix and a second low-rank decomposition matrix for the weight matrix of the base model; training the first low-rank decomposition matrix and the second low-rank decomposition matrix; merging the weights of the base model, the first low-rank decomposition matrix and the second low-rank decomposition matrix to obtain an inference model; inputting natural language text information into the inference model; specifying the task type and output format through natural language instructions, and outputting structured information extraction results. The beneficial effect of the present invention is that by training the LoRA matrix in stages and compounding parameters, the target task is learned while retaining the knowledge of the auxiliary task, thereby significantly reducing the loss of cross-stage knowledge transfer; natural language instructions drive the same model to process different tasks without switching the model architecture.

Description

Technical Field

[0001] The present invention belongs to the field of information processing technology, and in particular relates to a general information extraction method based on contrast supervision and cross-stage distillation. Background Art

[0002] General information extraction aims to develop a universal model capable of simultaneously handling multiple information extraction tasks, including core subtasks such as named entity recognition, relation extraction, and event extraction. Existing research has enhanced the extraction performance of the target task by designing auxiliary tasks closely related to the target task and constructing a training paradigm based on multi-stage continuous learning. This multi-stage training-based learning paradigm, which gradually improves extraction capabilities by sequentially learning extraction objectives of varying difficulty and importance, has achieved breakthrough progress in the field of general information extraction. Specifically, this progressive learning strategy can be referred to as continuous learning.

[0003] Continual learning, also known as lifelong learning or incremental learning, aims to learn different tasks sequentially in a certain order to cope with dynamically changing knowledge and environments. Although existing studies have achieved better extraction results than traditional multi-task learning architectures based on the continuous learning framework, they often face the problem of catastrophic forgetting (CF). Catastrophic forgetting refers to the phenomenon that when a model learns a new task, its reasoning performance on past tasks decreases significantly. In particular, when the new and old tasks have a high degree of overlap, the model may better retain the memory of previous knowledge. When the new and old tasks are significantly different, the model tends to update and overwrite the previously learned knowledge, causing the catastrophic forgetting problem to become more significant.

[0004] Auxiliary tasks usually provide underlying language representations for the target task and may serve as a key sub-link in the target task. For example, if the auxiliary task is to extract entity fragments, and the target task is to extract entity fragments and assign them to a predefined entity type label, then forgetting the auxiliary task will directly destroy the logical link between tasks, thereby negatively affecting the performance of the target task. In this case, the model tends to overfit the data distribution of the target task, resulting in poor performance in cross-task adaptive learning and task feature differentiation, ultimately affecting the overall extraction performance. However, existing general information extraction work based on multi-stage continuous learning often ignores the impact of catastrophic forgetting on the effectiveness of cross-stage knowledge transfer, resulting in suboptimal performance of the model during continuous learning. Therefore, there is an urgent need for a general information extraction method that solves the above problems. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a general information extraction method based on contrast supervision and cross-stage distillation, which is particularly suitable for coping with dynamically changing knowledge and environments through continuous learning in general information extraction.

[0006] The technical solution adopted by the present invention is: in a first aspect, a general information extraction method based on contrast supervision and cross-stage distillation is provided, comprising the following steps:

[0007] Select a pre-trained model as the base model and initialize it;

[0008] Initializing a first low-rank decomposition matrix and a second low-rank decomposition matrix for the weight matrix of the base model;

[0009] Training the first low-rank decomposition matrix and the second low-rank decomposition matrix;

[0010] Combining the weight of the base model, the first low-rank decomposition matrix, and the second low-rank decomposition matrix to obtain an inference model;

[0011] inputting natural language text information into the inference model;

[0012] The task type and output format are specified through natural language instructions, and the structured information extraction results are output.

[0013] Furthermore, training the first low-rank decomposition matrix includes the following steps:

[0014] freezing the weights of the base model;

[0015] A first input text is collected according to the task purpose of the first low-rank decomposition matrix;

[0016] The first input text is preprocessed to generate a first input vector;

[0017] inputting the first input vector into the base model;

[0018] According to the equation Compute the first output vector where is the first output vector, is the first input vector, is the original weight matrix of the base model, is the first low-rank decomposition matrix;

[0019] A first loss function is constructed to optimize the parameters of the first low-rank decomposition matrix.

[0020] Furthermore, training the second low-rank decomposition matrix includes the following steps:

[0021] Freezing the weights of the base model and the first low-rank decomposition matrix;

[0022] The first input text is sampled according to a sampling ratio by a random sampling method to form a playback sample;

[0023] A second input text is collected according to the task purpose of the second low-rank decomposition matrix;

[0024] The second input text and the playback sample are preprocessed to generate a second input vector;

[0025] According to the equation Compute the second output vector where is the second output vector, is the second input vector, is the original weight matrix of the base model, is the first low-rank decomposition matrix, is the second low-rank decomposition matrix;

[0026] A second loss function is constructed to optimize the parameters of the first low-rank decomposition matrix.

[0027] Furthermore, the first loss function The formula is ,in To minimize the negative log-likelihood loss, To supervise the contrast loss, is the weight of the supervised contrast loss term.

[0028] Furthermore, the second loss function The formula is ,in To minimize the negative log-likelihood loss, To supervise the contrast loss, is the weight of the supervised contrast loss term, is the playback sample loss, is the distillation loss, is the weight of the distillation loss term.

[0029] Furthermore, the supervised contrast loss The equation is ,in is the triplet loss, is the sample anchor point, represents one of the positive samples, represents one of the negative samples, is a marginal value greater than zero, Update the parameters for the model; the supervised contrast loss is introduced where the Euclidean distance quantifies the distance between two input example instructions ,in is the embedding vector of the instruction, represents the L2 norm.

[0030] Furthermore, the playback sample loss The equation is ,in are the weights of the base model and the first low-rank decomposition matrix, is the parameter updated on the second low-rank decomposition matrix, is the playback sample set size, is the input vector of the i-th sample; the distillation loss The equation is ,in is the probability distribution of the base model and the first low-rank decomposition matrix weights, is the probability distribution of the second low-rank decomposition matrix weights, is the output sequence length.

[0031] In a second aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the general information extraction method based on comparative supervision and cross-stage distillation provided in the present disclosure.

[0032] In a third aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the general information extraction method based on comparative supervision and cross-stage distillation provided by the present disclosure.

[0033] In a fourth aspect, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, provides a general information extraction method based on comparative supervision and cross-stage distillation provided by the present disclosure.

[0034] The advantages and positive effects of the present invention are: due to the adoption of the above-mentioned technical solution, by training the LoRA matrix in stages and compounding the parameters, the auxiliary task knowledge is retained while the target task is learned, which significantly reduces the loss of cross-stage knowledge transfer; natural language instructions drive the same model to process different tasks without switching the model architecture; freezing the base model weights ensures that the general language ability is not degraded, and focuses on optimizing the special features of the auxiliary task; independent training avoids noise interference of the target task and improves the accuracy of basic extraction tasks such as entities / relationships; by adding replay samples, the auxiliary task knowledge is forced to be recalled from the data level; the contrast loss enables the model to map semantically identical instructions to similar representation spaces; improves the model's fault tolerance for interfering instructions and reduces extraction errors caused by differences in user instruction expressions; replay loss retains the performance of auxiliary tasks, distillation loss constrains the new model output to approach the teacher model (old knowledge), and contrast loss enhances the discrimination of task instructions; the four-fold loss works synergistically to improve the learning speed of the target task. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of a general information extraction method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present disclosure is described more fully below with reference to the accompanying drawings, which illustrate exemplary embodiments of the present disclosure. The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present disclosure.

[0037] like Figure 1 As shown, the present invention provides a general information extraction method based on contrast supervision and cross-stage distillation, comprising the following steps:

[0038] S100, selecting a pre-trained model as a base model and initializing it;

[0039] The target tasks are clearly defined as named entity recognition (NER), relation extraction (RE), event trigger word extraction (EET), and event argument extraction (EEA). At the same time, a set of auxiliary tasks is constructed, including entity recognition, entity classification, entity pair recognition, entity pair classification, and event element extraction. It should be noted that the target tasks are not limited to the aforementioned naming and are determined through the model training settings. Named entity recognition (NER) is used to identify specific categories of entities in text, such as names of people, places, organizational names, time, diseases, and drugs; relation extraction (RE) is used to identify semantic relationships between two entities in text (such as "treat," "cause," and "belong to"); event trigger word extraction (EET) is used to identify words in text that mark the occurrence of events (for example, in "The company announced a new merger and acquisition plan yesterday," the word "announce" can be considered a trigger word because it marks the occurrence of a declaration event); and event argument extraction (EEA) is used to identify information such as entities or time related to the event.

[0040] S200, initializing a first low-rank decomposition matrix and a second low-rank decomposition matrix for the weight matrix of the base model;

[0041] S300, training a first low-rank decomposition matrix and a second low-rank decomposition matrix;

[0042] S400, merging the weight of the base model, the first low-rank decomposition matrix, and the second low-rank decomposition matrix to obtain an inference model;

[0043] S500, inputting natural language text information into the inference model;

[0044] S600: Specify the task type and output format through natural language instructions, and output the structured information extraction result.

[0045] Using the above method, by training the LoRA matrix in stages and compounding the parameters, the knowledge of the auxiliary task (i.e., the first low-rank decomposition matrix) is retained while learning the target task (i.e., the second low-rank decomposition matrix), significantly reducing the loss of cross-stage knowledge transfer; natural language instructions drive the same model to process different tasks (such as entity recognition → entity classification) without switching the model architecture; the merged single inference model reduces computing resource consumption and is suitable for edge device deployment.

[0046] In order to solve the problem of avoiding feature confusion caused by direct multi-task training, an implementation method is provided in this embodiment.

[0047] In one embodiment, training the first low-rank decomposition matrix includes the following steps:

[0048] Freeze the weights of the base model;

[0049] A first input text is collected according to the task purpose of the first low-rank decomposition matrix;

[0050] The first input text is preprocessed to generate a first input vector;

[0051] Inputting a first input vector into the base model;

[0052] According to the equation Compute the first output vector where is the first output vector, is the first input vector, is the original weight matrix of the base model, is the first low-rank decomposition matrix;

[0053] A first loss function is constructed to optimize the parameters of the first low-rank decomposition matrix.

[0054] Using the above method, the base model weights are frozen to ensure that the general language ability is not degraded, and the focus is on optimizing the dedicated features of the auxiliary tasks; independent training avoids noise interference from the target task and improves the accuracy of basic extraction tasks such as entities / relationships.

[0055] In order to solve the problem of catastrophic forgetting in multi-stage continuous learning, an implementation method is provided in this embodiment.

[0056] In one embodiment, training the second low-rank decomposition matrix comprises the following steps:

[0057] Freeze the weights and first low-rank factorization matrix of the base model;

[0058] The first input text is sampled according to a sampling ratio by random sampling to form a playback sample, the sampling ratio range is 10%-30%, and the sampling ratio can be adjusted according to specific circumstances;

[0059] A second input text is collected according to the task purpose of the second low-rank decomposition matrix;

[0060] The second input text and the playback sample are preprocessed to generate a second input vector;

[0061] According to the equation Compute the second output vector where is the second output vector, is the second input vector, is the original weight matrix of the base model, is the first low-rank decomposition matrix, is the second low-rank decomposition matrix;

[0062] A second loss function is constructed to optimize the parameters of the first low-rank decomposition matrix.

[0063] Using the above method, we add playback samples to force recall of auxiliary task knowledge from the data level.

[0064] In order to solve the problem of poor stability against instruction diversity and output fluctuations due to differences in instruction expressions, an implementation method is provided in this embodiment.

[0065] In one embodiment, the first loss function The formula is ,in To minimize the negative log-likelihood loss, To supervise the contrast loss, is the weight of the supervised contrast loss term.

[0066] By minimizing the negative log-likelihood loss Construct the basic loss function. For the auxiliary task learning process, the training loss function represents the pre-trained model weights, Indicates the change of the first low-rank decomposition matrix during the parameter update process. For the target task learning process, the training loss function represents the pre-trained model weights and the first low-rank decomposition matrix weights, Represents the change of the second low-rank decomposition matrix during the parameter update process.

[0067] Using the above method, contrastive loss enables the model to map semantically identical instructions to similar representation spaces, improves the model's tolerance to interfering instructions, and reduces extraction errors caused by differences in user instruction representations.

[0068] In order to solve the problems of forgetting old tasks, aligning feature spaces, and distinguishing new task instructions, an implementation method is provided in this embodiment.

[0069] In one embodiment, the second loss function The formula is ,in To minimize the negative log-likelihood loss, To supervise the contrast loss, is the weight of the supervised contrast loss term, is the playback sample loss, is the distillation loss, is the weight of the distillation loss term.

[0070] The knowledge distillation technology is applied to the target task (i.e., the second low-rank decomposition matrix) learning phase to alleviate the limitation of reduced effectiveness of cross-stage knowledge transfer. The fixed auxiliary task model weights (i.e., the first low-rank decomposition matrix) are regarded as the teacher model, and the continuously updated target task model weights (i.e., the second low-rank decomposition matrix) are regarded as the student model. The difference in probability distribution between the new and old models is quantified through the distillation regularization term based on Kullback-Leibler (KL) divergence. Contrastive loss is introduced for each of the two training phases to alleviate the catastrophic forgetting problem in the continuous learning process. By constructing a positive sample set of input examples and negative sample set The current input is defined as the anchor sample. Triplet loss is used to keep positive pairs close to each other in feature space and negative pairs farther apart. The contrastive loss designed in this paper improves the model's sensitivity to and ability to follow instructions from the perspective of embedded representation encoding, thereby alleviating confusion about past input instructions and ensuring that the model accurately outputs extraction results as required by the instructions.

[0071] Using this method, the replay loss preserves the performance of the auxiliary task, the distillation loss constrains the new model output to approach the teacher model (old knowledge), and the contrastive loss enhances the discriminability of task instructions. The four losses work synergistically to speed up the learning of the target task.

[0072] In order to solve the problem that traditional contrastive learning does not consider the logical relevance of cross-task instructions, resulting in instruction confusion during task switching, an implementation method is provided in this embodiment.

[0073] In one embodiment, the supervised contrastive loss The equation is ,in is the triplet loss, is the sample anchor point, represents one of the positive samples, represents one of the negative samples, is a marginal value greater than zero, Update the parameters for the model; supervised contrast loss is introduced to quantify the distance between two input example instructions. ,in is the embedding vector of the instruction, represents the L2 norm.

[0074] Using this method, the triplet loss explicitly brings together cross-task instructions with consistent outputs and pushes away irrelevant instructions; the model can seamlessly switch between task instructions, and the error rate caused by instruction confusion is reduced.

[0075] In order to solve the problem that simple data playback is difficult to solve the feature space offset and distillation lacks constraints on the original samples, an implementation method is provided in this embodiment.

[0076] In one embodiment, playback sample loss The equation is ,in are the weights of the base model and the first low-rank decomposition matrix, is the parameter updated on the second low-rank decomposition matrix, is the playback sample set size, Input vector for the i-th sample; distillation loss The equation is ,in is the probability distribution of the base model and the first low-rank decomposition matrix weights, is the probability distribution of the second low-rank decomposition matrix weights, is the output sequence length.

[0077] Using the above method, the average distance between new and old tasks in the representation space is effectively reduced, proving that feature degradation is effectively suppressed.

[0078] The following describes the contents involved in the above embodiment in conjunction with a preferred embodiment.

[0079] To verify the effectiveness of this invention, a case study was conducted on a standard general information extraction benchmark. The standard general information extraction benchmark covers 25 named entity recognition datasets, 10 relationship extraction datasets, and 3 event extraction datasets, totaling 200 entity categories, 81 relationship categories, and 40 event categories. Each dataset is divided into training, validation, and test sets. To ensure balanced samples in the corpus, a random sampling strategy is adopted, so that each dataset contains a maximum of 10,000 instances.

[0080] The model used is the flan-t5-xxl model, and the metric used for evaluation is Micro-F1 based on span shift, with a contrastive loss weight of 0.2 and a distillation loss weight of 0.7.

[0081] On the standard general information extraction benchmark, the flan-t5-xxl model achieved accuracies of 87.89%, 69.04%, and 72.97% on named entity recognition, relation extraction, and event extraction tasks, respectively.

[0082] Furthermore, in order to explore the effectiveness of the learning strategy of the present invention, the present invention also conducted an ablation experiment based on the flan-t5-large model. On the standard general information extraction benchmark, the continuous learning model with contrastive loss and distillation loss constraints surpassed the traditional continuous learning method with F1 scores of 3.22%, 4.27%, 5.94% and 6.79% in named entity recognition, relation extraction, event trigger word extraction and event argument extraction tasks, respectively, fully verifying the effectiveness of the present invention.

[0083] Based on the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0084] An electronic device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the general information extraction method based on contrast supervision and cross-stage distillation provided by the present disclosure.

[0085] Electronic device is intended to refer to various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device may also refer to various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are intended to be examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0086] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the general information extraction method based on contrastive supervision and cross-stage distillation provided by the present disclosure.

[0087] Various embodiments of the present disclosure may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which may be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0088] A computer program product includes a computer program / instruction, which, when executed by a processor, provides a general information extraction method based on contrastive supervision and cross-stage distillation provided by the present disclosure.

[0089] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0090] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0091] The embodiments of the present invention are described in detail above, but the contents described are only preferred embodiments of the present invention and should not be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A general information extraction method based on contrastive supervision and cross-stage distillation, characterized in that: The following steps are involved: Select a pre-trained model as the base model and initialize it; Initializing a first low-rank decomposition matrix and a second low-rank decomposition matrix for the weight matrix of the base model; Training the first low-rank decomposition matrix and the second low-rank decomposition matrix; Combining the weight of the base model, the first low-rank decomposition matrix, and the second low-rank decomposition matrix to obtain an inference model; inputting natural language text information into the inference model; Specify the task type and output format through natural language instructions, and output the structured information extraction results; Training the first low-rank decomposition matrix includes the following steps: freezing the weights of the base model; A first input text is collected according to the task purpose of the first low-rank decomposition matrix; The first input text is preprocessed to generate a first input vector; inputting the first input vector into the base model; According to the equation Compute the first output vector where is the first output vector, is the first input vector, is the original weight matrix of the base model, is the first low-rank decomposition matrix; Constructing a first loss function to optimize the parameters of the first low-rank decomposition matrix; Training the second low-rank decomposition matrix includes the following steps: Freezing the weights of the base model and the first low-rank decomposition matrix; The first input text is sampled according to a sampling ratio by a random sampling method to form a playback sample; A second input text is collected according to the task purpose of the second low-rank decomposition matrix; The second input text and the playback sample are preprocessed to generate a second input vector; According to the equation Compute the second output vector where is the second output vector, is the second input vector, is the second low-rank decomposition matrix; A second loss function is constructed to optimize the parameters of the first low-rank decomposition matrix.

2. The general information extraction method according to claim 1, characterized in that: The first loss function The formula is ,in To minimize the negative log-likelihood loss, To supervise the contrast loss, is the weight of the supervised contrast loss term.

3. The general information extraction method according to claim 1, wherein: The second loss function The formula is ,in To minimize the negative log-likelihood loss, To supervise the contrast loss, is the weight of the supervised contrast loss term, is the playback sample loss, is the distillation loss, is the weight of the distillation loss term.

4. The general information extraction method according to claim 3, characterized in that: The supervised contrastive loss The equation is ,in is the triplet loss, is the sample anchor point, represents one of the positive samples, represents one of the negative samples, is a marginal value greater than zero, Update the parameters for the model; the supervised contrast loss introduces the squared Euclidean distance to quantify the distance between the two input example instructions ,in is the embedding vector of the instruction, represents the L2 norm.

5. The general information extraction method according to claim 3, characterized in that: The playback sample loss The equation is ,in are the weights of the base model and the first low-rank decomposition matrix, is the parameter updated on the second low-rank decomposition matrix, is the playback sample set size, is the input vector of the i-th sample; the distillation loss The equation is ,in is the probability distribution of the base model and the first low-rank decomposition matrix weights, is the probability distribution of the second low-rank decomposition matrix weights, is the output sequence length.

6. An electronic device comprising: at least one processor; as well as a memory communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor. The instructions are executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 5.

7. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1 to 5.

8. A computer program product comprising a computer program / instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 5.

Images