A general information extraction method based on continuous low rank adaptation

Abstract

The application provides a general information extraction method based on continuous low rank adaptation, comprising selecting a pre-trained model as a base model and initializing; initializing at least two rank decomposition matrices for at least one weight matrix of the base model; training the first low rank decomposition matrix and the second low rank decomposition matrix respectively according to a hierarchical training mechanism; obtaining a corresponding inference model according to a hierarchical merging mechanism; inputting a natural language text and a natural language instruction into the second inference model; and outputting a structured information extraction result. The application can solve the problem that a low-level task enhancement method based on a multi-task learning strategy does not fully utilize the low-level task, thereby improving the performance of the model on the information extraction task; through a progressive training mechanism, simple tasks and complex tasks can be learned in order from simple to difficult; the model can further learn complex tasks on the basis of mastering simple task knowledge, thereby enhancing the learning effect.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of information processing, and particularly relates to a general information extraction method based on continuous low-rank adaptation. BACKGROUND

[0002] The general information extraction aims to realize a general model capable of simultaneously processing multiple information extraction tasks, including core subtasks such as named entity recognition, relation extraction and event extraction. The extraction method realizes the unified modeling of various information extraction tasks by constructing a unified link prediction framework according to a predefined structured extraction paradigm. However, such methods usually rely on fixed extraction paradigms and need to explicitly design the link relationship between labels, which makes them lack flexibility when dealing with complex information extraction tasks.

[0003] The existing generative general information extraction method based on multi-task instruction fine-tuning regards the task for downstream reasoning and evaluation as a target task, and enhances the learning process of the model by constructing a series of auxiliary tasks related to the target task. However, this method faces some challenges. First, the existing method only mixes training samples from different tasks in an unordered manner based on the multi-task learning strategy, ignoring the differences in difficulty and importance between auxiliary tasks and target tasks. Since there is no explicit modeling of the training order of tasks of different difficulties, if the model learns samples from the target task and auxiliary tasks in turn, it will be difficult for the model to fully utilize the knowledge learned from the auxiliary tasks to optimize the performance of the target task. At the same time, in the initial stage of target task training, the model may not have fully mastered the relevant capabilities of the auxiliary task, resulting in its inability to effectively utilize cross-task knowledge transfer to enhance the learning process. Therefore, there is an urgent need for a general information extraction method to solve the above problems. SUMMARY

[0004] To solve the above technical problems, the application provides a general information extraction method based on continuous low-rank adaptation, which is particularly suitable for processing multiple information extraction tasks.

[0005] The technical scheme adopted by the application is as follows: a general information extraction method based on continuous low-rank adaptation is provided, comprising the following steps:

[0006] A pre-trained model is selected as a base model and is initialized;

[0007] At least two rank decomposition matrices, including a first low-rank decomposition matrix and a second low-rank decomposition matrix, are initialized for at least one weight matrix of the base model;

[0008] The first low-rank decomposition matrix and the second low-rank decomposition matrix are trained according to a hierarchical training mechanism;

[0009] merge the weight of the base model, the first low-rank decomposition matrix and the second low-rank decomposition matrix according to a hierarchical merging mechanism to obtain a corresponding first inference model and a second inference model;

[0010] input the natural language instruction into the second inference model;

[0011] specify the task type and the output format by the natural language instruction, and output the structured information extraction result.

[0012] Further, the hierarchical training mechanism means that the high-level low-rank decomposition matrix and the low-level low-rank decomposition matrix comply with the accumulation characteristic to freeze the weight, calculate the output vector and determine the loss function.

[0013] Further, training the first low-rank decomposition matrix comprises the following steps:

[0014] freeze the weight of the base model;

[0015] collect a first input text according to the task purpose of the first low-rank decomposition matrix;

[0016] generate a first input vector after preprocessing the first input text;

[0017] input the first input vector into the base model;

[0018] calculate a first output vector according to the equation , wherein is the first output vector, is the first input vector, denotes the dimension of the first output vector, denotes the dimension of the first input vector, denotes a dimensional real space, is the original weight matrix of the base model, is the first low-rank decomposition matrix;

[0019] optimize the parameters of the first low-rank decomposition matrix based on minimizing the negative log-likelihood loss as the loss function.

[0020] Further, training the second low-rank decomposition matrix comprises the following steps:

[0021] freeze the weight of the base model and the first low-rank decomposition matrix;

[0022] collect a second input text according to the task purpose of the second low-rank decomposition matrix;

[0023] generate a second input vector after preprocessing the second input text;

[0024] inputting the second input vector into the base model;

[0025] computing a second output vector according to equation is the second output vector, is the second input vector, denotes a second output vector dimension, denotes a second input vector dimension, denotes a dimensional real space, is a raw weight matrix of the base model, is the first low-rank decomposition matrix, is the second low-rank decomposition matrix.

[0026] optimizing the parameters of the second low-rank decomposition matrix based on minimizing a negative log-likelihood loss as the loss function.

[0027] Further, the hierarchical merging mechanism means that the high-level low-rank decomposition matrix and the low-level low-rank decomposition matrix comply with the accumulation property for matrix merging to obtain the inference model.

[0028] The first inference model is obtained by merging the base model weight and the trained first low-rank decomposition matrix.

[0029] The second inference model is obtained by merging the base model weight, the trained first low-rank decomposition matrix, and the trained second low-rank decomposition matrix.

[0030] Further, the preprocessing method includes text cleaning, standardization, word segmentation, labeling, label mapping, and vector representation.

[0031] ​The present application has the advantages and positive effects that: due to the above technical scheme, the problem that the low-level task enhancement method based on the multi-task learning strategy at the present stage is not sufficient for the utilization of low-level tasks can be effectively solved, and the performance of the model on the general information extraction task is further improved; through the progressive training mechanism, simple tasks and complex tasks can be learned in turn according to the order from simple to difficult; the model can further learn complex tasks on the basis of mastering simple task knowledge comprehensively, and the learning effect is further enhanced; an extensible continuous learning framework is provided for unknown task types; the high-level model automatically inherits all low-level capabilities to form a pyramid knowledge structure; the freezing mechanism ensures that each layer of training is only focused on the task of this level, and parameter conflicts are completely avoided; the fixed basic capability makes the complex task training focus on the new characteristics, physically isolates the parameter update, and ensures that the basic task precision is maintained; the pre-merged model is accelerated compared with the dynamic calculation, and the shared base model weight is greatly reduced to reduce the storage occupation. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a flowchart of a general information extraction method according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] The present disclosure will be described more fully hereinafter with reference to the accompanying drawings, in which example embodiments of the present disclosure are shown. The technical solutions in the embodiments of the present disclosure will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present disclosure.

[0034] As shown in Figure 1 The present application provides a general information extraction method based on continuous low-rank adaptation, comprising the following steps:

[0035] S100, selecting a pre-trained model as a base model and initializing;

[0036] Select a suitable large language model and initialize, and specify the target task as named entity recognition (NER), relation extraction (RE), event trigger extraction (EET) and event argument extraction (EEA), and construct three types of auxiliary tasks for each target task, representing "extraction", "recognition" and "classification" capabilities. It should be noted that the target task is not limited to the above naming, and the target task is determined by the model training setting. Named entity recognition (NER) is used to identify specific categories of entities in text such as names, places, organizations, times, diseases, and drugs; relation extraction (RE) is used to identify semantic relationships between two entities in text (such as "treatment", "cause", "belong to"); Event trigger extraction (EET) is used to identify words that mark the occurrence of an event (such as "the company announced a new merger plan yesterday" in which the word "announce" can be considered a trigger word because it marks the occurrence of a declaration event); Event argument extraction (EEA) is used to identify entities or time information related to events.

[0037] S200, initialize at least two rank decomposition matrices for at least one weight matrix of the base model, the at least two rank decomposition matrices including a first low-rank decomposition matrix and a second low-rank decomposition matrix;

[0038] Query weight matrix of the Transformer attention head and value weight matrix Apply low-rank adaptation, where represents the query (Query), v represents the value (Value), the query weight matrix is used to transform the input vector to the query space, the value weight matrix is used to transform the input vector to the value space, the specific number of rank decomposition matrices is set according to actual conditions, the first level is the lowest level, the second level is higher than the first level, and other levels are higher than the second level, while maintaining compliance with the hierarchical training mechanism and the hierarchical merging mechanism.

[0039] S300, train the first low-rank decomposition matrix and the second low-rank decomposition matrix according to the hierarchical training mechanism;

[0040] S400, merge the weights of the base model, the first low-rank decomposition matrix and the second low-rank decomposition matrix according to the hierarchical merging mechanism, to obtain corresponding first reasoning model and second reasoning model;

[0041] For the case of multiple low-rank decomposition matrices, the hierarchical merging mechanism can obtain the corresponding reasoning model, that is, the high-level model has the set of all low-rank decomposition matrices, which embodies the scalability of the method.

[0042] S500, input the natural language instruction into the second reasoning model;

[0043] The natural language instruction includes an input sequence, an output format, a task target, and a predefined extraction mode.

[0044] S600, a task type and an output format are specified by a natural language instruction, and a structured information extraction result is output.

[0045] The format task description and the specified output format of the natural language instruction, a candidate type set, and a sentence to be extracted.

[0046] By using the above method, simple tasks and complex tasks are not interfered with each other in a unified framework through a hierarchical training mechanism; the optimal model is automatically selected according to the task type, and the waste of computing power caused by a complex model processing a simple task is avoided; an extensible hierarchical learning framework is provided for unknown task types.

[0047] In order to solve the problem that the existing LoRA technology cannot realize capability superposition, and the performance is reduced due to parameter conflict during multi-task training, an implementation manner is provided in the embodiment.

[0048] In an embodiment, the hierarchical training mechanism means that the low-rank decomposition matrix of the high level and the low-rank decomposition matrix of the low level comply with the accumulation characteristic to perform weight freezing, calculate an output vector, and determine a loss function.

[0049] By using the above method, the high-level model automatically inherits all low-level capabilities to form a pyramid knowledge structure; the freezing mechanism ensures that each layer of training focuses only on the task of this level, and completely avoids parameter conflict.

[0050] In order to solve the problem that a simple task (such as boundary annotation) needs lightweight adaptation, and the knowledge of the base model is easily destroyed, an implementation manner is provided in the embodiment.

[0051] In an embodiment, training the first low-rank decomposition matrix includes the following steps:

[0052] Freezing the weight of the base model;

[0053] The weight of the base model is frozen, only the first low-rank decomposition matrix is enabled for training, and after the training is completed, the first inference model is obtained.

[0054] The first input text is collected according to the task purpose of the first low-rank decomposition matrix;

[0055] The first input text generates a first input vector after preprocessing;

[0056] The first input vector is input into the base model;

[0057] According to the equation The first output vector is calculated, wherein is the first output vector, is a first input vector, denotes a first output vector dimension, denotes a first input vector dimension, denotes a one dimensional real space, is a base model's original weight matrix, is a first low-rank decomposition matrix;

[0058] The parameters of the first low-rank decomposition matrix are optimized based on minimizing a negative log-likelihood loss as a loss function.

[0059] The negative log-likelihood loss is minimized as a loss function, and the equation of the negative log-likelihood loss is where X represents an input text, I represents a corresponding instruction, denotes a frozen pre-training model weight, denotes an adjustable model parameter, denotes a negative log-likelihood loss, P is a conditional probability distribution, denotes a change amount of the low-rank adaptation module in the parameter updating process, denotes an output sequence length, and i represents an index of each word in the output sequence, denotes the i-th word of the output sequence, denotes the first i-1 words in the output sequence.

[0060] By using the above method, the frozen base weight avoids the loss of pre-training knowledge, and only a small amount of parameters needs to be trained to obtain the basic extraction capability, thereby providing a stable semantic understanding basis for high-level tasks.

[0061] In order to solve the problem that the low-level basic capability needs to be used in the training of a complex task, but the parameters of the basic capability cannot be modified, an implementation manner is provided in the embodiment.

[0062] In an embodiment, training the second low-rank decomposition matrix includes the following steps:

[0063] The weight of the base model and the trained first low-rank decomposition matrix are frozen;

[0064] That is, the first inference model is frozen, and the second low-rank decomposition matrix is enabled for training.

[0065] The second input text is collected according to the task purpose of the second low-rank decomposition matrix;

[0066] The second input text generates a second input vector after preprocessing;

[0067] The second input vector is input into the base model;

[0068] According to the equation computing a second output vector, wherein for the second output vector, for the second input vector, denotes a second output vector dimension, denotes a second input vector dimension, denotes a one dimensional real space, is a base model's original weight matrix, is a first low-rank decomposition matrix, is a second low-rank decomposition matrix.

[0069] The parameters of the second low-rank decomposition matrix are optimized based on minimizing a negative log-likelihood loss as a loss function.

[0070] By using the above method, the basic capability is fixed, the complex task training is focused on the new features, the parameter update is physically isolated, and the accuracy of the basic task is ensured to be maintained.

[0071] In order to solve the problem of increasing inference delay due to real-time calculation required by dynamic adaptation, and the problem of loading multiple models required by multi-task support, an implementation manner is provided in the embodiment.

[0072] In an embodiment, the hierarchical merging mechanism refers to that the high-level low-rank decomposition matrix and the low-level low-rank decomposition matrix comply with the accumulation property to perform matrix merging to obtain an inference model.

[0073] The first inference model is obtained by merging the base model weight and the trained first low-rank decomposition matrix.

[0074] The second inference model is obtained by merging the base model weight, the trained first low-rank decomposition matrix, and the trained second low-rank decomposition matrix.

[0075] By using the above method, the pre-merged model is accelerated compared with dynamic calculation, and the storage occupation of the shared base model weight is greatly reduced.

[0076] In order to solve the problem of cross-task error propagation caused by different task annotation formats and text noise, an implementation manner is provided in the embodiment.

[0077] In an embodiment, the preprocessing method includes text cleaning, standardization, word segmentation, annotation, annotation mapping, and vector representation.

[0078] By using the above method, the matrix training effectiveness of different types of tasks is greatly improved through annotation, and it is ensured that different task inputs are in the same semantic space.

[0079] In order to facilitate the use of the general information extraction method based on continuous low-rank adaptation provided by the present disclosure, the present disclosure further provides a general information extraction system based on continuous low-rank adaptation, comprising:

[0080] An initialization module is configured to load a base model and initialize the weight parameters thereof, initialize a low-rank decomposition matrix for the base model;

[0081] At least two task training modules are configured to train the corresponding low-rank decomposition matrix according to a hierarchical training mechanism;

[0082] A model merging module is configured to merge the base model weight and the low-rank decomposition matrix according to a hierarchical merging mechanism, and generate an inference model;

[0083] An inference execution module is configured to input the natural language text into the corresponding inference model according to the classification, and output a structured information extraction result.

[0084] With the above arrangement, different task modules can be enabled as needed, greatly reducing idle resources; from training to inference, end-to-end processing reduces the manual intervention link.

[0085] The content involved in the above embodiments will be described below in conjunction with a preferred embodiment.

[0086] To verify the effectiveness of the present application, an example verification was performed on the standard general information extraction benchmark. The standard general information extraction benchmark covers 25 named entity recognition datasets, 10 relation extraction datasets, and 3 event extraction datasets, a total of 200 entity categories, 81 relation categories, and 40 event categories. Each dataset is divided into a training set, a validation set, and a test set. To ensure the balance of samples in the corpus, a random sampling strategy is used, so that each dataset contains at most 10,000 instances.

[0087] The model used is the flan-t5-xxl model, and the index used for evaluation is the span-based Micro-F1. On the standard general information extraction benchmark, the flan-t5-xxl model achieved an accuracy of 86.11%, 67.30%, and 70.15% on the named entity recognition, relation extraction, and event extraction tasks, respectively.

[0088] An additional ablation experiment was performed based on the flan-t5-large model. On the standard general information extraction benchmark, the two-stage continuous learning model outperformed the single-stage multi-task learning model by 1.21%, 0.57%, 1.25%, and 1.98% F1 scores on the named entity recognition, relation extraction, event trigger extraction, and event argument extraction tasks, respectively.

[0089] Based on the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0090] An electronic device includes at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the general information extraction method based on persistent low-rank adaptation provided by the present disclosure.

[0091] The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0092] A non-transitory computer readable storage medium storing computer instructions, wherein the computer instructions are used to make a computer perform the general information extraction method based on persistent low-rank adaptation provided by the present disclosure.

[0093] The various implementations in the present disclosure can be implemented in digital electronic circuitry, integrated circuitry, field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), system on a chip systems (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0094] A computer program product includes computer programs / instructions, which, when executed by a processor, perform the general information extraction method based on persistent low-rank adaptation provided by the present disclosure.

[0095] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0096] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0097] The above detailed description of the embodiments of the present application has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the application to the precise form disclosed. Many modifications and variations will be apparent to practitioners skilled in this art. Embodiments were chosen and described in order to best explain the principles of the application and its best mode, and to enable others skilled in the art to best utilize the application.

Claims

1. A general information extraction method based on continuous low-rank adaptation, characterized in that, The method comprises the following steps: selecting a pre-trained model as a base model and initializing the base model; initializing at least two rank decomposition matrices for at least one weight matrix of the base model, the at least two rank decomposition matrices comprising a first low-rank decomposition matrix and a second low-rank decomposition matrix; training the first low-rank decomposition matrix and the second low-rank decomposition matrix according to a hierarchical training mechanism; training the first low-rank decomposition matrix comprises the following steps: freezing the weights of the base model; collecting first input texts according to the task purpose of the first low-rank decomposition matrix; generating a first input vector after preprocessing the first input texts; inputting the first input vector into the base model; According to equation computing a first output vector, wherein is the first output vector, is the first input vector, denotes the first output vector dimension, denotes the first input vector dimension, denotes a dimensional real space, is the original weight matrix of the base model, is the first low-rank decomposition matrix; optimizing the parameters of the first low-rank decomposition matrix based on minimizing the negative log-likelihood loss as a loss function; training the second low-rank decomposition matrix comprises the following steps: freezing the weights of the base model and the first low-rank decomposition matrix; collecting second input texts according to the task purpose of the second low-rank decomposition matrix; generating a second input vector after preprocessing the second input texts; inputting the second input vector into the base model; According to equation computing a second output vector, wherein is the second output vector, is the second input vector, is the second input vector, denotes the second output vector dimension, denotes the second input vector dimension, denotes a dimensional real space, is the original weight matrix of the base model, is the first low-rank decomposition matrix, is the second low-rank decomposition matrix; optimizing the parameters of the second low-rank decomposition matrix based on minimizing the negative log-likelihood loss as the loss function; merging the weights of the base model, the first low-rank decomposition matrix and the second low-rank decomposition matrix according to a hierarchical merging mechanism to obtain corresponding first inference models and second inference models; inputting natural language text information into the second inference model; specifying the task type and output format through natural language instructions to output structured information extraction results; the hierarchical training mechanism refers to that the high-level low-rank decomposition matrix and the low-level low-rank decomposition matrix comply with the accumulation characteristic to freeze the weights, calculate the output vector and determine the loss function; the hierarchical merging mechanism refers to that the high-level low-rank decomposition matrix and the low-level low-rank decomposition matrix comply with the accumulation characteristic to merge the matrices to obtain the inference model; the first inference model is obtained by merging the base model weights and the trained first low-rank decomposition matrix; the second inference model is obtained by merging the base model weights, the trained first low-rank decomposition matrix and the trained second low-rank decomposition matrix.

2. The general information extraction method of claim 1, wherein: The preprocessing method comprises text cleaning, standardization, word segmentation, labeling, label mapping and vector representation.

Images