General information extraction method based on continuous low-rank adaptation

Abstract

The invention provides a general information extraction method based on continuous low-rank adaptation. The general information extraction method comprises the steps of selecting a pre-trained model as a base model and performing initialization; initializing at least two rank decomposition matrixes for at least one weight matrix of the pedestal model; respectively training the first low-rank decomposition matrix and the second low-rank decomposition matrix according to a hierarchical training mechanism; obtaining a corresponding reasoning model according to a hierarchical merging mechanism; inputting the natural language text and the natural language instruction into a second inference model; and outputting a structured information extraction result. The method has the beneficial effects that the problem that a low-level task is insufficiently utilized by a low-level task enhancement method based on a multi-task learning strategy can be solved, so that the performance of a model on an information extraction task is improved; through a progressive training mechanism, the simple tasks and the complex tasks can be learned in sequence from simple to difficult; the model can further learn complex tasks on the basis of mastering simple task knowledge, so that the learning effect is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of information processing, and in particular relates to a general information extraction method based on continuous low-rank adaptation. Background Art

[0002] General information extraction aims to implement a general model that can handle multiple information extraction tasks simultaneously, including core subtasks such as named entity recognition, relation extraction, and event extraction. The extraction method, based on a predefined structured extraction paradigm, realizes unified modeling of various information extraction tasks by constructing a unified link prediction framework. However, such methods usually rely on a fixed extraction paradigm and require explicit design of the link relationships between tags, which makes them lack flexibility in dealing with information extraction tasks with complex structures.

[0003] Existing generative general information extraction methods based on multi-task instruction fine-tuning regard the tasks for downstream inference and evaluation as target tasks, and enhance the learning process of the model by constructing a series of auxiliary tasks related to the target tasks. However, this method faces some challenges. First, the existing methods only mix the training samples from different tasks disorderly based on the multi-task learning strategy, ignoring the differences in difficulty and importance between the auxiliary tasks and the target task. Due to the lack of explicit modeling of the training order of tasks with different difficulties, if the model learns the samples from the target task and the auxiliary tasks in sequence, it will be difficult for the model to fully utilize the knowledge learned from the auxiliary tasks to optimize the performance of the target task. At the same time, in the initial stage of training the target task, the model may not have fully mastered the relevant capabilities of the auxiliary tasks, resulting in its inability to effectively utilize cross-task knowledge transfer to enhance the learning process. Therefore, there is an urgent need for a general information extraction method to solve the above problems. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a general information extraction method based on continuous low-rank adaptation, which is particularly suitable for processing multiple information extraction tasks.

[0005] The technical solution adopted by the present invention is: to provide a general information extraction method based on continuous low-rank adaptation, including the following steps:

[0006] Select a pre-trained model as the base model and initialize it;

[0007] Initialize at least two rank decomposition matrices for at least one weight matrix of the base model, and the at least two rank decomposition matrices include a first low-rank decomposition matrix and a second low-rank decomposition matrix;

[0008] Train the first low-rank decomposition matrix and the second low-rank decomposition matrix respectively according to a hierarchical training mechanism;

[0009] Merge the weights of the base model, the first low-rank decomposition matrix, and the second low-rank decomposition matrix according to the hierarchical merging mechanism to obtain the corresponding first inference model and second inference model;

[0010] Input the natural language instruction into the second inference model;

[0011] Specify the task type and output format through the natural language instruction, and output the structured information extraction result.

[0012] Furthermore, the hierarchical training mechanism means that the high-level low-rank decomposition matrix and the low-level low-rank decomposition matrix follow the accumulation property to freeze weights, calculate the output vector, and determine the loss function.

[0013] Furthermore, training the first low-rank decomposition matrix includes the following steps:

[0014] Freeze the weights of the base model;

[0015] Collect the first input text according to the task purpose of the first low-rank decomposition matrix;

[0016] The first input text generates a first input vector after preprocessing;

[0017] Input the first input vector into the base model;

[0018] According to the equation Calculate the first output vector, where is the first output vector, is the first input vector, represents the dimension of the first output vector, represents the dimension of the first input vector, represents a dimensional real number space, is the original weight matrix of the base model, is the first low-rank decomposition matrix;

[0019] Optimize the parameters of the first low-rank decomposition matrix based on minimizing the negative log-likelihood loss as the loss function.

[0020] Furthermore, training the second low-rank decomposition matrix includes the following steps:

[0021] Freeze the weights of the base model and the first low-rank decomposition matrix;

[0022] Collect the second input text according to the task purpose of the second low-rank decomposition matrix;

[0023] The second input text generates a second input vector after preprocessing;

[0024] Input the second input vector into the base model;

[0025] According to the equation calculate the second output vector, where is the second output vector, is the second input vector, represents the dimension of the second output vector, represents the dimension of the second input vector, represents a dimensional real number space, is the original weight matrix of the base model, is the first low-rank decomposition matrix, is the second low-rank decomposition matrix;

[0026] Optimize the parameters of the second low-rank decomposition matrix based on minimizing the negative log-likelihood loss as the loss function.

[0027] Furthermore, the hierarchical merging mechanism refers to the high-level low-rank decomposition matrix and the low-level low-rank decomposition matrix complying with the accumulation property to perform matrix merging to obtain the inference model;

[0028] The first inference model is obtained by merging the weights of the base model and the trained first low-rank decomposition matrix;

[0029] The second inference model is obtained by merging the weights of the base model, the trained first low-rank decomposition matrix, and the trained second low-rank decomposition matrix.

[0030] Furthermore, the preprocessing method includes text cleaning, standardization, word segmentation, annotation, annotation mapping, and vectorized representation.

[0031] The advantages and positive effects of the present invention are: due to the adoption of the above-mentioned technical scheme, it can effectively solve the problem of insufficient utilization of low-level tasks by the current low-level task enhancement method based on multi-task learning strategy, thereby improving the performance of the model in general information extraction tasks; through the progressive training mechanism, simple tasks and complex tasks can be learned in sequence from simple to difficult; the model can further learn complex tasks on the basis of fully mastering the knowledge of simple tasks, thereby enhancing the learning effect; a scalable continuous learning framework is provided for unknown task types; high-level models automatically inherit all low-level capabilities to form a pyramid-like knowledge structure; the freezing mechanism ensures that each layer of training only focuses on the tasks at this level, and parameter conflicts are completely avoided; the basic capabilities are fixed, so that the training of complex tasks focuses on new features, and the parameter updates are physically isolated to ensure that the accuracy of basic tasks is maintained; the pre-merged model is faster than the dynamic calculation, and the shared base model weights greatly reduce storage occupancy. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flowchart of a general information extraction method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present disclosure is described more fully below with reference to the accompanying drawings, in which exemplary embodiments of the present disclosure are described. The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0034] like Figure 1 As shown, the present invention provides a general information extraction method based on continuous low-rank adaptation, comprising the following steps:

[0035] S100, selecting a pre-trained model as a base model and initializing it;

[0036] Select a suitable large language model and initialize it. Define the target tasks as named entity recognition (NER), relation extraction (RE), event trigger extraction (EET), and event argument extraction (EEA), and construct three types of auxiliary tasks for each target task, representing "extraction", "recognition", and "classification" capabilities respectively. It should be noted that the target tasks are not limited to the above names and are determined by the settings of model training. Named entity recognition (NER) is used to identify entities of specific categories such as person names, place names, organization names, time, diseases, drugs, etc. in the text; relation extraction (RE) is used to identify the semantic relationships (such as "treatment", "causing", "belonging") between two entities in the text; event trigger extraction (EET) is used to identify the words that mark the occurrence of events in the text (for example, in the sentence "The company announced a new merger and acquisition plan yesterday", the word "announced" can be regarded as a trigger word because it marks the occurrence of an announcement event); event argument extraction (EEA) is used to identify information such as entities or times related to the event.

[0037] S200. Initialize at least two rank decomposition matrices for at least one weight matrix of the base model. The at least two rank decomposition matrices include a first low-rank decomposition matrix and a second low-rank decomposition matrix;

[0038] Apply low-rank adaptation to the query weight matrix and the value weight matrix of the Transformer attention head, where represents the query, v represents the value, the query weight matrix is used to transform the input vector into the query space, and the value weight matrix is used to transform the input vector into the value space. The specific number of rank decomposition matrices is set according to the actual situation. The first level is the lowest level, the second level is higher than the first level, and so on for other levels, while adhering to the hierarchical training mechanism and the hierarchical merging mechanism.

[0039] S300. Train the first low-rank decomposition matrix and the second low-rank decomposition matrix respectively according to the hierarchical training mechanism;

[0040] S400. Merge the weights of the base model, the first low-rank decomposition matrix, and the second low-rank decomposition matrix according to the hierarchical merging mechanism to obtain the corresponding first inference model and second inference model;

[0041] For the case of multiple low-rank decomposition matrices, following the hierarchical merging mechanism can obtain the corresponding inference models, that is, the high-level models have the set of all previous low-rank decomposition matrices, reflecting the scalability of this method.

[0042] S500. Input the natural language instruction into the second inference model;

[0043] The natural language instructions include input sequence, output format, task goal, and predefined extraction pattern.

[0044] S600: Specify the task type and output format through natural language instructions, and output the structured information extraction result.

[0045] The format of natural language instructions, task description and specified output format, candidate type set, and sentences to be extracted.

[0046] By adopting the above method, through the hierarchical training mechanism, simple tasks and complex tasks will not interfere with each other under the same framework; the optimal model is automatically selected according to the task type to avoid the waste of computing power caused by complex models processing simple tasks; and a scalable hierarchical learning framework is provided for unknown task types.

[0047] In order to solve the problem that the existing LoRA technology cannot achieve capability superposition and parameter conflicts during multi-task training lead to performance degradation, an implementation method is provided in this embodiment.

[0048] In one embodiment, the hierarchical training mechanism refers to that the high-level low-rank decomposition matrix and the low-level low-rank decomposition matrix comply with the accumulation characteristic to freeze the weights, calculate the output vector and determine the loss function.

[0049] Using the above method, the high-level model automatically inherits all the low-level capabilities to form a pyramid-like knowledge structure; the freezing mechanism ensures that each layer of training only focuses on the tasks at this level, completely avoiding parameter conflicts.

[0050] In order to solve the problem that simple tasks (such as boundary annotation) require lightweight adaptation and the base model knowledge is easily destroyed, an implementation method is provided in this embodiment.

[0051] In one embodiment, training the first low-rank decomposition matrix comprises the following steps:

[0052] Freeze the weights of the base model;

[0053] Freeze the weights of the base model and only enable the first low-rank decomposition matrix for training. After the training is completed, it becomes the first inference model.

[0054] A first input text is collected according to the task purpose of the first low-rank decomposition matrix;

[0055] The first input text is preprocessed to generate a first input vector;

[0056] Inputting a first input vector into the base model;

[0057] According to the equation Compute the first output vector where is the first output vector, is the first input vector, represents the dimension of the first output vector, represents the dimension of the first input vector, represents a dimensional real number space, is the original weight matrix of the base model, is the first low-rank decomposition matrix;

[0058] Optimize the parameters of the first low-rank decomposition matrix based on minimizing the negative log-likelihood loss as the loss function.

[0059] Minimize the negative log-likelihood loss as the loss function, and the equation for minimizing the negative log-likelihood loss is where X represents the input text, I represents the corresponding instruction, represents the frozen pre-trained model weights, represents the adjustable model parameters, represents the negative log-likelihood loss, P is the conditional probability distribution, represents the change amount of the low-rank adaptation module during the parameter update process, represents the output sequence length, and i represents the index of each word in the output sequence, represents the i-th word of the output sequence, represents the first i - 1 words in the output sequence.

[0060] Using the above method, freezing the base weights can avoid the loss of pre-trained knowledge, and only training a small number of parameters can obtain the basic extraction ability, providing a stable semantic understanding basis for high-level tasks.

[0061] In order to solve the problem that in the training of complex tasks, the basic capabilities of the lower levels need to be utilized, but the parameters of the basic capabilities cannot be modified, an implementation method is provided in this embodiment.

[0062] In one embodiment, training the second low-rank decomposition matrix includes the following steps:

[0063] Freeze the weights of the base model and the trained first low-rank decomposition matrix;

[0064] That is, freeze the first inference model and enable the second low-rank decomposition matrix for training.

[0065] Collect the second input text according to the task purpose of the second low-rank decomposition matrix;

[0066] The second input text is preprocessed to generate a second input vector;

[0067] Input the second input vector into the base model;

[0068] According to the equation Calculate the second output vector, where is the second output vector, is the second input vector, represents the dimension of the second output vector, represents the dimension of the second input vector, represents a dimensional real space, is the original weight matrix of the base model, is the first low-rank decomposition matrix, is the second low-rank decomposition matrix;

[0069] Optimize the parameters of the second low-rank decomposition matrix based on minimizing the negative log-likelihood loss as the loss function.

[0070] Adopt the above method, fix the basic capabilities, make the complex task training focus on the new features, physically isolate the parameter updates, and ensure that the accuracy of the basic tasks is maintained.

[0071] To solve the problem that dynamic adaptation requires real-time calculation to increase the inference latency and multi-task support requires loading multiple models, an implementation method is provided in this embodiment.

[0072] In one embodiment, the hierarchical merging mechanism means that the high-level low-rank decomposition matrix and the low-level low-rank decomposition matrix comply with the accumulation property to perform matrix merging to obtain the inference model;

[0073] The first inference model is obtained by merging the weights of the base model and the trained first low-rank decomposition matrix;

[0074] The second inference model is obtained by merging the weights of the base model, the trained first low-rank decomposition matrix, and the trained second low-rank decomposition matrix.

[0075] Adopt the above method, the pre-merged model is faster than dynamic calculation, and sharing the weights of the base model greatly reduces the storage occupancy.

[0076] To solve the problems of different task annotation format conflicts and text noise causing cross-task error propagation, an implementation method is provided in this embodiment.

[0077] In one embodiment, the preprocessing methods include text cleaning, standardization, word segmentation, annotation, annotation mapping, and vector representation.

[0078] Adopt the above method, the effectiveness of matrix training for different types of tasks is greatly improved through annotation, ensuring that different task inputs are in the same semantic space.

[0079] To facilitate the use of a general information extraction method based on continuous low-rank adaptation provided by the present disclosure, the present disclosure also provides a general information extraction system based on continuous low-rank adaptation, including:

[0080] An initialization module, configured to load a base model and initialize its weight parameters, and initialize a low-rank decomposition matrix for the base model;

[0081] At least two task training modules, configured to train corresponding low-rank decomposition matrices according to a hierarchical training mechanism;

[0082] A model merging module, configured to merge the base model weights and the low-rank decomposition matrix according to a hierarchical merging mechanism to generate an inference model;

[0083] An inference execution module, configured to input natural language text into the corresponding inference model according to classification and output a structured information extraction result.

[0084] With the above settings, different task modules can be enabled as needed, significantly reducing idle resources; end-to-end processing from training to inference reduces the manual intervention link.

[0085] Next, in combination with a preferred embodiment, the content involved in the above embodiment will be described.

[0086] To verify the effectiveness of the present invention, an example verification was carried out on a standard general information extraction benchmark. The standard general information extraction benchmark covers 25 named entity recognition datasets, 10 relation extraction datasets, and 3 event extraction datasets, totaling 200 entity categories, 81 relation categories, and 40 event categories. Among them, each dataset is divided into a training set, a validation set, and a test set. To ensure the balance of samples in the corpus, a random sampling strategy is adopted, so that each dataset contains at most 10,000 instances.

[0087] The model used is the flan-t5-xxl model, and the metric used for evaluation is Micro-F1 based on span offset. On the standard general information extraction benchmark, the flan-t5-xxl model achieved accuracies of 86.11%, 67.30%, and 70.15% in the named entity recognition, relation extraction, and event extraction tasks, respectively.

[0088] An ablation experiment was additionally carried out based on the flan-t5-large model. On the standard general information extraction benchmark, the two-stage continuous learning model exceeded the single-stage multi-task learning model by 1.21%, 0.57%, 1.25%, and 1.98% in the named entity recognition, relation extraction, event trigger word extraction, and event argument extraction tasks, respectively, in terms of F1 score.

[0089] Based on the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0090] An electronic device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the general information extraction method based on continuous low-rank adaptation provided by the present disclosure.

[0091] The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0092] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the general information extraction method based on continuous low-rank adaptation provided by the present disclosure.

[0093] The various embodiments in the present disclosure can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, the programmable processor can be a dedicated or general-purpose programmable processor, and can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0094] A computer program product includes computer programs / instructions that, when executed by a processor, execute the general information extraction method based on continuous low-rank adaptation provided by the present disclosure.

[0095] The program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, a special purpose computer, or other programmable data processing device, such that the program codes, when executed by the processor or controller, cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0096] In the context of the present disclosure, a machine-readable medium may be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0097] The above has described the embodiments of the present invention in detail, but the above content is only the preferred embodiments of the present invention and should not be considered as limiting the scope of implementation of the present invention. All equivalent changes and improvements made in accordance with the scope of the present invention application should still fall within the scope covered by the patent of the present invention.

Claims

1. A general information extraction method based on continuous low-rank adaptation, characterized in that, It includes the following steps: Select a pre-trained model as the base model and initialize it; Initialize at least two rank decomposition matrices for at least one weight matrix of the base model, and the at least two rank decomposition matrices include a first low-rank decomposition matrix and a second low-rank decomposition matrix; Train the first low-rank decomposition matrix and the second low-rank decomposition matrix respectively according to a hierarchical training mechanism; Merge the weights of the base model, the first low-rank decomposition matrix and the second low-rank decomposition matrix according to a hierarchical merging mechanism to obtain corresponding first and second inference models; Input natural language text information into the second inference model; Specify the task type and output format through natural language instructions, and output the structured information extraction result.

2. The general information extraction method according to claim 1, wherein: The hierarchical training mechanism means that the high-level low-rank decomposition matrix and the low-level low-rank decomposition matrix follow the accumulation property to freeze weights, calculate output vectors and determine loss functions.

3. The general information extraction method according to claim 2, wherein Training the first low-rank decomposition matrix includes the following steps: Freeze the weights of the base model; Collect a first input text according to the task purpose of the first low-rank decomposition matrix; The first input text is preprocessed to generate a first input vector; Input the first input vector into the base model; According to the equation calculate the first output vector, where is the first output vector, is the first input vector, represents the dimension of the first output vector, represents the dimension of the first input vector, represents a dimensional real space, is the original weight matrix of the base model, is the first low-rank decomposition matrix; Optimize the parameters of the first low-rank decomposition matrix based on minimizing the negative log-likelihood loss as the loss function.

4. The general information extraction method according to claim 3, wherein Training the second low-rank decomposition matrix includes the following steps: Freeze the weights of the base model and the first low-rank decomposition matrix; Collect a second input text according to the task purpose of the second low-rank decomposition matrix; The second input text is preprocessed to generate a second input vector; Input the second input vector into the base model; According to the equation calculate the second output vector, where is the second output vector,[ is the second input vector,[ represents the dimension of the second output vector,[ represents the dimension of the second input vector,[ represents a dimensional real space,[ is the original weight matrix of the base model,[ is the first low-rank decomposition matrix,[ is the second low-rank decomposition matrix;[ Optimize the parameters of the second low-rank decomposition matrix based on minimizing the negative log-likelihood loss as the loss function.

5. The general information extraction method according to claim 1, characterized in that: The hierarchical merging mechanism means that the high-level low-rank decomposition matrix and the low-level low-rank decomposition matrix follow the accumulation property to merge matrices to obtain an inference model; The first inference model is obtained by merging the weights of the base model and the trained first low-rank decomposition matrix; The second inference model is obtained by merging the weights of the base model, the trained first low-rank decomposition matrix and the trained second low-rank decomposition matrix.

6. The general information extraction method according to claim 3 or 4, characterized in that: The preprocessing method includes text cleaning, standardization, word segmentation, annotation, annotation mapping and vectorized representation.

Images