Generative information extraction method and device based on pre-training model

Abstract

The present application discloses a generative information extraction method and device based on a pre-trained model, the method comprising: obtaining historical annotation data of a structured information extraction task; generating a multi-round question template for scholars according to the dependency relationship between each target field, and processing the annotation data into a data set in the form of a multi-round question and answer; serializing the multi-round question template, and fine-tuning the generative training model in combination with the data set in the form of a multi-round question and answer; inputting the text to be extracted into the fine-tuned generative training model, and generating the answer to the field corresponding to each round of questions in the multi-round question template through the fine-tuned generative training model; forming the answer to each round of questions into structured data, and evaluating the model. The present invention fully considers the dependency relationship between different fields, reduces the dependence on entities through the generative model, and can accurately and quickly extract structured information.

Description

Technical Field

[0001] The present application relates to the technical field of information extraction, and in particular to a generative information extraction method and device based on a pre-trained model. Background Art

[0002] At present, the information resources on the Internet are rich, often unstructured and diverse in type, making it difficult to accurately retrieve relevant record information. For example, a certain type of event can be classified according to the location where it occurred, which cannot be accomplished by search engines. Therefore, there is an urgent need for technical means to quickly and accurately obtain information, and thus information extraction technology has emerged. Taking the academic circle as an example, there are hundreds of millions of experts and scholars in the world, and most of these expert and scholar data are presented on the Internet in heterogeneous and unstructured forms. These data contain a large amount of valuable data, including: basic information (such as email, title, work unit, etc.), educational experience (such as school, academic qualifications, etc.), and work resume (such as work unit, title, etc.). However, due to the different sources of data and unstructured storage, it is difficult to directly construct a multi-precision talent semantic portrait of a scholar, so as to meet the needs of intelligent talent analysis under various scenarios and data dimensions. Therefore, how to automatically, accurately and quickly extract valuable information from massive scattered and unstructured data and store it in a tabular format has become a hot issue of concern in academia and industry.

[0003] In the related art, text information is usually extracted using rule / regular-based text matching methods and sequence annotation-based deep learning methods. However, the applicant found that in actual scenarios, there is an entity that is a field in multiple records. For example, a person can study in different schools at different times, or study for different degrees in the same school at different times. This phenomenon will cause multiple records to appear in the same sentence, and the fields of multiple records share the same entity. This phenomenon is difficult to extract from normal text. The text extraction method in the related art cannot consider the dependency between different field information in the same experience, and it is difficult to capture syntactic and semantic information. In addition, traditional extractive question and answering relies heavily on the location information of each entity mention, and the effect is poor when the entity span is not considered. Therefore, there is an urgent need for a method that can consider syntactic and semantic information and integrate the dependency between fields to achieve structured information extraction. Summary of the invention

[0004] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, the first purpose of this application is to propose a generative information extraction method for a pre-trained model. The method regards the information extraction task as a multi-round generative question-answering task. By considering the dependency between fields and designing the question module, the dependency between different fields can be captured through multi-round question-answering. Then, the corresponding field information is extracted using a generative pre-trained model using the given context text and questions. This method can be applied to situations where multiple fields overlap or a field is missing. Generating answers through a generative model reduces the dependence on entity spans and improves the accuracy and convenience of extracting structured information.

[0006] The second objective of this application is to propose a generative information extraction method and device based on a pre-trained model.

[0007] A third object of the present application is to provide a non-transitory computer-readable storage medium.

[0008] To achieve the above objectives, the first embodiment of the present application proposes a generative information extraction method based on a pre-trained model, comprising:

[0009] Obtain historical annotation data for structured information extraction tasks;

[0010] Generate a multi-round question template according to the dependency relationship between each target field in the historical annotated data, and preprocess the historical annotated data into a data set in the form of a multi-round question and answer;

[0011] Serializing the multi-round question templates, and fine-tuning a preset generative training model in combination with the multi-round question-answering data set;

[0012] Inputting the text data to be extracted into the fine-tuned generative training model, and generating answers to the fields corresponding to each round of questions in the multiple rounds of question templates through the fine-tuned generative training model;

[0013] The answers to each round of questions are converted into structured data, and the fine-tuned generative training model is evaluated.

[0014] Optionally, in one embodiment of the present application, generating multiple rounds of question templates based on the dependency relationship between each target field in the historical annotation data includes: determining the dependency relationship of the structured information based on the dependency relationship and missing status between each target field in the annotation data; setting a corresponding question template for each target field based on the dependency relationship of the structured information; and combining each of the question templates according to the dependency relationship to generate the multiple rounds of question templates.

[0015] Optionally, in one embodiment of the present application, the serializing of the multi-round question templates and fine-tuning the preset generative training model in combination with the multi-round question-and-answer format data set includes: converting the multi-round question templates into a first sequence comprising context, questions and answers to be generated; obtaining a context representation vector of the first sequence through a preset language model; extracting data from the multi-round question-and-answer format data set, and training the preset generative training model in combination with the first sequence and the context representation vector to generate a generative training model that answers the multi-round question templates.

[0016] Optionally, in one embodiment of the present application, the generation of answers to fields corresponding to each round of questions in the multi-round question template by a fine-tuned generative training model includes: loading the fine-tuned generative training model and inputting text data to be extracted; combining the answers to the fields corresponding to the current question and the previous round of questions in the order of the multi-round question template, and inputting the answers to the fields into the fine-tuned generative training model to generate answers to the fields corresponding to the current question; combining the answers to the fields corresponding to the current question with the next round of questions in the multi-round question template, and inputting the answers to the fields into the fine-tuned generative training model to generate answers to the fields corresponding to the next round of questions.

[0017] Optionally, in one embodiment of the present application, after combining the answers to the fields corresponding to the current question and the previous round of questions, it also includes: obtaining the context of the current question; converting the current question and the context of the current question into a second sequence including the context of the current question, the current question and the answer to be generated for the current question.

[0018] Optionally, in one embodiment of the present application, after obtaining the historical annotated data of the scholar's educational experience or work experience, it also includes: dividing the historical annotated data into a training set, a verification set, and a test set according to a preset ratio.

[0019] Optionally, in one embodiment of the present application, the evaluating the fine-tuned generative training model includes: calculating the precision, recall and comprehensive evaluation value of the answers generated by the fine-tuned generative training model; and evaluating the fine-tuned generative training model based on the precision, recall and comprehensive evaluation value.

[0020] Optionally, in one embodiment of the present application, the precision rate, the recall rate and the comprehensive evaluation value are calculated by the following formula:

[0021]

[0022]

[0023]

[0024] in,

[0025] Among them, P is the precision, R is the recall, F1 is the comprehensive evaluation value, m is the number of extracted records, n is the number of labeled records, and k is the number of elements in record i in the labeled data.

[0026] To achieve the above-mentioned purpose, the second aspect of the present application proposes a generative information extraction device based on a pre-trained model, comprising the following modules:

[0027] The data acquisition module is used to obtain historical annotation data for structured information extraction tasks;

[0028] A first generating module is used to generate a multi-round question template according to the dependency relationship between each target field in the historical annotated data, and preprocess the historical annotated data into a data set in the form of a multi-round question and answer;

[0029] A training module, used to serialize the multi-round question templates and fine-tune a preset generative training model in combination with the multi-round question-answering format dataset;

[0030] A second generation module is used to input the text data to be extracted into the fine-tuned generative training model, and generate answers to the fields corresponding to each round of questions in the multiple rounds of question templates through the fine-tuned generative training model;

[0031] The evaluation module is used to form the answers to each round of questions into structured data and evaluate the fine-tuned generative training model.

[0032] To achieve the above-mentioned purpose, the third aspect embodiment of the present application proposes a non-temporary computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the generative information extraction method based on the pre-trained model described in the first aspect embodiment of the present application.

[0033] The generative information extraction method, device and storage medium based on the pre-trained model of the embodiment of the present application regard the information extraction task as a multi-round generative question-answering task, and design the question module by considering the dependency between fields. The dependency between different fields can be captured through multi-round question-answering, and then the corresponding field information is extracted using the generative pre-trained model using the given context text and question. It can be applicable to situations where multiple fields overlap or a field is missing. Generating answers through the generative model reduces the dependence on entities and improves the accuracy and convenience of extracting structured information.

[0034] Additional aspects and advantages of the present application will be given in part in the following description, and in part will become apparent from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0036] Figure 1 A schematic diagram of a flow chart of a generative information extraction method based on a pre-trained model according to an embodiment of the present application;

[0037] Figure 2 A flowchart of a specific method for generating a multi-round question template according to an embodiment of the present invention;

[0038] Figure 3 A flowchart of a specific method for extracting scholar's educational experience according to an embodiment of the present invention;

[0039] Figure 4 It is a structural schematic diagram of a generative information extraction device based on a pre-trained model according to an embodiment of the present application. DETAILED DESCRIPTION

[0040] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0041] The following describes a generative information extraction method and device based on a pre-trained model according to an embodiment of the present invention with reference to the accompanying drawings.

[0042] Figure 1 FIG. 1 is a flow chart of a generative information extraction method based on a pre-trained model according to an embodiment of the present application. Figure 1 As shown, the method includes:

[0043] Step 101, obtaining historical annotation data for the structured information extraction task.

[0044] The historically annotated data is the data after the target field in the structured information extraction task is annotated. The structured information includes the educational experience, work experience, events to be extracted, and other information that can be represented in a structured manner. The historically annotated data of the structured information extraction task is the historically annotated data of the structured information. The extraction method of the present application can generate structured information based on a generative pre-trained model from a large amount of information obtained in various ways.

[0045] Specifically, in the embodiments of the present application, a large amount of information is first collected, and relevant data is extracted, and the target fields therein are annotated. Among them, the target field is the key information to be structured. For example, for the educational experience of scholars, the target field includes the school attended, academic qualifications and school time, etc. For the work experience of scholars, the target field includes the work unit, title and working time, etc. For events, the target field includes event type, time, place, person, etc. This application annotates the collected data of the scholar's educational experience or work experience through different annotation methods, such as manual annotation.

[0046] In one embodiment of the present application, after obtaining the historical annotated data, the historical annotated data can also be divided into a training set, a validation set, and a test set according to a preset ratio. For example, through a cross-validation operation, the historical annotated data can be divided into a training set, a validation set, and a test set in a ratio of 7:1.5:1.5, which is convenient for subsequent training models and verification of model extraction effects.

[0047] Step 102, generating a multi-round question template according to the dependency relationship between each target field in the historical annotation data, and preprocessing the historical annotation data into a data set in the form of a multi-round question and answer.

[0048] Among them, the multi-round question template is a template for performing multiple rounds of question and answering when extracting structured information.

[0049] It can be understood that there is an inevitable dependency between the various target fields in the structured information. For example, the target field "a certain major" depends on the target field "which school to study in", the target field "job title" depends on the target field "work unit", etc. The target field "event argument to be extracted" depends on the "trigger word of the event". Therefore, this application can generate multiple rounds of question templates based on the dependency between each target field and the characteristics of each field.

[0050] As a possible implementation method, the present application embodiment proposes a specific method for generative information extraction based on a pre-trained model, such as Figure 2 As shown, the method includes:

[0051] Step 201, determining the dependency relationship in the structured information according to the dependency relationship and missing status between each target field in the annotated data.

[0052] Specifically, since some target fields may not exist in actual applications, for example, the start time of studying for a degree is missing in the historical annotated data obtained, the dependencies and missing conditions between the target fields in the annotated data are comprehensively analyzed, and the target fields are sorted according to the dependencies at all levels to formulate the dependencies of the scholars' educational experience or work experience. For example, the dependency of educational experience is: scholar's name-school-degree-other (major / end time / start time), the dependency of work experience is: scholar's name-unit-position / job-other (start time / end time), and the dependency of event extraction is: event type-event trigger word-event argument (person / place / time, etc.).

[0053] Step 202: According to the dependency relationship between the fields in the structured information, a corresponding question template is set for each target field.

[0054] Specifically, since the sentence structure when asking questions about a certain field is relatively fixed when answering questions about structured information, a specific question template is formulated for each target field in combination with the dependency relationship between the fields in the determined structured information. In one embodiment of the present application, the question template formulated for each target field may include the target field on which the target field depends. For example, the question template formulated for the target field of "a certain major" may include the target field of "which school you studied in", so as to facilitate more accurate extraction of the scholar's educational experience through multiple rounds of questioning.

[0055] Step 203: Combine each question template according to the dependency relationship to generate multiple rounds of question templates.

[0056] Specifically, after generating the question template corresponding to each target field, the question templates are sorted according to the dependency relationship between the target fields and combined together to generate multiple rounds of question templates corresponding to the dependency relationship between the fields. In this way, multiple rounds of question templates are generated according to the dependency relationship between each target field in the historical annotation data.

[0057] Furthermore, the historical annotation data is preprocessed into a dataset in the form of multi-round question-answering. As an example, the historical annotation data is organized into a dataset containing a question set, a context set, and an answer set. For example, a dataset in the form of multi-round question-answering contains a given question Q = {q1, q2, ..., q n}, context C = {c1, c2, ..., c m}, answer A = [{a 11 ,a 12 ,…,a 1i},{a 21 ,a 22 ,…,a 1j},…], where if there is no answer, unknown is used instead.

[0058] Step 103, serialize the multi-round question templates, and fine-tune the preset generative training model in combination with the multi-round question-answering data set.

[0059] In one embodiment of the present application, the multi-round question templates are first serialized, that is, the multi-round question templates are converted into a sequence containing context, questions and answers to be generated. The sequence can be: sentence = [CLS] Context: C Question: Q answer: [MASK], wherein the answer to be generated in the converted sequence is unknown, and the answer to be generated is the answer generated by the preset generative training model for each question.

[0060] Then, the context representation vector of the first sequence is obtained through the preset language model. In this embodiment, the preset speech model can be a Bidirectional Encoder Representations from Transformers (BERT) model, and the context representation vector of the sequence is obtained from the labeled data through the BERT model.

[0061] Further, data is extracted from a data set in the form of multiple rounds of questions and answers, and a preset generative training model is trained in combination with the first sequence and the context representation vector to generate a generative training model that answers multiple rounds of question templates. Specifically, the generative training model in this embodiment can be a general language model (General Language Model, referred to as GLM), the first sequence and the context representation vector are input into the preset GLM model, the answer to each question in the serialized multiple rounds of question templates is generated by the GLM model, and the answer generated by the GLM model is verified based on the data extracted from the data set in the form of multiple rounds of questions and answers, and the GLM model is fine-tuned. As one possible implementation method, the answer generated by the GLM model is compared with the actual result of the answer extracted from the data set in the form of multiple rounds of questions and answers, the cross entropy loss of the two is calculated, the parameters of the GLM model are adjusted according to the calculated cross entropy loss, and the GLM model is converged by repeatedly executing multiple rounds of questions and answers.

[0062] Step 104, input the text data to be extracted into the fine-tuned generative training model, and generate answers to fields corresponding to each round of questions in the multi-round question template through the fine-tuned generative training model.

[0063] Among them, the text data to be extracted is the structured field information that actually needs to be extracted. After the generative training model is trained, questions are generated for the text data to be extracted through multiple rounds of question templates, and each round of questions is answered through the fine-tuned generative training model, so as to extract structured information from the text data to be extracted.

[0064] In specific implementation, in one embodiment of the present application, the fine-tuned generative training model is first loaded, and the text data to be extracted is input. Then, according to the order of each round of questions in the multi-round question template, the answers to the fields corresponding to the current question and the previous round of questions are combined and input into the fine-tuned generative training model to generate the answer to the field corresponding to the current question. Then, the answer to the field corresponding to the current question is combined with the next round of questions in the multi-round question template and input into the fine-tuned generative training model to generate the answer to the field corresponding to the next round of questions. And so on, the answer to each question is generated.

[0065] For example, the text to be extracted and the first question are input into the fine-tuned model to obtain the answer to the first target field, and then the answer to the first field is combined with the second question template in the multi-round question template to generate a new question, which is input into the model again to obtain the answer to the second target field, and so on.

[0066] It should be noted that after combining the answers to the fields corresponding to the current question and the previous round of questions, the present application can also obtain the context of the current question before inputting the generative training model, and then convert the current question and the context of the current question into a second sequence containing the context of the current question, the current question, and the answer to the current question to be generated, and input the sequence into the generative training model to facilitate the generative training model to accurately identify the question and determine the answer based on the context.

[0067] Step 105, the answers to each round of questions are converted into structured data, and the fine-tuned generative training model is evaluated.

[0068] It can be understood that since there is a dependency relationship between each question in the multi-round question template, there is also a corresponding dependency relationship after the answer corresponding to each question is obtained. In the embodiment of the present application, the answers corresponding to each question are arranged and combined together according to the order of the dependency relationship, and the answers of each round are preprocessed so that the target fields of each experience are set according to the dependency relationship to form structured data.

[0069] Furthermore, when evaluating the effectiveness of the fine-tuned generative training model in extracting structured information, as a possible implementation method, the data in the divided validation set and test set can be input into the generative training model in the above-mentioned manner to perform multiple rounds of question and answering, and the answers obtained by the generative training model are obtained. Then, the labeled data in the validation set and test set are obtained. According to the answers obtained by the generative training model and the corresponding labeled data, the precision, recall and comprehensive evaluation value of the answers generated by the fine-tuned generative training model are calculated. The effectiveness of the fine-tuned generative training model is evaluated according to the precision, recall and comprehensive evaluation value, that is, the accuracy of the answers generated by the model for each round of questions.

[0070] Specifically, the precision, recall and comprehensive evaluation value are calculated by the following formula:

[0071]

[0072]

[0073]

[0074] in,

[0075] Among them, P is the precision rate, R is the recall rate, F1 is the comprehensive evaluation value, m is the number of extracted records, n is the number of annotated records, and k is the number of elements of record i in the annotated data. After calculating the precision rate, recall rate and comprehensive evaluation value, the calculated value can be compared with the preset evaluation threshold, where the evaluation threshold can be the minimum threshold when the information extraction effect of the preset model meets the requirements. By comparing and judging whether the calculated above values ​​are greater than the evaluation threshold, whether the effect of the model meets the requirements can be evaluated.

[0076] In summary, the present application embodiment is based on a generative information extraction method of a pre-trained model, first obtains historical annotation data of a structured information extraction task, then generates a multi-round question template according to the dependency between each target field, and processes the annotation data into a multi-round question-answering data set, then serializes the multi-round question template, and fine-tunes the generative training model in combination with the multi-round question-answering data set, and inputs the text to be extracted into the fine-tuned generative training model, generates the answer to the field corresponding to each round of questions in the multi-round question template through the fine-tuned generative training model, and finally forms structured data with the answer to each round of questions, and evaluates the model. This method regards the scholar's education or work experience extraction task as a multi-round generative question-answering task, designs the question module by considering the dependency between fields, and can capture the dependency between different fields through the form of multi-round question-answering, and then uses the given context text and question to extract the corresponding field information with a generative pre-trained model, and this method can be applied to the situation where multiple fields overlap or a field is missing, and generates answers through a generative model to reduce the dependence on entities and improve the accuracy and convenience of extracting structured information.

[0077] In order to more clearly describe the generative information extraction method based on the pre-trained model of the embodiment of the present application, the following describes a specific embodiment of extracting a scholar's educational experience from a text to be extracted, taking the structured information to be extracted as a scholar's educational experience.

[0078] like Figure 3 As shown, in this embodiment, the fine-tuned generative model is first loaded, and then the text to be extracted is input into the generative model, and the text to be extracted includes multiple target fields representing the educational experience of the scholar.

[0079] The first question in the multi-round question template is set according to the dependency of the target field, and the context of the first question is serialized into Context+Question+[MASK], and then input into the fine-tuned model. The answer to the first question about the target field "school" is obtained, that is, "University of Bradford";

[0080] The answer is combined with the template of the next question in the multi-round question template to generate a second question containing the answer to the first question, such as "What major did the scholar study at the University of Bradford?" The generated new question is input into the model again to obtain the answer to the second question about the target field "major", namely "computer science".

[0081] Similarly, the obtained answers are returned to the step before the questions are input into the generative training model, and new questions are generated by combining the question templates in the multiple rounds of question templates, and then input into the generative training model to obtain the answers to each question until all the questions in the multiple rounds of question templates are input into the generative training model.

[0082] Finally, the answers to each round of questions are formatted to form a structured data package of the scholar's educational experience, thereby completing the task of extracting the scholar's educational experience from the text to be extracted.

[0083] The following describes a generative information extraction device based on a pre-training model proposed according to an embodiment of the present invention with reference to the accompanying drawings.

[0084] Figure 4 FIG. 1 is a schematic diagram of the structure of a generative information extraction device based on a pre-trained model according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0085] The data acquisition module 100 is used to acquire historical annotation data for the structured information extraction task.

[0086] The first generation module 200 is used to generate a multi-round question template according to the dependency relationship between each target field in the historical annotation data, and pre-process the historical annotation data into a data set in the form of a multi-round question and answer.

[0087] The training module 300 is used to serialize multiple rounds of question templates and fine-tune the preset generative training model in combination with a data set in the form of multiple rounds of question and answer.

[0088] The second generation module 400 is used to input the text data to be extracted into the fine-tuned generative training model, and generate answers to fields corresponding to each round of questions in the multi-round question template through the fine-tuned generative training model.

[0089] The evaluation module 500 is used to convert the answers to each round of questions into structured data and evaluate the fine-tuned generative training model.

[0090] Furthermore, in one embodiment of the present application, the data acquisition module 100 is also used to: divide the historical annotated data into a training set, a validation set, and a test set according to a preset ratio.

[0091] In one embodiment of the present application, the first generation module 200 is also used to: determine the dependencies in the structured information based on the dependencies and missing conditions between each target field in the annotated data; set a corresponding question template for each target field based on the dependencies in the structured information; and combine each question template according to the dependencies to generate multiple rounds of question templates.

[0092] In one embodiment of the present application, the training module 300 is also used to convert the multi-round question template into a first sequence including context, questions and answers to be generated, obtain the context representation vector of the first sequence through a preset language model, extract data from a data set in the form of multi-round question and answer, and train a preset generative training model in combination with the first sequence and the context representation vector to generate a generative training model that answers the multi-round question template.

[0093] In one embodiment of the present application, the second generation module 400 is also used to: load the fine-tuned generative training model and input the text data to be extracted; combine the answers to the fields corresponding to the current question and the previous round of questions in the order of each round of questions in the multi-round question template, and input them into the fine-tuned generative training model to generate the answer to the field corresponding to the current question; combine the answer to the field corresponding to the current question with the next round of questions in the multi-round question template and input them into the fine-tuned generative training model to generate the answer to the field corresponding to the next round of questions.

[0094] In one embodiment of the present application, the second generation module 400 is also used to: obtain the context of the current question; convert the current question and the context of the current question into a second sequence including the context of the current question, the current question and the answer to be generated for the current question.

[0095] In one embodiment of the present application, the evaluation module 500 is also used to: calculate the precision, recall and comprehensive evaluation value of the answers generated by the fine-tuned generative training model, and evaluate the fine-tuned generative training model based on the precision, recall and comprehensive evaluation value.

[0096] In one embodiment of the present application, the evaluation module 500 is specifically used to calculate the precision rate, the recall rate and the comprehensive evaluation value by the following formula:

[0097]

[0098]

[0099]

[0100] in,

[0101] Among them, P is the precision, R is the recall, F1 is the comprehensive evaluation value, m is the number of extracted records, n is the number of labeled records, and k is the number of elements in record i in the labeled data.

[0102] It should be noted that the aforementioned explanation of the embodiment of the generative information extraction method based on the pre-trained model is also applicable to the device of this embodiment and will not be repeated here.

[0103] In summary, the present invention is based on a generative information extraction device of a pre-trained model. The device first obtains historical annotation data of a structured information extraction task, then generates a multi-round question template according to the dependency between each target field, and processes the annotation data into a multi-round question-answering data set, then serializes the multi-round question template, and fine-tunes the generative training model in combination with the multi-round question-answering data set, and inputs the text to be extracted into the fine-tuned generative training model, generates the answer to the field corresponding to each round of questions in the multi-round question template through the fine-tuned generative training model, and finally forms structured data with the answer to each round of questions, and evaluates the model. The device regards the information extraction task as a multi-round generative question-answering task, designs the question module by considering the dependency between fields, and can capture the dependency between different fields through the form of multi-round question-answering, and then uses the given context text and question to extract the corresponding field information with the generative pre-trained model. The method can be applied to the situation where multiple fields overlap or a field is missing. Generating answers through the generative model reduces the dependence on the entity and improves the accuracy and convenience of extracting structured information.

[0104] To achieve the above-mentioned purpose, an embodiment of the present application also proposes a non-temporary computer-readable storage medium, on which a computer program is stored, characterized in that when the computer program is executed by a processor, a generative information extraction method based on a pre-trained model as described in any of the above embodiments is implemented.

[0105] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, if schematic expressions of the above terms are used in multiple embodiments or examples, it does not mean that these embodiments or examples are the same. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0106] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0107] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0108] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0109] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0110] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0111] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0112] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A generative information extraction method based on a pre-trained model, characterized in that: The following steps are involved: Obtain historical annotation data for structured information extraction tasks; Generate a multi-round question template according to the dependency relationship between each target field in the historical annotated data, and preprocess the historical annotated data into a data set in the form of a multi-round question and answer; Serializing the multi-round question templates, and fine-tuning a preset generative training model in combination with the multi-round question-answering data set; Inputting the text data to be extracted into the fine-tuned generative training model, and generating answers to the fields corresponding to each round of questions in the multiple rounds of question templates through the fine-tuned generative training model; The answers to each round of questions are converted into structured data, and the fine-tuned generative training model is evaluated.

2. The extraction method according to claim 1, characterized in that: The step of generating multiple rounds of question templates according to the dependency relationship between each target field in the historical annotation data includes: Determining the dependency relationship of the structured information according to the dependency relationship and missing status between each target field in the annotated data; According to the dependency relationship between the structured information, a corresponding question template is set for each of the target fields; Each of the question templates is combined according to the dependency relationship to generate the multiple rounds of question templates.

3. The extraction method according to claim 1 or 2, characterized in that: The step of serializing the multiple rounds of question templates and fine-tuning a preset generative training model in combination with the multiple rounds of question-answering data sets includes: Converting the multi-round question template into a first sequence including context, question, and answer to be generated; Acquire a context representation vector of the first sequence through a preset language model; Data is extracted from the multi-round question-answering format data set, and the preset generative training model is trained in combination with the first sequence and the context representation vector to generate a generative training model that answers the multi-round question template.

4. The extraction method according to claim 1, characterized in that: The generating of the answer to the field corresponding to each round of questions in the multi-round question template by the fine-tuned generative training model includes: Loading the fine-tuned generative training model and inputting the text data to be extracted; According to the order of each round of questions in the multi-round question template, the answers to the fields corresponding to the current question and the previous round of questions are combined and input into the fine-tuned generative training model to generate the answer to the field corresponding to the current question; The answer to the field corresponding to the current question is combined with the next round of questions in the multi-round question template and input into the fine-tuned generative training model to generate the answer to the field corresponding to the next round of questions.

5. The extraction method according to claim 4, characterized in that: After combining the answers to the fields corresponding to the current question and the previous round of questions, it also includes: Get the context of the current problem; The current question and the context of the current question are converted into a second sequence including the context of the current question, the current question and an answer to be generated for the current question.

6. The extraction method according to claim 1, characterized in that: After obtaining the historical annotated data for the structured information extraction task, the method further includes: dividing the historical annotated data into a training set, a validation set, and a test set according to a preset ratio.

7. The extraction method according to claim 1, characterized in that: The evaluating the fine-tuned generative training model includes: Calculate the precision, recall and comprehensive evaluation value of the answers generated by the fine-tuned generative training model; The fine-tuned generative training model is evaluated according to the precision, the recall and the comprehensive evaluation value.

8. The extraction method according to claim 7, characterized in that: The precision rate, the recall rate and the comprehensive evaluation value are calculated by the following formula: in, Among them, P is the precision, R is the recall, F1 is the comprehensive evaluation value, m is the number of extracted records, n is the number of labeled records, and k is the number of elements in record i in the labeled data.

9. A generative information extraction device based on a pre-trained model, characterized in that: include: The data acquisition module is used to obtain historical annotation data for structured information extraction tasks; A first generating module is used to generate a multi-round question template according to the dependency relationship between each target field in the historical annotated data, and preprocess the historical annotated data into a data set in the form of a multi-round question and answer; A training module, used to serialize the multi-round question templates and fine-tune a preset generative training model in combination with the multi-round question-answering format dataset; A second generation module is used to input the text data to be extracted into the fine-tuned generative training model, and generate answers to the fields corresponding to each round of questions in the multiple rounds of question templates through the fine-tuned generative training model; The evaluation module is used to form the answers to each round of questions into structured data and evaluate the fine-tuned generative training model.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the generative information extraction method based on the pre-training model as described in any one of claims 1-8.

Images