Document information extraction method oriented to industrial field
By performing three-stage fine-tuning and low-rank adjustment on the general model, the problems of flexibility and generalization in document information extraction in industrial field are solved, and more efficient and accurate information extraction effect is achieved.
Patent Information
- Application Number
- CN202311032603.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-16
- Publication Date
- 2025-02-21
AI Technical Summary
The existing technology lacks flexibility and generalization in document information extraction in the industrial field, requires frequent manual annotation and template definition, and general large models are difficult to directly adapt to the complex information extraction tasks in the industrial field.
By performing three-stage fine-tuning of the general big model: the first stage fine-tuning is the ability to extract simple information, the second stage fine-tuning is the ability to extract complex information, and adapt to specific target scenarios in the industrial field through low-rank adjustment.
It improves the accuracy and adaptability of general large models in document information extraction in industrial field, and reduces the need for manual annotation and template definition.
Smart Images
Figure CN119493820A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence and industry, and specifically relates to a document information extraction method oriented to the industrial field. Background Art
[0002] Industrial documents are an important source of knowledge that guides human industrial production. They record the operation procedures and basic principles of equipment or processes and are a type of text with extremely high knowledge density. How to extract correct and effective information with practical application value from industrial documents has always been a hot topic in the research of information extraction technology, and is also a representative pain point and difficult problem in the industrial field.
[0003] Early information extraction solutions for industrial documents basically relied on pre-trained language models as encoders to understand text and extract corresponding pattern knowledge. However, most of these deep neural networks are task-oriented, and the parameters of the models are mostly adapted to a single information extraction task, such as supervised learning extraction based on manual annotation and bootstrap extraction based on rule templates.
[0004] However, although the existing technical solutions can perform relatively well on specific corpora, they still lack flexibility and generalization for documents of different types and contents.
[0005] For early extraction models based on deep neural networks, documents of different types or contents need to be manually re-annotated or templates need to be defined, which is very time-consuming and labor-intensive. Once the annotation scale decreases, the extraction effect will decline catastrophically.
[0006] For large models, although they can solve complex information extraction tasks to a certain extent, it is often difficult to learn and train specifically for industrial domain knowledge and complex extraction tasks during the training process, which makes it difficult for general large models to directly handle industrial information extraction tasks. Summary of the invention
[0007] The present invention is made to solve the above-mentioned problem, and aims to provide a document information extraction method oriented to the industrial field.
[0008] The present invention provides a document information extraction method for the industrial field, which is used for extracting information from the original industrial documents of the target scene according to the existing general large model to obtain extracted information, and has the following characteristics, including the following steps: step S1, according to the existing general natural language processing tasks and simple extraction task instructions, the general large model is fine-tuned in the first stage, and the fine-tuned general large model is used as the simple fine-tuning model; step S2, according to the existing public data, a plurality of complex extraction instruction data are constructed to obtain a complex extraction instruction data set; step S3, according to the complex extraction instruction data set, the simple fine-tuning model is fine-tuned in the second stage, and the fine-tuned simple fine-tuning model is used as the complex fine-tuning model; step S4, the document data of the existing target scene is converted to obtain the corresponding industrial field extraction data set; step S5, according to the industrial field extraction data set, the model parameters of the complex fine-tuning model are low-rank adjusted, and the complex fine-tuning model with low-rank adjustment is used as the adaptation model; step S6, the original industrial document and the preset extraction task instructions and prompts are input into the adaptation model, and the output of the adaptation model is formatted, parsed and converted to obtain the extraction information.
[0009] The document information extraction method for the industrial field provided by the present invention may also have the following features: wherein, in step S2, the public data includes public academic data sets for research in the field of information extraction, public competition data sets proposed by enterprises and companies for scenarios with certain domain knowledge, general encyclopedia data sets based on various encyclopedia texts, and self-annotated data sets. The self-annotated data sets are obtained by designing extraction tasks for unannotated texts and generating corresponding answers through a closed-source large model according to artificial learning prompts.
[0010] The industrial-oriented document information extraction method provided by the present invention may also have the following features: for public academic data sets, public competition data sets and general encyclopedia data sets, templates are manually designed and corresponding annotations are converted into corresponding formats of complex extraction instruction data to obtain multiple complex extraction instruction data.
[0011] The industrial-oriented document information extraction method provided by the present invention may also have the following features: for the self-annotated data set, low-quality information extraction tasks in the self-annotated data set are eliminated through manual verification to obtain multiple complex extraction instruction data.
[0012] The industrial-oriented document information extraction method provided by the present invention may also have the following features: wherein, in step S2, the complex extraction instruction data is used for accurate and detailed extraction task description, including task description, attribute description, restriction conditions, output format description and extraction samples; the task description is used to briefly describe the goal of the extraction task corresponding to the complex extraction instruction data; the attribute description is used to give a specific and detailed extraction task, and to give a detailed introduction to the names and meanings of the elements to be extracted; the restriction conditions are used to limit the extraction of specified relationships and specified types of entities; the output format description is used to specify the output format of the extracted elements; the extraction samples include text similar to the extraction task, and are used to provide sample references for a simple fine-tuning model for performing the extraction task.
[0013] The industrial-oriented document information extraction method provided by the present invention may also have the following features: wherein the extraction tasks in the complex extraction instruction data set include: extraction tasks of complex schemas, extraction tasks of complex output formats, and extraction tasks of complex task descriptions.
[0014] The industrial document information extraction method provided by the present invention may also have the following features: wherein, in step S4, the industrial extraction data set includes the instruction description, prompts and formatted text form of the expected output of the extraction task.
[0015] The document information extraction method for the industrial field provided by the present invention may also have the following characteristics: wherein, in step S5, before the model parameters are low-rank adjusted, all parameters of the complex fine-tuning model are frozen, and then a bypass network with a small number of parameters is introduced.
[0016] Functions and Effects of the Invention
[0017] According to the document information extraction method for the industrial field involved in the present invention, first, the general large model is fine-tuned in the first stage so that the general large model has simple information extraction capabilities, then the general large model is fine-tuned in the second stage so that the general large model has complex information extraction capabilities, and finally, the general large model is low-rank adjusted according to the specific target scene to improve the general large model's ability to extract complex information from industrial documents in specific target scenes in the industrial field. By training the general large model in three stages, the trained large model can be used for document information extraction in specific industrial scenes. Therefore, the document information extraction method for the industrial field of the present invention can improve the accuracy of extracting industrial document information using the general large model. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 It is a schematic diagram of the principle of fine-tuning a general large model in an embodiment of the present invention;
[0019] Figure 2 is a flow chart of a method for extracting document information in an industrial field according to an embodiment of the present invention;
[0020] Figure 3 It is a schematic diagram of complex extraction instruction data in an embodiment of the present invention. DETAILED DESCRIPTION
[0021] In order to make the technical means, creative features, objectives and effects of the present invention easy to understand, the following embodiments and drawings specifically illustrate the document information extraction method for the industrial field of the present invention.
[0022] The existing general large models have certain capabilities for extracting complex information. However, for professional fields, especially industrial fields with extremely high knowledge density, it is impossible to directly use general large models for complex information extraction. Therefore, specific fine-tuning of the general large models is required.
[0023] Figure 1 It is a schematic diagram of the principle of fine-tuning the general large model in an embodiment of the present invention.
[0024] like Figure 1 As shown, in this implementation, the general large model is first fine-tuned in the first stage through natural language processing tasks and simple extraction task instructions to obtain a simple fine-tuned model. Then, the simple fine-tuned model is fine-tuned in the second stage through a complex extraction instruction data set to obtain a complex fine-tuned model. Finally, the complex fine-tuned model is low-rank adjusted according to the target scenario to obtain an adapted model. The adapted model can be used to accurately and efficiently extract information from industrial documents in the target scenario to obtain extracted information.
[0025] Figure 2 It is a flowchart of a method for extracting document information for industrial fields in an embodiment of the present invention.
[0026] like Figure 2 As shown, the document information extraction method for the industrial field in this embodiment includes the following steps:
[0027] Step S1, according to the existing general natural language processing tasks and simple extraction task instructions, the general large model is fine-tuned in the first stage, and the fine-tuned general large model is used as a simple fine-tuning model.
[0028] In this embodiment, the general natural language processing task is a task selected from multiple existing public data sets, and the simple extraction task instruction is a simple named entity extraction and relationship extraction data constructed manually. The general natural language processing task and simple extraction task instructions can improve the natural language processing task capabilities of the general large model and strengthen the large model's ability to recognize various elements of the text. Therefore, the first stage of fine-tuning can enable the general large model to have simple information extraction capabilities, namely entity recognition and relationship extraction.
[0029] Step S2, constructing a plurality of complex extraction instruction data as a complex extraction instruction data set based on existing public data.
[0030] Among them, public data includes public academic datasets for research in the field of information extraction, public competition datasets proposed by enterprises and companies for scenarios with certain domain knowledge, general encyclopedia datasets based on various encyclopedia texts, and self-annotated datasets.
[0031] The data in public academic datasets are mainly used to extract key information elements of general texts, including relationship extraction datasets that focus on triple extraction of limited common relations, and event extraction datasets that focus on the joint extraction of general events and event elements. These public academic datasets have high text richness and annotation accuracy, but lack diverse annotations.
[0032] Compared with the above-mentioned public academic datasets, public competition datasets have richer information extraction task types and richer annotated data annotations. However, due to the lack of sufficient cleaning of data sources and the heavy reliance on automatic annotation methods such as remote supervision in the annotation process, the quality of the extracted texts and the accuracy of annotations in these data sets have declined.
[0033] The general encyclopedia dataset is a dataset generated by large-scale annotation based on various encyclopedia texts through remote supervision and manual verification. In addition, existing high-quality knowledge graphs such as Wikidata and CN-DBpedia contain rich triple relationships and attribute annotations. By aligning them with general encyclopedia texts, they can automatically generate a large amount of structured information of different relationships, which can help large models identify and extract more diverse attributes and relationships, and improve the ability of large models to recognize more complex relationships.
[0034] The self-annotated dataset is obtained by designing the extraction task of unannotated text and generating the corresponding answers based on the artificial learning prompts of the closed-source large model. The specific steps of constructing the self-annotated dataset in this embodiment are:
[0035] First, some high-quality heterogeneous extraction tasks are provided manually, which cover various types of information extraction tasks and extraction requirements. Then, these heterogeneous extraction tasks are organized into few-sample learning prompts, namely artificial learning prompts. Finally, the few-sample learning ability of the closed-source large model is used to design extraction tasks and generate corresponding answers for the pre-collected unlabeled text according to the artificial learning prompts, thus obtaining the above-mentioned self-labeled dataset.
[0036] The above four data sets have different data characteristics and require different construction methods to construct complex extraction instruction data. The specific construction methods are as follows:
[0037] For public academic data sets, public competition data sets, and general encyclopedia data sets, we manually design templates and convert the corresponding annotations into the corresponding formats of complex extraction instruction data to obtain multiple complex extraction instruction data; for self-annotated data sets, we manually verify and eliminate low-quality information extraction tasks in the self-annotated data sets to obtain multiple complex extraction instruction data.
[0038] The constructed complex extraction instruction data is used for accurate and detailed extraction task description, so as to further enhance the extraction and comprehension capabilities of the simple fine-tuning model by using the complex extraction instruction data. Therefore, the complex extraction instruction data includes task description, attribute description, restriction conditions, output format description and extraction samples.
[0039] The task description is used to briefly describe the goal of the extraction task corresponding to the complex extraction instruction data.
[0040] The attribute description is used to give a specific and detailed extraction task and to introduce in detail the name and meaning of the elements to be extracted.
[0041] Restrictions are used to restrict the extraction of specified relationships and entities of specified types.
[0042] The output format description is used to specify the output format of the extracted features.
[0043] The extraction examples include texts similar to the extraction task, which are used to provide sample references for a simple fine-tuned model to perform the extraction task.
[0044] Figure 3 It is a schematic diagram of complex extraction instruction data in an embodiment of the present invention.
[0045] like Figure 3 As shown, the complex extraction instruction data includes task description, attribute description, output format description and extraction samples.
[0046] Step S3, performing second-stage fine-tuning on the simple fine-tuning model according to the complex extraction instruction data set, and using the fine-tuned simple fine-tuning model as the complex fine-tuning model.
[0047] Among them, the extraction tasks in the complex extraction instruction data set include: extraction tasks of complex schemas, extraction tasks of complex output formats, and extraction tasks of complex task descriptions.
[0048] In this embodiment, the second stage of fine-tuning enables the general large model to have more accurate instruction understanding, extraction, and formatting and organizing extraction results, providing a strong extraction capability guarantee for various downstream extraction tasks.
[0049] Although the general large model has a certain document extraction capability through two-stage fine-tuning, due to the large number of industrial scenarios and different task requirements, the large model still lacks sufficient industrial domain knowledge such as professional device names, fault names, etc., and it is difficult to ensure high-quality completion of domain extraction tasks. Therefore, further fine-tuning is needed to optimize the general large model in the task of extracting industrial documents.
[0050] Step S4, converting the existing document data of the target scene to obtain the corresponding industrial field extraction data set.
[0051] Among them, the industrial extraction data set includes the instruction description, prompts and formatted text form of the expected output of the extraction task.
[0052] Step S5, performing low-rank adjustment on the model parameters of the complex fine-tuning model according to the data set extracted from the industrial field, and using the complex fine-tuning model with low-rank adjustment as the adaptation model.
[0053] Before low-rank adjustment is performed on the model parameters, all parameters of the complex fine-tuning model are frozen, and then a bypass network with a small number of parameters is introduced.
[0054] In this embodiment, the general large model after two-stage fine-tuning is subjected to low-rank adjustment through steps S4 and S5, and the large model is adjusted using existing industrial documents in the target scenario, so that the large model has the accuracy of information extraction in the target scenario.
[0055] Step S6, inputting the original industrial document and the preset extraction task instructions and prompts into the adaptation model, and then formatting and parsing the output of the adaptation model to obtain the extraction information.
[0056] Functions and Effects of the Embodiments
[0057] According to the document information extraction method for the industrial field involved in this embodiment, the general large model is first fine-tuned in the first stage to enable the general large model to have simple information extraction capabilities. Then, the general large model is fine-tuned in the second stage to enable the general large model to have complex information extraction capabilities. Finally, the general large model is low-rank adjusted according to the specific target scenario to improve the general large model's ability to extract complex information from industrial documents in specific target scenarios in the industrial field. By training the general large model in three stages, the trained large model can be used for document information extraction in specific industrial scenarios. In short, this method can improve the accuracy of extracting industrial document information using a general large model.
[0058] The above-mentioned embodiments are preferred examples of the present invention and are not intended to limit the protection scope of the present invention.
Claims
1. A document information extraction method for the industrial field, which is used to extract information from the original industrial documents of the target scene according to the existing general large model to obtain extracted information, characterized in that: The following steps are involved: Step S1, performing first-stage fine-tuning on the general large model according to the existing general natural language processing tasks and simple extraction task instructions, and using the fine-tuned general large model as a simple fine-tuning model; Step S2, constructing a plurality of complex extraction instruction data as a complex extraction instruction data set according to the existing public data; Step S3, performing second-stage fine-tuning on the simple fine-tuning model according to the complex extraction instruction data set, and using the fine-tuned simple fine-tuning model as a complex fine-tuning model; Step S4, converting the existing document data of the target scene to obtain the corresponding industrial field extraction data set; Step S5, performing low-rank adjustment on the model parameters of the complex fine-tuning model according to the industrial field extraction data set, and using the complex fine-tuning model after low-rank adjustment as an adaptation model; Step S6, inputting the original industrial document and the preset extraction task instructions and prompts into the adaptation model, and then formatting and parsing the output of the adaptation model to obtain the extraction information.
2. The method for extracting document information from industrial fields according to claim 1, characterized in that: in, In step S2, the public data includes public academic datasets for research in the field of information extraction, public competition datasets proposed by enterprises and companies for scenarios with certain domain knowledge, general encyclopedia datasets based on various encyclopedia texts, and self-annotated datasets. The self-annotated dataset is obtained by designing extraction tasks for unannotated texts and generating corresponding answers through a closed-source large model according to artificial learning prompts.
3. The method for extracting document information for industrial fields according to claim 2, characterized in that: in, For the public academic data set, the public competition data set and the general encyclopedia data set, a plurality of the complex extraction instruction data are obtained by manually designing templates and converting corresponding annotations into corresponding formats of the complex extraction instruction data.
4. The method for extracting document information for industrial fields according to claim 2, characterized in that: in, For the self-annotated data set, low-quality information extraction tasks in the self-annotated data set are eliminated through manual verification to obtain a plurality of the complex extraction instruction data.
5. The method for extracting document information from industrial fields according to claim 1, characterized in that: in, In step S2, the complex extraction instruction data is used to accurately and in detail extract the task description, including task description, attribute description, restriction conditions, output format description and extraction samples. The task description is used to briefly describe the target of the extraction task corresponding to the complex extraction instruction data. The attribute description is used to give a specific and detailed description of the extraction task and to give a detailed introduction to the name and meaning of the elements to be extracted. The restriction condition is used to extract and limit the specified relationship and the specified type of entity. The output format description is used to specify the output format of the extracted elements. The extraction samples include texts similar to the extraction task, and are used to provide sample references for the simple fine-tuning model that performs the extraction task.
6. The method for extracting document information from industrial fields according to claim 1, characterized in that: in, The extraction tasks in the complex extraction instruction data set include: extraction tasks of complex schemas, extraction tasks of complex output formats, and extraction tasks of complex task descriptions.
7. The method for extracting document information from industrial fields according to claim 1, characterized in that: in, In step S4, the industrial field extraction data set includes a formatted text form of the instruction description, prompts and expected output of the extraction task.
8. The method for extracting document information from industrial fields according to claim 1, characterized in that: in, In step S5, before low-rank adjustment is performed on the model parameters, all parameters of the complex fine-tuning model are frozen, and then a bypass network with a small number of parameters is introduced.