Label data generation method and system based on large language model

Abstract

The invention provides an annotation data generation method and system based on a large language model, and the method comprises the steps: S1, obtaining a structured text of a representative contract of the same category, and generating a corresponding information extraction template based on the obtained structured text of the representative contract; s2, generating annotation data in batches based on the information extraction template and all contract texts in the same category; s3, summarizing different types of annotation data and corresponding contract texts, generating an annotation data set, and performing fine adjustment on the large language model by utilizing the annotation data set to obtain a fine-adjusted large language model; and S4, marking the contract text based on the fine-tuned large language model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for generating labeled data based on a large language model. Background Art

[0002] Traditional data annotation methods usually rely on manual review and annotation, which is not only time-consuming and laborious when processing large amounts of data, but also easily affected by human factors, resulting in a decrease in the quality and consistency of the annotated data. In addition, the diversity and professionalism of complex contract terms increase the difficulty of manual annotation, often resulting in deviations and errors in the annotation results.

[0003] Patent document CN110705225A (application number: 201910752445.8) discloses a contract annotation method and device, which relates to the field of artificial intelligence technology. The method includes: extracting at least one contract sample from each business type of contract to obtain an initial sample set; constructing and training an initial annotation model based on the contract sample of each business type; obtaining multiple pre-stored contracts of each business type, and dividing them into a sample expansion set and a test set; using the initial annotation model to annotate the contract elements in the contracts in the sample expansion set; merging the annotated sample expansion set and the initial sample set into a training sample set, and using the training sample set to optimize and train the initial annotation model to obtain a annotation model; inputting the test set into the annotation model, and obtaining the annotation results of the contracts in the test set output by the annotation model; judging whether it is necessary to continue to optimize the annotation model according to the annotation results of the test set. Summary of the invention

[0004] In view of the defects in the prior art, the object of the present invention is to provide a method and system for generating annotated data based on a large language model.

[0005] A method for generating annotated data based on a large language model according to the present invention includes:

[0006] Step S1: obtaining structured texts of representative contracts of the same category, and generating corresponding information extraction templates based on the obtained structured texts of the representative contracts;

[0007] Step S2: based on the information extraction template and all contract texts in the same category, batch generate annotation data;

[0008] Step S3: Summarize the annotated data of different categories and the corresponding contract texts, and generate an annotated data set, and use the annotated data set to fine-tune the large language model to obtain a fine-tuned large language model;

[0009] Step S4: Annotate the contract text based on the fine-tuned large language model.

[0010] Preferably, the information extraction template includes:

[0011] p i =(z i ,q i )

[0012] Among them, z i For structured text, q i is the prompt word for the i-th category used to guide GPT3.5.

[0013] Preferably, step S2 comprises: extracting content from all contract texts in the cluster according to a corresponding information extraction template in the cluster using GPT3.5, and annotating the extracted content according to the information extraction template to obtain annotated data;

[0014] The annotation data includes:

[0015] Y i =g(p i , x i );

[0016] Among them, p i is the information extraction template of the constructed i-th category, x i is the contract text, Y i is the labeled data of the i-th category, and g is the character set to character set mapping implemented by GPT3.5.

[0017] Preferably, the method further comprises: using GPT4 to generate data for data enhancement, constructing a new contract text and its corresponding annotation; the expression is Among them, Y gen To generate a new contract text dataset.

[0018] Preferably, step S3 includes: fine-tuning the large language model using the annotated data set, including: at least one of LoRA fine-tuning, full parameter adjustment, and QLoRA parameter fine-tuning.

[0019] According to the present invention, a system for generating annotated data based on a large language model includes:

[0020] Module M1: Obtain structured texts of representative contracts of the same category, and generate corresponding information extraction templates based on the structured texts of the representative contracts obtained;

[0021] Module M2: Batch generate annotation data based on the information extraction template and all contract texts in the same category;

[0022] Module M3: Aggregate the annotated data of different categories and the corresponding contract texts, generate an annotated data set, and use the annotated data set to fine-tune the large language model to obtain the fine-tuned large language model;

[0023] Module M4: Annotate the contract text based on the fine-tuned large language model.

[0024] Preferably, the information extraction template includes:

[0025] p i =(z i ,q i )

[0026] Among them, z i For structured text, q i is the prompt word for the i-th category used to guide GPT3.5.

[0027] Preferably, the module M2 comprises: using GPT3.5 to extract content from all contract texts in the cluster according to the corresponding information extraction template in the cluster, and annotating the extracted content according to the information extraction template to obtain annotated data;

[0028] The annotation data includes:

[0029] Y i =g(p i , x i );

[0030] Among them, p i is the information extraction template of the constructed i-th category, x i is the contract text, Y i is the labeled data of the i-th category, and g is the character set to character set mapping implemented by GPT3.5.

[0031] Preferably, the system further includes: using GPT4 to generate data for data enhancement, constructing a new contract text and its corresponding annotation; the expression is Among them, Y gen To generate a new contract text dataset.

[0032] Preferably, the module M3 includes: fine-tuning the large language model using the annotated data set, including: at least one of LoRA fine-tuning, full parameter adjustment, and QLoRA parameter fine-tuning.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The present invention performs cluster analysis on the contract text, classifies it, and then uses a stratified sampling method to extract representative samples, thereby ensuring the balanced distribution of various types of contract information in the data set, reducing data deviation, and effectively improving the accuracy and stability of annotation;

[0035] 2. In the data annotation process, the accuracy of the annotated data is ensured by using a large language model for automatic annotation. The present invention not only effectively reduces the workload of manual annotation, but also improves the system's ability to understand complex contract information, especially when dealing with diversified contract terms.

[0036] 3. The present invention realizes the automated batch information extraction of unlabeled contract texts through a template-based information extraction method. The information extraction template generated by the system, combined with the large language model, performs batch processing on the unlabeled contract texts. This method effectively improves the automation level of contract information processing, reduces the workload of manual participation, and significantly improves processing efficiency. It can quickly and accurately extract key information from complex contract texts, bringing great convenience to the contract management of enterprises.

[0037] 4. The output results of the present invention are presented in the form of structured text, including key information such as performance requirements, delivery location, settlement information, material information and additional terms. Through the structured output method, the storage, retrieval and analysis of contract terms become more efficient and convenient, greatly improving the utilization and readability of contract information; this method provides strong technical support for enterprises in contract management, information sharing and risk control, and ensures the accuracy and consistency of contract information. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0039] Figure 1 Flowchart of the method for generating labeled data based on a large language model.

[0040] Figure 2 Flowchart of the complex industrial contract information extraction method based on a large language model. DETAILED DESCRIPTION

[0041] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0042] Example 1

[0043] According to a method for generating annotated data based on a large language model provided by the present invention, Figure 1 As shown, including:

[0044] Step S1: obtaining structured texts of representative contracts of the same category, and generating corresponding information extraction templates based on the obtained structured texts of the representative contracts;

[0045] Step S2: based on the information extraction template and all contract texts in the same category, batch generate annotation data;

[0046] Step S3: Summarize the annotated data of different categories and the corresponding contract texts, and generate an annotated data set, and use the annotated data set to fine-tune the large language model to obtain a fine-tuned large language model;

[0047] Step S4: Annotate the contract text based on the fine-tuned large language model.

[0048] Specifically, the information extraction template includes:

[0049] p i =(z i ,q i )

[0050] Among them, z i For structured text, q i is the prompt word for the i-th category used to guide GPT3.5.

[0051] Specifically, step S2 includes: using GPT3.5 to extract content from all contract texts in the cluster according to the corresponding information extraction template in the cluster, and annotating the extracted content according to the information extraction template to obtain annotated data;

[0052] The annotation data includes:

[0053] Y i =g(p i , x i );

[0054] Among them, p i is the information extraction template of the constructed i-th category, x i is the contract text, Y i is the labeled data of the i-th category, and g is the character set to character set mapping implemented by GPT3.5.

[0055] Specifically, the method also includes: using GPT4 to generate data for data enhancement, constructing a new contract text and its corresponding annotations; its expression is Among them, Y genTo generate a new contract text dataset.

[0056] Specifically, step S3 includes: fine-tuning the large language model using the annotated data set, including: at least one of LoRA fine-tuning, full parameter adjustment, and QLoRA parameter fine-tuning.

[0057] The present invention also provides a labeling data generation system based on a large language model. The labeling data generation system based on a large language model can be implemented by executing the process steps of the labeling data generation method based on a large language model, that is, those skilled in the art can understand the labeling data generation method based on a large language model as a preferred implementation of the labeling data generation system based on a large language model.

[0058] Example 2

[0059] Embodiment 2 is a preferred embodiment of Embodiment 1

[0060] In order to solve the problems of insufficient accuracy and low efficiency of information extraction in the prior art, the present invention provides a complex industrial contract information extraction method based on a large language model, which can process contract texts with complex contents and automatically and efficiently extract structured information.

[0061] The complex industrial contract information extraction method based on the large language model is as follows: Figure 2 As shown, including:

[0062] Step 1: Collect industrial contract texts as the basic data set for system analysis.

[0063] Step 2: Perform cluster analysis on the data set collected in step 1 and calculate the cluster center c of each cluster i , determine the cluster C where each contract text is located i .

[0064] Step 3: From each cluster C i Stratified sampling was performed to extract representative samples.

[0065] Step 4: Use GPT4 to perform the computation on each cluster C in step 3. i The key content of representative samples is extracted from the contract, and different prompt words are set to make GPT4 output the structured text corresponding to each sample. The output result is manually modified to make the structured text more accurate. Among them, the key content is the specific provisions in the contract. By prompting GPT4, GPT4 is allowed to extract and organize the structured specific information from the unstructured contract text. The purpose of obtaining the key content is to filter out the descriptive and colloquial texts such as "should" and "if" in the original contract text, and directly organize the unstructured contract text into structured information.

[0066] Step 5: Based on the structured text extracted in step 4, for each cluster C i Generate different information extraction templates p i The template is in text form, and its content includes structured text and instruction prompt words corresponding to the sample text. In this embodiment, the form of the template is, for example, "The following is an example of a contract text extraction task: {structured text}, please extract information from the input contract text based on this example." Different templates can be set for different categories.

[0067] Step 6: Create the template p generated in step 5 i and the corresponding cluster C in step 2 i The internal samples are used as input and GPT3.5 is used to generate labeled data Y in batches i . The labeled data Y of all clusters i In summary, a high-quality annotated dataset is constructed for model training. In this embodiment, using GPT3.5 is less expensive.

[0068] Step 7: Select a large language model as the base model, use the labeled data set in step 6 as the training set, and fine-tune the large language model. By fine-tuning the model parameters, the large language model can output correct and reasonable structured text.

[0069] Step 8: Deploy the fine-tuned large language model to the production environment to extract key information from unlabeled contract texts.

[0070] Through the above steps, the present invention can effectively extract key information from a large number of complex industrial contracts, improve the automation level of contract management and the efficiency of information processing. In addition, the system design of the present invention allows for the flexible addition of new contract types, and quickly adapts to new contract texts by re-clustering and fine-tuning the model.

[0071] The industrial contract text in step one includes contracts of bidding, procurement, sales, transportation, and service types, and the contract content includes two parts: qualification text and requirement text.

[0072] The cluster analysis method in step 2 is the K-means method. Optionally, there is also a hierarchical analysis method and a labeled classification method. The indicator for determining the number of cluster centers is the sum of squares of the distances from the samples to the cluster centers. The mathematical expression of the K-means clustering algorithm is to minimize Among them, k is the number of cluster centers, C i is the i-th cluster, c i is the cluster center of the ith cluster, and x is the sample value.

[0073] The stratified sampling method in step 3 is to extract an equal number of samples for each layer. Optionally, there are also proportional allocation of sampling units and Nayman sampling unit allocation. The expression for allocating sampling units in equal numbers is n i =n / k, where n is the total number of samples drawn, n i is the number of samples drawn from the ith category, and k is the number of categories. The mathematical expression for allocating sampling units proportionally is n i =nN i / N, where N is the total number of sampling units, N i is the total number of sampling units in the ith category. The mathematical expression of the Neyman sampling unit allocation is Where S i is the standard deviation of the ith category.

[0074] The mathematical expression representing the sample annotation in step 4 is the expression of the structured text:

[0075] z i =f(x i );

[0076] Among them, z i is the structured text output by GPT4, x i is the contract text input into GPT4, and f is the character set to character set mapping implemented by GPT4.

[0077] Step 5 Template p i The constructed mathematical expression is:

[0078] p i =(z i ,q i );

[0079] Among them, z i The structured text annotated in step 4, q i is the prompt word for the i-th category used to guide GPT3.5, for example, “Please extract structured text from unstructured text”.

[0080] The mathematical expression of batch labeling in step 6 is:

[0081] Y i =g(p i , x i );

[0082] Among them, p i is the template of the i-th category constructed in step 5, x i is the contract text, Y i is the labeled data of the i-th category, and g is the character set to character set mapping implemented by GPT3.5;

[0083] The step six is ​​used to integrate all the labeled data sets as training sets. The mathematical expression is:

[0084] In this embodiment, GPT4 can be used to generate data for data enhancement, construct a new contract text and its corresponding annotation to supplement the training set. The mathematical expression is: where Y gen To generate a new contract text dataset.

[0085] The large language model in step 7 is Qianwen 1.5-14B-Chat, and optionally all open source large language models. The mathematical expression predicted by the large language model is in is the prediction result output by the model, x is the input contract text, is the character set, and h is the character set to character set mapping implemented by the selected large language model. The parameter fine-tuning method is LoRA fine-tuning, and optionally there are full parameter adjustment and QLoRA parameter fine-tuning.

[0086] The mathematical expression for extracting the key information of the unmarked contract text in step eight is: h is the character set to character set mapping implemented by the large language model after fine-tuning in step 7.

[0087] The present invention also provides a complex industrial contract information extraction system based on a large language model. The complex industrial contract information extraction system based on a large language model can be implemented by executing the process steps of the complex industrial contract information extraction method based on a large language model, that is, technical personnel in this field can understand the complex industrial contract information extraction method based on a large language model as a preferred implementation of the complex industrial contract information extraction system based on a large language model.

[0088] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0089] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for generating labeled data based on a large language model, characterized in that: include: Step S1: obtaining structured texts of representative contracts of the same category, and generating corresponding information extraction templates based on the obtained structured texts of the representative contracts; Step S2: based on the information extraction template and all contract texts in the same category, batch generate annotation data; Step S3: Summarize the annotated data of different categories and the corresponding contract texts, and generate an annotated data set, and use the annotated data set to fine-tune the large language model to obtain a fine-tuned large language model; Step S4: Annotate the contract text based on the fine-tuned large language model.

2. The method for generating annotated data based on a large language model according to claim 1, characterized in that: The information extraction template includes: p i =(z i ,q i ) Among them, z i For structured text, q i is the prompt word for the i-th category used to guide GPT3.

5.

3. The method for generating annotated data based on a large language model according to claim 1, characterized in that: The step S2 includes: using GPT3.5 to extract content from all contract texts in the cluster according to the corresponding information extraction template in the cluster, and annotating the extracted content according to the information extraction template to obtain annotated data; The annotation data includes: Y i =g(p i ,x i ); Among them, p i is the information extraction template of the constructed i-th category, x i is the contract text, Y i is the labeled data of the i-th category, and g is the character set to character set mapping implemented by GPT3.

5.

4. The method for generating annotated data based on a large language model according to claim 1, characterized in that: The method also includes: using GPT4 to generate data for data enhancement, constructing a new contract text and its corresponding annotation; its expression is Among them, Y gen To generate a new contract text dataset.

5. The method for generating annotated data based on a large language model according to claim 1, characterized in that: The step S3 includes: fine-tuning the large language model using the annotated data set, including: at least one of LoRA fine-tuning, full parameter adjustment, and QLoRA parameter fine-tuning.

6. A system for generating annotated data based on a large language model, characterized in that: include: Module M1: Obtain structured texts of representative contracts of the same category, and generate corresponding information extraction templates based on the structured texts of the representative contracts obtained; Module M2: Batch generate annotation data based on the information extraction template and all contract texts in the same category; Module M3: Aggregate the annotated data of different categories and the corresponding contract texts, generate an annotated data set, and use the annotated data set to fine-tune the large language model to obtain the fine-tuned large language model; Module M4: Annotate the contract text based on the fine-tuned large language model.

7. The method for generating annotated data based on a large language model according to claim 6, characterized in that: The information extraction template includes: p i =(z i ,q i ) Among them, z i For structured text, q i is the prompt word for the i-th category used to guide GPT3.

5.

8. The large language model-based annotation data generation system according to claim 6, characterized in that: The module M2 includes: using GPT3.5 to extract content from all contract texts in the cluster according to the corresponding information extraction template in the cluster, and annotating the extracted content according to the information extraction template to obtain annotated data; The annotation data includes: Y i =g(p i ,x i ); Among them, p i is the information extraction template of the constructed i-th category, x i is the contract text, Y i is the labeled data of the i-th category, and g is the character set to character set mapping implemented by GPT3.

5.

9. The system for generating annotated data based on a large language model according to claim 6, characterized in that: The system also includes: using GPT4 to generate data for data enhancement, constructing a new contract text and its corresponding annotation; its expression is Among them, Y gen To generate a new contract text dataset.

10. The system for generating annotated data based on a large language model according to claim 6, characterized in that: The module M3 includes: fine-tuning the large language model using the labeled data set, including: at least one of LoRA fine-tuning, full parameter adjustment, and QLoRA parameter fine-tuning.

Images