Intelligent data auditing method and system based on large language model fine tuning

Abstract

The invention provides an intelligent data auditing method and system based on large language model fine tuning. The method comprises the steps that A1, a cue word template is constructed for an auditing task; a2, acquiring original business data, and forming a labeling data set of each auditing task based on the cue word template; step A3, randomly disrupting and mixing the annotation data set of each auditing task to form a fine adjustment data set; step A4, performing fine tuning on the pre-trained large language model by using the fine tuning data set to obtain a fine-tuned large language model; step A5, assembling the business data to be audited according to the prompt word templates of the audit tasks to form prompt words of the audit tasks; and A6, reasoning the assembled cue word by using the fine-tuned large language model, and outputting an auditing result. The understanding of the large language model on industry knowledge is deepened, and the auditing accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of large models and bill review technology, and in particular to a data intelligent review method and system based on large language model fine-tuning. Background Art

[0002] Existing intelligent business review methods, such as Patent Application 1 (CN202311505324.6), disclose a judicial case quality intelligent evaluation system and method based on big data, with the following steps: (1) Establish an evaluation index database set and a judicial semantic element database set, build an index element association set between the evaluation index and the judicial semantic element, and add an association label to the judicial case, and establish a database first contact channel and a database second contact channel; (2) respectively analyze the importance of the judicial semantic element to the evaluation index, and calculate the evaluation label value of any evaluation index selected when evaluating any judicial case; (3) Find judicial cases similar to the judicial case to be evaluated, form a set of evaluation indicators to be selected, calculate the candidate value of any evaluation indicator selected from the set of evaluation indicators to be selected, and analyze and select all evaluation indicators of the judicial case to be evaluated based on the candidate value. The evaluation index corpus of Patent Application 1 is manually maintained. If the corpus is not rich, the indicators matched by the case are likely to be inaccurate; secondly, the case and the evaluation index are matched by building an index element association set, and there is no matching through the semantics of the text, so there may be matching errors.

[0003] For example, Patent Application 2 (202311721159.8) discloses a certificate review method and system based on a large language model, and the steps are as follows: (1) Import the main text and standard template into the constructed large prediction model to generate an industry procedure library; (2) Match the procedure category of the certificate to be reviewed with the procedures stored in the industry procedure library; (3) Segment each item to be tested of the certificate to be reviewed based on the corresponding procedures; (4) Review the segmented items to be tested separately; (5) Summarize the review results and draw a review conclusion. The large model in Patent Application 2 is not fine-tuned with industry data, and the review effect may not be good for some industry-specific terms. Summary of the invention

[0004] Based on the above problems, the present invention provides a data intelligent audit method and system based on large language model fine-tuning, aiming to solve the technical problem that the audit process of the prior art has poor effect on the audit of business in a specific industry.

[0005] The present invention provides a data intelligent audit method based on fine-tuning of a large language model, which is used to audit business data in a predetermined business field, including:

[0006] Step A1, constructing a prompt word template for each type of review task;

[0007] Step A2, obtaining original business data of several predetermined business fields, and forming a labeled data set for each audit task based on the original business data and the prompt word template;

[0008] Step A3, randomly shuffle and mix the labeled data sets of each audit task to form a fine-tuning data set;

[0009] Step A4, fine-tuning the pre-trained large language model using the fine-tuning dataset to obtain a fine-tuned large language model;

[0010] Step A5: input the business data to be reviewed, and assemble the business data to be reviewed according to the prompt word templates of each review task to form the prompt words of each review task;

[0011] Step A6, use the fine-tuned large language model to infer the prompt words of each assembled audit task, and output the audit results corresponding to each audit task.

[0012] Furthermore, in step A1, one of the types of the audit task is a spelling correction task. In step A2, the process of forming a labeled data set for the spelling correction task includes:

[0013] Step B11, obtaining original business data of a number of predetermined business fields, and randomly sampling the original business data to obtain a number of first original data items;

[0014] Step B12, randomly extracting a number of original spelling items from each first original data item, and replacing the extracted original spelling items with incorrectly spelled items to form incorrectly spelled data items;

[0015] Step B13, forming a spelling correction corpus according to the constructed spelling error data items;

[0016] Step B14, processing the spelling correction corpus according to the prompt word template of the spelling correction task to form a labeled data set for the spelling correction task.

[0017] Furthermore, in step A1, one of the types of the audit task is a grammatical error correction task. In step A2, the process of forming a labeled data set for the grammatical error correction task includes:

[0018] Step B21, obtaining original business data of a number of predetermined business fields, and randomly extracting a number of second original data items from the original business data;

[0019] Step B22, according to the pre-defined grammatical error form, modify each second original data item according to a randomly selected grammatical error form to form a grammatical error data item;

[0020] Step B23, forming grammatical error correction corpus according to the constructed grammatical error data items;

[0021] Step B24, processing the grammatical correction corpus according to the prompt word template of the grammatical correction task to form a labeled data set for the grammatical correction task.

[0022] Furthermore, in step A1, one of the types of audit tasks is a context logic error correction task. In step A2, the process of forming a labeled data set for the context logic error correction task includes:

[0023] Step B31, obtaining original business data of several predetermined business fields, and generating business draft data of several predetermined business fields based on the pre-trained large language model;

[0024] Step B32, extracting context-related text pairs from the original business data as first text pair data by using regular expressions, and extracting context-related text pairs from the business draft data as second text pair data by using regular expressions;

[0025] Step B33, annotating the first text pair data according to the prompt word template of the context logic error correction task to obtain a first annotation result, and annotating the second text pair data according to the prompt word template of the context logic error correction task to obtain a second annotation result;

[0026] Step B34, extract the annotation results that are positive samples from the first annotation results as the first positive samples, extract the annotation results that are negative samples from the second annotation results as the first negative samples, and use the first positive samples and the first negative samples to form an annotation data set for the contextual logic error correction task.

[0027] Furthermore, in step A1, one of the types of the audit task is a writing standard error correction task. In step A2, the process of forming a labeled data set for the writing standard error correction task includes:

[0028] Step B41, respectively obtaining original business data of predetermined business fields within the first predetermined range and the second predetermined range;

[0029] Step B42, annotating the original business data within the first predetermined range according to the prompt word template of the writing standard error correction task to obtain a third annotation result, and annotating the original business data within the second predetermined range according to the prompt word template of the writing standard error correction task to obtain a fourth annotation result;

[0030] Step B43, extract the annotation results that are positive samples from the third annotation results as the second positive samples, extract the annotation results that are negative samples from the fourth annotation results as the second negative samples, and use the second positive samples and the second negative samples to form an annotation data set for the writing standard error correction task.

[0031] Furthermore, in step B13, the spelling correction corpus also includes at least part of the data in the open source dataset SIGHAN2015.

[0032] Furthermore, in step B23, the grammatical error correction corpus also includes at least part of the data in the open source dataset FCGEC.

[0033] Furthermore, the prompt word template for the writing standard error correction task includes prompt word sub-templates for various chapter types.

[0034] Furthermore, the first predetermined range and the second predetermined range are both geographical ranges.

[0035] The present invention also provides a data intelligent audit system based on large language model fine-tuning, characterized in that it is used to execute the aforementioned data intelligent audit method based on large language model fine-tuning, comprising:

[0036] Template building module, used to build prompt word templates for each type of review task;

[0037] The annotation data forming module is connected to the template building module, and is used to obtain the original business data of several predetermined business fields, and form the annotation data set of each audit task based on the original business data and the prompt word template;

[0038] A fine-tuning data forming module is connected to the annotated data forming module and is used to randomly shuffle and mix the annotated data sets of each audit task to form a fine-tuning data set;

[0039] The model fine-tuning module is connected to the fine-tuning data to form a module, and uses the fine-tuning data set to fine-tune the pre-trained large language model to obtain and store the fine-tuned large language model;

[0040] The assembly module is connected to the template construction module and is used to assemble the inputted business data to be audited according to the prompt word templates of each audit task, and assemble them into prompt words of each audit task;

[0041] The audit reasoning module is connected to the assembly module and is used to use the fine-tuned large language model to infer the prompt words of each assembled audit task and output the audit results corresponding to each audit task.

[0042] The beneficial technical effect of the present invention is that the present invention collects industry annotation data based on business data of a specific industry, and fine-tunes the large language model through the industry annotation data, so that the large language model can deepen the understanding of the knowledge and terminology of the industry and improve the accuracy of the audit. In addition, by annotating the data with prompt words, the cost of manual annotation is greatly saved and the annotation cycle is shortened. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1-5 A flowchart of the steps of a data intelligent audit method based on fine-tuning of a large language model according to the present invention;

[0044] Figure 6 This is a module schematic diagram of a data intelligent audit system based on large language model fine-tuning of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0047] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0048] See also Figure 1 The present invention provides a data intelligent audit method based on fine-tuning of a large language model, which is used to audit business data in a predetermined business field, including:

[0049] Step A1, constructing a prompt word template for each type of review task;

[0050] Step A2, obtaining original business data of several predetermined business fields, and forming a labeled data set for each audit task based on the original business data and the prompt word template;

[0051] Step A3, randomly shuffle and mix the labeled data sets of each audit task to form a fine-tuning data set;

[0052] Step A4, fine-tuning the pre-trained large language model using the fine-tuning dataset to obtain a fine-tuned large language model;

[0053] Step A5: input the business data to be reviewed, and assemble the business data to be reviewed according to the prompt word templates of each review task to form the prompt words of each review task;

[0054] Step A6, use the fine-tuned large language model to infer the prompt words of each assembled audit task, and output the audit results corresponding to each audit task.

[0055] The present invention collects industry annotation data based on business data of a specific industry, and fine-tunes the large language model through the industry annotation data, so that the large language model can deepen the understanding of the knowledge and terminology of the industry and improve the audit accuracy. In addition, the data is annotated with prompt words, which greatly saves the cost of manual annotation and shortens the annotation cycle.

[0056] Specifically, the pre-trained large language model is the chatglm3-6b large model.

[0057] Furthermore, the predetermined business field is, for example, the judicial field, and the business data is a bill. For example, the original business data refers to an existing bill that has been used. The business data to be reviewed is, for example, a draft legislation. In order to improve the efficiency of legislators in reviewing bills in the process of drafting legislation, the chatglm3-6b model is used to assist legislators in reviewing bills, and the industry data is collected and annotated to fine-tune the model, so as to deepen their understanding of the industry knowledge in the judicial field and enable them to review more accurately.

[0058] As a preferred option, the types of audit tasks include spelling correction tasks, grammar correction tasks, contextual logic correction tasks, and industry standard correction tasks. In the judicial field, the fine-tuned large language model has the following audit capabilities:

[0059] 1. Review of spelling errors in the bill;

[0060] 2. Review of grammatical errors in the bill;

[0061] 3. Review of irregularities in the wording of the bill;

[0062] 4. Review of contextual logical errors in the bill.

[0063] Specifically, in step A1, the prompt word templates of different types of audit tasks must have a certain degree of distinction from each other, so that the large model can distinguish each audit task well. In step A2, the prompt word templates are labeled based on the large language model.

[0064] In general, the intelligent audit of the present invention mainly consists of three parts. The first is to collect business data and annotate it in some way to obtain annotated data, which is used to fine-tune a large language model for high-accuracy audits in the predetermined business field. Different types of audit tasks need to construct their own prompt word templates, and they must be distinguishable from each other, so that the large model can distinguish each audit task well. Then, the large language model is fine-tuned using the annotated data, and finally the fine-tuned large language model is deployed. For each input business data item to be audited, the large language model is used to audit various types of audit tasks.

[0065] The fine-tuning data set of the present invention is constructed by the original business data of the predetermined industry field. Through fine-tuning, the large language model can well understand the business information of the predetermined industry field, such as the bill field. In the model deployment reasoning part, the business data to be reviewed input in the present invention will be assembled with the prompt words of various types of review tasks respectively, and then the fine-tuned large language model will be used for reasoning review. For example, in the field of bills, each time a legal provision is input, four types of review tasks, spelling correction tasks, grammar correction tasks, contextual logic correction tasks, and industry specification correction tasks, will be performed at the same time, except that the prompt word assembly process is different.

[0066] See also Figure 2 Further, in step A1, one of the types of audit tasks is a spelling correction task. In step A2, the process of forming a labeled data set for the spelling correction task includes:

[0067] Step B11, obtaining original business data of a number of predetermined business fields, and randomly sampling the original business data to obtain a number of first original data items;

[0068] Step B12, randomly extracting a number of original spelling items from each first original data item, and replacing the extracted original spelling items with incorrectly spelled items to form incorrectly spelled data items;

[0069] Step B13, forming a spelling correction corpus according to the constructed spelling error data items;

[0070] Step B14, processing the spelling correction corpus according to the prompt word template of the spelling correction task to form a labeled data set for the spelling correction task.

[0071] By obtaining an annotated data set for the spelling correction task through business data in a predetermined business field for model fine-tuning, the fine-tuned large language model can be improved to better adapt to the spelling correction in the predetermined business field. In addition, in step B13, the spelling correction corpus also includes at least part of the data in the open source data set SIGHAN2015, and part of the general spelling correction corpus is added to construct an annotated data set to improve the versatility of spelling correction, so that the fine-tuned large language model can perform both proprietary domain spelling review and general spelling review.

[0072] In step B11, if it is the field of judicial bills, the first original data item is the legal provision.

[0073] In step B12, when the first original data item is in Chinese, the original spelling item refers to a single word in the first original data item. The specific process of replacing a single word with an incorrect word is: using a tool to pick up the pinyin of the word, randomly select a word from the words with the same pinyin as the word to replace it, forming a wrong word, that is, an incorrect spelling item, thereby obtaining an incorrect data item.

[0074] Specifically, the annotation field content of the prompt template for the spelling correction task includes the subscript of the misspelled word, the misspelled word, and the corrected word.

[0075] For the prompt template of the spelling correction task, for example, the marked json result is: {"instruction": "You are an expert in identifying misspelled words. Please find the misspelled words in the text. The text is:", "input": "Text to be corrected", "output": [{"wrong_id": 5, "wrong_word": "嗓", "correct_word": "噪"}]}, where instruction represents the system fine-tuning instruction, explaining the requirements of the review task; input is the text to be corrected; output is the correction result, including wrong_id (subscript id of the misspelled word), wrong_word (misspelled word), and correct_word (corrected word).

[0076] The present invention optimizes the data annotation method for spelling correction. Spelling correction requires outputting the subscript of the misspelled word. However, when the large language model outputs the corrected text, it often adds other words for modification, which causes the subscript of the original misspelled word to not accurately correspond in the corrected text. Therefore, the prompt template redesigned by the present invention only annotates the subscript and the misspelled word of the misspelled word, without annotating the complete corrected text. In this way, the fine-tuned large language model only needs to output the corrected misspelled word and the subscript of the misspelled word, so that the subscript and the misspelled word are not easily misaligned. The subscript of the misspelled word is mainly used to mark and indicate the position of the misspelled word, which helps to more accurately locate and correct the error.

[0077] As a preference, the instruction field can be used as the review task instruction. For example, for spelling correction, "instruction: Please identify the places with misspelled words in the text" and for grammar correction, "instruction: Please identify the places with grammar errors in the text". The instructions for grammar correction and spelling correction are relatively similar. In order to better enable the model to distinguish the two tasks according to the instructions, the spelling correction is changed to "You are an expert in identifying misspelled words. Please find the misspelled words in the text", and the grammar correction is changed to "You are an expert in grammar detection. Please detect whether the grammar in the text is unsmooth". After the adjustment of the prompt template task instruction, the large language model can well distinguish between grammar and spelling correction tasks. Since the large language model needs to perform multiple review tasks together, in this way, the large language model's ability to distinguish tasks can be achieved by optimizing the prompt template, such as optimizing the prompt instruction.

[0078] See Figure 3Furthermore, in step A1, one of the types of the audit task is a grammatical error correction task. In step A2, the process of forming a labeled data set for the grammatical error correction task includes:

[0079] Step B21, obtaining original business data of a number of predetermined business fields, and randomly extracting a number of second original data items from the original business data;

[0080] Step B22, according to the pre-defined grammatical error form, modify each second original data item according to a randomly selected grammatical error form to form a grammatical error data item;

[0081] Step B23, forming grammatical error correction corpus according to the constructed grammatical error data items;

[0082] Step B24, processing the grammatical correction corpus according to the prompt word template of the grammatical correction task to form a labeled data set for the grammatical correction task.

[0083] Obtaining an annotated data set for the grammatical correction task through business data in a predetermined business field for model fine-tuning can improve the fine-tuned large language model to better adapt to the grammatical correction in the predetermined business field. In addition, in step B23, the spelling correction corpus also includes at least part of the data in the open source data set FCGEC, and some general grammatical correction corpus is added to construct an annotated data set to improve the versatility of grammatical correction, so that the fine-tuned large language model can perform both proprietary domain grammar review and general grammar review.

[0084] In step B21, the predetermined business domain is, for example, the judicial bill domain, the original business data is a legal provision, and the second original data item is also a legal provision.

[0085] In step B22, several grammatical error forms are pre-defined, and for each second original data item, a grammatical error form is randomly selected to automatically modify the second original data item to form a grammatical error data item.

[0086] Specifically, the annotation field content of the prompt word template of the grammatical error correction task includes the text to be corrected, the text after correction, and the reason for explaining the error.

[0087] For the prompt word template of the grammar correction task, the json result is, for example: {"instrcution":"You are a grammar detection expert, please check whether the grammar in the text is not smooth. The text is: ","input":"Text to be corrected","output":{"label":1,"correct_text":"Text after correction","explain":"Reason for correction"}}, where instruction represents the system fine-tuning instructions, explaining the audit task requirements; input is the text to be corrected; output is the correction result, including label (whether there is a grammatical error), correct_text (corrected text), explain (explain the reason for the error). The prompt word template of the present invention adds explain (explain the reason for the error), which is better used for model fine-tuning and provides ideas to auditors when performing grammar audits later.

[0088] See also Figure 4 Further, in step A1, one of the types of audit tasks is a context logic error correction task. In step A2, the process of forming a labeled data set for the context logic error correction task includes:

[0089] Step B31, obtaining original business data of several predetermined business fields, and generating business draft data of several predetermined business fields based on the pre-trained large language model;

[0090] Step B32, extracting context-related text pairs from the original business data as first text pair data by using regular expressions, and extracting context-related text pairs from the business draft data as second text pair data by using regular expressions;

[0091] Step B33, annotating the first text pair data according to the prompt word template of the context logic error correction task to obtain a first annotation result, and annotating the second text pair data according to the prompt word template of the context logic error correction task to obtain a second annotation result;

[0092] Step B34, extract the annotation results that are positive samples from the first annotation results as the first positive samples, extract the annotation results that are negative samples from the second annotation results as the first negative samples, and use the first positive samples and the first negative samples to form an annotation data set for the contextual logic error correction task.

[0093] Optimize the effect of distinguishing positive and negative samples of contextual logic annotation data. The annotation data set of the contextual logic error correction audit task is extracted based on the original business data. However, given the limited effect of prompt word annotation and the limited accuracy of annotated positive and negative samples, we optimize the strategy to make the extracted positive and negative samples more accurate, so that the fine-tuning model can distinguish positive and negative samples more accurately and improve the audit effect. The specific strategy is shown in steps B31-B34.

[0094] For example, in the field of judicial bills, for the construction of the first positive sample, because the context-related text pairs in the real published bills must be logically correct, the logically related text pairs are extracted from the real bill data through regular expressions, and then the real bill data is extracted and annotated through the large model according to the written prompt word template. The accuracy of the positive samples (samples with correct context logic) in all the first annotation results are extracted to ensure the accuracy of the positive samples. For the construction of the first negative sample, the present invention generates a batch of draft bills through a large language model. Because the large language model generates a batch of draft bills without manual correction, it is inevitable that there will be logically incorrect text pairs. Therefore, the text pairs are extracted from the draft bills generated by the large language model through regular expressions, and then annotated with the large model, and all negative samples (context logic errors) in the second annotation results are extracted as negative samples, which can also improve the accuracy of negative samples to a certain extent.

[0095] Furthermore, the process of forming a labeled data set for the contextual logic error correction task of the present invention also includes manually performing contextual logic modification on some text pairs in the first text pair data to modify them into text pairs with contextual logic errors, and using them as the first negative samples in the labeled data set for the contextual logic error correction task after manual annotation, thereby increasing the number of first negative samples.

[0096] Specifically, the annotation field content of the prompt word of the contextual logic error correction task includes the judgment result of the contextual logic error and the reason for explaining the logic error.

[0097] For the prompt word template of the context logic error correction task, the json result is, for example: {"instrcution":"You are a context logic error detection expert. You need to judge whether there is a logical error in the content between the two given texts. The text is:","input":"The text spliced ​​in the context.","output":{"label":1,"explain":"The reason for the error"}}, where instruction represents the system fine-tuning instructions and explains the task requirements; input is the text to be corrected; output is the correction result, including label (whether there is a logical error) and explain (explaining the reason for the logical error). Label indicates the judgment result of the context logic error, that is, whether there is a context logic error.

[0098] If there is no context-related text pair and only the above text exists, the large language model will not be called and the result with no logical association will be directly returned.

[0099] See also Figure 5 Furthermore, in step A1, one of the types of the audit task is a standard text correction task. In step A2, the process of forming a labeled data set for the standard text correction task includes:

[0100] Step B41, respectively obtaining original business data of predetermined business fields within the first predetermined range and the second predetermined range;

[0101] Step B42, annotating the original business data within the first predetermined range according to the prompt word template of the writing standard error correction task to obtain a third annotation result, and annotating the original business data within the second predetermined range according to the prompt word template of the writing standard error correction task to obtain a fourth annotation result;

[0102] Step B43, extract the annotation results that are positive samples from the third annotation results as the second positive samples, extract the annotation results that are negative samples from the fourth annotation results as the second negative samples, and use the second positive samples and the second negative samples to form an annotation data set for the writing standard error correction task.

[0103] Furthermore, the process of forming a labeled data set for the writing standard error correction task of the present invention also includes manually modifying part of the original business data into data with writing standard errors through manual annotation, and using the data as the second negative samples in the labeled data set for the writing standard error correction task after manual annotation, thereby increasing the number of second negative samples.

[0104] The annotated data set of the writing standard error correction task also comes from the original business data and has the same inaccuracy problem as the writing standard. The optimization strategy for the positive and negative samples in the annotated data set of the writing standard error correction task is shown in steps B41-B43.

[0105] Further, the first predetermined range and the second predetermined range are both, for example, geographical ranges. The first predetermined range and the second predetermined range do not overlap.

[0106] For example, in the field of judicial bills, if the writing standards of bills in area A may be different from those in other areas, when reviewing the bills in area A, the bill data in area A is used to annotate the bills using the prompt words using the big model, and then the positive samples (i.e., those that meet the writing standards of area A) are extracted from all the annotated results as the final second positive samples. When constructing the second negative sample, the bill data in other areas except area A are selected to annotate the data using the big model according to the prompt words, and then all the negative samples (those that do not meet the writing standards of area A) are extracted from the annotated results as the final negative samples, so as to improve the discrimination of the correction of writing standards.

[0107] Specifically, the prompt word template for the writing standard error correction task includes prompt word sub-templates for various chapter types. Since the writing standards of different types of chapters are different, different prompt word templates are designed for different types of chapters. When annotating, different prompt words are automatically associated with the type of chapter for data annotation. For example, in the field of judicial bills, general chapters use a prompt word sub-template of the subset, the detailed rules use their own prompt word sub-template, and the appendix uses its own prompt word sub-template.

[0108] Specifically, the annotation field content of the prompt word template of the writing standard error correction task includes the behavior standard error judgment result, the corrected text, and an explanation of how to correct it.

[0109] For the prompt word template of the writing standard error correction task, the json result is, for example: {"instrcution":"You are a writing standard detection expert. You need to judge whether the writing conforms to the standard based on the given text. The text is:","input":"Text to be detected","output":{"label":1,"correct_text":"Corrected text","explain":"Explain how to correct it"}}. The result of the behavior standard error judgment is the label. The label of 1 indicates that there is non-standard behavior.

[0110] The types of audit tasks in the present invention mainly include spelling correction tasks, grammar correction tasks, contextual logic correction tasks and industry standard correction tasks. In order to make the fine-tuned model have a good task differentiation effect and be able to distinguish different types of audit tasks, the present invention has partially optimized the fine-tuned prompt words and the strategy for constructing positive and negative samples to make it clearer in expression and have a good instruction and reasoning differentiation effect, so that the large language model can perform reasoning better.

[0111] See also Figure 6 The present invention also provides a data intelligent audit system based on large language model fine-tuning, characterized in that it is used to execute the aforementioned data intelligent audit method based on large language model fine-tuning, including:

[0112] A template building module (1), used to build a prompt word template for each type of review task;

[0113] The annotated data forming module (2) is connected to the template building module (1) and is used to obtain original business data of a number of predetermined business fields and form an annotated data set for each audit task based on the original business data and the prompt word template;

[0114] A fine-tuning data forming module (3), connected to the annotated data forming module (2), is used to randomly shuffle and mix the annotated data sets of each audit task to form a fine-tuning data set;

[0115] A model fine-tuning module (4) is connected to the fine-tuning data forming module (3), and uses the fine-tuning data set to fine-tune the pre-trained large language model to obtain and store the fine-tuned large language model;

[0116] An assembly module (5) is connected to the template construction module (1) and is used to assemble the inputted business data to be audited according to the prompt word templates of each audit task, and assemble them into prompt words of each audit task;

[0117] The audit reasoning module (6) is connected to the assembly module (5) and is used to use the fine-tuned large language model to reason about the prompt words of each assembled audit task and output the audit results corresponding to each audit task.

[0118] Furthermore, the predetermined business field is, for example, the judicial field, and the business data is a bill. For example, the original business data refers to an existing bill that has been used. The business data to be reviewed is, for example, a draft legislation. In order to improve the efficiency of legislators in reviewing bills in the process of drafting legislation, the chatglm3-6b model is used to assist legislators in reviewing bills, and the industry data is collected and annotated to fine-tune the model, so as to deepen their understanding of the industry knowledge in the judicial field and enable them to review more accurately.

[0119] The above are only preferred embodiments of the present invention, and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the description and illustrations of the present invention should be included in the protection scope of the present invention.

Claims

1. A data intelligent audit method based on fine-tuning of a large language model, used to audit business data in a predetermined business field, characterized in that: include: Step A1, constructing a prompt word template for each type of review task; Step A2, obtaining a number of original business data of the predetermined business field, and forming a labeled data set for each audit task based on the original business data and the prompt word template; Step A3, randomly shuffle and mix the labeled data sets of each audit task to form a fine-tuning data set; Step A4, fine-tuning the pre-trained large language model using the fine-tuning dataset to obtain a fine-tuned large language model; Step A5, inputting the business data to be reviewed, and assembling the business data to be reviewed according to the prompt word templates of each review task to form the prompt words of each review task; Step A6, using the fine-tuned large language model to infer the prompt words of each of the assembled audit tasks, and outputting the audit results corresponding to each audit task.

2. The data intelligent audit method based on large language model fine-tuning according to claim 1, characterized in that: In the step A1, one of the types of audit tasks is a spelling correction task. In the step A2, the process of forming a labeled data set for the spelling correction task includes: Step B11, obtaining a number of original business data of the predetermined business field, and randomly sampling the original business data to obtain a number of first original data items; Step B12, randomly extracting a number of original spelling items from each of the first original data items, and replacing the extracted original spelling items with incorrectly spelled items to form incorrectly spelled data items; Step B13, forming a spelling correction corpus according to the constructed spelling error data items; Step B14, processing the spelling correction corpus according to the prompt word template of the spelling correction task to form a labeled data set for the spelling correction task.

3. The data intelligent audit method based on large language model fine-tuning according to claim 1, characterized in that: In the step A1, one of the types of audit tasks is a grammar correction task. In the step A2, the process of forming a labeled data set for the grammar correction task includes: Step B21, obtaining a number of original business data of the predetermined business field, and randomly extracting a number of second original data items from the original business data; Step B22, according to a pre-defined grammatical error form, modify each of the second original data items according to a randomly selected grammatical error form to form a grammatical error data item; Step B23, forming grammatical error correction corpus according to the constructed grammatical error data items; Step B24, processing the grammatical correction corpus according to the prompt word template of the grammatical correction task to form a labeled data set for the grammatical correction task.

4. The data intelligent audit method based on large language model fine-tuning according to claim 1, characterized in that: In the step A1, one of the types of audit tasks is a context logic error correction task. In the step A2, the process of forming a labeled data set for the context logic error correction task includes: Step B31, obtaining some original business data of the predetermined business field, and generating some business draft data of the predetermined business field based on the pre-trained large language model; Step B32, extracting context-related text pairs from the original business data as first text pair data by using a regular expression, and extracting context-related text pairs from the business draft data as second text pair data by using a regular expression; Step B33, annotating the first text pair data according to the prompt word template of the contextual logic error correction task to obtain a first annotation result, and annotating the second text pair data according to the prompt word template of the contextual logic error correction task to obtain a second annotation result; Step B34, extracting the annotation results that are positive samples from the first annotation results as the first positive samples, extracting the annotation results that are negative samples from the second annotation results as the first negative samples, and using the first positive samples and the first negative samples to form an annotation data set for the contextual logic error correction task.

5. The data intelligent audit method based on large language model fine-tuning according to claim 1, characterized in that: In the step A1, one of the types of audit tasks is a writing standard error correction task. In the step A2, the process of forming a labeled data set for the writing standard error correction task includes: Step B41, respectively obtaining original business data of the predetermined business field within the first predetermined range and the second predetermined range; Step B42, annotating the original business data within the first predetermined range according to the prompt word template of the writing standard error correction task to obtain a third annotation result, and annotating the original business data within the second predetermined range according to the prompt word template of the writing standard error correction task to obtain a fourth annotation result; Step B43, extracting the annotation results that are positive samples from the third annotation results as second positive samples, extracting the annotation results that are negative samples from the fourth annotation results as second negative samples, and combining the second positive samples and the second negative samples to form an annotation data set for the writing standard error correction task.

6. The data intelligent audit method based on large language model fine-tuning according to claim 2, characterized in that: In the step B13, the spelling correction corpus also includes at least part of the data in the open source dataset SIGHAN2015.

7. The data intelligent audit method based on large language model fine-tuning according to claim 3 is characterized in that: In the step B23, the grammatical error correction corpus also includes at least part of the data in the open source dataset FCGEC.

8. The data intelligent audit method based on large language model fine-tuning according to claim 5, characterized in that: The prompt word template for the writing standard error correction task includes prompt word sub-templates for various chapter types.

9. The method for intelligent data audit based on fine-tuning of a large language model as claimed in claim 5, characterized in that: The first predetermined range and the second predetermined range are both geographical ranges.

10. A data intelligent audit system based on large language model fine-tuning, characterized in that: The method for executing a data intelligent audit method based on fine-tuning of a large language model as described in any one of claims 1 to 9 comprises: Template building module, used to build prompt word templates for each type of review task; A labeling data forming module, connected to the template building module, for acquiring original business data of a number of predetermined business areas, and forming a labeling data set for each audit task based on the original business data and the prompt word template; A fine-tuning data forming module is connected to the annotated data forming module and is used to randomly shuffle and mix the annotated data sets of each audit task to form a fine-tuning data set; A model fine-tuning module, connected to the fine-tuning data forming module, uses the fine-tuning data set to fine-tune the pre-trained large language model, obtains the fine-tuned large language model and stores it; An assembly module, connected to the template construction module, for assembling the inputted business data to be audited according to the prompt word templates of each audit task, into prompt words of each audit task; The audit reasoning module is connected to the assembly module and is used to use the fine-tuned large language model to reason about the prompt words of each of the assembled audit tasks, and output the audit results corresponding to each audit task.

Images