Audit rectification opinion generation method and system based on large model efficient fine tuning

Abstract

The invention relates to the technical field of audit, and provides an audit rectification opinion generation system and method based on large model efficient fine tuning, the system comprises a corpus management module, a model training module and a rectification opinion generation module, the corpus management module is used for storing and managing an audit corpus; the model training module comprises a pre-training sub-module used for constructing a pre-training model and carrying out self-supervised pre-training; the fine tuning sub-module is used for performing supervised efficient fine tuning; the noise generation sub-module is used for generating a noise enhancement training database based on an audit corpus and samples generated in the training process; the noise optimization training sub-module is used for performing optimization training according to the audit corpus and the noise enhancement training database; and the rectification opinion generation module is used for inputting the to-be-rectified audit manuscripts into the large audit model to generate corresponding audit rectification opinions, so that the model can still generate stable, consistent and targeted audit rectification opinions when facing noise, and the method is suitable for complex and changeable audit scenes.

Description

Technical Field

[0001] The present invention relates to the field of audit technology, and in particular to a method and system for generating audit rectification opinions based on efficient fine-tuning of a large model. Background Art

[0002] With the rapid growth of enterprise data and increasingly stringent compliance requirements, the complexity and workload of generating audit rectification opinions continue to increase. Traditional manual summarization is not only time-consuming and cumbersome, but also prone to errors. Especially when dealing with a large number of audit opinions, it is difficult for auditors to quickly generate high-quality audit rectification opinions, resulting in low efficiency and failure to meet the audit needs of enterprises. When dealing with complex and changing audit scenarios, the rectification opinions generated by traditional rule-based or template-based generation methods lack pertinence and are difficult to effectively reflect the actual problems and risk points of the enterprise.

[0003] With the rapid development of artificial intelligence technology, especially the development of large-scale pre-trained language models (big models), the capabilities of automated text generation and natural language processing have been significantly improved. Through pre-training, big models can capture important audit language features from audit corpus and adapt to specific audit tasks during fine-tuning. However, although big models have brought significant potential to the audit field, there are still many challenges in practical applications. Big models need to gradually adapt to audit corpus, from preliminary audit language understanding to optimizing task performance and finally coping with complex scenarios, so the standardization requirements for input audit texts are high. However, there are often some noises in the audit texts in actual scenarios, such as input errors or missing content, which will significantly affect the accuracy and completeness of the audit rectification opinions generated by the model. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides an audit rectification opinion generation system based on large model efficient fine-tuning, which can ensure that the model can still generate stable, consistent and targeted audit rectification opinions when faced with noisy data in audit working papers, and is suitable for complex and changeable audit scenarios.

[0005] The technical solution adopted by the present invention is as follows:

[0006] The embodiment of the present invention proposes an audit rectification opinion generation system based on efficient fine-tuning of a large model, the system comprising: a corpus management module, the corpus management module is used to store and manage the audit corpus, the audit corpus includes audit report corpus and audit working paper corpus; a model training module, the model training module is used to train the audit large model, the audit large model includes a first rectification opinion generation sub-model, the first rectification opinion generation sub-model is used to generate corresponding audit rectification opinions according to a single audit working paper to be rectified, the model training module includes a pre-training sub-module, a fine-tuning sub-module, a noise generation sub-module and a noise optimization training sub-module, the pre-training sub-module is used to construct a pre-trained model, and according to the audit The pre-trained model is self-supervised pre-trained with the report corpus; the fine-tuning submodule is used to perform supervised and efficient fine-tuning on the pre-trained model according to the audit working paper corpus to obtain a first preset rectification opinion generation model; the noise generation submodule is used to generate a noise enhanced training database based on the audit corpus and the samples generated during the training process of the fine-tuning submodule; the noise optimization training submodule is used to optimize the training of the first preset rectification opinion generation model according to the audit corpus and the noise enhanced training database to obtain a first rectification opinion generation submodel; the rectification opinion generation module, the rectification opinion generation module is used to input the audit working papers to be rectified into the audit big model to generate corresponding audit rectification opinions.

[0007] In addition, the audit rectification opinion generation system proposed in the present invention may also have the following additional technical features:

[0008] According to one embodiment of the present invention, the corpus management module is used to obtain pre-training data based on the audit report corpus, and each group of pre-training data includes a single audit working paper opinion and a corresponding single audit rectification opinion; the pre-training sub-module is specifically used to construct a pre-training model, and perform low-rank matrix training on the parameter matrix of the pre-training model based on the pre-training data.

[0009] According to one embodiment of the present invention, the corpus management module is also used to obtain fine-tuning training data based on the audit working paper corpus, and each group of the fine-tuning training data includes a single real audit working paper opinion and a single real audit rectification opinion corresponding thereto; the fine-tuning submodule is specifically used to perform supervised and efficient fine-tuning on the pre-trained model based on the fine-tuning training data to obtain a first preset rectification opinion generation model, and when fine-tuning the pre-trained model at different levels, insert a learnable embedding vector into the real audit working paper opinion and use it as extended input data, so that the Transformer decoder of the pre-trained model takes the extended input data of each layer as input and generates output layer by layer.

[0010] According to one embodiment of the present invention, the noise generation submodule is specifically used to: perform noise processing on the audit working paper-like opinions extracted from the audit report corpus to obtain first noise training data; when the fine-tuning submodule performs supervised and efficient fine-tuning on the pre-trained model, based on the confidence of the audit rectification opinions generated by the pre-training model, determine whether the audit rectification opinions and their corresponding inputs are low-quality samples, and use the real audit working paper opinions of the generated low-quality samples and their corresponding real audit rectification opinions as second noise training data; based on the first noise training data and the second noise training data, generate a noise-enhanced training database, the noise-enhanced training database includes noise data, and each group of noise data includes noisy audit working paper opinions and corresponding audit rectification opinions.

[0011] According to one embodiment of the present invention, the noise optimization training submodule is specifically used to: input the audit working paper opinions of the audit corpus and the noise enhanced training database into a first preset rectification opinion generation model to obtain a predicted value of the audit rectification opinion, and determine a weighted loss function based on the predicted value and the true value of the audit rectification opinion; optimize and train the first preset rectification opinion generation model according to the weighted loss function, and in the initial training batch, the proportion of normal fine-tuning training data is 100%, and the proportion of audit working paper opinions of the noise enhanced training database introduced in each subsequent training batch gradually increases until the weighted loss function corresponding to the first preset rectification opinion generation model converges or the preset batch of training for the first preset rectification opinion generation model is completed.

[0012] According to one embodiment of the present invention, the weighted loss function for:

[0013]

[0014] in, is the cross entropy loss function of the noise optimization training submodule, α is the adaptive adjustment weight, and the calculation formula of the adaptive adjustment weight α is:

[0015]

[0016] Among them, x and They represent the audit working papers opinions input into the model and the audit rectification opinions generated by the model. The confidence level of the audit rectification opinions generated by the model, is the perplexity of the audit rectification opinions generated by the model, and λ1 and λ2 represent weight coefficients, which are used to balance the influence of confidence and perplexity.

[0017] According to one embodiment of the present invention, the audit big model also includes a second rectification opinion generation sub-model, which is used to generate corresponding audit rectification opinions based on multiple audit working papers of the same type to be rectified; the corpus management module is also used to merge the fine-tuning training data according to a single real audit rectification opinion of the same type in the fine-tuning training data to obtain merged fine-tuning training data, and each group of the merged fine-tuning data contains multiple real audit working paper opinions and a single real audit rectification opinion corresponding thereto; the model training module also includes a data enhancement sub-module, which is used to construct a data enhancement model, perform data enhancement on the merged fine-tuning training data, and obtain merged enhanced training data; the fine-tuning sub-module is also used to perform supervised and efficient fine-tuning on the first preset rectification opinion generation model based on the merged enhanced training data to obtain a second preset rectification opinion generation model; the noise generation submodule is used to generate a merged noise enhancement training database based on the samples generated during the training process of the first preset rectification opinion generation model by the audit corpus and the fine-tuning submodule; the noise optimization training submodule is also used to optimize the training of the second preset rectification opinion generation model according to the audit corpus and the merged noise enhancement training database to obtain a second rectification opinion generation submodel; when the audit working paper to be rectified is a single audit working paper, the rectification opinion generation module inputs the audit working paper to be rectified into the first rectification opinion generation submodel to generate a corresponding single audit rectification opinion; when the audit working paper with rectification is a plurality of audit working papers of the same type, the rectification opinion generation module inputs the audit working paper to be rectified into the second rectification opinion generation submodel to generate a corresponding single audit rectification opinion.

[0018] In addition, to achieve the above purpose, the present invention also proposes a method for generating audit rectification opinions.

[0019] A method for generating audit rectification opinions based on the audit rectification opinion generating system described above, the method comprising the following steps: S1, obtaining an audit corpus, the audit corpus comprising an audit report corpus and an audit working paper corpus; S2, constructing a pre-trained model, and performing self-supervised pre-training on the pre-trained model based on the audit report corpus; S3, performing supervised and efficient fine-tuning on the pre-trained model based on the audit working paper corpus to obtain a first preset rectification opinion generating model; S4, generating a noise-enhanced training database based on the audit corpus and samples generated during the training process; S5, optimizing and training the first preset rectification opinion generating model based on the audit corpus and the noise-enhanced training database to obtain a first rectification opinion generating sub-model; S6, inputting the audit working paper to be rectified into the audit big model to generate corresponding audit rectification opinions, wherein the audit big model comprises a first rectification opinion generating sub-model, and the first rectification opinion generating sub-model is used to generate corresponding audit rectification opinions based on a single audit working paper to be rectified.

[0020] In addition, to achieve the above objective, the present invention also provides a computer device.

[0021] A computer device comprises: a memory, a processor and an audit rectification opinion generation program stored in the memory and executable on the processor, wherein the audit rectification opinion generation program implements the steps of the audit rectification opinion generation method as described above when executed by the processor.

[0022] In addition, to achieve the above-mentioned purpose, the present invention also provides a storage medium.

[0023] A storage medium stores an audit rectification opinion generation program, which, when executed by a processor, implements the steps of the audit rectification opinion generation method as described above.

[0024] Beneficial effects of the present invention:

[0025] The audit rectification opinion generation system based on efficient fine-tuning of a large model in an embodiment of the present invention generates a noise enhanced training database based on samples generated during the training process of the audit corpus and the fine-tuning submodule, and obtains a first rectification opinion generation sub-model through training based on the audit corpus and the noise enhanced training database. This ensures that the model can still generate stable, consistent and targeted audit rectification opinions when faced with noisy data in the audit working papers, and is suitable for complex and changeable audit scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a block diagram of a system for generating audit rectification opinions based on efficient fine-tuning of a large model according to an embodiment of the present invention;

[0027] Figure 2 A block diagram of a system for generating audit rectification opinions based on efficient fine-tuning of a large model according to an embodiment of the present invention;

[0028] Figure 3 The present invention is a flowchart of a method for generating audit rectification opinions according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] like Figure 1 As shown, the audit rectification opinion generation system based on efficient fine-tuning of a large model according to an embodiment of the present invention includes: a corpus management module, a model training module 20 and a rectification opinion generation module 30.

[0031] The corpus management module 10 is used to store and manage the audit corpus, which includes the audit report corpus and the audit working paper corpus.

[0032] Audit report corpus and audit working paper corpus mainly come from audit reports and related working papers. Among them, audit reports can be saved in the form of word documents, and working papers can be saved in the form of excel. When generating audit report corpus and audit working paper corpus, it is necessary to pre-process the real audit reports and audit working papers after the enterprise's desensitization processing, clean up irrelevant information, format text, and process them in sentences to ensure that the audit big model can effectively learn language patterns based on the audit report corpus and audit working paper corpus in the audit corpus.

[0033] The model training module 20 is used to train the audit model, which includes a first rectification opinion generation sub-model and an optional second rectification opinion generation sub-model. The first rectification opinion generation sub-model is used to generate corresponding audit rectification opinions based on a single audit working paper to be rectified, and the second rectification opinion generation sub-model is used to generate corresponding audit rectification opinions based on multiple audit working papers of the same type to be rectified. The model training module 20 specifically includes a pre-training sub-module 21, a fine-tuning sub-module 22, a noise generation sub-module 23, and a noise optimization training sub-module 24.

[0034] The pre-training submodule 21 is used to construct a pre-training model and perform self-supervised pre-training on the pre-training model based on the audit report corpus.

[0035] In one embodiment of the present invention, the corpus management module 10 can obtain pre-training data based on the audit report corpus, and each set of pre-training data includes a single audit working paper opinion and a corresponding single audit rectification opinion.

[0036] Specifically, assuming that there are m sentences in an audit report corpus, n sentences (which may not be continuous) are automatically extracted based on the text semantic similarity and the text is regarded as a single audit working paper opinion, and the remaining (mn) sentences are regarded as a single audit working paper opinion to form a set of pre-training data. When generating pre-training data, the longest common subsequence should be as long as possible, which is conducive to the pre-training model to analyze complex audit scenarios. It can also reduce noise interference by manually filtering out texts that are too short and paired data that are completely unrelated to audit working paper opinions and audit rectification opinions.

[0037] The pre-training submodule 21 is specifically used to construct a pre-training model and perform low-rank matrix training on the parameter matrix of the pre-training model based on the pre-training data.

[0038] In one embodiment of the present invention, the pre-trained model can be based on the large-scale pre-trained language model ChatGLM2-6B, and the parameter matrix of the model can be decomposed into two low-rank matrices for training. Let the original parameter matrix be W0, and the dimension be Decompose the matrix of this dimension into a low-rank matrix A with dimension and a low-rank matrix B with dimension in Then the weight ΔW after training with the low-rank matrix is:

[0039] ΔW=A·B

[0040] Among them, the dimension of ΔW is the dimension after the dot product of matrix A and matrix B, that is, The dimensions are the same as the original weight W0 of the base model. The matrix addition gives the weight of the pre-trained model after low-rank training:

[0041] W=W0+ΔW

[0042] Pre-trained models can use the cross entropy loss function to calculate the loss

[0043]

[0044] Among them, θ is the model parameter, Is the audit working paper opinion of a given type Next, the model generation audit rectification opinions After low-rank matrix training, the pre-trained model can fully learn the language features in the audit report, strengthen its ability to understand the audit language, and build its preliminary understanding of the language features from audit working paper opinions to audit rectification opinions, laying the foundation for the subsequent generation of audit rectification opinions.

[0045] The fine-tuning submodule 22 is used to perform supervised and efficient fine-tuning on the pre-trained model according to the audit working paper corpus to obtain a first preset rectification opinion generation model.

[0046] In one embodiment of the present invention, the corpus management module 10 is also used to obtain fine-tuning training data based on the audit working paper corpus, and each set of fine-tuning training data includes a single real audit working paper opinion and a corresponding single real audit rectification opinion.

[0047] Specifically, text regularization technology can be used to automatically extract fine-tuning training data. i ={s i1 ,s i2 ,...,s im},s im For the mth sentence in the ith audit working paper corpus, a set of regularized extraction rules Regex can be defined for extracting the true audit working paper opinions and the true audit rectification opinions. draft = "Audit opinion: (.+?)(?=Correction opinion: |$)" and Regex suggest = "Corrective Opinion: (.+?)(?=$)", extract a single real audit working paper opinion x i and a single real audit rectification opinion i Get fine-tuning training data:

[0048] x i =Regex draft (d i )

[0049] y i =Regex suggest (d i )

[0050]

[0051] Eliminate invalid data such as audit working paper opinions that are the same as audit rectification opinions, or audit rectification opinions that are too short (such as "maintain the original opinion") from the automatically extracted paired data, and finally obtain the final fine-tuning training data.

[0052] The fine-tuning submodule 22 can be specifically used to perform supervised and efficient fine-tuning on the pre-trained model according to the fine-tuning training data to obtain a first preset rectification opinion generation model. When fine-tuning the pre-trained model at different levels, a learnable embedding vector is inserted into the actual audit working paper opinion and used as extended input data, so that the Transformer decoder of the pre-trained model takes the extended input data of each layer as input and generates output layer by layer.

[0053] Specifically, for each set of true audit working paper opinions in the fine-tuning training data, that is, input x i ={x i1 ,x i2 ,...,x ik}, where x ik For x i Insert learnable embedding vectors at different levels of the input Where v is the length of the sequence of embedding vectors and l is the number of the insertion layer. These embedding vectors are used to guide the model to optimize the impact of input on output at different levels. The extended input form of the model is:

[0054]

[0055] in, It is a learnable embedding vector used for fine-tuning optimization of different layers. For the Transformer decoder of the pre-trained model, it accepts the extended input data after the embedding vector is expanded. And generate output layer by layer:

[0056]

[0057] in, is the output after each layer of embedding optimization. The audit rectification opinions generated by the model are:

[0058]

[0059] is the final probability distribution generated based on the output of all layers.

[0060] Multi-layer embedding target optimization: Embedding vector optimization allows the model to keep most of its parameters unchanged and only optimizes the embedding vectors of each layer to achieve the goal of enhancing the model's generative ability. Fine-tuning the cross entropy loss function of submodule 22 It can be expressed as:

[0061]

[0062] Among them, θ is the model parameter, It is the final probability distribution generated based on the output of all layers. In order to facilitate training and improve the generation speed, the accuracy of the pre-trained model trained by the pre-training submodule 21 can be quantized to 4 bits. In this way, faster and more accurate audit rectification opinions are generated based on embedding optimization at each layer, improving the logic, accuracy and generalization of the model from a single audit working paper opinion to a single rectification opinion.

[0063] The noise generation submodule 23 is used to generate a noise enhancement training database based on the audit corpus and samples generated during the training process of the fine-tuning submodule 22.

[0064] In one embodiment of the present invention, the noise generation submodule 23 is specifically used to: perform noise processing on the audit working paper-like opinions extracted from the audit report corpus to obtain first noise training data; when the fine-tuning submodule 22 performs supervised and efficient fine-tuning on the pre-trained model, based on the confidence of the audit rectification opinions generated by the pre-training model, determine whether the audit rectification opinions and their corresponding inputs are low-quality samples, and use the real audit working paper opinions of the low-quality samples and their corresponding real audit rectification opinions as second noise training data; based on the first noise training data and the second noise training data, generate a noise-enhanced training database, the noise-enhanced training database includes noise data, and each group of noise data includes noisy audit working paper opinions and corresponding audit rectification opinions.

[0065] Specifically, the audit working paper-like opinions extracted from the audit report corpus can be subjected to format disorder, information missing or vocabulary replacement to generate noisy audit working paper-like opinions, obtain noisy pre-training data, and use it as the first noisy training data. It should be noted that in the process of adding noise, only the audit working paper-like opinions need to be added with noise, and there is no need to add noise to the audit rectification opinions, so as to ensure the accuracy of the rectification opinions.

[0066] To judge the quality of the audit rectification opinions generated by the pre-trained model, a confidence estimation mechanism can be introduced. Confidence indicates the "degree of certainty" of the model in the generated audit rectification opinions. Confidence Defined as:

[0067]

[0068] Among them, T is the length of the generated audit rectification opinion, that is, the total number of subwords in the audit rectification opinion, K is the number of candidate words for each subword in the generated audit rectification opinion, is the kth candidate word for the tth subword that generates the audit rectification opinion. are the 1st to t-1th subwords of the generated audit rectification opinions, It is the probability of generating the kth candidate word of the tth subword in the audit rectification opinion after t-1 words have been generated. It can be understood that when the pre-trained model generates a subword of the audit rectification opinion, there will be multiple candidate words as candidates. Selecting the topK candidate words and arithmetically averaging their selection probabilities can avoid selecting only the top1 candidate words for confidence evaluation and reduce the uncertainty of the results.

[0069] If the audit rectification opinions are generated If the confidence of is lower than the preset threshold τ, it is considered that the opinion may be affected by the noise data, and the opinion and its corresponding input are marked as low-quality samples, thereby obtaining the second noise training data. The preset threshold τ can be determined according to experience or actual needs.

[0070] The noise optimization training submodule 24 is used to optimize the training of the first preset rectification opinion generation model according to the audit corpus and the noise enhancement training database to obtain the first rectification opinion generation submodel.

[0071] In one embodiment of the present invention, the noise optimization training submodule 24 is specifically used to: input the audit working paper opinions of the audit corpus and the noise enhanced training database into the first preset rectification opinion generation model to obtain the predicted value of the audit rectification opinion, and determine the weighted loss function based on the predicted value and the true value of the audit rectification opinion; optimize the training of the first preset rectification opinion generation model according to the weighted loss function, in the initial training batch, the proportion of normal fine-tuning training data is 100%, and the proportion of audit working paper opinions of the noise enhanced training database introduced in each subsequent training batch gradually increases until the weighted loss function corresponding to the first preset rectification opinion generation model converges or the preset batch of training for the first preset rectification opinion generation model is completed.

[0072] In one embodiment of the present invention, the weighted loss function may consider the confidence and perplexity on the basis of the standard cross entropy loss to guide the model to pay more attention to the influence of noisy training data when calculating the loss. for:

[0073]

[0074] in, is the cross entropy loss function of the noise optimization training submodule 24, α is the adaptive adjustment weight, and the calculation formula of the adaptive adjustment weight α is:

[0075]

[0076] Among them, x and They represent the audit working papers opinions input into the model and the audit rectification opinions generated by the model. The confidence level of the audit rectification opinions generated by the model, The perplexity of the audit rectification opinions generated by the model, λ1 and λ2 represent weight coefficients, which are used to balance the influence of confidence and perplexity. The specific values ​​can be set based on experience. When the model has low confidence or high perplexity for a certain input, α increases to guide the model to pay more attention to such inputs when calculating losses.

[0077] Among them, perplexity can be used to measure the uncertainty of the model when generating a sequence, which reflects the difficulty encountered by the model when processing the input. By analyzing the perplexity, the model can perceive the existence of noisy information in the input audit working paper opinions. Used to measure the generation of text sequences The uncertainty of is defined as follows:

[0078]

[0079] Where T is the length of the generated text, is the probability of generating the kth candidate for the tth word in the text after t-1 words have been generated.

[0080] In a specific embodiment of the present invention, the learning rate of the first preset rectification opinion generation model can be initially set to 0.001 and then gradually decreased; the initial proportion of normal fine-tuning training data is 100%, and the 10% noise data introduced in each subsequent training batch causes the proportion of audit working paper opinions in the noise-enhanced training database to gradually increase, so as to ensure that the model gradually learns how to deal with these abnormal data, thereby improving its tolerance to noise.

[0081] In actual audit scenarios, similar audit rectification opinions can be used to deal with audit working papers of the same type but with different opinions on issues. Therefore, multiple audit working papers of the same type can be merged to obtain appropriate audit rectification opinions.

[0082] In one embodiment of the present invention, the audit big model may also include a second rectification opinion generation sub-model, which is used to generate corresponding audit rectification opinions based on multiple audit working papers of the same type to be rectified.

[0083] The corpus management module 10 is also used to merge the fine-tuning training data according to the same type of single real audit rectification opinions in the fine-tuning training data to obtain merged fine-tuning training data. Each group of merged fine-tuning data contains multiple real audit working paper opinions and the corresponding single real audit rectification opinion.

[0084] Specifically, the real audit rectification opinions that are completely the same or the real audit working papers with ROUGE-L index greater than the threshold of 0.85 can be considered as the corresponding real audit rectification opinions of the same type, and similar audit working papers are deduplicated to obtain the merged fine-tuning training data. The ROUGE-L index takes into account the similarity of sentence structure, so when evaluating, it not only focuses on the matching of subwords, but also their order, which can better identify the real audit rectification opinions of the same type.

[0085] Since the combined fine-tuning training data after the merger is relatively scarce, but the actual fine-tuning process relies on a large amount of high-quality data, when faced with scarce audit data, the model often finds it difficult to learn to cope with specific audit tasks. Therefore, it is necessary to perform data augmentation processing on the combined fine-tuning training data.

[0086] like Figure 3 As shown, the model training module 20 also includes a data enhancement submodule 25, which is used to construct a data enhancement model, perform data enhancement on the combined fine-tuning training data, and obtain combined enhanced training data.

[0087] In a specific embodiment of the present invention, the GLM-4-9b-Chat large model can be used as a data augmentation model, which is defined as Generate enhanced audit working paper opinions aug The process is as follows:

[0088]

[0089] Among them, x i ,...,x j y is the selected multiple single real audit working paper opinions, C is the context information that prompts the data enhancement model to generate new audit rectification opinions, such as "Please summarize and generate audit rectification opinions based on the following multiple single audit working paper opinions." It can be adjusted according to actual needs. Repeat multiple times to generate more diverse merged enhanced training data, where each group of merged enhanced training data includes a single enhanced audit working paper opinion y aug and the corresponding single audit rectification opinion.

[0090] The fine-tuning submodule 22 is further used to perform supervised and efficient fine-tuning on the first preset rectification opinion generation model according to the combined enhanced training data to obtain a second preset rectification opinion generation model.

[0091] Here, when fine-tuning the second preset rectification opinion generation model at different levels, a learnable embedding vector can be inserted into the enhanced audit working paper opinion and used as extended input data, so that the Transformer decoder of the second preset rectification opinion generation model takes the extended input data of each layer as input and generates output layer by layer, thereby improving the model's ability to summarize multiple audit working paper opinions to obtain a combined audit rectification opinion.

[0092] The noise generation submodule 23 is also used to generate a merged noise enhancement training database based on the samples generated during the training process of the first preset rectification opinion generation model based on the audit corpus and the fine-tuning submodule 22. It can be generated in a similar manner to the generation of the noise enhancement training database, which will not be repeated here.

[0093] The noise optimization training submodule 24 is also used to optimize the training of the second preset rectification opinion generation model based on the audit corpus and the merged noise enhancement training database to obtain the second rectification opinion generation submodel, which can be trained using a method similar to the training method of the first preset rectification opinion generation model, and will not be repeated here.

[0094] The rectification opinion generating module 30 is used to input the audit working papers to be rectified into the audit big model to generate corresponding audit rectification opinions.

[0095] Specifically, if the audit working papers to be rectified are a single audit working papers, the rectification opinion generation module 30 will input the audit working papers to be rectified into the first rectification opinion generation sub-model to generate a corresponding single audit rectification opinion; if the audit working papers to be rectified are multiple audit working papers of the same type, the rectification opinion generation module 30 will input the audit working papers to be rectified into the second rectification opinion generation sub-model to generate a corresponding single audit rectification opinion.

[0096] In a specific embodiment of the present invention, each deep learning model is implemented based on the deep learning framework Pytorch.

[0097] When training the first rectification opinion generation sub-model: In the pre-training stage, the corpus management module 10 extracted 2783 sets of pre-training data from the audit report corpus. The basic model of the pre-training model is ChatGLM2-6B. The pre-training sub-module 21 sets the learning rate to 5e -5 , the rank r of the low-rank matrix is ​​set to 8, and pre-training is performed based on pre-training data; in the efficient fine-tuning stage, the corpus management module 10 extracts 2446 sets of fine-tuning training data from the audit working paper corpus, and the fine-tuning submodule 22 sets the learning rate to 2e -3, the sequence length of the embedding vector is set to 128. In order to facilitate training, the model accuracy obtained in the pre-training stage is quantized to 4 bits before training to increase the generation speed. In the first noise optimization training stage, the noise optimization training submodule 24 sets the learning rate to 2e -3 The initial value of α is set to 0.1. The accuracy of the first preset rectification opinion generation model obtained in the efficient fine-tuning stage is quantized to 4 bits before training. During the training process, the learning rate is initially set to 0.001 and then gradually decreased. The amount of noise data introduced in each training batch is dynamically adjusted, and 10% of the noise data is introduced in each round of training. Finally, the first rectification opinion generation sub-model is obtained.

[0098] When training the second rectification opinion generation sub-model: the corpus management module 10 obtains 198 sets of combined fine-tuning training data based on the fine-tuning training data. In the data enhancement training stage, the data enhancement sub-module 25 builds a data enhancement model and performs data enhancement on the combined fine-tuning training data to obtain 958 sets of combined enhanced training data. The fine-tuning sub-module 22 sets the learning rate to 2e -3 , the sequence length of the embedding vector is set to 128, and the accuracy of the first preset rectification opinion generation model obtained in the efficient fine-tuning stage is quantized to 4 bits before training; in the second noise optimization training stage, the noise optimization training submodule 24 sets the learning rate to 2e -3 The initial value of α is set to 0.1. The accuracy of the second preset rectification opinion generation model obtained in the data enhancement training phase is quantized to 4 bits before training. During the training process, the learning rate is initially set to 0.001 and then gradually decreased. The amount of noise data introduced in each training batch is dynamically adjusted, and 10% of the noise data is introduced in each round of training. Finally, the first rectification opinion generation sub-model is obtained.

[0099] Through multi-stage training, the audit big model can be better used in real audit scenarios, so that the model has a good compression ratio and maintains high content accuracy when generating audit rectification opinions.

[0100] Two evaluation indicators, output compression ratio and ROUGE indicator, are selected to evaluate the generalization degree and generation quality of audit rectification opinions. Output compression ratio is the ratio of output length to output length. The smaller the output compression ratio, the more concise the generated audit rectification opinions are, that is, the higher the generalization degree is. ROUGE indicator is used to evaluate the similarity between the audit rectification opinions generated by the audit big model and the audit rectification opinions marked by standard auditors. The higher the ROUGE1 / 2 / L, the more consistent the content of the generated audit rectification opinions is with the content given by professional auditors, that is, the higher the generation quality is.

[0101] As shown in Table 1, in the scenario where a single audit working paper to be rectified generates a single audit rectification opinion in an actual audit, the first rectification opinion generation sub-model outperforms the first preset rectification opinion generation model after efficient fine-tuning in terms of output compression ratio and ROUGE1 / 2 / L indicators.

[0102] Table 1

[0103] Model Output compression ratio ROUGE-1 ROUGE-2 ROUGE-L The first preset rectification opinion generation model 83.33％ 37.69％ 25.16％ 28.22％ The first rectification opinion generation sub-model 54.46％ 43.97％ 29.49％ 39.77％

[0104] As shown in Table 2, in the actual audit scenario of generating a single audit rectification opinion from multiple audit working papers of the same type to be rectified, the second rectification opinion generation sub-model is superior to the third rectification opinion generation model obtained by directly performing efficient fine-tuning based on the merged fine-tuning training data in terms of output compression ratio and ROUGE1 / 2 / L indicators.

[0105] Table 2

[0106] Model Output compression ratio ROUGE-1 ROUGE-2 ROUGE-L The third rectification opinion generation model 94.43％ 31.99％ 18.42％ 23.49％ Second rectification opinion generation sub-model 28.76％ 36.22％ 21.00％ 31.57％

[0107] According to the audit rectification opinion generation system based on large model efficient fine-tuning according to the embodiment of the present invention, a noise enhanced training database is generated based on the samples generated during the training process of the audit corpus and the fine-tuning sub-module, and the first rectification opinion generation sub-model is obtained by training the audit corpus and the noise enhanced training database. This can ensure that the model can still generate stable, consistent and targeted audit rectification opinions when faced with noisy data in the audit working papers, and is suitable for complex and changeable audit scenarios.

[0108] Based on the audit rectification opinion generation system based on large model efficient fine-tuning in the above-mentioned embodiment, the present invention also proposes an audit rectification opinion generation method.

[0109] like Figure 3 As shown, the audit rectification opinion generation method of the embodiment of the present invention includes the following steps:

[0110] S1, obtain the audit corpus, which includes the audit report corpus and the audit working paper corpus.

[0111] S2, build a pre-training model and perform self-supervised pre-training on the pre-training model based on the audit report corpus.

[0112] In one embodiment of the present invention, step S2 comprises the following steps:

[0113] S21, obtaining pre-training data based on the audit report corpus, each set of pre-training data includes a single audit working paper opinion and a corresponding single audit rectification opinion.

[0114] S22, constructing a pre-training model, and performing low-rank matrix training on the parameter matrix of the pre-training model based on the pre-training data.

[0115] In one embodiment of the present invention, the pre-trained model can be based on the large-scale pre-trained language model ChatGLM2-6B, and the parameter matrix of the model can be decomposed into two low-rank matrices for training. Let the original parameter matrix be W0, and the dimension be Decompose the matrix of this dimension into a low-rank matrix A with dimension and a low-rank matrix B with dimension in Then the weight ΔW after training with the low-rank matrix is:

[0116] ΔW=A·B

[0117] Among them, the dimension of ΔW is the dimension after the dot product of matrix A and matrix B, that is, The dimensions are the same as the original weight W0 of the base model. The matrix addition gives the weight of the pre-trained model after low-rank training:

[0118] W=W0+ΔW

[0119] Pre-trained models can use the cross entropy loss function to calculate the loss

[0120]

[0121] Among them, θ is the model parameter, Is the audit working paper opinion of a given type Next, the model generation audit rectification opinions After low-rank matrix training, the pre-trained model can fully learn the language features in the audit report, strengthen its ability to understand the audit language, and build its preliminary understanding of the language features from audit working paper opinions to audit rectification opinions, laying the foundation for the subsequent generation of audit rectification opinions.

[0122] S3, based on the audit working papers corpus, the pre-trained model is fine-tuned in a supervised and efficient manner to obtain the first preset rectification opinion generation model.

[0123] In one embodiment of the present invention, step S3 may specifically include:

[0124] S31, obtaining fine-tuning training data based on the audit working paper corpus, each set of fine-tuning training data includes a single real audit working paper opinion and a corresponding single real audit rectification opinion.

[0125] Specifically, text regularization technology can be used to automatically extract fine-tuning training data. i ={s i1 ,s i2 ,...,s im},s imFor the mth sentence in the ith audit working paper corpus, a set of regularized extraction rules Regex can be defined for extracting the true audit working paper opinions and the true audit rectification opinions. draft = "Audit opinion: (.+?)(?=Correction opinion: |$)" and Regex suggest = "Corrective Opinion: (.+?)(?=$)", extract a single real audit working paper opinion x i and a single real audit rectification opinion i Get fine-tuning training data:

[0126] x i =Regex draft (d i )

[0127] y i =Regex suggest (d i )

[0128]

[0129] Eliminate invalid data such as audit working paper opinions that are the same as audit rectification opinions, or audit rectification opinions that are too short (such as "maintain the original opinion") from the automatically extracted paired data, and finally obtain the final fine-tuning training data.

[0130] S32, based on the fine-tuning training data, the pre-trained model is fine-tuned in a supervised and efficient manner to obtain a first preset rectification opinion generation model. When fine-tuning the pre-trained model at different levels, a learnable embedding vector is inserted into the actual audit working paper opinion and used as extended input data, so that the Transformer decoder of the pre-trained model takes the extended input data of each layer as input and generates output layer by layer.

[0131] Specifically, for each set of true audit working paper opinions in the fine-tuning training data, that is, input x i ={x i1 ,x i2 ,...,x ik}, where x ik For x i Insert learnable embedding vectors at different levels of the input Where v is the length of the sequence of embedding vectors and l is the number of the insertion layer. These embedding vectors are used to guide the model to optimize the impact of input on output at different levels. The extended input form of the model is:

[0132]

[0133] in, It is a learnable embedding vector used for fine-tuning optimization of different layers. For the Transformer decoder of the pre-trained model, it accepts the extended input data after the embedding vector is expanded. And generate output layer by layer:

[0134]

[0135] in, is the output after each layer of embedding optimization. The audit rectification opinions generated by the model are:

[0136]

[0137] is the final probability distribution generated based on the output of all layers.

[0138] Multi-layer embedding target optimization: Embedding vector optimization allows the model to keep most parameters unchanged and only optimize the embedding vectors of each layer to achieve the goal of enhancing the model's generation ability. It can be expressed as:

[0139]

[0140] Among them, θ is the model parameter, It is the final probability distribution generated based on the output of all layers. In order to facilitate training and improve the generation speed, the accuracy of the pre-trained model after training the pre-trained submodule can be quantized to 4 bits. In this way, faster and more accurate audit rectification opinions are generated based on embedding optimization at each layer, improving the logic, accuracy and generalization of the model from a single audit working paper opinion to a single rectification opinion.

[0141] S4, generates a noise-enhanced training database based on the audit corpus and samples generated during the training process.

[0142] In one embodiment of the present invention, step S4 may specifically include the following steps:

[0143] S41, performing noise processing on the audit working paper-like opinions extracted from the audit report corpus to obtain first noise training data.

[0144] Specifically, the audit working paper-like opinions extracted from the audit report corpus can be subjected to format disorder, information missing or vocabulary replacement to generate noisy audit working paper-like opinions, obtain noisy pre-training data, and use it as the first noisy training data. It should be noted that in the process of adding noise, only the audit working paper-like opinions need to be added with noise, and there is no need to add noise to the audit rectification opinions, so as to ensure the accuracy of the rectification opinions.

[0145] S42, when the pre-trained model is subjected to supervised and efficient fine-tuning, based on the confidence of the audit rectification opinion generated by the pre-trained model, it is determined whether the audit rectification opinion and its corresponding input are low-quality samples, and the real audit working paper opinions of the generated low-quality samples and their corresponding real audit rectification opinions are used as the second noise training data.

[0146] Confidence indicates the "degree of certainty" of the model in the audit rectification opinions generated. Confidence Defined as:

[0147]

[0148] Among them, T is the length of the generated audit rectification opinion, that is, the total number of subwords in the audit rectification opinion, K is the number of candidate words for each subword in the generated audit rectification opinion, is the kth candidate word for the tth subword that generates the audit rectification opinion. are the 1st to t-1th subwords of the generated audit rectification opinions, It is the probability of generating the kth candidate word of the tth subword in the audit rectification opinion after t-1 words have been generated. It can be understood that when the pre-trained model generates a subword of the audit rectification opinion, there will be multiple candidate words as candidates. Selecting the topK candidate words and arithmetically averaging their selection probabilities can avoid selecting only the top1 candidate words for confidence evaluation and reduce the uncertainty of the results.

[0149] If the audit rectification opinions are generated If the confidence of is lower than the preset threshold τ, it is considered that the opinion may be affected by the noise data, and the opinion and its corresponding input are marked as low-quality samples, thereby obtaining the second noise training data. The preset threshold τ can be determined according to experience or actual needs.

[0150] S43, generating a noise enhancement training database based on the first noise training data and the second noise training data, wherein the noise enhancement training database includes noise data, and each group of noise data includes noisy audit working paper opinions and corresponding audit rectification opinions.

[0151] S5, optimizing and training the first preset rectification opinion generation model according to the audit corpus and the noise enhancement training database to obtain a first rectification opinion generation sub-model.

[0152] In one embodiment of the present invention, step S5 may specifically include the following steps:

[0153] S51, input the audit working papers opinions of the audit corpus and the noise enhanced training database into the first preset rectification opinion generation model to obtain the predicted value of the audit rectification opinion, and determine the weighted loss function according to the predicted value and the true value of the audit rectification opinion.

[0154] In one embodiment of the present invention, the weighted loss function may consider the confidence and perplexity on the basis of the standard cross entropy loss to guide the model to pay more attention to the influence of noisy training data when calculating the loss. for:

[0155]

[0156] in, is the cross entropy loss function of the noise optimization training submodule, α is the adaptive adjustment weight, and the calculation formula of the adaptive adjustment weight α is:

[0157]

[0158] Among them, x and They represent the audit working papers opinions input into the model and the audit rectification opinions generated by the model. The confidence level of the audit rectification opinions generated by the model, The perplexity of the audit rectification opinions generated by the model, λ1 and λ2 represent weight coefficients, which are used to balance the influence of confidence and perplexity. When the model has low confidence or high perplexity for a certain input, α increases to guide the model to pay more attention to such inputs when calculating losses.

[0159] Among them, perplexity can be used to measure the uncertainty of the model when generating a sequence, which reflects the difficulty encountered by the model when processing the input. By analyzing the perplexity, the model can perceive the existence of noisy information in the input audit working paper opinions. Used to measure the generation of text sequences The uncertainty of is defined as follows:

[0160]

[0161] Where T is the length of the generated text, is the probability of generating the kth candidate for the tth word in the text after t-1 words have been generated.

[0162] S52, optimizing and training the first preset rectification opinion generation model according to the weighted loss function. In the initial training batch, the proportion of normal fine-tuning training data is 100%. The proportion of audit working paper opinions of the noise-enhanced training database introduced in each subsequent training batch gradually increases until the weighted loss function corresponding to the first preset rectification opinion generation model converges or the preset batch of training for the first preset rectification opinion generation model is completed.

[0163] In a specific embodiment of the present invention, the learning rate of the first preset rectification opinion generation model can be initially set to 0.001 and then gradually decreased; the initial proportion of normal fine-tuning training data is 100%, and the 10% noise data introduced in each subsequent training batch causes the proportion of audit working paper opinions in the noise-enhanced training database to gradually increase, so as to ensure that the model gradually learns how to deal with these abnormal data, thereby improving its tolerance to noise.

[0164] S6, input the audit working papers to be rectified into the audit big model to generate corresponding audit rectification opinions, wherein the audit big model includes a first rectification opinion generation sub-model.

[0165] In actual audit scenarios, similar audit rectification opinions can be used to deal with audit working papers of the same type but with different opinions on issues. Therefore, multiple audit working papers of the same type can be merged to obtain appropriate audit rectification opinions.

[0166] In one embodiment of the present invention, the audit model may further include a second rectification opinion generation sub-model, which is used to generate corresponding audit rectification opinions according to multiple audit working papers of the same type to be rectified. Before step S6, the following steps may also be included:

[0167] (i) The fine-tuning training data are merged according to the single real audit rectification opinions of the same type in the fine-tuning training data to obtain merged fine-tuning training data. Each group of merged fine-tuning data contains multiple real audit working paper opinions and the corresponding single real audit rectification opinion.

[0168] Specifically, the real audit rectification opinions that are completely the same or the real audit working papers with ROUGE-L index greater than the threshold of 0.85 can be considered as the corresponding real audit rectification opinions of the same type, and similar audit working papers are deduplicated to obtain the merged fine-tuning training data. The ROUGE-L index takes into account the similarity of sentence structure, so when evaluating, it not only focuses on the matching of subwords, but also their order, which can better identify the real audit rectification opinions of the same type.

[0169] Since the combined fine-tuning training data is relatively scarce, but the actual fine-tuning process relies on a large amount of high-quality data, when faced with scarce audit data, the model often finds it difficult to learn to cope with specific audit tasks. Therefore, it is necessary to perform data augmentation on the combined fine-tuning training data.

[0170] (ii) Build a data enhancement model and perform data enhancement on the combined fine-tuning training data to obtain combined enhanced training data.

[0171] In a specific embodiment of the present invention, the GLM-4-9b-Chat large model can be used as a data augmentation model. It is defined as The process of generating new data is as follows:

[0172]

[0173] Among them, x i ,...,x j are multiple selected single real audit working paper opinions, and C is the context information that prompts the data enhancement model to generate new audit rectification opinions, such as "Please summarize and generate audit rectification opinions based on the following multiple single audit working paper opinions." It can be adjusted according to actual needs. Repeat multiple times to generate more diverse paired data, and aggregate multiple samples to obtain combined enhanced training data.

[0174] (III) Based on the combined enhanced training data, the first preset rectification opinion generation model is fine-tuned in a supervised and efficient manner to obtain the second preset rectification opinion generation model.

[0175] Here, when fine-tuning the first preset rectification opinion generation model at different levels, learnable embedding vectors can be inserted into multiple real audit working paper opinions and used as extended input data, so that the Transformer decoder of the first preset rectification opinion generation model takes the extended input data of each layer as input and generates output layer by layer, thereby improving the model's ability to summarize and generalize multiple audit working paper opinions to obtain consolidated audit rectification opinions.

[0176] (IV) A noise-enhanced training database is generated based on the samples generated during the training of the first preset rectification opinion generation model based on the audit corpus and the fine-tuning submodule. The database can be generated in a similar manner to the noise-enhanced training database, which will not be described in detail here.

[0177] (V) The second preset rectification opinion generation model is optimized and trained based on the audit corpus and the merged noise enhanced training database to obtain the second rectification opinion generation sub-model. The training method can be similar to the training method of the first preset rectification opinion generation model, which will not be repeated here.

[0178] If the audit working papers to be rectified are a single audit working paper, the audit working papers to be rectified will be input into the first rectification opinion generation sub-model to generate a corresponding single audit rectification opinion; if the audit working papers to be rectified are multiple audit working papers of the same type, the audit working papers to be rectified will be input into the second rectification opinion generation sub-model to generate a corresponding single audit rectification opinion.

[0179] According to the audit rectification opinion generation method based on efficient fine-tuning of a large model according to an embodiment of the present invention, a noise enhancement training database is generated based on samples generated during the training process of the audit corpus and the fine-tuning sub-module, and a first rectification opinion generation sub-model is obtained by training the audit corpus and the noise enhancement training database. This ensures that the model can still generate stable, consistent and targeted audit rectification opinions when faced with noisy data in the audit working papers, and is suitable for complex and changeable audit scenarios.

[0180] Based on the audit rectification opinion generating method of the above embodiment, the present invention also proposes a computer device.

[0181] The computer device of an embodiment of the present invention includes: a memory, a processor, and an audit rectification opinion generation program stored in the memory and executable on the processor. When the audit rectification opinion generation program is executed by the processor, the steps of the audit rectification opinion generation method described above are implemented.

[0182] The computer device of the embodiment of the present invention also has the above-mentioned beneficial effects because the audit rectification opinion generation program implements the steps of the authentication method of the audit rectification opinion generation method as described above when the audit rectification opinion generation program is executed by the processor.

[0183] Based on the audit rectification opinion generating method of the above embodiment, the present invention also proposes a storage medium.

[0184] The storage medium of the embodiment of the present invention stores an audit rectification opinion generation program, and when the audit rectification opinion generation program is executed by the processor, the steps of the audit rectification opinion generation method as described above are implemented.

[0185] The storage medium of the embodiment of the present invention also has the above-mentioned beneficial effects because the audit rectification opinion generation program implements the steps of the audit rectification opinion generation method as described above when executed by the processor.

[0186] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. "Multiple" means two or more, unless otherwise clearly and specifically defined.

[0187] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0188] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0189] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0190] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0191] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0192] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0193] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. An audit rectification opinion generation system based on efficient fine-tuning of a large model, characterized by: The system comprises: A corpus management module, the corpus management module is used to store and manage the audit corpus, the audit corpus includes audit report corpus and audit working paper corpus; A model training module, the model training module is used to train the audit big model, the audit big model includes a first rectification opinion generation sub-model, the first rectification opinion generation sub-model is used to generate corresponding audit rectification opinions according to a single audit working paper to be rectified, the model training module includes a pre-training sub-module, a fine-tuning sub-module, a noise generation sub-module and a noise optimization training sub-module, the pre-training sub-module is used to construct a pre-training model, and perform self-supervised pre-training on the pre-training model according to the audit report corpus; the fine-tuning sub-module is used to perform supervised and efficient fine-tuning on the pre-training model according to the audit working paper corpus to obtain a first preset rectification opinion generation model; the noise generation sub-module is used to generate a noise enhancement training database based on the samples generated during the training process of the audit corpus and the fine-tuning sub-module; the noise optimization training sub-module is used to optimize the training of the first preset rectification opinion generation model according to the audit corpus and the noise enhancement training database to obtain a first rectification opinion generation sub-model; The rectification opinion generation module is used to input the audit working papers to be rectified into the audit big model to generate corresponding audit rectification opinions.

2. The audit rectification opinion generation system based on large model efficient fine-tuning according to claim 1 is characterized in that: The corpus management module is used to obtain pre-training data based on the audit report corpus, each set of pre-training data includes a single audit working paper opinion and a corresponding single audit rectification opinion; The pre-training submodule is specifically used to construct a pre-training model and perform low-rank matrix training on the parameter matrix of the pre-training model based on the pre-training data.

3. The audit rectification opinion generation system based on large model efficient fine-tuning according to claim 2 is characterized in that: The corpus management module is also used to obtain fine-tuning training data based on the audit working paper corpus, each set of the fine-tuning training data includes a single real audit working paper opinion and a single real audit rectification opinion corresponding thereto; The fine-tuning submodule is specifically used to perform supervised and efficient fine-tuning on the pre-trained model according to the fine-tuning training data to obtain a first preset rectification opinion generation model. When fine-tuning the pre-trained model at different levels, a learnable embedding vector is inserted into the real audit working paper opinion and used as extended input data, so that the Transformer decoder of the pre-trained model takes the extended input data of each layer as input and generates output layer by layer.

4. The audit rectification opinion generation system based on large model efficient fine-tuning according to claim 1 is characterized in that: The noise generation submodule is specifically used for: Performing noise processing on the audit working paper-like opinions extracted from the audit report corpus to obtain first noise training data; When the fine-tuning submodule performs supervised and efficient fine-tuning on the pre-trained model, based on the confidence of the audit rectification opinion generated by the pre-trained model, it is determined whether the audit rectification opinion and its corresponding input are low-quality samples, and the real audit working paper opinions of the generated low-quality samples and their corresponding real audit rectification opinions are used as the second noise training data; A noise enhancement training database is generated based on the first noise training data and the second noise training data. The noise enhancement training database includes noise data. Each group of noise data includes noisy audit working paper opinions and corresponding audit rectification opinions.

5. The audit rectification opinion generation system based on large model efficient fine-tuning according to claim 4 is characterized in that: The noise optimization training submodule is specifically used for: Inputting the audit working papers opinions of the audit corpus and the noise-enhanced training database into a first preset rectification opinion generation model to obtain a predicted value of the audit rectification opinion, and determining a weighted loss function according to the predicted value and the true value of the audit rectification opinion; The first preset rectification opinion generation model is optimized and trained according to the weighted loss function. In the initial training batch, the proportion of normal fine-tuning training data is 100%, and the proportion of audit working paper opinions of the noise enhanced training database introduced in each subsequent training batch gradually increases until the weighted loss function corresponding to the first preset rectification opinion generation model converges or the preset batch of training for the first preset rectification opinion generation model is completed.

6. The audit rectification opinion generation system based on large model efficient fine-tuning according to claim 5 is characterized in that: The weighted loss function for: in, is the cross entropy loss function of the noise optimization training submodule, α is the adaptive adjustment weight, and the calculation formula of the adaptive adjustment weight α is: Among them, x and They represent the audit working papers opinions input into the model and the audit rectification opinions generated by the model. The confidence level of the audit rectification opinions generated by the model, is the perplexity of the audit rectification opinions generated by the model, and λ1 and λ2 represent weight coefficients, which are used to balance the influence of confidence and perplexity.

7. The audit rectification opinion generation system based on large model efficient fine-tuning according to claim 3 is characterized in that: The audit big model also includes a second rectification opinion generation sub-model, which is used to generate corresponding audit rectification opinions based on multiple audit working papers of the same type to be rectified; The corpus management module is also used to merge the fine-tuning training data according to the single real audit rectification opinions of the same type in the fine-tuning training data to obtain merged fine-tuning training data, each group of the merged fine-tuning data contains multiple real audit working paper opinions and the single real audit rectification opinion corresponding thereto; The model training module also includes a data enhancement submodule, which is used to build a data enhancement model and perform data enhancement on the combined fine-tuning training data to obtain combined enhanced training data; The fine-tuning submodule is further used to perform supervised and efficient fine-tuning on the first preset rectification opinion generation model according to the combined enhanced training data to obtain a second preset rectification opinion generation model; The noise generation submodule is further used to generate a merged noise enhancement training database based on the audit corpus and the samples generated in the training process of the first preset rectification opinion generation model by the fine-tuning submodule; The noise optimization training submodule is further used to optimize and train the second preset rectification opinion generation model according to the audit corpus and the combined noise enhancement training database to obtain a second rectification opinion generation submodel; When the audit working papers to be rectified are a single audit working paper, the rectification opinion generation module inputs the audit working papers to be rectified into the first rectification opinion generation sub-model to generate a corresponding single audit rectification opinion; when the audit working papers with rectification are multiple audit working papers of the same type, the rectification opinion generation module inputs the audit working papers to be rectified into the second rectification opinion generation sub-model to generate a corresponding single audit rectification opinion.

8. A method for generating audit rectification opinions based on the audit rectification opinion generating system according to any one of claims 1 to 7, characterized in that: The audit big model includes a first rectification opinion generation sub-model, and the first rectification opinion generation sub-model is used to generate a corresponding audit rectification opinion according to a single audit working paper to be rectified. The method includes the following steps: S1, obtaining an audit corpus, wherein the audit corpus includes an audit report corpus and an audit working paper corpus; S2, constructing a pre-training model, and performing self-supervised pre-training on the pre-training model according to the audit report corpus; S3, performing supervised and efficient fine-tuning on the pre-trained model according to the audit working paper corpus to obtain a first preset rectification opinion generation model; S4, generating a noise enhancement training database based on the audit corpus and samples generated during the training process; S5, optimizing and training the first preset rectification opinion generation model according to the audit corpus and the noise enhancement training database to obtain a first rectification opinion generation sub-model; S6, input the audit working papers to be rectified into the audit big model to generate corresponding audit rectification opinions.

9. A computer device, characterized in that: include: A memory, a processor, and an audit rectification opinion generation program stored in the memory and executable on the processor, wherein the audit rectification opinion generation program implements the steps of the audit rectification opinion generation method as described in claim 8 when executed by the processor.

10. A storage medium, characterized in that: The storage medium stores an audit rectification opinion generation program, and when the audit rectification opinion generation program is executed by the processor, the steps of the audit rectification opinion generation method according to claim 8 are implemented.

Images