An audit question and answer large model optimization method based on prompt fine-tuning

Abstract

The application relates to the technical field of electric power auditing, in particular to an audit question and answer large model optimization method based on prompt fine tuning, which comprises the following steps: collection and storage of audit data, wherein the audit data of a current viewport buffer is cleared before collection, and the audit data is separated during collection; audit data text analysis, wherein in an encoding layer, audit data text processing is converted into a text vector sequence; a prompt fine tuning instruction is introduced to obtain an audit data text subject and object; in a decoding layer, a subject mask sequence is obtained; in a translation layer, a subject vector sequence and a relationship-object vector sequence are obtained; summarization and release, wherein an intermediate layer in the encoding layer is taken out as an input of entity sequence labeling; and the results of audit data text analysis are combined to perform summarization and release. The audit question and answer large model optimization method based on prompt fine tuning can improve the reasoning ability and response speed of a model by fusing prior knowledge and a mixed prompt fine tuning strategy.

Description

Technical Field

[0001] This invention relates to the field of power auditing technology, specifically to a method for optimizing a large audit question-and-answer model based on prompting and fine-tuning. Background Technology

[0002] A language model is a model based on statistics, deep learning, or other machine learning techniques used to process and generate natural language text. Its main tasks are to understand, generate, predict, and analyze text data. Modern language models, especially those based on deep learning, are capable of handling complex language patterns and generating fluent and meaningful text.

[0003] Pre-trained language models refer to language models that are first pre-trained on large-scale text data to learn the general rules and knowledge of language, and then fine-tuned for application to specific tasks. Currently, pre-trained language models have achieved great success in the field of natural language processing, significantly improving machines' ability to understand and generate text. The basic process of pre-trained language models typically includes two main stages: pre-training and fine-tuning. Pre-training involves training the model on massive amounts of unlabeled text data to learn general language knowledge and patterns. The goal of pre-training is to enable the model to master the basic language abilities of word grammatical structure, semantic relationships, and contextual dependencies. The pre-training stage does not involve specific tasks; therefore, the training objectives are usually general language modeling tasks, such as autoregressive language modeling and autoencoder language modeling.

[0004] Fine-tuning involves applying the general knowledge gained in the pre-training phase to specific tasks; the pre-trained model is further trained on labeled datasets to adapt to specific downstream tasks; the fine-tuning phase usually requires less data because the pre-training phase has already endowed the model with rich language capabilities and knowledge.

[0005] Cue fine-tuning involves inputting specific cue words into a pre-trained model to guide its output performance on a particular task. In the application of large-scale pre-trained language models, the sheer number of parameters leads to slow inference speeds and enormous computational resource consumption, especially in auditing scenarios where models are required to respond quickly and accurately to complex auditing question-answering tasks. However, traditional cue fine-tuning methods often suffer from language bias issues, resulting in poor model performance when handling multimodal question-answering tasks. Summary of the Invention

[0006] To address the technical problems existing in the background art, this invention provides an audit question-answering large model optimization method based on prompt fine-tuning, which improves the model's reasoning ability and response speed by integrating prior knowledge and a hybrid prompt fine-tuning strategy.

[0007] The technical solution adopted by this invention is:

[0008] An audit question and answer large model optimization method based on prompt fine-tuning, comprising:

[0009] S1, collection and storage of audit data:

[0010] Before collection, the audit data of the current viewport buffer is cleared, and the audit data is separated during collection.

[0011] S2, audit data text analysis:

[0012] In the encoding layer, the audit data text is processed into a text vector sequence T N ; introduce prompt fine-tuning instructions, get audit data text subject T K , audit data text object O K from the text vector sequence T N ;

[0013] In the decoding layer, the subject mask sequence is obtained from the audit data text subject T K ;

[0014] In the decoding layer, the subject vector sequence is obtained from the subject mask sequence , combined with the text vector sequence T N ; on the basis of the subject vector sequence , combined with the text vector sequence T N , get the relation-object vector sequence .

[0015] S3, summary and release:

[0016] Take the intermediate layer in the encoding layer as the input of entity sequence labeling E C ;

[0017] Combine the results of audit data text analysis to summarize and release.

[0018] Further, the prompt fine-tuning instructions include: subject vector norm , object vector norm ;

[0019] The text vector sequence T N converges to the subject vector norm , that is: , thereby obtaining the audit data text subject T K ;

[0020] The text vector sequence T N converges to the object vector norm , that is: Thus, the audit data text object O is obtained. K .

[0021] Furthermore, the text vector sequence T N for: ,

[0022] The audit data text body T K for: ,

[0023] The audit data text object O K for: ;

[0024] in:

[0025] For the text vector sequence T N component groups,

[0026] For audit data text body T K The amount,

[0027] For auditing data text objects O K The amount,

[0028] T is the transpose of the vector.

[0029] Furthermore, the entity sequence label E C For: Text vector sequence T N The single-row sub-forms in the text.

[0030] Furthermore, the entity sequence label E C For: (T) K ∪O K Single-row sub-forms in )

[0031] The main body T of the audit data text K The object of audit data text O K The union improves the correlation between the components.

[0032] Furthermore, the audit data text body T K The padding is performed in the decoding layer to obtain the main mask sequence. for:

[0033] ,

[0034] in:

[0035] In K, (1, 2, ..., k) represent different subject masks, and are used to distinguish them.

[0036] T is the transpose of the vector.

[0037] Furthermore, the main body mask sequence In the decoding layer, the audit data text body T is combined with... K Obtain the main vector sequence for:

[0038] ,

[0039] in:

[0040] In K, (1, 2, ..., k) represent different principal vectors and are used to distinguish them.

[0041] T is the transpose of the vector;

[0042] The main vector sequence In the decoding layer, with the text vector sequence T N Adding them together yields a relation-object vector sequence. for:

[0043] ,

[0044] in:

[0045] In K, (1, 2, ..., k) represent different text vectors and are used to distinguish them.

[0046] T is the transpose of the vector.

[0047] Furthermore, a body pointer network is constructed in the encoding layer to transmit the audit data text body T. K After transforming traditional sequences into sequence models, a fixed sequence length is maintained;

[0048] The decoding layer constructs a relation-object pointer network to convert the relation-object vector sequence After transforming the traditional sequence into a sequence model, a fixed sequence length is maintained.

[0049] Furthermore, the main vector sequence To relation-object vector sequence During the transformation process, in addition to the text vector sequence T N In addition to performing basic operations, a position-relation function is constructed to obtain the intermediate vector sequence P. e for:

[0050] ,in:

[0051] V i For the main vector sequence The amount,

[0052] j is the frequency coefficient of the position-relationship function.

[0053] d represents the dimension.

[0054] Furthermore, an object pointer network is constructed in the encoding layer to store audit data text objects O. K After transforming from a traditional sequence to a sequence model, a fixed sequence length is maintained; thereby determining the audit data text object O. K The number of floors;

[0055] Audit data text body T K With audit data text object O K The relationship between the audit data text object O K The number of floors.

[0056] The beneficial effects of this invention's large-scale audit question-answering model optimization method based on prompt fine-tuning are as follows:

[0057] 1. By constructing a mask-patterned cloze template, more precise hints and fine-tuning instructions are provided for the model;

[0058] 2. By using mask templates and fine-tuning instructions suitable for audit scenarios, the model is guided to understand audit knowledge and answer questions.

[0059] 3. A method for generating subject vector sequences based on subject masking, which utilizes attention and masking mechanisms to generate subject vectors.

[0060] 4. Entity sequence labeling tasks are used to assist in entity decoding. Attached Figure Description

[0061] Figure 1 This is a flowchart illustrating an example of the present invention;

[0062] Figure 2 This is a schematic diagram of the encoding and decoding process of an example of the present invention;

[0063] Figure 3 This is a schematic diagram of the decoding process in an example of the present invention. Detailed Implementation

[0064] To more clearly and explicitly illustrate the specific objectives and implementation methods of this invention, a complete description of the technical solution of this invention will be provided below. The described embodiments are only a portion of, not all, of the embodiments of this invention. Without any inventive effort, all other embodiments based on the described embodiments of this invention are within the protection scope of this invention.

[0065] This invention provides a method for optimizing a large-scale audit question-and-answer model based on prompt-based fine-tuning, such as... Figure 1 As shown, it includes:

[0066] S1. Collection and storage of audit data:

[0067] Before data acquisition, the audit data in the current viewport buffer is cleared, and the audit data is segmented during acquisition.

[0068] S2. Audit Data Text Analysis:

[0069] S2.1 In the coding layer:

[0070] The audit data text is processed into a text vector sequence T. N for: ;

[0071] In the formula:

[0072] For the text vector sequence T N The component groups.

[0073] Introducing prompts for fine-tuning, including: the subject vector norm. Object vector norm ;

[0074] The text vector sequence T N According to the subject vector norm Convergence, that is: Thus, the audit data text body T is obtained. K ;

[0075] The text vector sequence T N Based on object vector norm Convergence, that is: Thus, the audit data text object O is obtained. K ;

[0076] The fine-tuning instructions can guide the model in audit knowledge understanding and question-answering tasks, making it more suitable for audit scenarios.

[0077] In the encoding layer, a main pointer network is constructed to store the audit data text body T. K After transforming from a traditional sequence to a sequence model, a fixed sequence length is maintained; the audit data text body T K for:

[0078] ;

[0079] In the coding layer, an object pointer network is constructed to audit the data text object O. K After transforming from a traditional sequence to a sequence model, a fixed sequence length is maintained; audit data text object O K for:

[0080] ;

[0081] In the above two formulas:

[0082] For audit data text body T K The amount;

[0083] For auditing data text objects O K The amount;

[0084] O k In this context, k represents the audit data text O. K The number of floors;

[0085] T is the transpose of the vector.

[0086] Audit data text body T K With audit data text object O K The relationship between the audit data text object O K The number of floors.

[0087] S2.2, In the decoding layer:

[0088] For the audit data text body T K By constructing a cloze template based on a mask pattern, more precise hints and fine-tuning instructions are provided to the model to obtain the main mask sequence. for:

[0089] ;

[0090] In the formula:

[0091] In K, (1, 2, ..., k) represent different subject masks and are used to distinguish them;

[0092] T is the transpose of the vector.

[0093] S2.3.1 In the decoding layer:

[0094] From the main mask sequence Combined with the text vector sequence T N Obtain the main vector sequence for:

[0095] ;

[0096] In the formula:

[0097] In K, (1, 2, ..., k) represent different principal vectors and are used to distinguish them;

[0098] T is the transpose of the vector.

[0099] S2.3.2, In the decoding layer:

[0100] In the main vector sequence Based on this, and the text vector sequence T N In addition, a relation-object pointer network is constructed in the decoding layer, which combines the relation-object vector sequences. After transforming from a traditional sequence to a sequence model, a fixed sequence length is maintained; relation-object vector sequence for:

[0101] ;

[0102] In the formula:

[0103] In K, (1, 2, ..., k) represent different text vectors and are used to distinguish them;

[0104] T is the transpose of the vector.

[0105] S2.3.3, In the decoding layer:

[0106] Main vector sequence To relation-object vector sequence During the transformation process, in addition to the text vector sequence T N In addition to performing basic operations, a position-relation function is constructed to obtain the intermediate vector sequence P. e for:

[0107] ;

[0108] in:

[0109] V i For the main vector sequence The amount;

[0110] j is the frequency coefficient of the position-relation function;

[0111] d represents the dimension.

[0112] S3, Summary and Release:

[0113] Extract the intermediate layer from the coding layer as entity sequence label E C Input.

[0114] Entity sequence labeling E C For: Text vector sequence T N The single-line sub-format in the text; to improve the main body T of the audit data text K The object of audit data text O K The relationships between the various components, from the main body T of the audit data text K Object O of the audit data text KSelecting from the set of unions, we get: entity sequence label E C For: (T) K ∪O K The single-line sub-form in ); through entity sequence label E C Subtasks that assist in entity decoding.

[0115] The results of the textual analysis of the audit data will be summarized and published.

[0116] In summary, the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the invention. Based on the above description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of the present invention is not limited to the contents of the specification; all equivalent variations and modifications of the shape, structure, features, and spirit described within the scope of the claims should be included within the scope of the claims.

Claims

1. A method for optimizing a large-scale audit question-and-answer model based on prompt fine-tuning, characterized in that: S1. Collection and storage of audit data: Before data acquisition, the audit data in the current viewport buffer is cleared, and the audit data is segmented during acquisition. S2. Audit Data Text Analysis: In the encoding layer, the audit data text is processed into a text vector sequence T. N ; Introducing prompts for fine-tuning, including: subject vector norm. Object vector norm , Text vector sequence T N According to the subject vector norm Convergence, that is: Thus, the audit data text body T is obtained. K , Text vector sequence T N Based on object vector norm Convergence, that is: Thus, the audit data text object O is obtained. K ; In the decoding layer, the audit data text body T K Fill in the blanks to obtain the main mask sequence. for: , in: In K, (1, 2, ..., k) represent different subject masks, which are used for differentiation. T is the transpose of the vector; In the decoding layer, the main mask sequence Combined with the audit data text body T K Obtain the main vector sequence for: , in: In K, (1, 2, ..., k) represent different principal vectors and are used to distinguish them. T is the transpose of the vector; In the decoding layer, a position-relation function is constructed to obtain the intermediate vector sequence P. e for: ,in: V i For the main vector sequence The amount, j is the frequency coefficient of the position-relationship function. d represents the dimension; In the decoding layer, the main vector sequence With text vector sequence T N Adding them together yields a relation-object vector sequence. for: , in: In K, (1, 2, ..., k) represent different text vectors and are used to distinguish them. T is the transpose of the vector; S3, Summary and Release: Extract the intermediate layer from the coding layer as entity sequence label E C The input, combined with the results of textual analysis of audit data, is summarized and published.

2. The method for optimizing a large-scale audit question-and-answer model based on prompt fine-tuning according to claim 1, characterized in that: The text vector sequence T N for: , The audit data text body T K for: , The audit data text object O K for: ; in: For the text vector sequence T N component groups, (T1, T2, ..., T) k (T) is the main text of the audit data. K The amount, (O1, O2, ..., O) k ) is the audit data text object O K The amount, T is the transpose of the vector.

3. The method for optimizing a large-scale audit question-and-answer model based on prompt fine-tuning according to claim 2, characterized in that: The entity sequence label E C For: Text vector sequence T N The single-row sub-forms in the text.

4. The method for optimizing a large-scale audit question-and-answer model based on prompt fine-tuning according to claim 2, characterized in that: The entity sequence label E C For: (T) K ∪O K Single-row sub-forms in ) The main body T of the audit data text K The object of audit data text O K The union improves the correlation between the components.

5. The method for optimizing a large-scale audit question-and-answer model based on prompt fine-tuning according to claim 4, characterized in that: The encoding layer constructs a main pointer network to encode the audit data text body T. K Maintain a fixed sequence length; The decoding layer constructs a relation-object pointer network to convert the relation-object vector sequence Maintain a fixed sequence length.

6. The method for optimizing a large-scale audit question-and-answer model based on prompt fine-tuning according to claim 4, characterized in that: The encoding layer constructs an object pointer network to store audit data text objects O. K Maintaining a fixed sequence length; thereby determining the audit data text object O. K The number of floors; Audit data text body T K With audit data text object O K The relationship between the audit data text object O K The number of floors.

Images