Large language model method and system fusing pre-training and dynamic fine tuning technologies

Abstract

The invention relates to the technical field of large language models, in particular to a large language model method and system integrating pre-training and dynamic fine tuning technologies, and provides the following scheme: firstly, an audit language pool is established through audit knowledge, and the audit language pool is pre-trained; secondly, searching the most matched auditing knowledge in the auditing language pool according to the auditing information for pairing, finally constructing a fine tuning model, and performing fine tuning on the auditing language pool according to a pairing result. According to the method, the overall information of the audit object is comprehensively, dynamically and truly obtained, internal auditing personnel are helped to efficiently identify the audit information, and a better reference basis is provided for an enterprise to avoid operation risks through cross validation and combined analysis of internal and external audit data.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a large language model method and system that integrates pre-training and dynamic fine-tuning technologies. Background Art

[0002] Internal audit is an important system design for scientific decision-making, standardized management, and normalized risk prevention and control. It is not only an indispensable means for enterprises to strengthen internal control, but also a basic link and important part of the national governance system. Accelerating the pace of digital transformation of enterprises, strengthening the reuse of data audit methods, and realizing intelligent auditing on this basis, from "human audit" to "machine audit" and then to "intelligent audit" as a change of thinking, has gradually become a new trend. The era of big data, cloud computing, and artificial intelligence have transformed the data basis of internal audits of enterprises from a small amount of "sample data" to a massive amount of "all data", realizing cross-professional, cross-regional, and cross-type full-volume data audit analysis and application, and realizing a new decision-making from limited cases to "speaking with data".

[0003] For example, a Chinese patent with authorization announcement number CN112214609B discloses an audit method and system based on knowledge graph, which belongs to the field of knowledge graph technology and solves the problem that existing audit methods cannot be reused and have poor versatility. The audit method includes: using basic data sources to build a knowledge graph based on the knowledge graph Schema; configuring rule expressions in the knowledge graph; and automatically auditing knowledge graph data based on rule expressions, including: looping through multiple starting nodes and obtaining a starting node; splitting the rule expression into multiple rule sub-expressions; looping through and executing multiple rule sub-expressions, when an error occurs in the execution of the rule sub-expression, all nodes involved in the execution of the rule sub-expression are recorded as rule violation data; looping through the recorded rule violation data and assigning values ​​to multiple rule sub-expressions; and executing the assigned rule expression, and when an error occurs in the execution of the rule expression, displaying the rule violation data through a visual knowledge graph. Enhance versatility and scalability.

[0004] The above patents all have the problems raised by this background technology: auditing work requires a lot of manpower and material support, and the rules are not detailed enough and difficult to explain and understand. The use of logical rule reasoning technology requires manual pre-definition of accurate logical rules. To solve the above problems, this application designs a large language model method that integrates pre-training and dynamic fine-tuning technology. Summary of the invention

[0005] The technical problem to be solved by the present invention is to address the deficiencies of the existing technology and provide a large language model method that integrates pre-training and dynamic fine-tuning technology. First, an audit language pool is established through audit knowledge and the audit language pool is pre-trained. Secondly, the most matching audit knowledge is found in the audit language pool according to the audit information for pairing. Finally, a fine-tuning model is constructed and the audit language pool is fine-tuned according to the pairing results.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A large language model method integrating pre-training and dynamic fine-tuning technology, comprising the following steps;

[0008] S1: Establish an audit language pool with audit knowledge as the research content, and the audit knowledge includes typical audit indicators and audit expert experience base;

[0009] S2: obtaining audit information input by the user, enhancing the audit information, and obtaining enhanced audit information, wherein the enhancing of the audit information includes back translation, professional synonym replacement, and question-answer pair exchange;

[0010] S3: Generate audit triplet training data according to the enhanced audit information and the audit language pool;

[0011] S4: Construct a fine-tuning model, input the audit triplet training data into the fine-tuning model, obtain a fine-tuning instruction set, and perform instruction fine-tuning on the audit language pool according to the fine-tuning instruction set.

[0012] The audit language pool established based on audit knowledge as research content includes:

[0013] Extract the audit project name and confirm the field and scope of the audit entity based on the audit project name;

[0014] According to the domain and scope of the audit ontology, keywords and key terms in the domain are determined, and an audit ontology knowledge model is established from top to bottom. The audit ontology knowledge model includes a top-level ontology, a concept subtree, and a sub-layer instance. The concept subtree includes financial statements, engineering projects, approval results, internal assessments, and project implementation logs.

[0015] Add keywords and key terms in the domain as knowledge features to sub-layer instances;

[0016] Conduct a logical check on the audit ontology knowledge model to determine whether it meets the audit domain ontology construction principles. If so, instantiate the audit ontology. If not, continue to add sub-layer instances.

[0017] According to the knowledge entry strategy, the instantiated audit ontology is entered into the audit language pool.

[0018] The specific steps of the knowledge entry strategy are as follows:

[0019] S2.1: Obtain the number of concept branches, keywords and keyword attributes of the input instantiated audit ontology;

[0020] S2.2: Determine the semantic tree where the instantiated audit ontology is located in the audit language pool, traverse all nodes of the semantic tree, extract the keyword pairs with the smallest semantic distance between the node and the instantiated audit ontology as similar node pairs, and calculate the semantic similarity between the node and the instantiated audit ontology;

[0021] S2.3: Compare the highest semantic similarity with the similarity threshold. If it is less than the similarity threshold, the instantiated audit ontology is used as a sibling node of the node with the highest semantic similarity. If it is greater than or equal to the similarity threshold, the instantiated audit ontology is merged with the node with the highest semantic similarity.

[0022] The specific steps of S3 are as follows:

[0023] S3.1: Establish a feature sequence according to the enhanced audit information, and calculate a fitting value of the feature sequence;

[0024] S3.2: converting the fitting value of the feature sequence into a high-dimensional space, extracting features layer by layer in the high-dimensional space through multiple convolutional layers and pooling layers to obtain high-level semantic features, performing relationship analysis on the high-level semantic features, and calculating audit keyword features;

[0025] S3.3: Calculate the similarity between the audit keyword feature and the audit ontology in the audit language pool according to the audit keyword feature and the audit language pool;

[0026] S3.4: Match the audit keyword features with the highest similarity with the audit ontology in the audit language pool to generate audit triplet training data.

[0027] The calculation formula for the similarity between the audit keyword feature and the audit ontology in the audit language pool is:

[0028]

[0029] Among them, R s represents the similarity between the audit keyword feature and the audit ontology in the audit language pool, γ represents the learning rate, u represents a single node of the audit keyword feature, U represents the total number of audit keyword feature nodes, v represents a single node of the audit ontology in the audit language pool, V represents the total number of audit ontology nodes in the audit language pool, H u Represents the feature vector of the u-th node of the audit keyword feature, ω uv represents the connection weight between the audit keyword feature and the audit ontology in the audit language pool, h vrepresents the feature vector of the vth node of the audit ontology in the audit language pool, b v represents the node threshold of the audit ontology in the audit language pool, and E represents the error.

[0030] The constructing of the fine-tuning model includes calling a low-rank decomposition matrix in combination with a channel and spatial attention mechanism to train the fine-tuning training model and generate a fine-tuning instruction set.

[0031] A large language model system integrating pre-training and dynamic fine-tuning technology, the system comprises an audit language pool training module, an audit training data generation module and an audit language pool fine-tuning module;

[0032] The audit language pool training module is used to establish an audit language pool with audit knowledge as research content;

[0033] The audit training data generation module is used to obtain the audit information input by the user, process the audit information, and obtain the audit triplet training data;

[0034] The audit language pool fine-tuning module is used to construct a fine-tuning model, input the audit triplet training data into the fine-tuning model, obtain a fine-tuning instruction set, and perform instruction fine-tuning on the audit language pool according to the fine-tuning instruction set.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. The present invention obtains the overall information of the audit object in a comprehensive, dynamic and real manner, helps internal auditors to efficiently identify audit information, cross-validates and combines internal and external audit data, and also provides a better reference for enterprises to avoid business risks;

[0037] 2. The present invention realizes the self-learning function through the large language model technology, improves the audit speed and audit accuracy, realizes the efficient management of internal audit and the local consumption of engineering workload, and enhances the operational flexibility of the internal audit system. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0039] Figure 1 This is a flow chart of a large language model method integrating pre-training and dynamic fine-tuning technology according to Embodiment 1 of the present invention;

[0040] Figure 2 This is a flow chart of the knowledge entry audit language pool of Example 1 of the present invention;

[0041] Figure 3 This is a schematic diagram of the fine-tuning model structure of Example 1 of the present invention;

[0042] Figure 4 This is a module diagram of a large language model system that integrates pre-training and dynamic fine-tuning technology in Example 2 of the present invention. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0044] Embodiment 1:

[0045] See also Figure 1 , an embodiment provided by the present invention: a large language model method integrating pre-training and dynamic fine-tuning technology, comprising the following steps;

[0046] S1: Establish an audit language pool with audit knowledge as the research content, and the audit knowledge includes typical audit indicators and audit expert experience base;

[0047] S2: obtaining audit information input by the user, enhancing the audit information, and obtaining enhanced audit information, wherein the enhancing of the audit information includes back translation, professional synonym replacement, and question-answer pair exchange;

[0048] S3: Generate audit triplet training data according to the enhanced audit information and the audit language pool;

[0049] S4: Construct a fine-tuning model, input the audit triplet training data into the fine-tuning model, obtain a fine-tuning instruction set, and perform instruction fine-tuning on the audit language pool according to the fine-tuning instruction set.

[0050] The audit language pool established based on audit knowledge as research content includes:

[0051] Extract the audit project name and confirm the field and scope of the audit entity based on the audit project name;

[0052] According to the domain and scope of the audit ontology, keywords and key terms in the domain are determined, and an audit ontology knowledge model is established from top to bottom. The audit ontology knowledge model includes a top-level ontology, a concept subtree, and a sub-layer instance. The concept subtree includes financial statements, engineering projects, approval results, internal assessments, and project implementation logs.

[0053] Add keywords and key terms in the domain as knowledge features to sub-layer instances;

[0054] Conduct a logical check on the audit ontology knowledge model to determine whether it meets the audit domain ontology construction principles. If so, instantiate the audit ontology. If not, continue to add sub-layer instances.

[0055] According to the knowledge entry strategy, the instantiated audit ontology is entered into the audit language pool.

[0056] The specific steps of the knowledge entry strategy are as follows:

[0057] S2.1: Obtain the number of concept branches, keywords and keyword attributes of the input instantiated audit ontology;

[0058] S2.2: Determine the semantic tree where the instantiated audit ontology is located in the audit language pool, traverse all nodes of the semantic tree, extract the keyword pairs with the smallest semantic distance between the node and the instantiated audit ontology as similar node pairs, and calculate the semantic similarity between the node and the instantiated audit ontology;

[0059] S2.3: Compare the highest semantic similarity with the similarity threshold. If it is less than the similarity threshold, the instantiated audit ontology is used as the sibling node of the node with the highest semantic similarity. If it is greater than or equal to the similarity threshold, the instantiated audit ontology is merged with the node with the highest semantic similarity.

[0060] The calculation formula of the semantic similarity between the node and the instantiated audit ontology is:

[0061] Sem s =dist α *des β *deg γ ,

[0062] Among them, Sem s Represents the semantic similarity between the node and the instantiated audit ontology, dist represents the semantic distance similarity of similar nodes, α represents the influence weight of semantic distance similarity, des represents the depth similarity of similar nodes, β represents the influence weight of depth similarity, deg represents the degree similarity of similar nodes, and γ represents the influence weight of degree similarity;

[0063] The calculation formula of the semantic distance similarity of similar nodes is:

[0064]

[0065] Among them, di(·) represents the semantic distance calculation function, A and B represent similar node pairs, i represents a single keyword of the instantiated disease ontology, n represents the total number of keywords of the instantiated audit ontology, j represents a single keyword of the node, and m represents the total number of keywords of the node;

[0066] The calculation formula for the deep similarity of similar nodes is:

[0067]

[0068] Where dep(·) represents the depth-finding function, d1 represents the average depth of the node, and d2 represents the average depth of the instantiated audit ontology;

[0069] The calculation formula of degree similarity of similar nodes is:

[0070]

[0071] Where E{·} represents the Euclidean distance function, represents the degree(·) measurement function, and A an represents the parent node of keyword A, O represents the ancestor node of the semantic tree, and B an Indicates the parent node of keyword B;

[0072] See also Figure 2 , the flowchart of the knowledge entry audit language pool of the embodiment of the present invention, firstly, according to the instantiated audit ontology, the instantiated audit ontology is converted into the number of concept branches, keywords and keyword attribute knowledge for addition, and then similarity detection is performed. The mechanism of similarity detection is to detect whether there is knowledge in the audit language pool whose similarity with the knowledge concept to be entered is greater than the threshold set by the system. The detection process needs to calculate the semantic relevance between the keywords of the knowledge to be entered and the keywords of each record in the database. The system expert sets the entry threshold. The knowledge entries above this threshold will be regarded as records with high similarity, indicating that such records already exist in the knowledge base, and this knowledge will not be allowed to be directly entered into the knowledge base. If it is lower than this threshold, it means that this knowledge entry does not exist in the knowledge base, and this knowledge can be entered into the system knowledge base after review;

[0073] The specific steps of S3 are as follows:

[0074] S3.1: Establish a feature sequence according to the enhanced audit information, and calculate a fitting value of the feature sequence;

[0075] S3.2: converting the fitting value of the feature sequence into a high-dimensional space, extracting features layer by layer in the high-dimensional space through multiple convolutional layers and pooling layers to obtain high-level semantic features, performing relationship analysis on the high-level semantic features, and calculating audit keyword features;

[0076] S3.3: Calculate the similarity between the audit keyword feature and the audit ontology in the audit language pool according to the audit keyword feature and the audit language pool;

[0077] S3.4: Match the audit keyword features with the highest similarity with the audit ontology in the audit language pool to generate audit triplet training data.

[0078] The calculation formula for the similarity between the audit keyword feature and the audit ontology in the audit language pool is:

[0079]

[0080] Among them, R s represents the similarity between the audit keyword feature and the audit ontology in the audit language pool, Y represents the learning rate, u represents a single node of the audit keyword feature, U represents the total number of audit keyword feature nodes, v represents a single node of the audit ontology in the audit language pool, V represents the total number of audit ontology nodes in the audit language pool, and H u Represents the feature vector of the u-th node of the audit keyword feature, ω uv represents the connection weight between the audit keyword feature and the audit ontology in the audit language pool, h v represents the feature vector of the vth node of the audit ontology in the audit language pool, b v represents the node threshold of the audit ontology in the audit language pool, and E represents the error;

[0081] The construction of the fine-tuning model includes calling the low-rank decomposition matrix in combination with the channel and spatial attention mechanism to train the fine-tuning training model and generate a fine-tuning instruction set; see Figure 3 , a schematic diagram of the structure of a fine-tuning model according to an embodiment of the present invention, wherein the fine-tuning model comprises an input layer, a hidden layer and an output layer;

[0082] The input layer performs word segmentation and part-of-speech tagging on the audit triple training data according to the pkuseg toolkit, calculates the weight of each word in the audit triple training data, and merges adjacent words whose weights are higher than the merging threshold to obtain a candidate keyword sequence;

[0083] The hidden layer calculates the relevance score of each candidate keyword in the candidate keyword sequence through the key information key value in the high-dimensional space, normalizes the relevance score by calling the low-rank decomposition matrix in combination with the channel, performs weighted summation based on the normalized relevance score, obtains the attention value of each candidate keyword, and performs training through the spatial attention mechanism to obtain the output of the hidden layer;

[0084] Calculating the output error of the hidden layer according to the error function of the output layer, judging whether the output error of the hidden layer meets the convergence requirement of the output layer, if not, updating the key information key value and the spatial attention mechanism, and returning to the hidden layer for retraining;

[0085] If the convergence requirement is met, extracting the fine-tuning instruction;

[0086] Embodiment 2:

[0087] See also Figure 4 , the present invention provides an embodiment: a large language model system integrating pre-training and dynamic fine-tuning technology, the system comprising an audit language pool training module, an audit training data generation module and an audit language pool fine-tuning module;

[0088] The audit language pool training module is used to establish an audit language pool with audit knowledge as research content;

[0089] The audit training data generation module is used to obtain the audit information input by the user, process the audit information, and obtain the audit triplet training data;

[0090] The audit language pool fine-tuning module is used to construct a fine-tuning model, input the audit triplet training data into the fine-tuning model, obtain a fine-tuning instruction set, and perform instruction fine-tuning on the audit language pool according to the fine-tuning instruction set;

[0091] The audit language pool training module includes:

[0092] The audit ontology knowledge model building unit is used to confirm the field and scope of the audit ontology according to the audit name, determine the keywords and key terms, and establish the audit ontology knowledge model;

[0093] The knowledge entry unit is used to enter the instantiated audit ontology into the audit language pool;

[0094] The audit training data generation module includes:

[0095] Audit information enhancement unit, used for back-translation of audit information, replacement of professional synonyms, and exchange of question and answer pairs;

[0096] A training data generating unit is used to generate audit triplet training data according to the enhanced audit information and the audit language pool.

[0097] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A large language model method integrating pre-training and dynamic fine-tuning technology, characterized in that: The steps include: S1: Establish an audit language pool with audit knowledge as the research content, and the audit knowledge includes typical audit indicators and audit expert experience base; S2: obtaining audit information input by the user, enhancing the audit information, and obtaining enhanced audit information, wherein the enhancing of the audit information includes back translation, professional synonym replacement, and question-answer pair exchange; S3: Generate audit triplet training data according to the enhanced audit information and the audit language pool; S4: Construct a fine-tuning model, input the audit triplet training data into the fine-tuning model, obtain a fine-tuning instruction set, and perform instruction fine-tuning on the audit language pool according to the fine-tuning instruction set.

2. According to claim 1, a large language model method integrating pre-training and dynamic fine-tuning technology is characterized in that: The audit language pool established based on audit knowledge as research content includes: Extract the audit project name and confirm the field and scope of the audit entity based on the audit project name; According to the domain and scope of the audit ontology, keywords and key terms in the domain are determined, and an audit ontology knowledge model is established from top to bottom. The audit ontology knowledge model includes a top-level ontology, a concept subtree, and a sub-layer instance. The concept subtree includes financial statements, engineering projects, approval results, internal assessments, and project implementation logs. Add keywords and key terms in the domain as knowledge features to sub-layer instances; Conduct a logical check on the audit ontology knowledge model to determine whether it meets the audit domain ontology construction principles. If so, instantiate the audit ontology. If not, continue to add sub-layer instances. According to the knowledge entry strategy, the instantiated audit ontology is entered into the audit language pool.

3. According to claim 2, a large language model method integrating pre-training and dynamic fine-tuning technology is characterized in that: The specific steps of the knowledge entry strategy are as follows: S2.1: Obtain the number of concept branches, keywords and keyword attributes of the input instantiated audit ontology; S2.2: Determine the semantic tree where the instantiated audit ontology is located in the audit language pool, traverse all nodes of the semantic tree, extract the keyword pairs with the smallest semantic distance between the node and the instantiated audit ontology as similar node pairs, and calculate the semantic similarity between the node and the instantiated audit ontology; S2.3: Compare the highest semantic similarity with the similarity threshold. If it is less than the similarity threshold, the instantiated audit ontology is used as a sibling node of the node with the highest semantic similarity. If it is greater than or equal to the similarity threshold, the instantiated audit ontology is merged with the node with the highest semantic similarity.

4. According to claim 3, a large language model method integrating pre-training and dynamic fine-tuning technology is characterized in that: The specific steps of S3 are as follows: S3.1: Establish a feature sequence according to the enhanced audit information, and calculate a fitting value of the feature sequence; S3.2: converting the fitting value of the feature sequence into a high-dimensional space, extracting features layer by layer in the high-dimensional space through multiple convolutional layers and pooling layers to obtain high-level semantic features, performing relationship analysis on the high-level semantic features, and calculating audit keyword features; S3.3: Calculate the similarity between the audit keyword feature and the audit ontology in the audit language pool according to the audit keyword feature and the audit language pool; S3.4: Match the audit keyword features with the highest similarity with the audit ontology in the audit language pool to generate audit triplet training data.

5. According to claim 4, a large language model method integrating pre-training and dynamic fine-tuning technology is characterized in that: The calculation formula for the similarity between the audit keyword feature and the audit ontology in the audit language pool is: Among them, R s represents the similarity between the audit keyword feature and the audit ontology in the audit language pool, γ represents the learning rate, u represents a single node of the audit keyword feature, U represents the total number of audit keyword feature nodes, v represents a single node of the audit ontology in the audit language pool, V represents the total number of audit ontology nodes in the audit language pool, H u Represents the feature vector of the u-th node of the audit keyword feature, ω uv represents the connection weight between the audit keyword feature and the audit ontology in the audit language pool, h v represents the feature vector of the vth node of the audit ontology in the audit language pool, b v represents the node threshold of the audit ontology in the audit language pool, and E represents the error.

6. According to claim 5, a large language model method integrating pre-training and dynamic fine-tuning technology is characterized in that: The constructing of the fine-tuning model includes calling a low-rank decomposition matrix in combination with a channel and spatial attention mechanism to train the fine-tuning training model and generate a fine-tuning instruction set.

7. A large language model system integrating pre-training and dynamic fine-tuning technology, which is implemented based on a large language model method integrating pre-training and dynamic fine-tuning technology as described in any one of claims 1 to 6, characterized in that: The system includes an audit language pool training module, an audit training data generation module and an audit language pool fine-tuning module; The audit language pool training module is used to establish an audit language pool with audit knowledge as research content; The audit training data generation module is used to obtain the audit information input by the user, process the audit information, and obtain the audit triplet training data; The audit language pool fine-tuning module is used to construct a fine-tuning model, input the audit triplet training data into the fine-tuning model, obtain a fine-tuning instruction set, and perform instruction fine-tuning on the audit language pool according to the fine-tuning instruction set.

Images