Auditing method and system based on large language model

Abstract

The invention particularly relates to an auditing method and system based on a large language model. The auditing method based on the large language model comprises the following steps: receiving a conference summary and an issue material document uploaded by a user, extracting text contents in the document, segmenting the text contents, and classifying segmented texts by using an issue classification model; prompt cue words are set according to the type of the issue, and key information in the issue is extracted by using the trained large language model and stored in a database; performing normalization processing on the issue information, and displaying specific contents of each item, decision and problem according to time nodes; existing problems and risks are intelligently examined according to the subject content and the subject type, and suggestions are given. According to the auditing method and system based on the large language model, a large number of auditing documents can be quickly processed, potential risk points and compliance problems can be accurately identified, and an auditing tracking timeline is automatically generated, so that the efficiency and accuracy of auditing work are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an audit method and system based on a large language model. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the breakthroughs in deep learning in natural language processing (NLP), computer vision (CV) and speech recognition (ASR), and especially the rise of large language models, new vitality has been injected into more and more traditional industries.

[0003] The big language model draws knowledge from massive text data on the Internet through deep learning and neural network architecture, achieving a deep understanding of complex language structures and semantic relationships.

[0004] In auditing work, traditional document review, data analysis and risk assessment processes are often time-consuming and laborious. In order to provide users with more intelligent, convenient and personalized audit support, the present invention proposes an audit method and system based on a large language model. Summary of the invention

[0005] In order to make up for the deficiencies of the prior art, the present invention provides a simple and efficient audit method and system based on a large language model.

[0006] The present invention is achieved through the following technical solutions:

[0007] An audit method based on a large language model, which extracts key information, tracks processes, and audits issues for custom projects, decisions, and issues based on meeting minutes and agenda materials; including the following steps:

[0008] Step S1: Classification of topic types

[0009] Receive meeting minutes and agenda material documents uploaded by users, extract text content from the documents, segment them according to major headings, and use the agenda classification model to classify the segmented text;

[0010] Step S2: Extracting topic information

[0011] Receive the classified topic text content, set prompt words according to the topic type, use the trained large language model to extract the key information in the topic, and store it in the database;

[0012] Step S3: Process tracking timeline

[0013] Normalize the topic information, form a timeline for each project, decision and issue, and display the specific content of each project, decision and issue according to the time node;

[0014] Step S4: Topic Audit

[0015] Intelligently review existing problems and risks based on topic content and topic type, and provide suggestions.

[0016] The step S1 specifically includes the following steps:

[0017] Step S1.1: Receive file

[0018] Receive files uploaded by users through the visualization page, including PDF, Word, Excel and image files; Step S1.2, preprocessing

[0019] Convert pdf and doc files to docx, extract text content from docx files, and divide the text content into several topic text contents according to the main title;

[0020] Step S1.3: Topic classification

[0021] Use the topic classification model to classify the segmented topic content.

[0022] In step S1.3, the process of constructing and training the topic classification model includes the following steps:

[0023] Step S1.3.1, according to the needs of topic classification and the characteristics of the data set, customize and select the topic classification model;

[0024] Step S1.3.2, load the dataset, convert the data into <text, label> format and randomly shuffle it, segment and encode the text, including adding special tags, truncation and padding;

[0025] Step S1.3.3, create data loaders for training and validation sets, and define model loss functions and optimizers;

[0026] Step S1.3.4, in the training loop, batch data, calculate loss, backpropagate and update model parameters;

[0027] At the end of each training cycle, the validation set is used to evaluate the model performance, and the learning rate scheduler and early stopping mechanism are used to optimize the training process;

[0028] Step S1.3.5. Calculate and print the accuracy, precision, recall, and F1 score metrics to obtain the final performance metrics.

[0029] In step S2, extracting key information of topics according to types specifically includes the following steps:

[0030] Step S2.1, customize and determine the key information to be extracted according to the topic type;

[0031] Step S2.2, designing prompt words for the large model question and answer based on the key information, and using the prompt words to extract the key information in the topic;

[0032] Step S2.3, customizing the evaluation criteria for key information extraction and preparing the evaluation set;

[0033] Step S2.4: Customize and evaluate the effect of extracting key information. If the extraction accuracy is lower than the custom threshold, optimize the prompt words or adjust and train the parameters of the large language model to improve the extraction accuracy.

[0034] If the extraction accuracy is lower than the custom threshold, the extracted key information of the topic will be stored in the database.

[0035] In step S2.4, the parameters of the large language model are adjusted and trained, specifically including the following steps:

[0036] Step S2.4.1, according to the nature of the target task and the characteristics of the dataset, custom select the pre-trained model;

[0037] Step S2.4.2: Customize the adjustment scope and adjustment level according to task requirements and available resources;

[0038] Step S2.4.3, determine the hyperparameters in the tuning process, including learning rate, batch size and number of training rounds; the selection of these hyperparameters has an important impact on the performance and convergence speed of fine-tuning;

[0039] Step S2.4.4, initialize the parameters of the large prediction model according to the weights of the pre-trained model; for full fine-tuning, all model parameters will be randomly initialized; for partial fine-tuning, only the parameters of the top layer or a few layers will be randomly initialized;

[0040] Step S2.4.5, training the large language model using the training data set and the adjustment strategy, and during the training process, adjusting the model parameters according to the pre-set hyperparameters and optimization algorithm to minimize the loss function;

[0041] Step S2.4.6: During the training process, the large language model is evaluated regularly using the validation set, and the hyperparameters or adjustment strategies are adjusted according to the evaluation results;

[0042] Step S2.4.7: After the adjustment is completed, the final large language model is evaluated using the test set to obtain the final performance indicators.

[0043] The step S3 specifically includes the following steps:

[0044] Step S3.1: Topic Normalization

[0045] Normalize the topic names extracted by the large language model and classify those with text similarity higher than 90% as the same topic;

[0046] Step S3.2: Timeline tracking

[0047] Track the process of forming a timeline of the same topic in chronological order;

[0048] The step S4 specifically includes the following steps:

[0049] Step S4.1, respectively obtain the contents of three types of topics including projects, decisions and problems;

[0050] Step S4.2: Design corresponding prompt words according to the topic type, use the large language model to extract the problems in the topic, and use the large language model to give corresponding modification suggestions.

[0051] A system for implementing the audit method based on a large language model of the present invention, comprising:

[0052] The topic type classification module is responsible for receiving the meeting minutes and topic material documents uploaded by users, extracting the text content in the documents, segmenting them according to the main headings, and classifying the segmented text using the topic classification model;

[0053] The topic information extraction module is responsible for receiving the classified topic text content, setting prompt words according to the topic type, using the trained large language model to extract the key information in the topic, and storing it in the database;

[0054] The process tracking timeline module is responsible for normalizing the topic information, forming a timeline for each project, decision and issue, and displaying the specific content of each project, decision and issue according to the time node;

[0055] The topic audit module is responsible for intelligently reviewing existing problems and risks based on topic content and topic type, and providing recommendations.

[0056] An audit device based on a large language model, characterized in that it includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above-mentioned method steps when executing the computer program.

[0057] A readable storage medium, characterized in that: a computer program is stored on the readable storage medium, and the computer program implements the above method steps when executed by a processor.

[0058] The beneficial effects of the present invention are: the audit method and system based on the large language model can quickly process a large number of audit documents, accurately identify potential risk points and compliance issues, and automatically generate an audit tracking timeline, thereby effectively reducing the workload of auditors and greatly improving the efficiency and accuracy of audit work. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Attached Figure 1 Schematic diagram of the audit method based on the large language model of the present invention. DETAILED DESCRIPTION

[0061] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0062] This audit method based on a large language model extracts key information, tracks processes, and audits issues for custom projects, decisions, and issues based on meeting minutes and agenda materials. It includes the following steps:

[0063] Step S1: Classification of topic types

[0064] Receive meeting minutes and agenda material documents uploaded by users, extract text content from the documents, segment them according to major headings, and use the agenda classification model to classify the segmented text;

[0065] Step S2: Extracting topic information

[0066] Receive the classified topic text content, set prompt words according to the topic type, use the trained large language model to extract the key information in the topic, and store it in the database;

[0067] Step S3: Process tracking timeline

[0068] Normalize the topic information, form a timeline for each project, decision and issue, and display the specific content of each project, decision and issue according to the time node;

[0069] Step S4: Topic Audit

[0070] Intelligently review existing problems and risks based on topic content and topic type, and provide suggestions.

[0071] The step S1 specifically includes the following steps:

[0072] Step S1.1: Receive file

[0073] Receive files uploaded by users through the visualization page, including PDF, Word, Excel and image files; Step S1.2, preprocessing

[0074] Convert pdf and doc files to docx, extract text content from docx files, and divide the text content into several topic text contents according to the main title;

[0075] Step S1.3: Topic classification

[0076] Use the topic classification model to classify the segmented topic content.

[0077] In step S1.3, the process of constructing and training the topic classification model includes the following steps:

[0078] Step S1.3.1, according to the needs of topic classification and the characteristics of the data set, customize and select the topic classification model;

[0079] Step S1.3.2, load the dataset, convert the data into <text, label> format and randomly shuffle it, segment and encode the text, including adding special tags, truncation and padding;

[0080] Step S1.3.3, create data loaders for training and validation sets, and define model loss functions and optimizers;

[0081] Step S1.3.4, in the training loop, batch data, calculate loss, backpropagate and update model parameters;

[0082] At the end of each training cycle, the validation set is used to evaluate the model performance, and the learning rate scheduler and early stopping mechanism are used to optimize the training process;

[0083] Step S1.3.5. Calculate and print the accuracy, precision, recall, and F1 score metrics to obtain the final performance metrics.

[0084] In step S2, extracting key information of topics according to types specifically includes the following steps:

[0085] Step S2.1, customize and determine the key information to be extracted according to the topic type;

[0086] Step S2.2, designing prompt words for the large model question and answer based on the key information, and using the prompt words to extract the key information in the topic;

[0087] Step S2.3, customizing the evaluation criteria for key information extraction and preparing the evaluation set;

[0088] Step S2.4: Customize and evaluate the effect of extracting key information. If the extraction accuracy is lower than the custom threshold, optimize the prompt words or adjust and train the parameters of the large language model to improve the extraction accuracy.

[0089] If the extraction accuracy is lower than the custom threshold, the extracted key information of the topic will be stored in the database.

[0090] In step S2.4, the parameters of the large language model are adjusted and trained, specifically including the following steps:

[0091] Step S2.4.1, according to the nature of the target task and the characteristics of the dataset, custom select the pre-trained model;

[0092] Step S2.4.2: Customize the adjustment scope and adjustment level according to task requirements and available resources;

[0093] Step S2.4.3, determine the hyperparameters in the tuning process, including learning rate, batch size and number of training rounds; the selection of these hyperparameters has an important impact on the performance and convergence speed of fine-tuning;

[0094] Step S2.4.4, initialize the parameters of the large prediction model according to the weights of the pre-trained model; for full fine-tuning, all model parameters will be randomly initialized; for partial fine-tuning, only the parameters of the top layer or a few layers will be randomly initialized;

[0095] Step S2.4.5, training the large language model using the training data set and the adjustment strategy, and during the training process, adjusting the model parameters according to the pre-set hyperparameters and optimization algorithm to minimize the loss function;

[0096] Step S2.4.6: During the training process, the large language model is evaluated regularly using the validation set, and the hyperparameters or adjustment strategies are adjusted according to the evaluation results;

[0097] Step S2.4.7: After the adjustment is completed, the final large language model is evaluated using the test set to obtain the final performance indicators.

[0098] The step S3 specifically includes the following steps:

[0099] Step S3.1: Topic Normalization

[0100] Normalize the topic names extracted by the large language model and classify those with text similarity higher than 90% as the same topic;

[0101] Step S3.2: Timeline tracking

[0102] Track the process of forming a timeline of the same topic in chronological order;

[0103] The step S4 specifically includes the following steps:

[0104] Step S4.1, respectively obtain the contents of three types of topics including projects, decisions and problems;

[0105] Step S4.2: Design corresponding prompt words according to the topic type, use the large language model to extract the problems in the topic, and use the large language model to give corresponding modification suggestions.

[0106] A system for implementing the audit method based on a large language model, comprising:

[0107] The topic type classification module is responsible for receiving the meeting minutes and topic material documents uploaded by users, extracting the text content in the documents, segmenting them according to the main headings, and classifying the segmented text using the topic classification model;

[0108] The topic information extraction module is responsible for receiving the classified topic text content, setting prompt words according to the topic type, using the trained large language model to extract the key information in the topic, and storing it in the database;

[0109] The process tracking timeline module is responsible for normalizing the topic information, forming a timeline for each project, decision and issue, and displaying the specific content of each project, decision and issue according to the time node;

[0110] The topic audit module is responsible for intelligently reviewing existing problems and risks based on topic content and topic type, and providing recommendations.

[0111] The audit device based on the large language model includes a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the above method steps when executing the computer program.

[0112] The readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method steps are implemented.

[0113] This audit method and system based on a large language model, with its powerful natural language processing capabilities and deep learning algorithms, can efficiently parse complex audit documents, accurately capture key information, and automatically perform risk assessments and compliance checks, thereby significantly improving the efficiency and accuracy of audit work; through intelligent analysis, it can promptly identify potential risk points and problems, and provide auditors with valuable clues and suggestions to help them make more informed decisions; in addition, it can also automatically generate a process tracking timeline, greatly reducing the workload of auditors and allowing them to focus more on core audit tasks.

[0114] In short, this audit method and system based on the large language model has brought intelligent and automated changes to the audit industry and promoted the modernization of audit work.

[0115] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. An audit method based on a large language model, characterized by: Extract key information, track processes and audit issues for custom projects, decisions and issues based on meeting minutes and agenda materials; including the following steps: Step S1: Classification of topic types Receive meeting minutes and agenda material documents uploaded by users, extract text content from the documents, segment them according to major headings, and use the agenda classification model to classify the segmented text; Step S2: Extracting topic information Receive the classified topic text content, set prompt words according to the topic type, use the trained large language model to extract the key information in the topic, and store it in the database; Step S3: Process tracking timeline Normalize the topic information, form a timeline for each project, decision and issue, and display the specific content of each project, decision and issue according to the time node; Step S4: Topic Audit Intelligently review existing problems and risks based on topic content and topic type, and provide suggestions.

2. The audit method based on a large language model according to claim 1, characterized in that: The step S1 specifically includes the following steps: Step S1.1: Receive file Receive files uploaded by users through the visualization page; Step S1.2: Preprocessing Convert pdf and doc files to docx, extract text content from docx files, and divide the text content into several topic text contents according to the main title; Step S1.3: Topic classification Use the topic classification model to classify the segmented topic content.

3. The audit method based on a large language model according to claim 2 is characterized in that: In the step S1.3, The construction and training process of the topic classification model includes the following steps: Step S1.3.1, according to the needs of topic classification and the characteristics of the data set, customize and select the topic classification model; Step S1.3.2, load the dataset, convert the data into <text, label> format and randomly shuffle it, segment and encode the text, including adding special tags, truncation and padding; Step S1.3.3, create data loaders for training and validation sets, and define model loss functions and optimizers; Step S1.3.4, in the training loop, batch data, calculate loss, backpropagate and update model parameters; At the end of each training cycle, the validation set is used to evaluate the model performance, and the learning rate scheduler and early stopping mechanism are used to optimize the training process; Step S1.3.

5. Calculate and print the accuracy, precision, recall, and F1 score metrics to obtain the final performance metrics.

4. The audit method based on a large language model according to claim 1, characterized in that: In step S2, extracting key information of topics according to types specifically includes the following steps: Step S2.1, customize and determine the key information to be extracted according to the topic type; Step S2.2, designing prompt words for the large model question and answer based on the key information, and using the prompt words to extract the key information in the topic; Step S2.3, customizing the evaluation criteria for key information extraction and preparing the evaluation set; Step S2.4: Customize and evaluate the effect of extracting key information. If the extraction accuracy is lower than the custom threshold, optimize the prompt words or adjust and train the parameters of the large language model to improve the extraction accuracy. If the extraction accuracy is lower than the custom threshold, the extracted key information of the topic will be stored in the database.

5. The audit method based on a large language model according to claim 4 is characterized in that: In step S2.4, Adjust and train the parameters of the large language model, including the following steps: Step S2.4.1, according to the nature of the target task and the characteristics of the dataset, custom select the pre-trained model; Step S2.4.2: Customize the adjustment scope and adjustment level according to task requirements and available resources; Step S2.4.3, determine the hyperparameters in the adjustment process, including learning rate, batch size and number of training rounds; Step S2.4.4, initializing the parameters of the large prediction model according to the weights of the pre-trained model; Step S2.4.5, training the large language model using the training data set and the adjustment strategy, and during the training process, adjusting the model parameters according to the pre-set hyperparameters and optimization algorithm to minimize the loss function; Step S2.4.6: During the training process, the large language model is evaluated regularly using the validation set, and the hyperparameters or adjustment strategies are adjusted according to the evaluation results; Step S2.4.7: After the adjustment is completed, the final large language model is evaluated using the test set to obtain the final performance indicators.

6. The audit method based on a large language model according to claim 1, characterized in that: The step S3 specifically includes the following steps: Step S3.1: Topic Normalization Normalize the topic names extracted by the large language model and classify those with text similarity higher than 90% as the same topic; Step S3.2: Timeline tracking Organize the same topic into a chronological order to form a process tracking timeline.

7. The audit method based on a large language model according to claim 1, characterized in that: The step S4 specifically includes the following steps: Step S4.1, respectively obtain the contents of three types of topics including projects, decisions and problems; Step S4.2: Design corresponding prompt words according to the topic type, use the large language model to extract the problems in the topic, and use the large language model to give corresponding modification suggestions.

8. A system for implementing the audit method based on a large language model according to any one of claims 1 to 7, characterized in that: include: The topic type classification module is responsible for receiving the meeting minutes and topic material documents uploaded by users, extracting the text content in the documents, segmenting them according to the main headings, and classifying the segmented text using the topic classification model; The topic information extraction module is responsible for receiving the classified topic text content, setting prompt words according to the topic type, using the trained large language model to extract the key information in the topic, and storing it in the database; The process tracking timeline module is responsible for normalizing the topic information, forming a timeline for each project, decision and issue, and displaying the specific content of each project, decision and issue according to the time node; The topic audit module is responsible for intelligently reviewing existing problems and risks based on topic content and topic type, and providing recommendations.

9. An audit device based on a large language model, characterized in that: The method comprises a memory and a processor; the memory is used to store a computer program, and the processor is used to implement the method according to any one of claims 1 to 7 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images