Auditing information extraction method and system based on large language model

Abstract

The invention provides an audit information extraction method and system based on a large language model, and belongs to the technical field of intelligent information processing. Accurate extraction of unstructured documents is achieved through dynamic matching of an intelligent agent, the ultra-long text processing bottleneck is broken through in combination with natural segment slices and knowledge graph association, and document content clustering analysis is completed by adopting a preprocessing-semantic representation-dimension reduction-clustering process; meanwhile, extracted content self-definition is supported, and a complex auditing scene can be adapted without improving model configuration; the system constructed based on the method covers the functions of structured task management, agent scheduling, batch processing, result query and the like, the pain points of low efficiency and error proneness when auditing personnel manually process unstructured data are effectively solved, the labor cost is remarkably reduced in the whole process, the auditing information extraction efficiency and accuracy are improved, and the auditing efficiency is improved. And a powerful support is provided for intelligent auditing work.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent information processing, and particularly relates to an audit information extraction method and system based on a large language model. BACKGROUND

[0002] In the daily work of audit authorities, audit decision and doubt discovery are highly dependent on data support, and data types can be divided into two categories: structured data and unstructured data. Among them, structured data such as business data and financial data can quickly mine audit doubts through preset comparison and analysis models; but unstructured data accounts for a higher proportion and a wider range, including not only the notices, announcements, audit reports, party committee documents and other documents generated by the audit authorities themselves, but also the three-fixed schemes, company charters, work summaries, policies and regulations, meeting minutes and leadership speeches provided by the audited units; Currently, the processing of unstructured audit documents is mainly based on "manual opening, content recognition and classification extraction". For example, a CN118657154A intelligent audit report information extraction method based on LLM technology reads the audit report text, divides the modules through regular matching technology, extracts the cover page and text module information through the audit report structured LLM model, and collects labeled data and divides the data set for fine-tuning training when building the model; the fine-tuning training and deployment reasoning of the LLM model can be completed on a single consumer-level graphics card, the key information of the audit report can be automatically extracted, and the audit result reuse rate can be improved; however, the technical solution of the application relies on a large amount of audit reports in previous years to build a database and labeled data, and the data collection and labeling workload is large; the complex format and special circumstances of the audit report are not fully considered; The processing of unstructured audit documents represented by this method has the following problems: first, the document style is diverse, the type is dynamically increasing, and the manual adaptation cost is high; second, the document content lacks orderly organization, clustering analysis is difficult, and a single document may extract one or more records; these problems result in low efficiency of audit data processing, and human judgment is easy to introduce errors, which restricts the intelligent level of audit work. SUMMARY

[0003] The application provides an audit information extraction method and system based on a large language model to solve the problems of poor adaptability, low efficiency, limited context of large models, clustering difficulty, and poor adaptability of multi-record extraction in the processing of unstructured documents in the audit, through intelligent agent matching, super-long text slicing association, content clustering optimization and other technologies, and realizes efficient and accurate extraction of unstructured audit information.

[0004] In order to achieve the above purpose, the technical scheme adopted by the present application is: In a first aspect, a method and system for extracting audit information based on a large language model are provided, comprising the following steps: S1: obtaining an unstructured audit document; S2: automatically matching the unstructured audit document with an agent based on a large language model; S3: determining whether the unstructured audit document exceeds the context token limit of the large language model, and if so, slicing the document by natural paragraphs; identifying entities in each slice and constructing the association between slices through a knowledge graph; S4: calling the large language model from the unstructured audit document or slice to extract structured data based on the agent; S5: performing cluster analysis on the extracted structured data, cleaning the text and removing stop words before clustering, generating text semantic vectors through a large language model, then dimension reduction and algorithm clustering; S6: storing the extracted structured data and clustering results to an audit database.

[0005] As a preferred embodiment, in step S1, the unstructured audit document includes notices and announcements issued by audit authorities, official documents of superiors and subordinates, audit reports, and three-fixed schemes, company charters, meeting minutes, and leadership speeches provided by audited units.

[0006] As a preferred embodiment, the automatic matching of the unstructured audit document with the agent based on the large language model is implemented as follows: A plurality of agents are predefined in an agent library, and the agents are preconfigured with ID, document type label, core semantic features, and extraction information item template; The text content of the unstructured audit document is read by the ReadFileUtil tool, and the format symbols are filtered; The ID, document type label, and core semantic features of each agent are spliced into a structured text, and the structured text format is {|ID| document type label | core semantic features |}; Then, a prompt is constructed, and the content of the prompt is: playing the role of a text classification expert, based on the document content and the above-mentioned agent features, selecting the most matching agent and returning its ID, and the output format is {"id":"xxx"}, and if there is no matching agent, the output format is {"id":"no match"}; the large language model is called to perform classification; The returned result of the large language model is parsed, and if the matching degree of the matched agent ID is ≥80%, the agent is called; if the matching degree is <80% or the return is {"id":"no match"}, manual intervention is triggered, and the auditor selects an agent or creates a new agent.

[0007] As a preferred method, the document is sliced ​​according to natural paragraphs; entities in each slice are identified, and relationships between slices are constructed using a knowledge graph, specifically: The document is split into segments based on natural paragraphs, and the token length of each segment is less than or equal to 80% of the context token limit of the large language model; each segment is labeled with a tag including document ID, segment number, and context summary; The core entities in each slice are extracted by calling a large language model; a knowledge graph is constructed with entities as nodes and semantic relationships as edges to realize the association between slices. When extracting information, the knowledge graph is searched by keywords to locate relevant segments, and the segments are then integrated and input into the large language model for extraction.

[0008] As a preferred option, specifically: Split the document into slices using a combination of period and newline or semicolon and newline as delimiters.

[0009] Preferably, structured data is extracted from the unstructured audit document or slice based on the intelligent agent invoking a large language model, specifically as follows: The matching agent invokes a large language model to extract information from the unstructured audit document or slice. If a single document contains multiple audit records, it is automatically split into multiple structured data.

[0010] As a preferred approach, cluster analysis is performed on the extracted structured data. Before clustering, the text is cleaned and stop words are removed. Then, a large language model is used to generate text semantic vectors. Subsequently, dimensionality reduction and clustering algorithms are applied, specifically as follows: The structured data above is subjected to text cleaning and stop word removal to obtain preprocessed text; text cleaning includes removing special characters or removing duplicate spaces; The preprocessed text is transformed into a high-dimensional semantic vector using a large language model. Then, the high-dimensional vectors are reduced to 20-50 dimensions using PCA or UMAP algorithms; If the number of clusters is known, the K-means algorithm is used for clustering; if the number of clusters is unknown, hierarchical clustering is used, and the clustering results are output. The clustering results include cluster labels, document list, and core topics.

[0011] Secondly, an audit information extraction system based on a large language model includes: The document retrieval module is configured to retrieve unstructured audit documents; The agent matching module is configured to automatically match the unstructured audit document with the agent based on a large language model. The ultra-long text processing module is configured to: determine whether the unstructured audit document exceeds the context token limit of the large language model; if so, slice the document into segments; identify entities in each segment; and construct the relationship between segments through a knowledge graph. The information extraction module is configured to extract structured data from the unstructured audit document or slice based on the intelligent agent calling the large language model; The clustering module is configured to perform clustering analysis on the extracted structured data. Before clustering, the text is cleaned and stop words are removed. Then, text semantic vectors are generated through a large language model, followed by dimensionality reduction and clustering algorithms. The storage module is configured to store the extracted structured data and clustering results in the audit database.

[0012] Compared with the prior art, the advantages and positive effects of the present invention are as follows: This application presents a method and system for extracting audit information based on a large language model. It achieves accurate extraction of unstructured documents through dynamic matching by intelligent agents, overcomes the bottleneck of processing ultra-long texts by combining natural paragraph slicing with knowledge graph association, and completes document content clustering analysis using a preprocessing-semantic representation-dimensionality reduction-clustering process. It also supports customized extraction content, adapting to complex audit scenarios without requiring model configuration upgrades. The system built based on this method encompasses structured task management, intelligent agent scheduling, batch processing, and result querying, effectively addressing the pain points of low efficiency and error-proneness when auditors manually process unstructured data. The overall process significantly reduces labor costs, improves the efficiency and accuracy of audit information extraction, and provides strong support for the intelligentization of audit work. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This refers to the software interface developed based on the audit information extraction method and system of this application; Figure 2 This is the query interface for the software described in this application; Figure 3 This is the intelligent agent list interface for the software in this application; Figure 4 , Figure 5 This application provides an interface for extracting information from newly created user-defined rules in the software. Figures 6-9Here are screenshots of the relevant code for this application. Detailed Implementation

[0015] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0016] Numerous specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways than those described herein, and therefore the invention is not limited to the specific embodiments disclosed in the following specification.

[0017] like Figure 1 , Figure 2 , Figures 6-9 As shown, this application discloses a method for extracting audit information based on a large language model, which includes the following steps: S1: Obtain unstructured audit documents; the unstructured audit documents include notices and announcements generated by the auditing authority, official documents from superiors and subordinates, audit reports, as well as the three-fixed plan, articles of association, meeting minutes, and speeches by leaders provided by the audited entity; S2: Automatic matching of the unstructured audit documents with the intelligent agent is achieved based on the large language model; S3: Determine whether the unstructured audit document exceeds the context token limit of the large language model. If so, slice the document into segments by natural paragraphs; identify the entities in each segment and construct the relationship between segments through a knowledge graph. S4: Extract structured data from the unstructured audit document or slice based on the large language model invoked by the intelligent agent; S5: Perform cluster analysis on the extracted structured data. Before clustering, clean the text and remove stop words. Then, generate text semantic vectors through a large language model, followed by dimensionality reduction and clustering algorithms. S6: Store the extracted structured data and clustering results in the audit database.

[0018] An audit information extraction system based on a large language model includes: The document retrieval module is configured to retrieve unstructured audit documents; The agent matching module is configured to automatically match the unstructured audit document with the agent based on a large language model. The ultra-long text processing module is configured to: determine whether the unstructured audit document exceeds the context token limit of the large language model; if so, slice the document into segments; identify entities in each segment; and construct the relationship between segments through a knowledge graph. The information extraction module is configured to extract structured data from the unstructured audit document or slice based on the intelligent agent calling the large language model; The clustering module is configured to perform clustering analysis on the extracted structured data. Before clustering, the text is cleaned and stop words are removed. Then, text semantic vectors are generated through a large language model, followed by dimensionality reduction and clustering algorithms. The storage module is configured to store the extracted structured data and clustering results in the audit database.

[0019] like Figure 3 As shown, the automatic matching of the unstructured audit document and the intelligent agent is achieved based on a large language model, specifically as follows: Multiple intelligent agents are predefined in the intelligent agent library. Each intelligent agent is pre-configured with an ID, document type label, core semantic features, and templates for extracting information items. For example, the meeting minutes intelligent agent: the document type label is meeting minutes, the core semantic features are attendees, meeting time, resolutions, and tasks to be done, and the extracted information item template includes meeting topic, resolution content, responsible department, and completion deadline; Audit Report - Issue List Intelligent Agent: The document type tag is Report - Issue List, and the core semantic features are issue number, issue type, basis for type, and audit recommendation. The extracted information item template includes issue number, issue description, basis for type, and audit recommendation. The unstructured audit document is read using the ReadFileUtil tool, and formatting symbols such as headers, footers, and watermarks are filtered out. The IDs, document type tags, and core semantic features of each agent are concatenated into structured text, with the structured text format being {|ID|Document Type Tag|Core Semantic Features|}; for example, the structured text of the meeting minutes agent is {|1|Meeting Minutes|Attendees, Meeting Time|}; and the structured text of the audit report-problem list agent is {|2|Report-Problem List|Problem Number, Qualitative Basis|}. Then, a prompt is constructed, the content of which is: act as a text classification expert, select the best matching agent based on the document content and the above-mentioned agent features, and return its ID, with the output format being {"id":"xxx"}. If no matching agent is found, return or output the format {"id":"No matching"}; call the large language model to perform classification; for example, in this application, the large language model uses the assistantAgent.extract interface; the parameters are set to temperature=0.2, top_p=0.9; If the matching degree of the agent ID is ≥80%, the agent is invoked; if the matching degree is <80% or the result is {"id":"No matching"}, manual intervention is triggered, and the auditor selects an agent or creates a new agent. The document is sliced ​​into segments based on natural paragraphs; entities within each segment are identified, and relationships between segments are constructed using a knowledge graph. Specifically: The document is split into segments by paragraphs, specifically by using a combination of period and newline or semicolon and newline as delimiters. The token length of each slice is ≤ 80% of the context token limit of the large language model; if the model supports 16k tokens, the maximum slice length is 12.8k. Each slice is labeled with a tag including document ID, slice number, and context summary; for example, if there is slice 1, it can be written as: DOC20250325_001_Slice 1: The previous document introduced the funding situation of XX project, and this document continues to discuss the violation issues; The large language model is called to extract the core entities in each slice, such as project name, funding amount, responsible unit or problem type; a knowledge graph is constructed with entities as nodes and semantic relationships as edges to realize the association between slices; specifically, if there is funding situation of project xx in slice 1 and funding violation of project xx in slice 3; slice 1 and slice 3 are connected by the association edge of project xx. When extracting information, the knowledge graph is searched by keywords to locate relevant segments, and the segments are then integrated and input into the large language model for extraction, thus avoiding information fragmentation. like Figure 4 , Figure 5 As shown, the structured data is extracted from the unstructured audit document or slice based on the intelligent agent calling the large language model, specifically as follows: The intelligent agent also includes user-defined rules for extracting information. These user-defined rules allow adding or deleting document type tags, core semantic features, or information extraction item templates in the newly created structured text interface. For example, in the audit report – issue list intelligent agent, a new information extraction item template for rectification deadlines can be added, and information attributes including field name, field type, and field length can be configured. Based on the matching intelligent agent, a large language model is invoked to extract information from the unstructured audit document or slice. If a single document contains multiple audit records, it is automatically split into multiple structured data entries. For example, an audit report containing multiple violations can be split into multiple structured data entries by the large language model, such as: Issue 1: No. 101, qualitative violation payment; Issue 2: No. 102, qualitative asset idleness. Cluster analysis is performed on the extracted structured data. Before clustering, the text is cleaned and stop words are removed. Then, semantic vectors of the text are generated using a large language model. Subsequently, dimensionality reduction and clustering algorithms are applied. Specifically: The structured data above is subjected to text cleaning and stop word removal to obtain preprocessed text; text cleaning includes removing special characters or removing duplicate spaces; The preprocessed text is transformed into a high-dimensional semantic vector using a large language model. Then, PCA or UMAP algorithms are used to reduce the dimensionality of the high-dimensional vector to 20-50 dimensions to reduce computational complexity; If the number of clusters is known, the K-means algorithm is used for clustering; if the number of clusters is unknown, hierarchical clustering is used, and the clustering results are output. The clustering results include cluster labels, document list, and core topics. For example, in this application, cluster 1: the document list contains the category of fund misconduct, including DOC20250301 and DOC20250305, and the core topics are misappropriation of project funds and irregularities in subsidy issuance.

[0020] Example 1, using an audit report - issue list type file as an example: In the structured task management interface, click "Create New Agent" and upload a document named "202X Year XXXX Special Audit Report.pdf". Alternatively, enter the agent name as "202X Year XXXX Special Audit Report.pdf", select "Special Audit" from the parent menu, and complete the creation of the new agent. The document text is read using ReadFileUtil, the header and watermark are filtered, and the information of each agent is concatenated. The structure is {|1|Meeting Minutes|Attendees, Meeting Time|}, {|2|Report - Problem List|Problem Number, Qualitative Basis, Audit Recommendation|}, {|3|Contract|Contract Number, Contracting Party|}. Construct a prompt and call a large language model to analyze the document content. For example, in the document "202X XXXX Special Audit Report.pdf", the content includes "Issue 1: No. 202X001, Qualitative nature: Misappropriation of funds, in accordance with the 'XX Enterprise Fund Management Measures', Recommendation: Rectification within a time limit". Return the matching agent ID 2 of the above: {|2|Report-Issue List|Issue No., Qualitative basis, Audit Recommendation|}. After the matching degree calculation, the matching degree exceeds 80%, and call the agent named Report-Issue List: {|2|Report-Issue List|Issue No., Qualitative basis, Audit Recommendation|}. If the information is clearly collected: Users can modify the information item interface by adding a rectification deadline field to the default template of the intelligent agent, which already includes the problem number, problem type, basis for the type, and audit recommendations. The configuration type is varchar and the length is 500. The large language model extracts information according to the configuration, identifies documents containing 3 audit issue records, and automatically splits them into 3 structured data records; then, it performs cluster analysis on the structured data and stores it.

[0021] Example 2: Taking a document with over 16k tokens named "Minutes of the 2025 Economic Responsibility Audit Meeting.docx" as an example: The document is classified as an ultra-long text. The ultra-long text processing module splits the document into 5 slices according to natural paragraphs. The token length of each slice is ≤12.8k. The slice tags are "DOC20250410_002_Slice 1: Meeting Time and Participants" to "DOC20250410_002_Slice 5: Audit Rectification Requirements"; The large language model extracts entities from each slice, such as "XX unit, meeting time: 202X year X month X day" in slice 1; "XX unit's financial irregularities" in slice 3; and "XX unit's rectification deadline" in slice 5. Subsequently, the extracted structured data was used to construct relationships between slices using a knowledge graph; for example, XX unit was associated with the meeting date: 202X year X month X day; XX unit was associated with XX unit's financial irregularities; XX unit was associated with XX unit's rectification deadline; When a user queries the financial irregularities of XX unit, the system uses a knowledge graph to locate slices 3 and 5, integrates the text from the two slices, and inputs it into a large language model to obtain relevant information.

[0022] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for extracting audit information based on a large language model, characterized in that, The method comprises the following steps: S1: obtaining an unstructured audit document; S2: realizing automatic matching of the unstructured audit document and an intelligent agent based on a large language model; S3: judging whether the unstructured audit document exceeds the context token limit of the large language model, and if so, slicing the document according to natural paragraphs; identifying entities in each slice, and constructing an association relationship between the slices through a knowledge graph; S4: calling the large language model based on the intelligent agent to extract structured data from the unstructured audit document or the slices; S5: performing cluster analysis on the extracted structured data, cleaning the text and removing stop words before clustering, generating a text semantic vector through the large language model, then performing algorithm dimension reduction and algorithm clustering; S6: storing the extracted structured data and the clustering results to an audit database.

2. The method of claim 1, wherein, In step S1, the unstructured audit document includes notices and announcements generated by audit authorities, official documents of superiors and subordinates, audit reports, and three-fixed schemes, company charters, meeting minutes, and leadership speeches provided by audited units.

3. The method of claim 1, wherein the method further comprises: Realizing automatic matching of the unstructured audit document and an intelligent agent based on a large language model, specifically: Predefining a plurality of intelligent agents in an intelligent agent library, the intelligent agents being preconfigured with an ID, a document type label, core semantic features, and an extraction information item template; Reading the text content of the unstructured audit document through a ReadFileUtil tool and filtering format symbols; Splicing the ID, the document type label, and the core semantic features of each intelligent agent into a structured text in the format of {|ID|document type label|core semantic features|}; Then constructing a prompt, the content of the prompt being: acting as a text classification expert, based on the document content and the above intelligent agent features, selecting the most matching intelligent agent and returning its ID, the output format being {"id":"xxx"}, and if there is no matching intelligent agent, the output format being {"id":"no match"}; calling the large language model to perform classification; Analyzing the returned result of the large language model, if the matching degree of the ID of the matched intelligent agent is greater than or equal to 80%, the intelligent agent is called; If the matching degree is less than 80% or the {"id":"no match"} is returned, manual intervention is triggered, and an audit personnel selects an intelligent agent or creates a new intelligent agent.

4. The method of claim 1, wherein the method is based on a large language model. If so, the document is sliced according to natural paragraphs; entities in each slice are identified, and an association relationship between the slices is constructed through a knowledge graph, specifically: The document is split into slices according to natural paragraphs, and the token length of each slice is less than or equal to 80% of the context token limit of the large language model; a label including a document ID, a slice serial number, and a context abstract is added to each slice; The core entities in each slice are extracted by calling the large language model; a knowledge graph is constructed with the entities as nodes and semantic relationships as edges to realize the association between the slices; When extracting information, the knowledge graph is searched through keywords to locate related slices, and the slices are integrated again and input into the large language model for extraction.

5. The method of claim 4, wherein the method further comprises: The document is split into slices according to natural paragraphs, specifically: The document is split into slices according to the combination of period and line feed symbols or the combination of semicolon and line feed symbols as delimiters.

6. The method of claim 1, wherein the method is based on a large language model. The agent calls a large language model to extract structured data from the unstructured audit document or slice, specifically: The agent calls a large language model to extract information from the unstructured audit document or slice based on matching. If a single document contains multiple audit records, it is automatically split to generate multiple structured data.

7. The method of claim 1, wherein the method further comprises: Cluster analysis is performed on the extracted structured data. Before clustering, the text is cleaned and stop words are removed. Then, a large language model is used to generate a text semantic vector. Subsequently, dimensionality reduction and algorithm clustering are performed. Specifically: Text cleaning and stop word removal are performed on the structured data to obtain preprocessed text. Text cleaning includes removing special symbols or removing duplicate spaces. The preprocessed text is converted into a high-dimensional semantic vector using a large language model. The high-dimensional vector is then reduced to 20-50 dimensions using PCA or UMAP algorithms. If the number of clusters is known, K-means algorithm is used for clustering. If the number of clusters is unknown, hierarchical clustering is used, and the clustering results are output. The clustering results include cluster labels, document lists, and core topics.

8. A large language model-based audit information extraction system applied to the large language model-based audit information extraction method of any one of claims 1-7, characterized in that, It includes: A document acquisition module configured to acquire unstructured audit documents. An agent matching module configured to automatically match the unstructured audit documents with agents based on a large language model. A super-long text processing module configured to determine whether the unstructured audit document exceeds the context token limit of the large language model. If so, the document is sliced by natural paragraphs. The entities in each slice are identified, and the association between slices is constructed through a knowledge graph. An information extraction module configured to call a large language model from the unstructured audit document or slice based on an agent to extract structured data. A clustering module configured to perform cluster analysis on the extracted structured data. Before clustering, the text is cleaned and stop words are removed. Then, a large language model is used to generate a text semantic vector. Subsequently, dimensionality reduction and algorithm clustering are performed. A storage module configured to store the extracted structured data and clustering results in an audit database.

Images