Intelligent auditing method, device and equipment and storage medium

Abstract

The invention discloses an intelligent auditing method, apparatus and device, and a storage medium. The method comprises the steps of collecting a financial auditing document file; converting the financial audit document file into a structured text by using an OCR model; classifying the structured text through a classification model to obtain a classification result; based on a classification result, key fields are extracted from the structured text through a large language model and a multi-modal model; obtaining an auditing rule and constructing a knowledge graph system based on the auditing rule; and inputting the key field into a knowledge graph system for auditing and decision making to obtain an auditing result. According to the method, the document text is accurately identified through the OCR model, the documents are intelligently classified through the classification model, the document content is deeply analyzed through the large language model and the multi-modal model, and meanwhile, the knowledge graph system is combined for automatic auditing, so that automatic auditing of the documents is realized. The auditing efficiency and accuracy are greatly improved, and the labor cost is reduced. And objectivity and consistency of examination and approval decisions are ensured.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an intelligent audit method, device, equipment and storage medium. Background Art

[0002] In corporate administrative operations, traditional approval processes (such as expense reimbursement, contract review, and credit risk control) have long relied on manual operations, resulting in significant efficiency bottlenecks and inconsistent standards, which have severely hampered the progress of digital transformation. 1. Traditional approval processes rely heavily on manual verification of invoices, validation of business rules, and cross-departmental collaboration. For example, expense reimbursements require item-by-item checks for invoice authenticity, amount compliance, and approval levels; contract reviews require manual comparison of clauses with legal policies; and credit risk control requires analysis of complex data such as borrower qualifications and credit history. These operations often take days or even longer, especially in batch processing scenarios (such as quarterly financial settlements and large-scale contract approvals). Manual processing efficiency cannot meet the needs of rapidly growing businesses. 2. Due to the lack of a unified intelligent rule base, different auditors have varying understandings of policies and implementation standards, leading to inconsistent approval results. For example, the determination of "compliance" in expense reimbursements can vary due to varying auditor experience; and sensitivity to risk clauses in contract reviews can fluctuate due to subjective judgment. This subjectivity affects the fairness of approvals. Summary of the Invention

[0003] The purpose of the present invention is to provide an intelligent review method, device, equipment and storage medium, aiming to solve the problems of low efficiency and inconsistent approval results of traditional approval processes.

[0004] In a first aspect, an embodiment of the present invention provides an intelligent audit method, comprising: Collect financial audit documents; Using an OCR model to convert the financial audit document file into structured text; Classifying the structured text using a classification model to obtain a classification result; Based on the classification result, extract key fields from the structured text using a large language model and a multimodal model; Obtaining audit rules and building a knowledge graph system based on the audit rules; The key fields are input into the knowledge graph system for review and decision making to obtain review results.

[0005] In a second aspect, an embodiment of the present invention provides an intelligent auditing device, comprising: Collection unit, used to collect financial audit documents; A conversion unit, configured to convert the financial audit document file into structured text using an OCR model; A classification unit, configured to classify the structured text using a classification model to obtain a classification result; an extraction unit, configured to extract key fields from the structured text using a large language model and a multimodal model based on the classification result; A construction unit, configured to obtain audit rules and construct a knowledge graph system based on the audit rules; The audit unit is used to input the key fields into the knowledge graph system for audit decision-making to obtain audit results.

[0006] In a third aspect, an embodiment of the present invention further provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the intelligent audit method described in the first aspect when executing the computer program.

[0007] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the intelligent audit method described in the first aspect is implemented.

[0008] The present invention discloses an intelligent audit method, device, equipment and storage medium, the method comprising: collecting financial audit document files; converting the financial audit document files into structured text using an OCR model; classifying the structured text using a classification model to obtain a classification result; extracting key fields from the structured text using a large language model and a multimodal model based on the classification result; obtaining audit rules and constructing a knowledge graph system based on the audit rules; inputting the key fields into the knowledge graph system for audit decision-making to obtain an audit result. The present invention accurately identifies document text through an OCR model, intelligently classifies documents through a classification model, deeply analyzes document content through a large language model and a multimodal model, and automatically audits in combination with a knowledge graph system, thereby realizing automated audit of documents. The audit efficiency and accuracy are greatly improved, and labor costs are reduced. The objectivity and consistency of approval decisions are ensured. The embodiments of the present invention also provide an intelligent audit device, a computer-readable storage medium and a computer device, which have the above-mentioned beneficial effects and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 This is a flowchart of the intelligent audit method; Figure 2 It is a sub-process diagram of the intelligent audit method; Figure 3 is another sub-process diagram of the intelligent audit method; Figure 4 It is a structural diagram of the PP-OCR model; Figure 5 Schematic diagram of the structure of the ViT-PPOCR model; Figure 6 This is a schematic diagram of the structure of the text detection module of the ViT-PPOCR model; Figure 7 This is a schematic diagram of the structure of the text recognition module of the ViT-PPOCR model; Figure 8 It is a structural diagram of a heterogeneous graph; Figure 9 This is a flowchart of the DoubleMix strategy; Figure 10 Schematic diagram of the process of automatic expansion of example fields; Figure 11 A schematic block diagram of an intelligent audit device. DETAILED DESCRIPTION

[0011] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0012] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0013] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in the specification and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0014] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0015] See also Figure 1-Figure 3 , Figure 3 for Figure 2 The subsequent process, Figure 3 The structured text within the LLM parsing module is derived from Figure 2 Structured text acquisition within file categories, Figure 3 The fields to be extracted in the LLM parsing module are Figure 2 The alphabetical mapping within the file classification is then followed by obtaining the fields to be extracted corresponding to the categories. This embodiment provides an intelligent audit method, including: S101: Collect financial audit documents; In this example, financial audit documents are first collected and annotated using an online audit data platform. Specifically, the platform collects various financial documents, including invoices (e.g., VAT invoices and standard invoices), purchase lists (e.g., shopping lists and restaurant bills), and online payment receipts (e.g., Alipay payment receipts and WeChat Pay bills). These documents are then converted into jpg-format images. Financial experts then perform data cleansing on the collected images, scrutinizing their content and eliminating duplicate uploads and blank images without valid information. Only images containing clear financial information that meet the requirements for business scenarios such as expense reimbursement and compliance audits are retained.

[0016] Then, based on the actual business needs of corporate financial audits, such as extracting key information from invoices, such as the invoice code, invoice date, and amount, professional annotation tools are used to annotate the cleaned images. The tools precisely select the areas containing all text within the image, determine the coordinates of the text boxes, and convert the content within the text boxes into structured text (e.g., labeling "Amount: 1,000 yuan" as "Amount: 1,000"). Once the annotation is complete, the text box location information and corresponding structured text content for all images are organized and stored in a JSON-formatted file. Each entry contains the image number, a text box coordinate array, and a structured text field, providing an annotated dataset for subsequent model training.

[0017] S102: Convert the financial audit document file into structured text using an OCR model; Specifically, using the OCR model to convert financial audit documents into structured text includes: Determine the file format of financial audit documents; If the file format of the financial audit document is a standard format file, the file content is directly read to obtain the structured text; If the file format of the financial audit document file is a non-standard format file, the financial audit document file is converted into an image file, and text is extracted from the converted image file using an OCR model to obtain structured text; If the file format of the financial audit document file is an image file, the text of the financial audit document file is extracted using an OCR model to obtain structured text.

[0018] Differentiated processing strategies are adopted for files of different formats. Direct reading of standardized files preserves original structured information and reduces conversion errors. The OCR model focuses on processing image-based text, and combined with optimization algorithms (such as ViT-PPOCR), it enhances the recognition of complex, handwritten text, improving overall structured text accuracy. Text conversion is achieved through a unified "format determination-targeted processing" process, avoiding operational inconsistencies caused by differences in file types during manual processing and providing a standardized structured data foundation for subsequent classification and review.

[0019] Specifically, the system first identifies the file type based on file extensions (e.g., PDF, DOCX, JPG, PNG) and content characteristics (e.g., whether it is a scanned document or handwritten). For example, standard format files (e.g., PDF electronic invoices or Word contracts) can be directly identified based on file metadata, while non-standard formats (e.g., scanned paper receipts or handwritten expense reports) require further processing.

[0020] If the file is in a standard format (such as PDF, CSV, XLS, XLSX, Doc, or DocX), the system directly calls the document parsing interface to extract the text content. For example, table fields in PDF files (such as amounts and dates) are automatically parsed by the table recognition module and mapped to a pre-set JSON structure (e.g., {Invoice Code: 044001900111, Amount: 800 RMB}). This process requires no OCR and relies solely on the document's editable properties, achieving millisecond-level efficiency.

[0021] If the file is in a non-standard format (such as a scanned document or a composite of multiple images), the system converts it to a unified image format (e.g., PNG, JPEG). For example, the scanned document is split into single-page images using Zerox's file conversion function (file-to-image conversion can also be implemented using Python code), and then an OCR model (such as Keras-OCR and PaddleOCR) is called to locate and recognize text. For handwritten documents, Tencent Cloud OCR's handwriting recognition capabilities are used (PaddleOCR can also be used), combined with contextual logic validation (e.g., the "amount" field must match a numeric regular expression), to ultimately output structured text and store it in a JSON file.

[0022] If the file itself is an image (such as a photo of an invoice taken with a mobile phone), the system directly calls the OCR model for processing. For example, PaddleOCR's mobile-optimized model can be used to quickly extract text, and key fields (such as tax ID number and invoice issuer name) can be verified using Tencent Cloud OCR's universal print recognition interface. PaddleOCR can also be used to verify key fields. The recognition results are filtered by the rule engine (for example, excluding ambiguous text and verifying the amount range), generating structured text and associating it with business fields.

[0023] For further information, see Figure 4 and Figure 5 , using the OCR model to convert financial audit document files into structured text includes: Input the financial audit document file into the text detection model for text detection to obtain a text box; Correct the text box; The text recognition model is used to perform text recognition on the content in the corrected text box to obtain structured text.

[0024] The text detection model can accurately identify and locate all text boxes in documents with complex backgrounds, avoiding interference from irrelevant information, ensuring that subsequent processing targets only valid text, and improving the targetedness of structured text. Correcting text boxes can correct text tilt and deformation caused by shooting angle deviation, paper wrinkles, etc., so that the text enters the recognition process in a correct posture, providing better input for the text recognition model and reducing recognition errors. In addition, for complex scenarios such as handwritten fonts, blurred printing, and multi-sized text, step-by-step processing can optimize the models of each link (such as using the ViT-PPOCR model to enhance recognition capabilities), so that the system can stably process various financial documents and ensure the consistency and reliability of structured text.

[0025] In this embodiment, the process of using the OCR model to convert financial audit document files into structured text is as follows: First, the financial audit document files (such as handwritten invoice images, scanned contracts, etc.) are input into the text detection model (i.e., text detection module) based on the CML collaborative mutual learning distillation framework. Through the collaborative training of the teacher model and the student model, the model can accurately detect the area where all the texts in the document are located, and output the detection results containing the text box coordinate information. For example, in a restaurant invoice image, the text boxes containing key information such as "invoice code", "invoice date", and "amount" are accurately selected.

[0026] Next, the detected text frame is corrected. To address text frame deformation caused by tilted shooting angles, wrinkled paper, and other factors, the detection frame correction module adjusts the angle and shape of the text frame, ensuring that the text within the text frame is presented horizontally and correctly, ensuring the integrity and standardization of the text during subsequent recognition.

[0027] Finally, the corrected text box is fed into a text recognition model (i.e., the text recognition module) based on the NaViT-MLP framework. This model uses the GTC-NRTR training strategy, combined with AttentionLoss and CTCLoss to optimize recognition accuracy. It recognizes the content within the text box. For example, it accurately recognizes the handwritten numerals "¥568.00" in the corrected "Amount" text box as "Amount: 568.00 yuan" in structured text. This process ultimately yields structured text containing all key invoice information, providing standardized data for subsequent review.

[0028] Specifically, the OCR model of this embodiment adopts the ViT-PPOCR model, which is based on the PP-OCR model (such as Figure 4 The PP-OCRv4 model in the PP-OCR is improved. PP-OCRv4 is an efficient and practical OCR model launched by Baidu PaddlePaddle. It is the fourth version of the PP-OCR series. The SVTR_LCNetV3 model in PP-OCR is replaced with the NaViT model. Compared with SVTR_LCNetV3, NaViT has stronger recognition capabilities for complex fonts, handwritten text, and long texts; it can handle image inputs of different sizes without resizing; and it has stronger generalization capabilities for complex scenarios (such as documents such as bills). The ViT-PPOCR model is as follows: Figure 5 As shown, Figure 5 Query represents query, key represents key, Image represents image, and Pad represents padding.

[0029] The overall structure of ViT-PPOCR can be divided into three parts: text detection module TextDetection, detection box correction module DetectionBoxesRectify, and text recognition module TextRecognition. The following explains these three parts respectively: See also Figure 6 The Text Detection module uses the Collaborative Mutual Learning (CML) distillation framework and consists of a teacher model and a student model. During the training phase, the teacher model is trained using the Deep Mutual Learning (DML) method. Simultaneously, the student model is trained using a combination of distillation loss and KL divergence loss to ensure that the distribution of its output response maps closely matches that of the teacher model.

[0030] Specifically, the training method of the text detection model includes: Get the image training set and true labels; The image training set is input into the teacher model and two student models to obtain the teacher response graph and two student response graphs respectively; Calculate the KL divergence loss between two student response maps; Calculate the true label loss between the student response map and the true label; Calculate the distillation loss between the student response map and the teacher response map; Update the parameters of the student model according to the KL divergence loss, true label loss and distillation loss to obtain the trained student model; The trained student model is used as the text detection model.

[0031] More specifically, we obtained a training set of images and their corresponding true labels. The training set contained images of various financial documents (such as invoices in different formats and scanned copies of consumption lists). The true labels were the exact coordinates and bounding box information of each text region in the image. We then divided the dataset into 70% for training, 20% for validation, and 10% for testing.

[0032] The image training set is then input into the teacher model and two student models at the same time. After training using the DML method, the teacher model outputs a teacher response graph, and the two student models output their own student response graphs. These response graphs reflect the model's predicted probability distribution for the presence of text at each location in the image.

[0033] Then, the KL divergence loss between the two student response maps is calculated to measure the difference in the predicted distributions of the two models; the true label loss between each student response map and the true label is calculated to ensure that the model prediction fits the actual text area; at the same time, the distillation loss between the student response map and the teacher response map is calculated to enable the student model to learn the knowledge of the teacher model.

[0034] Based on the above KL divergence loss, true label loss and distillation loss, the parameters of the two student models are iteratively updated through the back-propagation algorithm until the model converges. The trained student model is obtained and used as the final text detection model for text detection.

[0035] Then input the image file to be detected into the text detection model to obtain the text box.

[0036] DetectionBoxesRectify: To address the traditional Vision Transformer (ViT)'s reliance on images with fixed resolutions and aspect ratios, this application introduces factorized positional embeddings and a dynamic mask mechanism into the ViT architecture. By decomposing spatial coordinates and controlling the attention receptive field, it achieves adaptive processing of images with arbitrary resolutions and aspect ratios. The specific implementation is as follows: For any resolution of the input image, the system divides the image into non-overlapping patches and assigns a horizontal index x and a vertical index y to each patch. Position encoding function Φx: R→R D and Фy: R→R D Map x and y to D-dimensional vectors respectively, and fuse the spatial information through sum operation: ;

[0037] Where x represents the horizontal index (or relative position) of the image patch, and y represents the vertical index (or relative position) of the image patch. D It is the horizontal position encoding function, which maps the horizontal position to a D-dimensional vector. D It is the position encoding function in the vertical direction, mapping the vertical position to a D-dimensional vector.

[0038] For example, for an image with a resolution of W×H, the number of image blocks is N = W / P×H / P (P is the block size). The position code for each image block is calculated by independently calculating its horizontal position Φx(x) and vertical position Φy(y) and then adding them together. This design avoids the traditional absolute position coding's dependence on a fixed resolution and, by separating the encoding of the x and y axes, supports image inputs with dynamically changing aspect ratios (such as 4:3 and 16:9).

[0039] In order to control the range of attention calculation, the system introduces a mask matrix M∈{0, 1} N×N , where M i，j =1 means that the i-th token is allowed to follow the j-th token, M i，j = 0, the association is masked. For example, for test images whose aspect ratio deviates significantly from the training set (such as ultra-wide screens), the mask matrix is ​​dynamically generated to limit each token to focus only on the local neighborhood (such as the token within a 3×3 window) to avoid redundant calculations. The mask matrix can be generated based on the spatial position of the image block: if the Euclidean distance between two tokens exceeds a preset threshold (such as 20 pixels), the corresponding M i，j Set to 0. This mechanism significantly reduces the computational complexity of self-attention while suppressing interference from irrelevant areas.

[0040] In the pooling stage, the system uses the pooling mask P∈{0, 1} S (S is the number of tokens in the current stage) Selectively retain key tokens. For example, in the hierarchical ViT architecture, when the input image resolution is higher than the training set, the pooling mask P k =1 means that the kth token participates in the pooling, and only tokens in high-frequency semantic areas (such as text and edges) are allowed to participate in the pooling operation, while low-frequency background areas are ignored (P k =0). Pooling operations combined with residual connections further enhance feature transfer.

[0041] See also Figure 7 , Text Recognition Module Text Recognition: This module adopts the GTC-NRTR training strategy, and replaces the original PP-LCNetV3-SVTR-G framework with the NaViT-MLP framework. Specifically, the NaViT model is first used to extract features from the input financial document image (such as handwritten reimbursement forms, printed invoices, etc.), capturing the global and local features of the text in the image, and then the MLP (multi-layer perceptron) is used to complete the linear conversion of features to text recognition results. During the training process, the loss function of this section is jointly composed of CTC loss and Attention loss: CTC loss is used to calculate the alignment loss of the input image sequence and the output text sequence to solve the sequence length mismatch problem caused by the different lengths of financial document text; Attention loss is used as the cross-entropy loss between the predicted characters and the labeled characters to enhance the model's accurate recognition of each character, such as ensuring that the prediction of the number after the "amount" field is exactly the same as the label. The total loss Ltotal is obtained by the weighted sum of the two ( , λ is the weight coefficient), and the entire text recognition module is trained by minimizing Ltotal, so that the model can ensure the overall alignment of the sequence and improve the accuracy of single character recognition when processing various financial document texts. Among them, Figure 7 GTC_1 and GTC_2 are global text classifiers, NaviT_MLP_1 and NaviT_MLP_2 are multi-layer perceptrons, UDML stands for unsupervised domain adaptation multimodal learning; score represents the confidence score of the model's prediction results for text images.

[0042] More specifically, the training method for the text recognition model includes: Get the image training set and its corresponding labeled characters; Perform multi-scale transformation and data enhancement on the image training set to generate multi-scale input images; Use the NaViT model to extract features from the input image and generate image feature representation; The image feature representation is transformed through a multi-layer perceptron to obtain a transformed feature representation; Calculate the connectionist temporal classification loss based on the transformed feature representation and structured text; Recognize the image feature representation through a non-recurrent neural network model (such as NRTR) to obtain the predicted character; Calculate the cross entropy loss based on the predicted characters and the labeled characters; The parameters of the NaViT model are updated according to the connectionist temporal classification loss and cross entropy loss to obtain the trained NaViT model; The trained NaViT model is used as the text recognition model.

[0043] In this embodiment, the training method of the text recognition model is as follows: first, an image training set and its corresponding annotated characters are obtained. The image training set contains images of various financial audit documents (such as scanned copies of invoices in different formats, photos of handwritten expense lists, etc.), and the annotated characters are the accurate character sequences of the text content in the image (such as "Amount: 1,200 yuan" on the invoice corresponds to the annotated characters "Amount: 1,200 yuan").

[0044] The image training set is processed with multi-scale transformation (generating images of different resolutions) and data augmentation (such as rotation, scaling, text content perturbation, etc.) to generate multi-scale input images to enhance the model's adaptability to complex scenes.

[0045] The multi-scale input image is input into the NaViT model for feature extraction to generate an image feature representation containing text details and spatial information; the image feature representation is transformed through a multi-layer perceptron to obtain a transformed feature representation, and the connectionist temporal classification loss (CTC Loss) is calculated based on the transformed feature representation and the corresponding structured text to handle the alignment problem of the input image sequence and the output text sequence.

[0046] At the same time, the image feature representation is recognized through a non-recurrent neural network model (such as NRTR) to obtain the predicted characters; the cross entropy loss (Attention Loss) is calculated based on the predicted characters and the labeled characters to optimize the accuracy of character prediction.

[0047] According to the connectionist temporal classification loss and cross entropy loss, the parameters of the NaViT model are iteratively updated through the back propagation algorithm until the model converges. The trained NaViT model is obtained and used as the final text recognition model.

[0048] In this embodiment, Attention DML Loss (Attention Deep Mutual Learning Loss) is introduced between two non-recurrent neural network models. The core goal of Attention DML Loss is to quantify the differences in the attention mechanisms of different models (or different branches of the same model), prompting them to learn from each other and converge on feature attention patterns, thereby enhancing the collaborative ability between models. Its calculation process can be divided into the following key steps: For each non-recurrent neural network model, extract the attention weight matrix in its attention mechanism module (such as the self-attention layer of Transformer, the attention module of CNN, etc.). Assume that the input sequence length processed by the model is n (such as the number of text tokens or image patches), and the weight matrix of a single attention head is A∈R n×n , where A i，j Indicates the attention strength of the i-th position to the j-th position.

[0049] If the model contains multiple attention heads (such as the Transformer's multi-head attention), it is necessary to extract the attention matrix of each head, denoted as {A1, A2, ..., Ah} (h is the number of heads); if it contains multi-layer attention modules, it is also necessary to extract the matrices of each layer.

[0050] Next, to ensure that the attention matrices of different models are comparable on the same scale, they need to be normalized: Convert each row of each attention matrix into a probability distribution, i.e. , so that the sum of each row element is 1. This step converts the original attention intensity into relative attention probability, eliminating the deviation caused by the output scale difference of different models. Represents the normalized matrix elements; represents the attention matrix, It means to sum all elements in row i.

[0051] If the matrix has extreme values ​​(such as 0 or too large values), you can add small perturbations (such as ) to avoid computational instability.

[0052] Then, a metric function is used to quantify the difference between the attention matrices of the two models (or branches). Common metrics include KL divergence (Kullback-Leibler Divergence), Frobenius distance, and cosine similarity.

[0053] Then, since the model usually contains multiple attention heads and multi-layer attention modules, the difference losses of all heads and layers need to be aggregated: For each of the h attention heads of a single model, calculate the difference loss L for each head head，k (k=1,...,h), take the average to get the single layer loss: ; For the L attention layers of the model, calculate the loss L for each layer layer，l (l=1,...,L), take the average to get the final Attention DML Loss: .

[0054] In this example, Attention DML Loss serves as an auxiliary loss, weightedly combined with the primary task loss (such as classification loss and CTC loss) to jointly optimize model parameters. By normalizing the attention matrix, quantizing differences, and aggregating multi-scale losses, Attention DML Loss enables deep mutual learning of attention mechanisms across different models, ultimately improving the collaborative performance of the model group.

[0055] In this embodiment, SVTR DML Loss (Deep Mutual Learning Loss) is calculated to achieve cross-branch knowledge transfer and collaborative optimization by constraining the output features or prediction distributions of the two MLPs to be consistent. The specific steps are as follows: MLP1 outputs the feature vector F1∈R N×D , MLP2 outputs feature vector F2∈R N×D (N is the number of samples, D is the feature dimension); If MLP is used as the prediction head, MLP1 outputs the character probability distribution P1∈R N×T×V , MLP2 output P2∈R N×T×V (T is the length of the text sequence, V is the character set size).

[0056] For the feature outputs F1 and F2 of the two MLPs, the difference in their feature distribution is measured to ensure that the conversion mode of the two input features is consistent. Frobenius distance and cosine similarity loss can be used.

[0057] If two MLPs are used as prediction heads (outputting character probability distributions), it is necessary to constrain the consistency of their prediction results to avoid excessive prediction deviations between branches. KL divergence or cross entropy loss can be used.

[0058] Then, according to the functions of the two MLPs (feature conversion or prediction), the above losses are selected for weighted aggregation to obtain SVTR DML Loss.

[0059] Then use SVTR DML Loss as an auxiliary loss and train it jointly with SVTR's main task loss (such as CTC Loss and Attention Loss).

[0060] The SVTR DML Loss of the two MLPs achieves knowledge sharing between branches by constraining the consistency of feature distribution and prediction results, ultimately improving the accuracy and stability of SVTR text recognition.

[0061] In this example, Feature Distillation Loss is calculated between two NaViT models. Feature Distillation Loss is a core component of knowledge distillation (KD). It aims to improve the performance of the student model by allowing it to learn the intermediate-level features (hidden features) of the teacher model. For two NaViT models (assuming one is the teacher model T and the other is the student model S), the goal of Feature Distillation Loss is to align the intermediate-level feature distributions of the teacher and student models and to leverage the teacher model's robustness at variable resolutions to guide the student model's feature extraction capabilities.

[0062] Then, the text recognition model is used to perform text recognition on the content in the corrected text box to obtain structured text.

[0063] S103: Classifying the structured text using a classification model to obtain a classification result; Specifically, the structured text is classified by the classification model, and the classification results include: Preliminary classification of structured text through keyword matching; If the preliminary classification is successful, the preliminary classification results are further classified through the XLG-Net model; If the XLG-Net model classification is successful, the final classification result is obtained; If the XLG-Net model fails to classify or the preliminary classification fails, the structured text is classified through the multimodal model to obtain the final classification result.

[0064] Initial classification is quickly completed through keyword matching, reducing the frequency of complex model calls and improving overall classification efficiency. The XLG-Net model then performs in-depth analysis to achieve precise classification, balancing speed and accuracy. When initial classification or XLG-Net model classification fails, a multimodal model is introduced as a supplement to address blind spots in the previous classification process, ensuring that all structured text is effectively classified and avoiding classification failures due to the limitations of a single model. The XLG-Net model, combining XLNet and GCNII, excels at processing text semantic associations. The multimodal model can fuse text with underlying visual information, making it more advantageous for classifying ambiguous, cross-domain financial documents (such as mixed-format expense reports), improving the system's adaptability to complex business scenarios.

[0065] In this embodiment, the process of classifying the structured text of financial audit documents through the classification model is as follows: First, the structured text is preliminarily classified using keyword matching, and a keyword library in the financial field is preset (such as "VAT special invoice" corresponds to the "input tax invoice" category, and "travel expense reimbursement" corresponds to the "expense reimbursement" category). The field content in the structured text is matched with the keyword library. If the match is successful (such as the "VAT general invoice" keyword is identified), the preliminary classification is completed.

[0066] If the initial classification is successful, the results are fed into the XLG-Net model for further classification. This model combines XLNet's ability to capture textual semantics with GCNII's graph modeling strengths to analyze the deep semantics of structured text (for example, the relationship between the "taxable service name" in an invoice and the business type). If the classification confidence score output by the model exceeds a preset threshold (for example, 0.85), the classification is considered successful, resulting in the final classification result (for example, "Office VAT General Invoice").

[0067] If the XLG-Net model's classification confidence falls below the threshold (classification failure) or the initial classification fails to match any keywords (preliminary classification failure), the multimodal model is called for classification. The multimodal model simultaneously inputs structured text and the corresponding original document image features. It combines the text information with visual features such as the layout and seal in the image (for example, identifying the header style of a "Travel Expense Reimbursement Form" in an image), and after comprehensive judgment, outputs the final classification result. This ensures accurate classification even in the absence of keywords or ambiguous text (such as non-standard keyword expressions in handwritten documents).

[0068] In this embodiment, on the basis of obtaining the structured text, the structured text is categorized and labeled according to the business needs of the financial audit (such as distinguishing the types of expenses, types of bills, etc.). For example, for structured text containing information such as "VAT special invoice" and "Name of goods: office computer", its category is labeled as "purchase input invoice"; for structured text containing information such as "accommodation fee" and "travel expense reimbursement form", its category is labeled as "travel expense reimbursement form". After the labeling is completed, these labeled information are associated with the corresponding structured text data and stored in a json format file. Each json file entry contains the unique identifier of the structured text, the specific text content and the labeled category information, providing a data set with accurate category labels for the training of the subsequent classification model.

[0069] In this example, the XLG-Net model is a hybrid fusion model for text classification that combines XLNet (XLNet is a pre-trained model in the field of natural language processing, whose core goal is to integrate the advantages of autoregressive and autoencoder language models) and GCNII (graph neural network). It mainly consists of four modules: (1) graph construction; (2) feature extraction based on XLNetMix; (3) feature extraction based on GCNII; and (4) feature aggregation.

[0070] (1) Graph construction: See also Figure 8 , we construct a heterogeneous graph for the dataset, represented by G = (V, E), where V represents the set of all nodes in the graph and E represents the set of edges between nodes. Nodes are divided into two categories: document nodes and word nodes. TF-IDF is used to measure the connection between document nodes and word nodes. PMI measures the connection between word nodes: ; Among them, #W(i, j) represents the number of sliding windows containing word i and word j, #W(i) represents the number of sliding windows containing word i, #W represents the total number of sliding windows; #W(j) represents the number of sliding windows containing word j.

[0071] The weight calculation formula of the edge between nodes is as follows: ; In this example, a document set D and a sliding window size w are input, and the goal is to output an adjacency matrix A and a feature matrix X. The specific steps are as follows: First, the total number of documents |D| and the number of word nodes |word_set| are determined. Next, a zero matrix A of size (|D| + |word_set|) × (|D| + |word_set|) is created as the adjacency matrix, and a zero matrix X of the same size but with the third dimension being the embedding size is created as the feature matrix.

[0072] Then, for each document di in the set D, we first calculate the document vector for that document, which is the average of all the word vectors in the document. After obtaining the document vector, we write it into the corresponding position of the feature matrix X, specifically X[|word_set|+i].

[0073] Next, for each document di, we iterate over each word wj in the document. For each word, we calculate its TF-IDF value in the document and then add this value to the A[|word_set|+i][word_index(wj)] and A[word_index(wj)][|word_set|+i] positions in the adjacency matrix A to represent the strength of the connection between the document and the word.

[0074] Finally, for each document di, we traverse each pair of words (wj, wk) obtained by sliding the window size w in the document. For each pair of words, we calculate the PMI (point mutual information) value between them and then fill this value into the A[word_index(wj)][word_index(wk)] and A[word_index(wk)][word_index(wj)] positions of the adjacency matrix A to represent the strength of the association between the words.

[0075] Through the above four steps, we finally obtain the filled adjacency matrix A and feature matrix X, which respectively describe the "document-word" and "word-word" relationships in the document collection and the feature representation of each document and word. This completes the entire process from input document collection to output adjacency matrix and feature matrix; (2) Feature extraction based on XLNetMix: See also Figure 9 This application proposes a text processing method that combines back-translation, XLNet contextual embedding, and DoubleMix strategy to improve the model's robustness and generalization ability for both original text and its perturbation samples. The specific steps are as follows: For the original text sample x in the heterogeneous graph, two perturbed samples x1 and x2 are generated through two independent back-translation operations. For example, a Chinese text is first translated into English and then translated back into Chinese, using different language pairs or translation models (e.g., Google Translate in the first stage and DeepL in the second stage) to introduce diverse semantic perturbations.

[0076] The original sample x and the perturbed samples x1 and x2 are then fed into the pre-trained XLNet model (a bidirectional autoregressive language model based on the Transformer architecture). Its hidden layer extracts contextualized word embeddings to produce document embedding vectors. XLNet uses a permutation language modeling (PLM) mechanism to dynamically adjust word order to capture bidirectional contextual information, ensuring the contextual awareness of the embedded features.

[0077] The DoubleMix strategy is used in the hidden layer of XLNet for two-step interpolation mixing: The first step is to use the Dirichlet distribution α to generate the mixing weights λ = (λ1, λ2) for the perturbed samples x1 and x2, where λ1 + λ2 = 1. Based on the weights λ, the mixed sample xmix = λ1x1 + λ2x2 is calculated. The semantic features of the two perturbed samples are fused through linear interpolation to enhance data diversity.

[0078] The parameter α of the Dirichlet distribution can adjust the concentration of the mixture ratio. For example, when α=1, the weights are evenly distributed, and when α<1, they tend to be extreme weights (such as one sample dominates).

[0079] The second step is to use the Beta distribution β(γ, δ) to generate the mixing coefficient θ for the mixed sample xmix and the original sample x, where γ>δ to give greater weight to the original sample (for example, γ=2, δ=1). Then, the final mixed sample xfinal=θx+(1-θ)xmix is ​​calculated. The weighted average suppresses the noise influence in the perturbed sample and retains the original semantic core information. (3) Feature extraction based on GCNII: Each layer of GCNII is calculated according to the following formula: ; Among them, H(0) is the 0th layer feature representation, α n , β n is a hyperparameter, is the graph convolution matrix; and is the weight matrix.

[0080] GCNII outputs the updated feature representation Z GCNII , used to capture document structure information, and obtain the final prediction through the softmax layer: ; Where g is the GCNII model, X is the input feature matrix, and A is the normalized adjacency matrix; (4) Feature aggregation: After softmax transformation, the final result is the linear combination Z of XLNetMix and GCNII. The formula is as follows: ; X represents the document embedding vector, and W is the corresponding weight matrix; ;

[0081] The loss function for training is: ; Y i , c∈{0,1} is the true label of sample i belonging to category c, and y^i,c is the predicted probability distribution.

[0082] In this embodiment, the multimodal model uses the qwen-vl-max model. This model integrates text semantics and image visual features to identify the text content in the document, analyze the layout structure, and classify the text using multimodal understanding capabilities. For example, for a handwritten travel expense reimbursement form with a unique layout, failed keyword matching, and unsuccessful XLG-Net classification, qwen-vl-max recognizes multimodal information in the image, such as the handwritten font, the position of the amount column, and the itinerary description text, and outputs a classification result (such as "Travel Expense Reimbursement Form - City Transportation") as the final classification conclusion, ensuring the accuracy and comprehensiveness of the classification of financial audit documents.

[0083] S104: Based on the classification result, extract key fields from the structured text using a large language model and a multimodal model; Specifically, based on the classification results, key fields are extracted from structured text using large language models and multimodal models, including: Parse and extract fields from structured text using a large language model; If the extraction is successful, the final key field is obtained; If the extraction fails, the financial audit document file will be converted into image format; The multimodal model is used to extract fields from the financial audit document files converted into image format to obtain the final key fields.

[0084] The large language model prioritizes parsing structured text, enabling rapid extraction of key fields (such as invoice headers, amounts, and dates) from standardized text. This approach is applicable to most standard financial documents, improving overall processing efficiency. Furthermore, the large language model's powerful semantic understanding capabilities accurately identify logical relationships between fields, ensuring extraction accuracy. When the large language model fails (e.g., for structured text containing mixed handwriting or severely malformed formatting), the multimodal model is used to process documents in image format. Leveraging the model's comprehensive analysis of visual layout and text morphology, it addresses field extraction challenges in complex scenarios (e.g., identifying amounts next to blurred seals or project names within tilted text boxes), addressing the extraction needs of a wide range of specialized documents.

[0085] In this example, key fields are extracted using a large language model and a multimodal model based on the classification results of financial audit documents (such as "VAT Special Invoice" and "Travel Expense Reimbursement Form"). First, a large language model (such as Qwen-Plus) is invoked to parse the structured text classified as "VAT Special Invoice." The model extracts the corresponding information from the structured text based on a preset invoice field template (including invoice code, invoice date, purchaser name, amount, tax rate, etc.). If the large language model successfully extracts fields such as "Invoice Code: 110023456789" and "Amount: 1500.00 RMB," and if the format of each field complies with financial standards, this information is used as the final key fields.

[0086] If the large language model fails extraction (for example, due to the presence of garbled handwritten text in the structured text, preventing the model from recognizing the "Price and Tax Total" field), the financial audit document is converted into a jpg image. A multimodal model (such as qwen-vl-max) is then used to process the image. The model combines visual information such as the text layout (such as the position of the amount column) and font style (such as the shape of the handwritten numerals) to accurately locate and extract key fields. For example, for a handwritten travel expense report image where extraction failed due to garbled structured text, the multimodal model recognizes the handwritten numeral "680" in the "Accommodation Fee" column and the printed text "2025-08-01" in the "Date" column, ultimately extracting key fields such as "Accommodation Fee: 680 Yuan" and "Date: 2025-08-01" as the final result.

[0087] See also Figure 10In this embodiment, a field automatic expansion mechanism is introduced to address the synonym problem that exists when extracting key fields of financial audit documents. When the structured text is parsed by the large language model to extract fields, the field description is added to the extraction instruction. For example, for the "handler" field, the additional description "refers to the person in charge of handling the financial business. Possible expressions include but are not limited to the person in charge, the person on the business trip, the name, etc." During the parsing process, the large language model will identify the expressions in the text that are synonymous with the target field based on these descriptions. For example, in the structured text of a travel expense reimbursement form, it is recognized that "business traveler: Zhang San" is synonymous with "handler", and the corresponding relationship of "business traveler → handler" is added to the field mapping table.

[0088] The next time the field table is loaded, the system automatically incorporates the new entry in the mapping table, ensuring that the field table includes "handler" and its synonyms, "handler," "business trip," and "name." Now, when processing structured text containing "handler: Li Si," the large language model can directly identify the corresponding field as "handler" using the expanded field table, efficiently completing extraction and avoiding field omissions due to differences in synonym representations. This improves the comprehensiveness and accuracy of key field extraction in financial documents.

[0089] S105: Acquire audit rules and build a knowledge graph system based on the audit rules; Specifically, in this embodiment, the process of obtaining financial audit rules and building a knowledge graph system is as follows: First, collect the company's internal financial audit system, national fiscal and tax regulations and clauses related to document audit (such as "the invoice amount shall not exceed the reimbursement application amount", "travel accommodation standards must meet the upper limit corresponding to the job grade", etc.), and implicit rules formed in historical audit cases (such as "cross-month invoices must be accompanied by a statement of the situation"), and organize these rules into structured audit items.

[0090] Next, a knowledge graph system is constructed based on these audit rules: determine the core entities of the knowledge graph, including "audit rules", "document types", "key fields", "constraints", etc. (for example, the "audit rules" entity can be specifically "travel accommodation reimbursement rules", and the "key fields" entity includes "accommodation amount", "position", etc.); define the relationship between entities, such as "audit rules apply to document types", "audit rules contain constraints", and "constraints are associated with key fields" (for example, "travel accommodation reimbursement rules apply to travel expense reimbursement forms", "travel accommodation reimbursement rules contain constraints - accommodation amount ≤ accommodation standard corresponding to position").

[0091] Finally, the organized audit rules are entered into the knowledge graph according to the definitions of entities and relationships, forming a knowledge network centered on rules and linking document types and key fields. For example, when the document type "VAT Special Invoice" is linked to the "Input Tax Deduction Rules," the knowledge graph clearly displays the constraint in the rule that "the invoice issuance date must be within the deduction period" and the key field "Invoice Date" associated with this constraint, providing a structured rule basis for subsequent intelligent audits.

[0092] It should be noted that the construction of the knowledge graph system in this embodiment is completed once before the keyword is input, and there is no need to construct it every time the keyword is obtained.

[0093] S106: Input the key fields into the knowledge graph system for review and decision making to obtain review results.

[0094] In this embodiment, the review and decision-making process is carried out based on a constructed business rule knowledge graph system. This knowledge graph models financial review rules in the form of ontology and triples. Each rule is an independent rule node, associated with attributes such as check fields and judgment conditions. For example, rule node "R001" corresponds to "Travel and accommodation expenses must not exceed the standard." Its associated check fields are "Accommodation amount" and "Rank," and the judgment condition is "Accommodation amount ≤ the upper limit of accommodation corresponding to the rank."

[0095] The financial document to be reviewed is extracted as an entity node, and the key fields obtained in step 5.3 (such as "Accommodation Amount: 800 Yuan" and "Position: Department Manager") are embedded as attributes in this entity node. The system uses the knowledge graph to match the rule nodes corresponding to the entity node and intelligently determines whether the content of each field is compliant. For example, the system compares the value of the "Accommodation Amount" field with the accommodation limit for the "Department Manager" position (assuming it is 700 Yuan). If 800 Yuan is greater than 700 Yuan, the field is deemed to be in violation.

[0096] The final output is a non-compliant result, with the non-compliant field clearly marked as "accommodation amount", the corresponding rule number "R001" and the condition item "accommodation amount ≤ the upper limit of accommodation corresponding to the job grade", providing auditors with a clear basis for violations and improving the accuracy and efficiency of financial audits.

[0097] For example: In the invoice review scenario, (1) Construction of business rule knowledge graph: {Rule R001: [Check fields: invoice amount, tax rate, judgment condition: invoice amount × tax rate = tax amount], {Rule R002: [Check fields: Invoice issue date, Reimbursement application date, Judgment condition: Invoice issue date must not be later than reimbursement application date]}; (2) Entity and field embedding of audit files: {Entity node: Invoice F001; Field embedding: [Invoice amount: 1000 yuan, Tax rate: 13%, Tax amount: 130 yuan, Issue date: July 10, 2025, Reimbursement application date: July 8, 2025]}; (3) Intelligent compliance audit: Verification rule R001: Calculate "invoice amount × tax rate" (1000 × 13% = 130 yuan), which is consistent with the tax amount (130 yuan) and is compliant; Verification rule R002: The invoice issue date (July 10, 2025) is later than the reimbursement application date (July 8, 2025), which is a violation; (4) Output audit results: Non-compliant fields: invoice issue date, reimbursement application date; Association rule number: R002; Violation condition item: The invoice issuance date must not be later than the reimbursement application date.

[0098] In this embodiment, the entire document recognition and review process is encapsulated as an API interface, encompassing the complete logic from financial document upload, OCR recognition, structured processing, classification, key field extraction, and rule-based review. After integrating this API into the online financial review platform, when a user adds a new document (such as uploading a scanned VAT invoice or completing and submitting a travel expense reimbursement form), the platform calls the API in real time to trigger an automated review process. The API first detects and recognizes the newly added document, converting it into structured text. It then classifies and extracts key fields such as "amount" and "invoice date." Finally, based on the review rules in the knowledge graph, it verifies the compliance of these fields. If all fields comply with the rules (e.g., the invoice amount matches the reimbursement application amount and the invoice date is within the validity period), the API returns a "review approved" decision. If any fields violate the rules (e.g., the accommodation amount exceeds the standard for the position level), the API returns a "review failed" decision along with detailed violation details (including the violating field, the corresponding rule number, and the conditional item). The online platform displays the decision opinions returned by the API to users and auditors in real time, enabling instant automatic review of new documents, significantly shortening the review cycle, and improving the convenience and timeliness of financial audits.

[0099] This embodiment uses an OCR model to accurately identify document text, a classification model to intelligently classify documents, and a large language model and multimodal model to deeply analyze document content. Combined with a knowledge graph system for automatic review, this system achieves automated document review. This significantly improves review efficiency and accuracy, reduces labor costs, and ensures the objectivity and consistency of approval decisions.

[0100] See also Figure 11 This embodiment provides an intelligent auditing device 200, including: The collection unit 201 is used to collect financial audit documents; A conversion unit 202 is configured to convert the financial audit document file into structured text using an OCR model; The classification unit 203 is configured to classify the structured text using a classification model to obtain a classification result; An extraction unit 204 is configured to extract key fields from the structured text using a large language model and a multimodal model based on the classification result; A construction unit 205 is used to obtain audit rules and construct a knowledge graph system based on the audit rules; The review unit 206 is used to input the key fields into the knowledge graph system for review decision and obtain the review result.

[0101] Furthermore, the conversion unit 202 includes: A format determination subunit, configured to determine the file format of the financial audit document file; A reading subunit is used to directly read the file content to obtain a structured text if the file format of the financial audit document file is a standard format file; a format conversion subunit, configured to convert the financial audit document file into an image file if the file format of the financial audit document file is a non-standard format file, and perform text extraction on the converted image file using the OCR model to obtain structured text; The text extraction subunit is configured to extract text from the financial audit document file using the OCR model to obtain structured text if the file format of the financial audit document file is an image file.

[0102] Furthermore, the conversion unit 202 further includes: A text detection subunit, configured to input the financial audit document file into a text detection model for text detection to obtain a text box; A correction subunit, configured to correct the text box; The text recognition subunit is used to perform text recognition on the content in the corrected text box through a text recognition model to obtain structured text.

[0103] Furthermore, the training method of the text detection model includes: Get the image training set and true labels; Inputting the image training set into a teacher model and two student models to obtain a teacher response graph and two student response graphs respectively; Calculate the KL divergence loss between two student response maps; Calculate the true label loss between the student response map and the true label; Calculate the distillation loss between the student response map and the teacher response map; Updating the parameters of the student model according to the KL divergence loss, the true label loss, and the distillation loss to obtain a trained student model; The trained student model is used as the text detection model.

[0104] Furthermore, the training method of the text recognition model includes: Get the image training set and its corresponding labeled characters; Performing multi-scale transformation and data enhancement processing on the image training set to generate a multi-scale input image; Use the NaViT model to extract features from the input image and generate image feature representation; Converting the image feature representation using a multi-layer perceptron to obtain a converted feature representation; Calculating a connectionist temporal classification loss based on the transformed feature representation and the structured text; Recognizing the image feature representation through a non-recurrent neural network model to obtain a predicted character; Calculating a cross entropy loss based on the predicted characters and the labeled characters; Updating parameters of the NaViT model according to the connectionist temporal classification loss and the cross entropy loss to obtain a trained NaViT model; The trained NaViT model is used as the text recognition model.

[0105] Furthermore, the classification unit 203 includes: A preliminary classification subunit, configured to perform preliminary classification of the structured text by keyword matching; The further classification subunit is used to further classify the preliminary classification results through the XLG-Net model if the preliminary classification is successful; The result acquisition subunit is used to obtain the final classification result if the XLG-Net model classification is successful; The text classification subunit is used to classify the structured text through the multimodal model to obtain the final classification result if the XLG-Net model fails to classify or the preliminary classification fails.

[0106] Furthermore, the extraction unit 204 includes: a parsing subunit, configured to parse and extract fields from the structured text using the large language model; The field acquisition subunit is used to obtain the final key field if the extraction is successful; A file format conversion subunit, configured to convert the financial audit document file into an image format if the extraction fails; The field extraction subunit is used to extract fields from the financial audit document file converted into an image format through the multimodal model to obtain the final key fields.

[0107] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-mentioned devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0108] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When executed, the computer program can implement the methods provided in the above embodiments. The storage medium may include a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0109] The present invention further provides a computer device that may include a memory and a processor. The memory stores a computer program, and the processor, when invoking the computer program in the memory, can implement the method provided in the above embodiment. Of course, the computer device may also include various network interfaces, a power supply, and other components.

[0110] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0111] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprising" or any other variations thereof are intended to cover non-exclusive.

[0112] Inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a..." does not preclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

Claims

1. An intelligent audit method, characterized in that: include: Collect financial audit documents; Using an OCR model to convert the financial audit document file into structured text; Classifying the structured text using a classification model to obtain a classification result; Based on the classification result, extract key fields from the structured text using a large language model and a multimodal model; Obtaining audit rules and building a knowledge graph system based on the audit rules; The key fields are input into the knowledge graph system for review and decision making to obtain review results.

2. The intelligent audit method according to claim 1, characterized in that: The converting of the financial audit document file into structured text using the OCR model includes: Determining the file format of the financial audit document; If the file format of the financial audit document file is a standard format file, the file content is directly read to obtain a structured text; If the file format of the financial audit document file is a non-standard format file, converting the financial audit document file into an image file, and performing text extraction on the converted image file using the OCR model to obtain structured text; If the file format of the financial audit document file is an image file, text extraction is performed on the financial audit document file using the OCR model to obtain structured text.

3. The intelligent audit method according to claim 1, characterized in that: The converting of the financial audit document file into structured text using the OCR model includes: Input the financial audit document file into the text detection model for text detection to obtain a text box; Correcting the text box; The text recognition model is used to perform text recognition on the content in the corrected text box to obtain structured text.

4. The intelligent audit method according to claim 3, characterized in that: The training method of the text detection model includes: Get the image training set and true labels; Inputting the image training set into a teacher model and two student models to obtain a teacher response graph and two student response graphs respectively; Calculate the KL divergence loss between two student response maps; Calculate the true label loss between the student response map and the true label; Calculate the distillation loss between the student response map and the teacher response map; Updating the parameters of the student model according to the KL divergence loss, the true label loss, and the distillation loss to obtain a trained student model; The trained student model is used as the text detection model.

5. The intelligent audit method according to claim 3, characterized in that: The training method of the text recognition model includes: Get the image training set and its corresponding labeled characters; Performing multi-scale transformation and data enhancement processing on the image training set to generate a multi-scale input image; Use the NaViT model to extract features from the input image and generate image feature representation; Converting the image feature representation using a multi-layer perceptron to obtain a converted feature representation; Calculating a connectionist temporal classification loss based on the transformed feature representation and the structured text; Recognizing the image feature representation through a non-recurrent neural network model to obtain a predicted character; Calculating a cross entropy loss based on the predicted characters and the labeled characters; Updating parameters of the NaViT model according to the connectionist temporal classification loss and the cross entropy loss to obtain a trained NaViT model; The trained NaViT model is used as the text recognition model.

6. The intelligent audit method according to claim 1, characterized in that: The classification of the structured text by the classification model to obtain the classification result includes: Preliminarily classifying the structured text by keyword matching; If the preliminary classification is successful, the preliminary classification results are further classified through the XLG-Net model; If the XLG-Net model classification is successful, the final classification result is obtained; If the XLG-Net model fails to classify or the preliminary classification fails, the structured text is classified by the multimodal model to obtain the final classification result.

7. The intelligent audit method according to claim 1, characterized in that: The extracting key fields from the structured text based on the classification result by using a large language model and a multimodal model includes: Parsing and extracting fields from the structured text using the large language model; If the extraction is successful, the final key field is obtained; If the extraction fails, converting the financial audit document file into an image format; The multimodal model is used to extract fields from the financial audit document file converted into an image format to obtain the final key fields.

8. An intelligent audit device, characterized in that: include: Collection unit, used to collect financial audit documents; A conversion unit, configured to convert the financial audit document file into structured text using an OCR model; A classification unit, configured to classify the structured text using a classification model to obtain a classification result; an extraction unit, configured to extract key fields from the structured text using a large language model and a multimodal model based on the classification result; A construction unit, configured to obtain audit rules and construct a knowledge graph system based on the audit rules; The audit unit is used to input the key fields into the knowledge graph system for audit decision-making to obtain audit results.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the intelligent audit method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the intelligent audit method according to any one of claims 1 to 7.

Images