An intelligent auditing method, device, equipment and storage medium

Abstract

The application discloses an intelligent auditing method, device and equipment and a storage medium. The method comprises the following steps: collecting a financial auditing document file; converting the financial auditing document file into a structured text by using an OCR model; classifying the structured text by using a classification model to obtain a classification result; extracting a key field from the structured text by using a large language model and a multi-modal model based on the classification result; obtaining an auditing rule and constructing a knowledge graph system based on the auditing rule; inputting the key field into the knowledge graph system to make an auditing decision and obtain an auditing result. The application accurately identifies a document text by using an OCR model, intelligently classifies a document by using a classification model, deeply analyzes a document content by using a large language model and a multi-modal model, and automatically audits in combination with a knowledge graph system, so that the automatic auditing of the document is realized. The auditing efficiency and accuracy are greatly improved, and the labor cost is reduced. The objectivity and consistency of the approval decision are ensured.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an intelligent auditing method, apparatus, device, and storage medium. Background Technology

[0002] In corporate administration, traditional approval processes (such as expense reimbursement, contract review, and credit risk control) have long relied on manual operation, resulting in significant efficiency bottlenecks and inconsistent standards, which seriously hinders the progress of corporate digital transformation.

[0003] 1. Traditional approval processes heavily rely on manual verification of invoices, validation of business rules, and cross-departmental collaboration. For example, expense reimbursements require item-by-item checks of invoice authenticity, amount compliance, and approval level; contract review requires manual comparison of terms with laws and policies; and credit risk control requires analysis of complex data such as borrower qualifications and credit records. These operations typically take several days or even longer, especially in batch task scenarios (such as quarterly financial settlements and large-scale contract approvals), where manual processing efficiency cannot meet the needs of rapidly developing enterprises.

[0004] 2. Due to the lack of a unified intelligent rule base, different auditors have varying understandings and application standards of policies, leading to inconsistencies in approval results. For example, the determination of "compliance" in expense reimbursement may deviate due to differences in auditor experience; the sensitivity to risk clauses in contract review may also fluctuate due to subjective judgment. This subjectivity affects the fairness of the approval process. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent review method, apparatus, device, and storage medium to address the problems of low efficiency and inconsistent approval results in traditional approval processes.

[0006] In a first aspect, embodiments of the present invention provide an intelligent auditing method, comprising:

[0007] Collect financial audit documents;

[0008] The financial audit document was converted into structured text using an OCR model.

[0009] The structured text is classified using a classification model to obtain the classification results;

[0010] Based on the classification results, key fields are extracted from the structured text using a large language model and a multimodal model;

[0011] Obtain the review rules and construct a knowledge graph system based on the review rules;

[0012] The key fields are input into the knowledge graph system for review and decision-making, and the review results are obtained.

[0013] Secondly, embodiments of the present invention provide an intelligent auditing device, comprising:

[0014] The data acquisition unit is used to collect financial audit documents.

[0015] A conversion unit is used to convert the financial audit document file into structured text using an OCR model;

[0016] A classification unit is used to classify the structured text using a classification model to obtain a classification result;

[0017] An extraction unit is used to extract key fields from the structured text based on the classification results using a large language model and a multimodal model.

[0018] The construction unit is used to acquire the review rules and construct a knowledge graph system based on the review rules;

[0019] The review unit is used to input the key fields into the knowledge graph system for review decision-making and obtain the review result.

[0020] Thirdly, embodiments of the present invention provide a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the intelligent auditing method described in the first aspect.

[0021] Fourthly, embodiments of the present invention also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the intelligent auditing method described in the first aspect.

[0022] This invention discloses an intelligent auditing method, apparatus, device, and storage medium. The method includes: collecting financial auditing documents; converting the financial auditing documents into structured text using an OCR model; classifying the structured text using a classification model to obtain classification results; extracting key fields from the structured text based on the classification results using a large language model and a multimodal model; obtaining auditing rules and constructing a knowledge graph system based on the auditing rules; and inputting the key fields into the knowledge graph system for auditing decision-making to obtain audit results. This invention accurately identifies document text using an OCR model, intelligently classifies documents using a classification model, deeply analyzes document content using a large language model and a multimodal model, and automatically audits documents using a knowledge graph system, thereby achieving automated document auditing. This significantly improves auditing efficiency and accuracy, reduces labor costs, and ensures the objectivity and consistency of approval decisions. This invention also provides an intelligent auditing device, a computer-readable storage medium, and a computer device, which have the above-mentioned beneficial effects, and will not be elaborated further here. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the intelligent review method.

[0025] Figure 2 This is a schematic diagram of the sub-processes of the intelligent review method;

[0026] Figure 3 This is a schematic diagram of another sub-process of the intelligent review method;

[0027] Figure 4 This is a schematic diagram of the PP-OCR model.

[0028] Figure 5 This is a schematic diagram of the ViT-PPOCR model.

[0029] Figure 6 This is a schematic diagram of the text detection module in the ViT-PPOCR model.

[0030] Figure 7 This is a schematic diagram of the text recognition module in the ViT-PPOCR model.

[0031] Figure 8 This is a schematic diagram of the structure of a heterogeneous graph;

[0032] Figure 9 This is a flowchart illustrating the DoubleMix strategy.

[0033] Figure 10 A flowchart illustrating the automatic field expansion process for example;

[0034] Figure 11 This is a schematic block diagram of an intelligent auditing device. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] It should be understood that, when used in this specification and the appended claims, the terms “comprising” and “including” indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more of its features, integrals, steps, operations, elements, components and / or collections thereof.

[0037] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0038] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0039] Please see Figures 1-3 , Figure 3 for Figure 2 The subsequent process, Figure 3 The structured text within the inner LLM parsing module is derived from... Figure 2 Structured text obtained from document categories Figure 3 The fields to be extracted in the internal LLM parsing module are from Figure 2 This embodiment provides an intelligent review method, which uses the alphabetical mapping within the file category to obtain the fields to be extracted corresponding to the category.

[0040] S101: Collect financial audit documents;

[0041] In this embodiment, the first step is to collect and annotate financial audit documents through an online audit data platform. Specifically, various financial-related documents, such as invoices (e.g., VAT invoices, general invoices), expense lists (e.g., shopping lists, restaurant expense details), and online payment receipts (e.g., Alipay payment vouchers, WeChat Pay bills), are collected from the platform and then converted into JPG images. Next, financial experts perform data cleaning on the collected images, reviewing each image's content and removing duplicate uploads and blank images without valid information, retaining only those images containing clear financial information that meet the requirements of business scenarios such as expense reimbursement and compliance audits.

[0042] Subsequently, based on the actual business needs of the company's financial audit, such as extracting key information from invoices including invoice codes, invoice dates, and amounts, professional annotation tools were used to annotate the cleaned images. This involved precisely selecting the regions containing all text within the images, determining the coordinates of the text boxes, and converting the content within the text boxes into structured text (e.g., annotating "Amount: 1000 yuan" as "Amount: 1000"). After annotation, the text box position information and corresponding structured text content for all images were compiled into a JSON file and stored. Each entry contained an image number, a text box coordinate array, and structured text fields, providing an annotated dataset for subsequent model training.

[0043] S102: Use an OCR model to convert the financial audit document file into structured text;

[0044] Specifically, using an OCR model to convert financial audit documents into structured text includes:

[0045] Determine the file format of financial audit documents;

[0046] If the financial audit document is in a standard format, the file content can be read directly to obtain structured text.

[0047] If the financial audit document file is in a non-standard format, the financial audit document file will be converted into an image file, and the text will be extracted from the converted image file using an OCR model to obtain structured text.

[0048] If the financial audit document is in image format, then the OCR model is used to extract the text from the financial audit document to obtain structured text.

[0049] Differentiated processing strategies are adopted for files of different formats. Standardized files can be directly read to retain the original structured information, reducing conversion errors. The OCR model focuses on processing image-based text, and combined with optimization algorithms (such as ViT-PPOCR), it enhances the recognition ability of complex and handwritten text, improving the overall accuracy of structured text. Text conversion is achieved through a unified "format judgment - targeted processing" process, avoiding inconsistencies in operation caused by differences in file types during manual processing, and providing a standardized structured data foundation for subsequent classification, review, and other processes.

[0050] Specifically, the system first identifies the file type by its file extension (such as PDF, DOCX, JPG, PNG, etc.) and content characteristics (such as whether it is a scanned document or handwritten). For example, standardized format files (such as electronic invoices in PDF format or Word contracts) can be directly identified through file metadata, while non-standard formats (such as scanned copies of paper documents or handwritten expense reports) require further processing.

[0051] If the file is in a standard format (such as standard PDF, CSV, XLS, XLSX, DOC, or DOCX formats), the system directly calls the document parsing interface to extract the text content. For example, table fields in a PDF file (such as amount and date) are automatically parsed by the table recognition module and mapped to a preset JSON structure (such as {Invoice Code: 044001900111, Amount: 800 RMB}). This process requires no OCR, relying only on the editable attributes of the document itself, achieving millisecond-level efficiency.

[0052] If the file is in a non-standard format (such as a scanned document or a document composed of multiple images), the system converts it to a uniform image format (such as PNG or JPEG). For example, the scanned document can be split into single-page images using Zerox's file conversion function (or a file-to-image function can be implemented using written Python code). Then, an OCR model (such as Keras-OCR or PaddleOCR) is called to locate and recognize the text. For handwritten documents, Tencent Cloud OCR's handwritten recognition capability is used (PaddleOCR can also be used), combined with contextual logic validation (such as matching the "amount" field with numeric regular expressions), and finally, structured text is output and stored in a JSON file.

[0053] If the file itself is an image (such as a photo of an invoice taken with a mobile phone), the system directly calls the OCR model for processing. For example, it can use PaddleOCR's mobile-optimized model to quickly extract text and verify key fields (such as tax ID and invoice issuer name) through Tencent Cloud OCR's general printed text recognition interface. Alternatively, PaddleOCR can be used to verify key fields. After the recognition results are filtered by the rule engine (such as excluding fuzzy text and verifying the range of monetary values), structured text is generated and associated with business fields.

[0054] For further details, please refer to Figure 4 and Figure 5 Using an OCR model to convert financial audit documents into structured text includes:

[0055] The financial audit documents are input into a text detection model for text detection, resulting in text boxes.

[0056] Correct the text box;

[0057] The text recognition model is used to perform text recognition on the content within the corrected text box to obtain structured text.

[0058] The text detection model can accurately identify and locate all text boxes in documents with complex backgrounds, avoiding interference from irrelevant information and ensuring that subsequent processing focuses only on valid text, thus improving the relevance of structured text. Text box correction can rectify text tilting and distortion caused by factors such as shooting angle deviations and paper wrinkles, ensuring the text enters the recognition stage in a correct posture, providing higher-quality input for the text recognition model and reducing recognition errors. Furthermore, for complex scenarios such as handwritten fonts, blurred printing, and multi-size text, step-by-step processing allows for targeted optimization of the model at each stage (e.g., using the ViT-PPOCR model to enhance recognition capabilities), enabling the system to stably process various financial documents and ensuring the consistency and reliability of structured text.

[0059] In this embodiment, the process of converting financial audit documents into structured text using an OCR model is as follows: First, the financial audit documents (such as handwritten invoice images, scanned contracts, etc.) are input into a text detection model (i.e., text detection module) based on the CML collaborative learning distillation framework. Through collaborative training between the teacher model and the student model, this model can accurately detect the areas where all text is located in the document and output detection results containing text box coordinate information. For example, in a restaurant invoice image, the text boxes containing key information such as "invoice code", "invoice date", and "amount" can be accurately selected.

[0060] Next, the detected text boxes are corrected. To address text box distortion caused by factors such as tilted shooting angles and paper wrinkles, the detection box correction module adjusts the angle and shape of the text boxes, ensuring that the text within them is presented in a horizontal and upright manner, thus guaranteeing the integrity and standardization of the text during subsequent recognition.

[0061] Finally, the corrected text boxes are input into a text recognition model (i.e., the text recognition module) based on the NaViT-MLP framework. This model uses the GTC-NRTR training strategy, combined with AttentionLoss and CTCLoss to optimize recognition accuracy, to recognize the content within the text boxes. For example, it accurately recognizes the handwritten number "¥568.00" in the corrected "Amount" text box as "Amount: 568.00 yuan" in the structured text. Through the above process, a structured text containing various key information of the document is finally obtained, providing standardized data for subsequent review processes.

[0062] Specifically, the OCR model used in this embodiment is the ViT-PPOCR model, which is based on the PP-OCR model (such as...). Figure 4 The PP-OCRv4 model (the fourth version in the PP-OCR series, a high-efficiency and practical OCR model launched by Baidu PaddlePaddle) has been improved by replacing the SVTR_LCNetV3 model with the NaViT model. Compared to SVTR_LCNetV3, NaViT has stronger recognition capabilities for complex fonts, handwritten text, and long texts; it can handle image inputs of different sizes without resizing; and it has stronger generalization ability in handling complex scenarios (such as invoices and other documents). The ViT-PPOCR model is as follows... Figure 5 As shown, where, Figure 5 Within this text, Query represents a query, key represents a key, Image represents an image, and Pad represents padding.

[0063] The overall structure of ViT-PPOCR can be divided into three parts: TextDetection, DetectionBoxesRectify, and TextRecognition. These three parts are explained below:

[0064] Please see Figure 6 The text detection module, Text Detection, uses a CML (Collaborative Mutual Learning) distillation framework, consisting of a teacher model and a student model. During the training phase, the teacher model is trained using the DML (Deep Mutual Learning) method; simultaneously, when training the student model, distillation loss and KL divergence loss are combined to make its output response maps as close as possible to the distribution of the teacher model.

[0065] Specifically, the training methods for text detection models include:

[0066] Obtain the image training set and ground truth labels;

[0067] The image training set is input into the teacher model and two student models to obtain the teacher response map and the two student response maps, respectively.

[0068] Calculate the KL divergence loss between two student response plots;

[0069] Calculate the true label loss between the student response map and the true label;

[0070] Calculate the distillation loss between the student response map and the teacher response map;

[0071] The parameters of the student model are updated based on KL divergence loss, true label loss, and distillation loss to obtain the trained student model.

[0072] The trained student model was used as a text detection model.

[0073] More specifically, we obtain the image training set and its corresponding ground truth labels. The image training set includes images of various financial audit documents (such as invoices of different formats, scanned copies of expense lists, etc.), and the ground truth labels are the accurate coordinates and bounding box information of each text region in the image. Next, the dataset is divided proportionally, with 70% used as the training set, 20% as the validation set, and 10% as the test set.

[0074] The image training set was then simultaneously input into the teacher model and two student models. The teacher model was trained using the DML method and output a teacher response map, while the two student models output their respective student response maps. These response maps reflect the model's predicted probability distribution of the presence of text at various locations in the image.

[0075] Then, the KL divergence loss between the two student response maps is calculated to measure the difference in the predicted distributions of the two models; the true label loss between each student response map and the true label is calculated to ensure that the model predictions fit the actual text region; at the same time, the distillation loss between the student response map and the teacher response map is calculated to enable the student model to learn the knowledge of the teacher model.

[0076] Based on the KL divergence loss, true label loss, and distillation loss mentioned above, the parameters of the two student models are iteratively updated through the backpropagation algorithm until the models converge, resulting in the trained student models, which are then used as the final text detection model for text detection.

[0077] Next, input the image file to be detected into the text detection model to obtain the text box.

[0078] The DetectionBoxesRectify module addresses the dependency of traditional VisionTransformer (ViT) on images with fixed resolution and aspect ratios. This application introduces Factorized Positional Embeddings and a dynamic masking mechanism into the ViT architecture. By decomposing spatial coordinates and controlling the attention receptive field, it achieves adaptive processing of images with arbitrary resolution and aspect ratios. The specific implementation is as follows:

[0079] For any resolution of the input image, the system divides the image into non-overlapping image patches and assigns a horizontal index x and a vertical index y to each patch. The positional encoding function Фx: R→R D And Фy: R→R D Map x and y to D-dimensional vectors respectively, and fuse spatial information through addition:

[0080] ;

[0081] Where x represents the horizontal index (or relative position) of the image patch, and y represents the vertical index (or relative position) of the image patch. Фx: R→R D It is a horizontal position encoding function that maps horizontal positions to D-dimensional vectors. Фy: R→R D It is a vertical position encoding function that maps vertical positions to D-dimensional vectors.

[0082] For example, for an image with a resolution of W×H, the number of image blocks is N=W / P×H / P (where P is the block size). The positional code of each image block is calculated independently and then added together by its horizontal position Φx(x) and vertical position Φy(y). This design avoids the dependence of traditional absolute positional coding on a fixed resolution and supports image input with dynamically changing aspect ratios (such as 4:3 and 16:9) by separating the encoding of the x and y axes.

[0083] To control the scope of attention calculation, the system introduces a mask matrix M∈{0,1}. N×N M i，j =1 indicates that the i-th token is allowed to follow the j-th token, M i，j =0 indicates that associations are masked. For example, for test images with aspect ratios significantly deviating from the training set (such as ultrawide screens), a dynamically generated mask matrix restricts each token to focus only on its local neighborhood (such as tokens within a 3×3 window), avoiding redundant computation. The mask matrix can be generated based on the spatial location of image patches: if the Euclidean distance between two tokens exceeds a preset threshold (such as 20 pixels), then the corresponding M... i，jSet to 0. This mechanism significantly reduces the computational complexity of self-attention while suppressing interference from irrelevant regions.

[0084] During the pooling phase, the system uses a pooling mask P∈{0,1} S (S represents the number of tokens in the current stage) Selectively retain key tokens. For example, in the hierarchical ViT architecture, when the input image resolution is higher than the training set, the pooling mask P... k =1 indicates that the k-th token participates in pooling, allowing only tokens from high-frequency semantic regions (such as text and edges) to participate in the pooling operation, while ignoring low-frequency background regions (P). k =0). Pooling operations combined with residual connections further enhance feature transfer.

[0085] Please see Figure 7 Text Recognition Module: This module adopts the GTC-NRTR training strategy and replaces the original PP-LCNetV3-SVTR-G framework with the NaViT-MLP framework. Specifically, the NaViT model is first used to extract features from the input financial document images (such as handwritten expense reports, printed invoices, etc.), capturing the global and local features of the text in the image. Then, the MLP (Multilayer Perceptron) is used to complete the linear transformation from features to text recognition results. During training, the loss function of this module is composed of CTC loss and Attention loss: CTC loss is used to calculate the alignment loss between the input image sequence and the output text sequence, solving the problem of sequence length mismatch caused by the varying lengths of financial document text; Attention loss is used as the cross-entropy loss between predicted characters and labeled characters, strengthening the model's accurate recognition of each character, for example, ensuring that the predicted number after the "amount" field is completely consistent with the label. The total loss Ltotal is obtained by weighted summation of the two. (where λ is the weighting coefficient). By minimizing Ltotal, the entire text recognition module is trained, enabling the model to ensure overall sequence alignment while improving single-character recognition accuracy when processing various financial document texts. Figure 7 GTC_1 and GTC_2 are both global text classifiers, NaviT_MLP_1 and NaviT_MLP_2 are both multilayer perceptrons, UDML represents unsupervised domain adaptation multimodal learning, and score represents the confidence score of the model's prediction results for text images.

[0086] More specifically, the training methods for text recognition models include:

[0087] Obtain the image training set and its corresponding labeled characters;

[0088] Multi-scale transformation and data augmentation are performed on the image training set to generate multi-scale input images;

[0089] The NaViT model is used to extract features from the input image and generate image feature representations.

[0090] The image feature representation is transformed using a multilayer perceptron to obtain the transformed feature representation;

[0091] A connectionist temporal classification loss is calculated based on transformed feature representations and structured text;

[0092] Image feature representations are identified using a recurrent neural network model (such as NRTR) to obtain predicted characters;

[0093] The cross-entropy loss is calculated based on the predicted and labeled characters;

[0094] The parameters of the NaViT model are updated based on connectionist temporal classification loss and cross-entropy loss to obtain the trained NaViT model.

[0095] The trained NaViT model was used as the text recognition model.

[0096] In this embodiment, the training method of the text recognition model is as follows: First, obtain the image training set and its corresponding labeled characters. The image training set includes images of various financial audit documents (such as scanned copies of invoices of different formats, photos of handwritten expense lists, etc.). The labeled characters are the accurate character sequences of the text content in the image (such as the labeled characters "Amount: 1200 yuan" corresponding to "Amount: 1200 yuan" on the invoice).

[0097] Multi-scale transformations (generating images at different resolutions) and data augmentation (such as rotation, scaling, and text content perturbation) are performed on the image training set to generate multi-scale input images, thereby enhancing the model's adaptability to complex scenes.

[0098] Multi-scale input images are fed into the NaViT model for feature extraction, generating image feature representations that include textual details and spatial information. These image feature representations are then transformed using a multilayer perceptron to obtain transformed feature representations. Based on these transformed feature representations and the corresponding structured text, a connectionist temporal classification loss (CTC Loss) is calculated to address the alignment problem between the input image sequence and the output text sequence.

[0099] Meanwhile, the image feature representation is identified using an acyclic neural network model (such as NRTR) to obtain predicted characters; cross-entropy loss (Attention Loss) is calculated based on the predicted characters and labeled characters to optimize the accuracy of character prediction.

[0100] Based on connectionist temporal classification loss and cross-entropy loss, the parameters of the NaViT model are iteratively updated through backpropagation until the model converges, resulting in the trained NaViT model, which is then used as the final text recognition model.

[0101] In this embodiment, Attention DML Loss (Attention Deep Mutual Learning Loss) is introduced between two acyclic neural network models. The core objective of Attention Deep Mutual Learning Loss is to quantify the differences in attention mechanisms between different models (or different branches of the same model), prompting them to learn from each other and converge on feature attention patterns, thereby enhancing the collaborative ability between models. Its calculation process can be divided into the following key steps:

[0102] For each acyclic neural network model, extract the attention weight matrix from its attention mechanism module (such as the self-attention layer of a Transformer, the attention module of a CNN, etc.). Assuming the input sequence processed by the model has a length of n (e.g., the number of text tokens, the number of image patches), the weight matrix of a single attention head is A∈R. n×n A i，j This represents the intensity of attention from position i to position j.

[0103] If the model contains multiple attention heads (such as the multi-head attention of Transformer), the attention matrix of each head needs to be extracted, denoted as {A1, A2, ..., Ah} (where h is the number of heads); if it contains multiple attention modules, the matrices of each layer also need to be extracted.

[0104] Next, to ensure that the attention matrices of different models are comparable on the same scale, standardization is required:

[0105] Transform each row of each attention matrix into a probability distribution, i.e. This step converts the original attention intensity into relative attention probability, eliminating bias caused by differences in output scale between different models. Specifically, Represents the normalized matrix elements; Represents the attention matrix. This represents the summation of all elements in the i-th row.

[0106] If the matrix has extreme values ​​(such as 0 or excessively large values), a small perturbation can be added (e.g., To avoid computational instability.

[0107] Then, the differences between the attention matrices of the two models (or branches) are quantified using a metric function. Commonly used metrics include: Kullback-Leibler Divergence, Frobenius distance, and cosine similarity.

[0108] Subsequently, since the model typically contains multiple attention heads and multiple attention modules, the differential losses of all heads and layers need to be aggregated:

[0109] For a single model with h attention heads, calculate the differential loss L for each head. head，k (k=1,...,h), taking the average yields the single-layer loss: ;

[0110] For the model's L attention layers, calculate the loss L for each layer. layer，l (l=1,...,L), taking the average yields the final Attention DML Loss: .

[0111] In this embodiment, Attention DML Loss is used as an auxiliary loss, weighted and combined with the main task loss (such as classification loss, CTC loss, etc.) to jointly optimize model parameters. Attention DML Loss achieves deep mutual learning between different model attention mechanisms by standardizing the attention matrix, quantizing differences, and aggregating multi-scale losses, ultimately improving the collaborative performance of the model group.

[0112] In this embodiment, the SVTR DML Loss (Deep Mutual Learning Loss) is calculated to achieve cross-branch knowledge transfer and collaborative optimization by constraining the output features or prediction distributions of the two MLPs to tend to be consistent. The specific steps are as follows:

[0113] MLP1 outputs feature vector F1∈R N×D MLP2 outputs feature vector F2∈R N×D (N is the number of samples, D is the feature dimension);

[0114] If the MLP is used as the prediction head, the probability distribution of the output character P1∈R of MLP1 N×T×V MLP2 outputs P2∈R N×T×V (T is the length of the text sequence, and V is the size of the character set).

[0115] For the feature outputs F1 and F2 of two MLPs, measure the difference in their feature distributions to ensure that their transformation patterns for input features are consistent. Frobenius distance and cosine similarity loss can be used.

[0116] If two MLPs are used as prediction heads (output character probability distributions), it is necessary to constrain the consistency of their prediction results to avoid excessive prediction bias between branches. KL divergence or cross-entropy loss can be used.

[0117] Then, based on the functions of the two MLPs (feature transformation or prediction), the above losses are selected and weighted to obtain the SVTR DML Loss.

[0118] Then, the SVTR DML Loss is used as an auxiliary loss and trained together with the main task loss of SVTR (such as CTC Loss, AttentionLoss).

[0119] By constraining the consistency between feature distribution and prediction results, the SVTR DML Loss of the two MLPs achieves knowledge sharing between branches, ultimately improving the accuracy and stability of SVTR text recognition.

[0120] In this embodiment, the Feature Distillation Loss between two NaViT models is calculated. Feature Distillation Loss is a core component of Knowledge Distillation (KD), aiming to improve the performance of the student model by having it learn the hidden features of the teacher model. For two NaViT models (assuming one is the teacher model T and the other is the student model S), the goal of Feature Distillation Loss is to align the distribution of hidden features between the teacher and student models and to leverage the robustness of the teacher model at variable resolution to guide the feature extraction capabilities of the student model.

[0121] Then, the text recognition model is used to perform text recognition on the content within the corrected text box to obtain structured text.

[0122] S103: Classify the structured text using a classification model to obtain the classification result;

[0123] Specifically, the classification model is used to classify structured text, and the classification results include:

[0124] Preliminary classification of structured text is performed using keyword matching;

[0125] If the initial classification is successful, the XLG-Net model is used to further classify the initial classification results.

[0126] If the XLG-Net model successfully classifies the data, the final classification result is obtained.

[0127] If the XLG-Net model fails to classify or fails to classify initially, the structured text is classified using a multimodal model to obtain the final classification result.

[0128] First, keyword matching is used to quickly complete the initial classification, reducing the frequency of calling complex models and improving overall classification efficiency. Then, the XLG-Net model performs in-depth analysis to complete the accurate classification, balancing speed and accuracy. When the initial classification or the XLG-Net model fails, a multimodal model is introduced as a supplement to cover the classification blind spots of the previous steps, ensuring that all structured text can be effectively classified and avoiding classification failure due to the limitations of a single model. The XLG-Net model combines XLNet and GCNII, excelling at handling semantic associations in text; the multimodal model can integrate text and potential visual information, and is more advantageous in classifying ambiguous and cross-domain financial documents (such as mixed-format expense reports), improving the system's adaptability to complex business scenarios.

[0129] In this embodiment, the process of classifying the structured text of financial audit documents using a classification model is as follows: First, the structured text is initially classified using keyword matching. A keyword library for the financial field is preset (e.g., "VAT special invoice" corresponds to the "input tax invoice" category, and "travel expense reimbursement" corresponds to the "expense reimbursement" category). The field content in the structured text is matched with the keyword library. If the match is successful (e.g., the keyword "VAT general invoice" is identified), the initial classification is completed.

[0130] If the initial classification is successful, the results are input into the XLG-Net model for further classification. This model combines XLNet's text semantic capture capabilities with GCNII's graph structure modeling advantages to analyze the deep semantics of structured text (such as the association between "taxable service name" and business type in an invoice). If the model's classification confidence score is higher than a preset threshold (such as 0.85), the classification is considered successful, and the final classification result (such as "office-related VAT general invoice") is obtained.

[0131] If the classification confidence of the XLG-Net model is below the threshold (classification failure), or if the initial classification fails to match any keywords (initial classification failure), the multimodal model is invoked for classification. The multimodal model simultaneously inputs structured text and the corresponding original document image features, combining text information with visual features in the image such as layout and seals (e.g., recognizing the header style of "travel expense reimbursement form" in the image), and outputs the final classification result after comprehensive judgment, ensuring accurate classification even when keywords are missing or the text semantics are ambiguous (e.g., keywords are not expressed in a standardized way in handwritten documents).

[0132] In this embodiment, based on the obtained structured text, the structured text is categorized according to the business needs of financial auditing (such as distinguishing expense types, invoice types, etc.). For example, structured text containing information such as "VAT special invoice" and "goods name: office computer" is categorized as "purchased input invoice"; structured text containing information such as "accommodation fee" and "travel expense reimbursement form" is categorized as "travel expense reimbursement form". After the categorization is completed, these categorized information are associated with the corresponding structured text data and stored in a unified JSON file. Each JSON file entry contains a unique identifier for the structured text, the specific text content, and the categorized information, providing a dataset with accurate categorized labels for subsequent training of the classification model.

[0133] In this embodiment, the XLG-Net model is a hybrid fusion model for text classification, combining XLNet (XLNet is a pre-trained model in the field of natural language processing, whose core goal is to integrate the advantages of autoregressive and autoencoder language models) and GCNII (graph neural network). It mainly includes four modules: (1) graph construction; (2) feature extraction based on XLNetMix; (3) feature extraction based on GCNII; and (4) feature aggregation.

[0134] (1) Graph construction:

[0135] Please see Figure 8 A heterogeneous graph is constructed for the dataset, denoted by G = (V, E), where V represents the set of all nodes in the graph, and E is the set of edges between nodes. Nodes are divided into document nodes and word nodes, and TF-IDF is used to measure the connections between document nodes and word nodes. PMI measures the connections between word nodes.

[0136] ;

[0137] Where #W(i,j) represents the number of sliding windows containing words i and j, #W(i) represents the number of sliding windows containing word i, #W represents the total number of sliding windows, and #W(j) represents the number of sliding windows containing word j.

[0138] The formula for calculating the weight of the edges between nodes is as follows:

[0139] ;

[0140] In this embodiment, the input is a document set D and a sliding window size w, and the goal is to output an adjacency matrix A and a feature matrix X. The specific steps are as follows:

[0141] First, determine the total number of documents |D| and the number of word nodes |word_set|. Then, create a zero matrix A of size (|D| + |word_set|) × (|D| + |word_set|) as the adjacency matrix, and a zero matrix X of the same size but with the third dimension being the embedding size as the feature matrix.

[0142] Then, for each document di in set D, the document vector of the document is first calculated, which is the average of all word vectors in the document. After obtaining the document vector, it is written into the corresponding position of the feature matrix X, specifically X[|word_set|+i].

[0143] Next, for each document di, we iterate through each word wj in the document. For each word, we calculate its TF-IDF value in the document, and then fill the value into the positions A[∣word_set∣+i][word_index(wj)] and A[word_index(wj)][∣word_set∣+i] of the adjacency matrix A to represent the connection strength between "document-word".

[0144] Finally, for each document di, iterate through each pair of words (wj, wk) obtained by the sliding window size w. For each pair of words, calculate their PMI (point mutual information) value, and then fill the A[word_index(wj)][word_index(wk)] and A[word_index(wk)][word_index(wj)] positions in the adjacency matrix A to represent the association strength between words.

[0145] Through the above four steps, we finally obtain the filled adjacency matrix A and feature matrix X, which respectively describe the relationships between "document-word" and "word-word" pairs in the document set, as well as the feature representations of each document and word. This completes the entire process from input document set to output adjacency matrix and feature matrix.

[0146] (2) Feature extraction based on XLNetMix:

[0147] Please see Figure 9 This application proposes a text processing method that combines reverse translation, XLNet contextualized embedding, and the DoubleMix strategy, aiming to improve the model's robustness and generalization ability to the original text and its perturbed samples. The specific steps are as follows:

[0148] For the original text sample x within the heterogeneous graph, two perturbation samples x1 and x2 are generated through two independent back-translation operations. For example, Chinese text is first translated into English and then back into Chinese. The language pairs or translation models used in the two translations are different (e.g., GoogleTranslate is used in the first stage and DeepL is used in the second stage) to introduce diverse semantic perturbations.

[0149] Next, the original sample x, perturbation samples x1 and x2 are input into the pre-trained XLNet model (a bidirectional autoregressive language model based on the Transformer architecture). The hidden layers are used to extract contextualized word embeddings to obtain document embedding vectors. XLNet dynamically adjusts the word order to capture bidirectional contextual information through the Permutation Language Modeling (PLM) mechanism, ensuring the context-awareness of the embedded features.

[0150] A DoubleMix strategy is used in the hidden layers of XLNet for two-step interpolation mixing:

[0151] The first step is to generate a mixture weight λ = (λ1, λ2) using a Dirichlet distribution α for perturbed samples x1 and x2, where λ1 + λ2 = 1. Based on the weight λ, calculate the mixture sample xmix = λ1x1 + λ2x2, and fuse the semantic features of the two perturbed samples through linear interpolation to enhance data diversity.

[0152] The parameter α of the Dirichlet distribution can adjust the concentration of the mixing ratio. For example, when α=1, the weights are evenly distributed, and when α<1, the weights tend to be extreme (such as one sample being dominant).

[0153] The second step is to generate a mixing coefficient θ for the mixed sample xmix and the original sample x using a Beta distribution β(γ, δ), where γ > δ to give the original sample a greater weight (e.g., γ = 2, δ = 1). Then, the final mixed sample xfinal = θx + (1 - θ)xmix is ​​calculated, and the influence of noise in the perturbed sample is suppressed by weighted averaging, while retaining the original semantic core information.

[0154] (3) Feature extraction based on GCNII:

[0155] Each layer of GCNII is calculated using the following formula:

[0156] ;

[0157] Where H(0) is the feature representation of the 0th layer, α n β n For hyperparameters, It is a graph convolution matrix; and This is the weight matrix.

[0158] GCNII outputs the updated feature representation Z GCNII It is used to capture document structure information, and the final prediction is obtained through the softmax layer:

[0159] ;

[0160] Where g is the GCNII model, X is the input feature matrix, and A is the normalized adjacency matrix;

[0161] (4) Feature aggregation:

[0162] After softmax transformation, the final result is a linear combination Z of XLNetMix and GCNII. The formula is as follows:

[0163] ;

[0164] X represents the document embedding vector, and W is the corresponding weight matrix;

[0165] ;

[0166] The training loss function is:

[0167] ;

[0168] Y i Let c∈{0,1} be the true label of sample i belonging to category c, and y^i,c be the predicted probability distribution.

[0169] In this embodiment, the multimodal model adopted is the qwen-vl-max model. This model integrates text semantics and image visual features to identify the text content and parsing the layout structure of documents, and classifies the text by combining multimodal understanding capabilities. For example, for a handwritten travel expense reimbursement form with an unusual layout, ineffective keyword matching, and no classification result from XLG-Net, qwen-vl-max can identify multimodal information such as handwritten font, the position of the amount column, and itinerary description text in the image, and output a classification result (such as "Travel Expense Reimbursement Form - Local Transportation") as the final classification conclusion, ensuring the accuracy and comprehensiveness of the classification of financial audit documents.

[0170] S104: Based on the classification results, extract key fields from the structured text using a large language model and a multimodal model;

[0171] Specifically, based on the classification results, key fields are extracted from structured text using large language models and multimodal models, including:

[0172] Structured text is parsed and fields are extracted using a large language model;

[0173] If the extraction is successful, the final key fields will be obtained;

[0174] If extraction fails, convert the financial audit document file to image format;

[0175] The financial audit documents converted to image format are processed using a multimodal model to extract the final key fields.

[0176] The system prioritizes parsing structured text using a large language model, enabling rapid extraction of key fields (such as invoice headers, amounts, and dates) from standardized text. This is applicable to most standard-format financial documents, improving overall processing efficiency. Simultaneously, the large language model possesses powerful semantic understanding capabilities, accurately identifying logical relationships between fields and ensuring extraction accuracy. When the large language model fails to extract data (e.g., encountering mixed handwriting or severely formatted structured text), the system switches to using a multimodal model to process image-formatted documents. Leveraging the model's comprehensive analysis capabilities of visual layout and text morphology, it solves the challenges of field extraction in complex scenarios (e.g., recognizing amounts next to blurred seals or project names within slanted text boxes), covering the extraction needs of various special documents.

[0177] In this embodiment, based on the classification results of financial audit documents (such as "Value-Added Tax Invoice" and "Travel Expense Reimbursement Form"), key fields are extracted using a large language model and a multimodal model. First, for structured text classified as "Value-Added Tax Invoice," a large language model (such as qwen-plus) is invoked for parsing. The model extracts corresponding information from the structured text based on preset invoice field templates (including invoice code, invoice date, buyer's name, amount, tax rate, etc.). If the large language model successfully extracts fields such as "Invoice Code: 110023456789" and "Amount: 1500.00 yuan," and the format of each field conforms to financial regulations, this information is used as the final key fields.

[0178] If the large language model fails to extract the data (for example, due to a large amount of handwritten garbled text in the structured text, causing the model to be unable to recognize the "total price and tax" field), the financial audit document file is converted into a JPG image. Subsequently, a multimodal model (such as qwen-vl-max) is called to process the image. The model combines visual information such as text layout (e.g., the position of the amount field) and font style (e.g., the shape of handwritten numbers) in the image to accurately locate and extract key fields. For example, for a handwritten travel expense reimbursement form image that fails to extract data due to garbled structured text, the multimodal model identifies the handwritten number "680" in the "accommodation fee" field and the printed "2025-08-01" in the "date" field, ultimately extracting key fields such as "accommodation fee: 680 yuan" and "date: 2025-08-01" as the final result.

[0179] Please see Figure 10 In this embodiment, to address the synonym problem encountered when extracting key fields from financial audit documents, an automatic field expansion mechanism is introduced. When extracting fields from structured text using a large language model, field descriptions are added to the extraction instructions. For example, for the "handler" field, the description is supplemented as "refers to the person in charge of handling this financial transaction; possible expressions include, but are not limited to, handler, traveler, name, etc." During the parsing process, the large language model identifies synonymous expressions in the text based on these descriptions. For instance, in the structured text of a travel expense reimbursement form, if "traveler: Zhang San" is identified as synonymous with "handler," the correspondence between "traveler → handler" is added to the field mapping table.

[0180] When the field table is loaded again, the system will automatically include new entries in the mapping table, making the field table contain "person in charge" and its synonyms "handler," "traveler," and "name." At this point, when processing structured text containing "handler: Li Si," the large language model can directly identify the corresponding "person in charge" through the expanded field table, efficiently completing the extraction and avoiding field omissions due to differences in synonym expressions, thus improving the comprehensiveness and accuracy of extracting key fields from financial documents.

[0181] S105: Obtain the audit rules and construct a knowledge graph system based on the audit rules;

[0182] Specifically, in this embodiment, the process of obtaining financial audit rules and building a knowledge graph system is as follows: First, collect the company's internal financial audit system, clauses related to document audit in national financial and tax regulations (such as "the invoice amount shall not exceed the reimbursement application amount" and "travel accommodation standards shall meet the upper limit corresponding to the job level"), and implicit rules formed in historical audit cases (such as "invoices spanning months must be accompanied by an explanation"), and organize these rules into structured audit items.

[0183] Next, a knowledge graph system is constructed based on these audit rules: the core entities of the knowledge graph are determined, including "audit rules", "document types", "key fields", and "constraints" (for example, the "audit rules" entity can be specifically "travel and accommodation reimbursement rules", and the "key fields" entity includes "accommodation amount" and "job level"); the relationships between entities are defined, such as "audit rules apply to document types", "audit rules contain constraints", and "constraints are associated with key fields" (for example, "travel and accommodation reimbursement rules apply to travel expense reimbursement forms" and "travel and accommodation reimbursement rules contain the constraint that accommodation amount ≤ accommodation standard corresponding to job level").

[0184] Finally, the compiled audit rules are entered into the knowledge graph according to the definitions of entities and relationships, forming a knowledge network with rules as the core and related document types and key fields. For example, when the document type "Value-Added Tax Special Invoice" is associated with "Input Tax Deduction Rules", the knowledge graph will clearly display the constraint condition "the invoice issuance date must be within the deduction period" in the rule, as well as the key field "Invoice Date" associated with this condition, providing a structured rule basis for subsequent intelligent auditing.

[0185] It should be noted that the knowledge graph system in this embodiment is constructed once before the keyword input, and there is no need to construct it every time a keyword is obtained thereafter.

[0186] S106: Input the key fields into the knowledge graph system for review and decision-making, and obtain the review result.

[0187] In this embodiment, the review and decision-making process is carried out based on the constructed business rule knowledge graph system. This knowledge graph models financial review rules in the form of ontologies and triples. Each rule is an independent rule node, associated with attributes such as inspection fields and judgment conditions. For example, the rule node "R001" corresponds to "travel accommodation expenses shall not exceed the standard", and its associated inspection fields are "accommodation amount" and "job level", and the judgment condition is "accommodation amount ≤ accommodation limit corresponding to job level".

[0188] The financial documents awaiting review are extracted into entity nodes. The key fields obtained in step 5.3 (such as "Accommodation Amount: 800 yuan" and "Job Level: Department Manager") are embedded as attributes into these entity nodes. The system uses a knowledge graph to match the rule nodes corresponding to the entity nodes and intelligently determines whether the content of each field is compliant: by comparing the value of the "Accommodation Amount" field with the accommodation limit corresponding to the "Department Manager" job level (assuming it is 700 yuan), it is found that 800 yuan > 700 yuan, and the field is determined to be non-compliant.

[0189] The final output shows non-compliant results, clearly marking the non-compliant field as "accommodation amount", the corresponding rule number "R001" and the condition "accommodation amount ≤ accommodation limit corresponding to job level", providing auditors with clear evidence of violations and improving the accuracy and efficiency of financial audits.

[0190] For example, in an invoice verification scenario,

[0191] (1) Construction of business rule knowledge graph:

[0192] {Rule R001:[Check fields: Invoice amount, tax rate; Decision condition: Invoice amount × tax rate = tax amount],

[0193] {Rule R002:[Check fields: Invoice issuance date, reimbursement application date; Decision condition: Invoice issuance date must not be later than reimbursement application date]};

[0194] (2) Embedding of entities and fields in audit documents:

[0195] {Entity node: Invoice F001; Field embedding: [Invoice amount: 1000 yuan, tax rate: 13%, tax amount: 130 yuan, issuance date: July 10, 2025, reimbursement application date: July 8, 2025]};

[0196] (3) Intelligent compliance audit:

[0197] Verification rule R001: The calculation of "invoice amount × tax rate" (1000 × 13% = 130 yuan) is consistent with the tax amount (130 yuan), which is compliant;

[0198] Verification rule R002: The invoice issuance date (July 10, 2025) is later than the reimbursement application date (July 8, 2025), which is a violation;

[0199] (4) Output the audit results:

[0200] Non-compliant fields: Invoice issuance date, reimbursement application date;

[0201] Association rule number: R002;

[0202] Violation condition: The invoice issuance date must not be later than the reimbursement application date.

[0203] In this embodiment, the entire document recognition and review process is encapsulated as an API interface, covering the complete logic from uploading financial documents, OCR recognition, structured processing, classification, key field extraction to rule review. After integrating this API into the online financial review platform, when a user adds a new document (such as uploading a scanned copy of a VAT invoice, filling out a travel expense reimbursement form and submitting it), the platform calls the API in real time to trigger an automatic review process: The API first performs text detection and recognition on the new document, converting it into structured text; then it completes the classification and extracts key fields such as "amount" and "invoice date"; subsequently, it verifies the compliance of the fields based on the review rules in the knowledge graph. If all fields comply with the rules (such as the invoice amount matching the reimbursement application amount, and the invoice date being within the validity period), the API returns a "review passed" decision; if there are any non-compliant fields (such as the accommodation amount exceeding the job level standard), it returns "review failed" and specific violation details (including the non-compliant field, the corresponding rule number, and the condition item). The online platform displays the decision opinions returned by the API to users and auditors in real time, enabling instant automatic review of new documents, significantly shortening the review cycle and improving the convenience and timeliness of financial review.

[0204] This embodiment accurately recognizes document text using an OCR model, intelligently categorizes documents using a classification model, deeply analyzes document content using a large language model and a multimodal model, and automatically reviews documents using a knowledge graph system. This achieves automated document review, significantly improving review efficiency and accuracy while reducing labor costs. It also ensures the objectivity and consistency of approval decisions.

[0205] Please see Figure 11 This embodiment provides an intelligent auditing device 200, including:

[0206] Data acquisition unit 201 is used to collect financial audit documents;

[0207] The conversion unit 202 is used to convert the financial audit document file into structured text using an OCR model;

[0208] The classification unit 203 is used to classify the structured text using a classification model to obtain a classification result;

[0209] Extraction unit 204 is used to extract key fields from the structured text based on the classification results using a large language model and a multimodal model;

[0210] Construction unit 205 is used to acquire review rules and construct a knowledge graph system based on the review rules;

[0211] The review unit 206 is used to input the key fields into the knowledge graph system for review decision-making and obtain the review result.

[0212] Furthermore, the conversion unit 202 includes:

[0213] The format determination subunit is used to determine the file format of the financial audit document file;

[0214] The reading subunit is used to directly read the file content and obtain structured text if the file format of the financial audit document is a standard format file;

[0215] The format conversion subunit is used to convert the financial audit document file into an image file if the file format of the financial audit document file is a non-standard format file, and to extract the text from the converted image file using the OCR model to obtain structured text;

[0216] The text extraction subunit is used to extract text from the financial audit document file using the OCR model if the file format of the financial audit document file is an image file, thereby obtaining structured text.

[0217] Furthermore, the conversion unit 202 also includes:

[0218] The text detection subunit is used to input the financial audit document into the text detection model for text detection and obtain a text box;

[0219] A correction subunit is used to correct the text box;

[0220] The text recognition subunit is used to perform text recognition on the content within the corrected text box using a text recognition model to obtain structured text.

[0221] Furthermore, the training method for the text detection model includes:

[0222] Obtain the image training set and ground truth labels;

[0223] The image training set is input into the teacher model and two student models to obtain the teacher response map and the two student response maps, respectively.

[0224] Calculate the KL divergence loss between two student response plots;

[0225] Calculate the true label loss between the student response map and the true label;

[0226] Calculate the distillation loss between the student response map and the teacher response map;

[0227] The parameters of the student model are updated based on the KL divergence loss, the true label loss, and the distillation loss to obtain the trained student model.

[0228] The trained student model is used as the text detection model.

[0229] Furthermore, the training method for the text recognition model includes:

[0230] Obtain the image training set and its corresponding labeled characters;

[0231] The image training set is subjected to multi-scale transformation and data augmentation processing to generate multi-scale input images;

[0232] The NaViT model is used to extract features from the input image and generate image feature representations.

[0233] The image feature representation is transformed using a multilayer perceptron to obtain the transformed feature representation;

[0234] The connectionist temporal classification loss is calculated based on the transformed feature representation and the structured text;

[0235] The image feature representation is identified using an acyclic neural network model to obtain predicted characters;

[0236] The cross-entropy loss is calculated based on the predicted character and the labeled character;

[0237] The parameters of the NaViT model are updated based on the connectionist temporal classification loss and the cross-entropy loss to obtain the trained NaViT model.

[0238] The trained NaViT model was used as the text recognition model.

[0239] Furthermore, the classification unit 203 includes:

[0240] The preliminary classification subunit is used to perform preliminary classification of the structured text through keyword matching;

[0241] Further classification sub-units are used to further classify the preliminary classification results using the XLG-Net model if the preliminary classification is successful;

[0242] The result acquisition sub-unit is used to obtain the final classification result if the XLG-Net model successfully classifies the data.

[0243] The text classification subunit is used to classify the structured text using a multimodal model if the XLG-Net model fails to classify it or if the initial classification fails, so as to obtain the final classification result.

[0244] Furthermore, the extraction unit 204 includes:

[0245] The parsing subunit is used to parse the structured text and extract fields using the large language model;

[0246] The field retrieval sub-unit is used to obtain the final key field if the extraction is successful;

[0247] The file format conversion subunit is used to convert the financial audit document file into an image format if the extraction fails.

[0248] The field extraction subunit is used to extract fields from the financial audit document file converted into image format using the multimodal model to obtain the final key fields.

[0249] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0250] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, can implement the methods provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0251] The present invention also provides a computer device, which may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the methods provided in the above embodiments. Of course, the computer device may also include various network interfaces, power supplies, and other components.

[0252] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

[0253] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusivity.

[0254] The term "comprises" implies that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. An intelligent auditing method, characterized in that, include: Collect financial audit documents; The financial audit document was converted into structured text using an OCR model. The structured text is classified using a classification model to obtain the classification results; Based on the classification results, key fields are extracted from the structured text using a large language model and a multimodal model; Obtain the review rules and construct a knowledge graph system based on the review rules; The key fields are input into the knowledge graph system for review and decision-making, and the review results are obtained. The process of converting the financial audit document into structured text using an OCR model includes: The financial audit document is input into a text detection model for text detection, resulting in a text box. The text box is corrected; The text recognition model is used to perform text recognition on the content within the corrected text box to obtain structured text; The training method for the text detection model includes: Obtain the image training set and ground truth labels; The image training set is input into the teacher model and two student models to obtain the teacher response map and the two student response maps, respectively. Calculate the KL divergence loss between two student response plots; Calculate the true label loss between the student response map and the true label; Calculate the distillation loss between the student response map and the teacher response map; The parameters of the student model are updated based on the KL divergence loss, the true label loss, and the distillation loss to obtain the trained student model. The trained student model is used as the text detection model. The training method for the text recognition model includes: Obtain the image training set and its corresponding labeled characters; The image training set is subjected to multi-scale transformation and data augmentation processing to generate multi-scale input images; The NaViT model is used to extract features from the input image and generate image feature representations. The image feature representation is transformed using a multilayer perceptron to obtain the transformed feature representation; The connectionist temporal classification loss is calculated based on the transformed feature representation and the structured text; The image feature representation is identified using an acyclic neural network model to obtain predicted characters; The cross-entropy loss is calculated based on the predicted character and the labeled character; The parameters of the NaViT model are updated based on the connectionist temporal classification loss and the cross-entropy loss to obtain the trained NaViT model. The trained NaViT model was used as a text recognition model. The classification of the structured text using a classification model to obtain the classification results includes: The structured text is initially classified using keyword matching; If the initial classification is successful, the XLG-Net model is used to further classify the initial classification results. If the XLG-Net model successfully classifies the data, the final classification result is obtained. If the XLG-Net model fails to classify or fails to classify initially, the structured text is classified using a multimodal model to obtain the final classification result.

2. The intelligent auditing method according to claim 1, characterized in that, The process of converting the financial audit document into structured text using an OCR model includes: Determine the file format of the financial audit document; If the financial audit document file is in a standard format, the file content is read directly to obtain structured text; If the financial audit document file is in a non-standard format, the financial audit document file will be converted into an image file, and the text will be extracted from the converted image file using the OCR model to obtain structured text. If the financial audit document is in image format, then the OCR model is used to extract the text from the financial audit document to obtain structured text.

3. The intelligent auditing method according to claim 1, characterized in that, The extraction of key fields from the structured text based on the classification results, using a large language model and a multimodal model, includes: The structured text is parsed and fields are extracted using the large language model. If the extraction is successful, the final key fields will be obtained; If extraction fails, the financial audit document file will be converted into an image format. The multimodal model is used to extract fields from the financial audit documents converted into image format to obtain the final key fields.

4. An intelligent auditing device, characterized in that, include: The data acquisition unit is used to collect financial audit documents. A conversion unit is used to convert the financial audit document file into structured text using an OCR model; A classification unit is used to classify the structured text using a classification model to obtain a classification result; An extraction unit is used to extract key fields from the structured text based on the classification results using a large language model and a multimodal model. The construction unit is used to acquire the review rules and construct a knowledge graph system based on the review rules; The review unit is used to input the key fields into the knowledge graph system for review decision-making and obtain the review result; The conversion unit further includes: The text detection subunit is used to input the financial audit document into the text detection model for text detection and obtain a text box; A correction subunit is used to correct the text box; The text recognition subunit is used to perform text recognition on the content within the corrected text box using a text recognition model to obtain structured text. The training method for the text detection model includes: Obtain the image training set and ground truth labels; The image training set is input into the teacher model and two student models to obtain the teacher response map and the two student response maps, respectively. Calculate the KL divergence loss between two student response plots; Calculate the true label loss between the student response map and the true label; Calculate the distillation loss between the student response map and the teacher response map; The parameters of the student model are updated based on the KL divergence loss, the true label loss, and the distillation loss to obtain the trained student model. The trained student model is used as the text detection model. The training method for the text recognition model includes: Obtain the image training set and its corresponding labeled characters; The image training set is subjected to multi-scale transformation and data augmentation processing to generate multi-scale input images; The NaViT model is used to extract features from the input image and generate image feature representations. The image feature representation is transformed using a multilayer perceptron to obtain the transformed feature representation; The connectionist temporal classification loss is calculated based on the transformed feature representation and the structured text; The image feature representation is identified using an acyclic neural network model to obtain predicted characters; The cross-entropy loss is calculated based on the predicted character and the labeled character; The parameters of the NaViT model are updated based on the connectionist temporal classification loss and the cross-entropy loss to obtain the trained NaViT model. The trained NaViT model was used as a text recognition model. The classification unit includes: The preliminary classification subunit is used to perform preliminary classification of the structured text through keyword matching; Further classification sub-units are used to further classify the preliminary classification results using the XLG-Net model if the preliminary classification is successful; The result acquisition sub-unit is used to obtain the final classification result if the XLG-Net model successfully classifies the data. The text classification subunit is used to classify the structured text using a multimodal model if the XLG-Net model fails to classify it or if the initial classification fails, so as to obtain the final classification result.

5. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the intelligent auditing method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the intelligent auditing method as described in any one of claims 1 to 3.

Images