Intelligent auditing method based on multi-modal large model

Abstract

The invention relates to the technical field of artificial intelligence and intelligent auditing, and discloses an intelligent auditing method based on a multi-modal large model, and the method comprises the following steps: S1, carrying out the image preprocessing; s2, performing depth feature extraction based on a multi-modal large model; s3, dynamically adjusting OCR identification parameters; s4, performing context analysis on an OCR recognition result through a semantic understanding module; s5, expanding the training data set by adopting a data enhancement technology; s6, parallel computing and hardware acceleration technologies are utilized; and S7, converting an OCR recognition result into a structured data format. According to the method and the device, the technical effects of unified recognition and structural understanding of multi-modal document information are achieved by adopting the technical scheme based on image-text fusion modeling and hierarchical semantic analysis, and compared with the technical scheme of semantic dislocation and context disjunction caused by image and text processing separation in the prior art, the technical scheme has the advantages that the method and the device are easy to implement; the problems of low heterogeneous modal information fusion degree and weak context reasoning capability are solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and intelligent auditing technology, and in particular to an intelligent auditing method based on a multimodal large model. Background Art

[0002] As the development of artificial intelligence and intelligent auditing continues to deepen, the automated processing of audit documents has become a crucial step in promoting financial compliance management and risk identification. Currently, in this technical field, methods based on OCR recognition and rule-driven processing are commonly used to extract text and identify fields in audit documents. Some systems have begun to introduce image recognition and semantic modeling to enhance document comprehension capabilities. However, in existing technical practices, image and text modalities are often processed separately, and there is a lack of effective fusion mechanisms between images and text. This makes it difficult for the model to accurately perceive the correlation structure between content. This is especially true in document scenarios where images and text are mixed and printed in a variety of formats. The integrity and semantic consistency of information extraction are difficult to guarantee.

[0003] Furthermore, traditional technologies often rely on fixed templates, predefined rules, or simple lexical matching for field labeling and data structured output. In complex audit scenarios with diverse field types and flexible text expression, these techniques are prone to problems such as inaccurate label attribution and non-standard field mapping, impacting the data quality of downstream financial analysis systems. Existing methods typically employ static output formats, making it difficult to support highly adaptable output requirements across systems and business scenarios, limiting the broad applicability of structured results.

[0004] At the same time, some technical solutions have problems such as low computing efficiency and system response delay when processing large quantities of documents, high-definition images or deep semantic models, and cannot meet the requirements of high-concurrency processing and real-time interaction in practical audit scenarios. Summary of the Invention

[0005] The purpose of the present invention is to provide an intelligent auditing method based on a multimodal large model, which solves the problem in existing technical practices that image and text modalities are often processed separately, and there is a lack of an effective fusion mechanism between images and texts, which makes it difficult for the model to accurately perceive the correlation structure between the content.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: Intelligent auditing methods based on multimodal large models include: S1. Image preprocessing, including grayscale conversion, binarization, contrast enhancement and denoising of the input audit working paper image; S2. Deep feature extraction based on a multimodal large model. In the image modality, a convolutional neural network (CNN) is used to extract the spatial features of the image, and in the text modality, a Transformer model is used to extract the linguistic features of the text. The image and text information are then fused through a self-attention mechanism. S3. Dynamically adjust OCR recognition parameters, including dynamically adjusting OCR recognition thresholds based on the complexity of the document area and automatically adapting to different text types; S4, using the semantic understanding module to perform contextual analysis on the OCR recognition results and correct misrecognition content based on domain knowledge; S5. Use data enhancement technology to expand the training data set; S6. Utilize parallel computing and hardware acceleration technology; S7. Convert the OCR recognition result into a structured data format.

[0007] Preferably, the image preprocessing step further comprises: Grayscale the input audit working paper image to reduce unnecessary color information and highlight text features; The Otsu method is used for automatic binarization, and the optimal threshold is determined by maximizing the inter-class variance. Enhance the contrast of the image by using histogram equalization to improve the contrast between text and background; Use median filtering method to denoise and reduce noise interference in the image; Otsu binarization formula: in, represents the inter-class variance, T is the optimal threshold for binarization, τ is the candidate threshold, The optimal threshold is determined by calculating the inter-class variance. It represents the value of τ that maximizes a certain objective function among all possible values ​​of τ.

[0008] Preferably, the deep feature extraction step is performed in the following manner: Use convolutional neural network (CNN) to extract the spatial features of the image modality and generate the feature vector of the image; Use the Transformer model to extract the language features of the text modality and generate the feature vector of the text; Fuse image and text features and jointly represent image and text information through the self-attention mechanism; Convolutional neural network operation formula: Among them, F conv(x, y) is the image feature obtained after convolution, W(i, j) is the weight of the convolution kernel, I(x+i, y+j) is the pixel value of the image at position (x+i, y+j), k is the radius of the convolution window, which indicates the size of the convolution window, (x, y) is the coordinate position in the output feature map of the convolution operation, (i, j) is the offset coordinate of the convolution kernel relative to the center point, and the value range is (-k, k), and k is the "radius" of the convolution kernel.

[0009] Preferably, the step of dynamically adjusting OCR recognition parameters includes: Dynamically calculate the complexity of the text region based on the complexity of the document region; Adjust the OCR recognition threshold through adaptive algorithms and automatically adjust parameters according to regional complexity to improve the recognition accuracy of different regions; Adaptive adjustment of OCR threshold formula: T adjusted =f(C complexity ); Among them, T adjusted is the adjusted OCR threshold, C complexity represents the complexity of the text area, f(·) is an adaptive algorithm, according to the complexity C complexity Automatically adjust the OCR recognition threshold.

[0010] Preferably, the semantic understanding module corrects the OCR recognition results in the following ways: Perform contextual analysis on OCR recognition results and use deep learning models to analyze the semantic relevance of text; Combined with expertise in the financial auditing field, knowledge graphs are used to correct common recognition errors, including misidentification of numbers or handwriting errors.

[0011] Preferably, the data enhancement technique comprises the following steps: Rotate, scale, and crop the training dataset to simulate different document formats and blurry conditions; Noise injection technology is used to improve the robustness of the model to noise interference and achieve diversity in training data.

[0012] Preferably, parallel computing and hardware acceleration technology are performed in the following ways: Parallel execution of OCR recognition tasks on multi-core processors; Use GPU or TPU to accelerate the OCR calculation process and shorten processing time.

[0013] Preferably, the step of converting structured data includes: Convert OCR-recognized text and table data into JSON or CSV format for subsequent analysis and processing; Transmit structured data to financial analysis software through standardized data output interface.

[0014] Preferably, the image preprocessing step includes: Automatically identify and enhance areas of audit working papers containing handwritten annotations and complex table structures through adaptive algorithms; Automatically adjust the OCR recognition threshold in handwritten areas to accommodate irregular fonts or blurred text.

[0015] Preferably, the model training process further includes the following methods: Adopting reinforcement learning strategy and adjusting model parameters through feedback mechanism to achieve recognition capability of complex documents; Use transfer learning methods to transfer the trained model to different types of audit working paper documents.

[0016] In summary, the present invention includes at least one of the following beneficial technical effects: 1. The present invention adopts a technical solution based on image-text fusion modeling and hierarchical semantic analysis, achieving the technical effect of unified recognition and structural understanding of multimodal document information. Compared with the technical solution in the prior art that separates image and text processing, resulting in semantic misalignment and disconnection from context, it solves the problems of low fusion of heterogeneous modal information and weak contextual reasoning ability.

[0017] 2. The present invention adopts a technical solution that combines a field classification network with a self-attention mechanism and context-guided attention weight allocation, achieving the technical effect of high-precision attribution of semantic elements and robust discrimination of field labels. Compared with the technical solution of using fixed templates or rule-based matching for field mapping in the existing technology, it solves the shortcomings of poor generalization ability and low adaptability in semantic change scenarios.

[0018] 3. The present invention adopts a technical solution of hierarchical mapping functions based on structural depth annotation and an extensible output template system, which achieves the technical effect of flexible structured information generation and compatibility with multiple business system interfaces. Compared with the technical solution of the existing technology with a single output form and fixed structure definition, it solves the shortcomings of poor cross-platform liquidity and weak structural reconstruction capability of audit data.

[0019] 4. The present invention adopts a technical solution of parallel computing scheduling strategy and hardware inference acceleration mechanism, which achieves the technical effect of improving system execution efficiency and meeting large-scale audit processing needs. Compared with the technical solutions of serial execution or single-threaded computing with low resource utilization in the existing technology, it solves the shortcomings of poor real-time performance and weak engineering availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0021] The following is combined with Figure 1 , the present invention is described in further detail.

[0022] The present invention provides an intelligent auditing method based on a multimodal large model, comprising: S1. Image preprocessing, including grayscale conversion, binarization, contrast enhancement and denoising of the input audit working paper image; S2. Deep feature extraction based on a multimodal large model. In the image modality, a convolutional neural network (CNN) is used to extract the spatial features of the image, and in the text modality, a Transformer model is used to extract the linguistic features of the text. The image and text information are then fused through a self-attention mechanism. S3. Dynamically adjust OCR recognition parameters, including dynamically adjusting OCR recognition thresholds based on the complexity of the document area and automatically adapting to different text types; S4, using the semantic understanding module to perform contextual analysis on the OCR recognition results and correct misrecognition content based on domain knowledge; S5. Use data augmentation technology to expand the training data set to enhance the model's generalization ability for complex format documents; S6. Use parallel computing and hardware acceleration technology to optimize the processing speed and efficiency of the OCR recognition process; S7. Convert the OCR recognition results into a structured data format to facilitate subsequent financial data analysis and processing.

[0023] For step S1: image preprocessing In this embodiment, for the intelligent audit method based on the multimodal large model provided by the present invention, the image preprocessing step serves as a key front-end module of the entire system. Its purpose is to standardize, clarify and structure the input audit working paper image, and provide a stable and clear input basis for subsequent feature extraction and OCR recognition.

[0024] In practical applications, audit working document images often come from diverse sources, have varying quality, and are complex in format. They also present issues such as color interference, noise pollution, low resolution, and mixed handwriting. Therefore, this step preprocesses the input image through a series of image enhancement and optimization processes to improve the feasibility and stability of the overall system processing.

[0025] In this example, the input image is first grayscaled, converting the original color image into a single-channel grayscale image. This process removes color redundancy by weighting the brightness information of the RGB channels, effectively highlighting the structural features of the text and its edge contours. This reduces the computational complexity of the subsequent image analysis model and preserves sufficient contrast information for subsequent operations.

[0026] Next, a binarization process is performed. In the embodiment of the present invention, the Otsu automatic threshold segmentation algorithm is preferably used to adaptively binarize the image. The algorithm is based on the maximization of the inter-class variance criterion and adaptively selects the optimal global threshold T to divide the image into foreground (usually text) and background areas. Its mathematical definition is as follows: in, represents the inter-class variance, T is the optimal threshold for binarization, τ is the candidate threshold, The optimal threshold is determined by calculating the inter-class variance. It represents the value of τ that maximizes a certain objective function among all possible values ​​of τ.

[0027] Subsequently, contrast enhancement processing is performed. Preferably, the image brightness is redistributed through histogram equalization technology, so that the pixel values ​​of the image are more evenly distributed throughout the grayscale range, thereby enhancing the contrast between the text and the background. This processing helps to improve text blurring caused by uneven lighting or printing quality, and improves the clarity of text edges and the prominence of local contrast features.

[0028] To further enhance the robustness of subsequent models, this embodiment also includes image denoising. Considering that audit working papers may contain interference factors such as scanning artifacts, paper stains, and image compression noise, a median filter algorithm is preferably used for image denoising. Median filtering effectively suppresses salt-and-pepper noise by sorting the pixel values ​​within each pixel's neighborhood in the image and selecting the median value as the new value for that pixel. This effectively reduces the impact of random noise while preserving image edges and contours.

[0029] This embodiment specifically considers the frequent presence of handwritten annotations or complex tables in audit images. These areas often have non-standard structures and significant style variability, making them difficult to directly identify using conventional OCR methods. To address this, this embodiment introduces a handwritten region detection mechanism based on region segmentation, automatically marking areas in the image that exhibit handwriting characteristics. This detection can be combined with features such as edge strength variations, stroke width differences, and text density for comprehensive judgment.

[0030] After identifying areas that may contain handwritten content, the system will automatically adjust the recognition strategy for these areas in the subsequent OCR stage, including using adaptive recognition models or modifying recognition parameters, so that the entire system still has high stability and parsability when facing manuscripts with complex structures or significant handwritten interference.

[0031] Furthermore, all processing modules in the image preprocessing phase can be automatically executed within a modular pipeline or combined with pre-set templates based on user needs, creating a flexible and scalable image standardization solution. This process can be implemented via scripts or by calling an image processing framework, adapting to audit images of varying resolutions, scanning angles, and paper materials.

[0032] In summary, this embodiment constructs a preprocessing module for multi-source heterogeneous audit images through multiple processing steps, including grayscale conversion, Otsu binarization, histogram equalization, median filtering denoising, and handwritten region detection and adaptation. This module exhibits strong adaptability and processing integrity. This image preprocessing solution provides a clear and standardized input foundation for subsequent modules such as deep feature extraction, multimodal modeling, and OCR recognition, and is a key prerequisite for implementing the method of the present invention.

[0033] Step S2: Deep feature extraction and multimodal fusion In this embodiment, after image preprocessing is completed, in order to achieve a deep understanding and semantic fusion of the graphic and text information in the audit working papers, it is necessary to extract features of the image modality and text modality separately, and to fuse them through a multimodal mechanism to support subsequent core processing steps such as OCR recognition correction, semantic analysis, and structured data output.

[0034] In this embodiment of the present invention, the image modality and text modality are modeled separately using specially designed deep neural network models. The former focuses on extracting the spatial structure and visual elements of the image region, while the latter is used to capture the contextual semantics and linguistic structure of the text content. Ultimately, an attention mechanism is used to semantically align and fuse the two modalities to generate a unified multimodal feature representation.

[0035] In terms of image modality processing, a convolutional neural network (CNN) structure is preferably used to extract spatial features from audit images. CNN has excellent edge detection and local structure capture capabilities in image processing tasks, and can effectively extract key image patterns such as table lines, handwriting, and seal outlines in audit working papers. The convolutional feature extraction process can be described as follows: Among them, F conv (x, y) is the image feature obtained after convolution, W(i, j) is the weight of the convolution kernel, I(x+i, y+j) is the pixel value of the image at position (x+i, y+j), k is the radius of the convolution window, which indicates the size of the convolution window, (x, y) is the coordinate position in the output feature map of the convolution operation, (i, j) is the offset coordinate of the convolution kernel relative to the center point, and the value range is (-k, k), and k is the "radius" of the convolution kernel.

[0036] To further enhance the expressive power of the image modality, multiple convolutional layers and downsampling modules can be superimposed in this embodiment, and combined with activation functions and normalization mechanisms to extract multi-level image features, thereby providing sufficient representation capabilities for areas such as text, markings, stamps, signatures, etc. in the image.

[0037] In terms of text modality processing, this embodiment preferably uses the Transformer architecture to perform contextual modeling on the initially recognized OCR text. Based on a self-attention mechanism, the Transformer can capture long-range dependencies between words, making it suitable for analyzing text content with dense semantic associations and strong contextual coupling in audit scenarios. This network can establish effective semantic connections between keywords such as time, amount, subject, and project name within the same text, thereby supporting subsequent structural recovery and semantic correction.

[0038] The input for text modalities is typically a sequence of text snippets output from the OCR stage. After tokenization, it is fed into the Transformer encoder structure to generate a corresponding embedding feature vector. This embedding vector expresses the semantic position of each character or word within the entire document, enabling recognition modeling of information such as syntactic structure, domain meaning, and entity relationships.

[0039] To achieve joint modeling between image and text modalities, this embodiment further introduces the self-attention mechanism as a fundamental means of multimodal fusion. This mechanism calculates the relative weights between the modalities to achieve semantic alignment between image regions and text content. The fusion process establishes a correspondence between image and text through joint training, thereby representing image and text information in the same feature space.

[0040] During implementation, a Cross-Attention module or a Shared Embedding Space can be used to integrate image and text features. This fusion mechanism achieves a weighted coupling between the pixel feature vector at a certain position in the image and the semantic representation of the text at that position, enabling the system to understand the correspondence between "position in the image" and "text semantics."

[0041] This multimodal representation is not only suitable for improving the understanding accuracy of text recognition results, but also can provide a unified data support interface for subsequent modules such as structured data reconstruction and domain term verification.

[0042] In this embodiment, the multimodal feature extraction and fusion process can be encapsulated as a unified neural network model, which can be optimized end-to-end using a multi-task objective function during the training phase. Feature extraction and semantic matching can also be performed using a pre-trained model during the inference phase. The output of this module is typically a multimodal representation in the form of a unified vector, which can be passed to subsequent semantic understanding modules or data structuring modules for further processing.

[0043] To sum up, this embodiment constructs a multimodal feature extraction and expression method for audit scenarios by modeling the image modality and text modality separately and using the attention mechanism to effectively integrate them. It provides a semantically consistent and information-complete input basis for subsequent OCR parameter adjustment, semantic correction and structure output tasks, and constitutes one of the key technical support modules in the method system of the present invention.

[0044] For step S3: Dynamically adjust OCR recognition parameters In this embodiment, after completing image preprocessing and multimodal feature extraction, in order to improve the system's adaptability when processing different document areas, a mechanism for dynamically adjusting OCR recognition parameters is further introduced, especially in areas with complex structures or uneven image quality, which can specifically improve recognition accuracy and robustness.

[0045] In traditional OCR systems, character recognition typically relies on fixed global parameter configurations, including threshold settings, font templates, and noise tolerance. However, audit working papers often exhibit regional heterogeneity, meaning the same image may contain a variety of information structures, including printed text, handwritten content, stamp overlays, and table line interference.

[0046] In this embodiment, the image content is first segmented into regions. This segmentation can be based on structural features extracted in the aforementioned image feature extraction phase, such as table cell boundaries, text density distribution, and outline texture intensity. After the segmentation is complete, the system performs a complexity assessment on each region to quantify the difficulty of processing the text content in that region.

[0047] Regional complexity (denoted as C complexity ) can be determined by a combination of multiple factors. Preferred factors include but are not limited to: pixel grayscale variance within the region, character connectivity, edge contour density, font change frequency, etc. This complexity measure provides a basis for OCR parameter adjustment.

[0048] The process can be formalized as follows: T adjusted =f(C complexity ); Among them, T adjusted is the adjusted OCR threshold, C complexityrepresents the complexity of the text area, f(·) is an adaptive algorithm, according to the complexity C complexity Automatically adjust the OCR recognition threshold.

[0049] In actual operation, when the system detects that the complexity index of a certain area exceeds a set threshold, it will trigger corresponding adjustment mechanisms. For example, it can appropriately increase the sensitivity of the OCR model to low-contrast characters, adjust the tolerance range of character segmentation, or switch to a recognition model that is more robust to blurred fonts in the area.

[0050] Preferably, this adjustment process can be embedded in the OCR inference pipeline, processing the image block by block in a sliding window or region-parallel manner, thereby dynamically optimizing recognition accuracy while ensuring overall processing efficiency. This mechanism is particularly critical for areas with dense handwriting content, edge occlusion, or mixed multilingual text.

[0051] Furthermore, to ensure the stability of recognition parameters across the entire image, embodiments of the present invention incorporate a coordinated strategy of local and global thresholds. This strategy involves adjusting the thresholds in local regions while also applying upper and lower bounds to the parameter adjustment range based on the average complexity of the entire image. This prevents the problem of global consistency degradation caused by over-adjustment in a single region.

[0052] The above-mentioned dynamic OCR recognition parameter adjustment method complements the aforementioned image preprocessing and deep feature extraction modules, which not only enhances the recognition adaptability of the system of the present invention in documents with different structures, but also establishes a higher-quality text recognition foundation for subsequent semantic correction and structured output.

[0053] The dynamic parameter adjustment mechanism described in this embodiment has strong scalability and platform independence, can adapt to a variety of OCR engines and hardware deployment environments, and can also continuously iteratively adjust the function model through online training to meet the actual needs in different application scenarios.

[0054] Step S4: Semantic understanding and knowledge correction of OCR results In this embodiment, after completing the OCR recognition and dynamic parameter adjustment of the image, the recognition result is usually output in the form of unstructured or weakly structured text. Due to image complexity, regional ambiguity, and model limitations, problems such as character errors, field misrecognition, and semantic fragmentation may occur in the OCR results. In order to further improve the usability and semantic accuracy of recognized text, the present invention provides a processing method that combines a semantic understanding mechanism with an expertise correction module to perform context-based understanding and knowledge-based correction on the preliminary recognition results.

[0055] In this embodiment, a contextual semantic representation is first constructed based on the OCR recognition results. Specifically, the OCR results are divided into continuous text sequence segments and input into the language modeling module. This module preferably uses a pre-trained Transformer language model to model the semantic relationships between texts through a self-attention mechanism and learn the contextual dependency structure between text items. For example, for the "subject-amount-explanation" structure commonly found in audit reports, contextual analysis can identify subordinate or parallel semantic relationships between fields.

[0056] During this semantic modeling process, the model not only considers the sequential dependencies between terms but also incorporates auxiliary information such as recognition confidence and text layout (e.g., paragraph and line number) to improve understanding of non-standard expressions. Furthermore, text embedding vectors can be used to calculate semantic similarity between words, identifying cases where structurally similar words may be misidentified in terms of character form, such as "1000" being misidentified as "100D" or "1OOO."

[0057] To further enhance the specialized targeting of semantic recognition, this embodiment introduces a domain knowledge correction mechanism. This mechanism relies on a predefined financial audit domain knowledge base or knowledge graph to verify and correct key terms, structural fields, and standard expressions in the OCR results. This knowledge base may include commonly used charts of accounts, common financial fields (such as "accounts receivable" and "total assets"), standard unit symbols (such as "yuan" and "10,000 yuan"), and their semantic attributes.

[0058] During operation, the system first performs entity recognition and candidate matching on key fields in the OCR results. For example, the named entity recognition (NER) model extracts entity tags such as amounts, dates, project numbers, and organization names from the text, and then matches them with standard entities in the knowledge base to correct recognition errors or standardize expressions.

[0059] Furthermore, this embodiment uses a semantic similarity scoring mechanism to prioritize candidate corrections, ensuring contextual rationality and semantic consistency. For example, if the recognition result shows "Accounts Receivable 1", the system will use a similarity function to merge it into "Accounts Receivable", removing redundant punctuation or interfering characters.

[0060] Integrating the above mechanisms, this embodiment preferably implements the following processing flow: First, a semantic context tensor is constructed based on the OCR text, and a contextual representation of each token is obtained through a Transformer architecture. Second, semantic entities are verified using an expertise matching module, and incorrect characters or fields are replaced or corrected using contextual attention weights. The resulting text output has greater semantic integrity and expression standardization, facilitating subsequent structured processing.

[0061] Furthermore, the semantic correction mechanism described in this embodiment also supports field structure recovery for table areas. For table cells that have been mis-segmented or misidentified, the system can restore their logical arrangement and reconstruct the header-body structure by jointly modeling the spatial position information and context sequence information of the text blocks. This process relies on the joint encoding results of the aforementioned image and text modalities, especially the spatial semantic joint features extracted by the image-text fusion model, which serve as an important input for table structure reasoning.

[0062] For step S5: data enhancement and training set expansion In this embodiment, to improve the generalization and robustness of the multimodal large model in the audit document processing process, the system introduces a data augmentation and training set expansion mechanism. This mechanism not only expands the scale and distribution of the original training data, but also simulates the complex interference factors of images and text commonly found in audit scenarios, thereby enhancing the model's adaptability and fault tolerance to images of different formats and qualities.

[0063] In actual deployments, audit working document image samples come from a wide range of sources, including scanned documents, digitally captured images, and PDF screenshots. These samples often exhibit interfering features such as angle offset, local blur, occlusion, and uneven lighting. Furthermore, document content structures are highly diverse, including mixed handwriting and printed content, nested tables, and floating stamps. To address these challenges, a systematic data augmentation strategy is required to construct a training dataset with extensive feature coverage.

[0064] In this embodiment, a variety of geometric enhancement operations are first performed on the original image samples, including rotation, affine transformation, random scaling, horizontal / vertical mirroring, etc., to simulate scenarios such as shooting angle error and scanning offset. Geometric enhancement improves the stability of the model to changes in the image coordinate system by perturbing the image spatial arrangement.

[0065] Secondly, image blur and occlusion simulation technology is introduced. By adding Gaussian blur, motion blur, and compression artifacts, the degradation effects commonly seen in scanned or compressed images can be recreated. To simulate stamp overlays and text erasures, artificial occlusion regions can also be introduced, with interference blocks or pattern layers superimposed around key document fields. This allows the model to be robust against partially incomplete inputs.

[0066] Furthermore, to simulate the noise issues encountered in actual image acquisition, this embodiment preferably introduces a noise injection strategy, including but not limited to salt and pepper noise, Gaussian white noise, and Poisson noise. This noise perturbation improves the model's anti-interference capabilities and is particularly suitable for input scenarios involving old manuscript paper, severely compressed images, and pixel-level degradation.

[0067] To further expand the structural distribution of training data, this example also introduces a typesetting deformation and layout reconstruction strategy. This strategy generates simulated audit documents with non-standard formats by recombining text blocks, simulating irregular table boundaries, inserting virtual stamps, and changing text fonts, thereby enhancing the model's ability to recognize complex structures.

[0068] In addition to image modality enhancements, the text modality training data has also been expanded. Specifically, the system introduces character-level replacement, word order perturbation, and field truncation on the original OCR output text to construct typical misrecognition samples for training the semantic understanding module's error-tolerant recognition capabilities in noisy OCR environments.

[0069] In order to ensure that the training process remains convergent and effective on the basis of enhanced samples, in this embodiment, it is preferred to use a soft label mechanism to label the synthetic data, or to introduce pre-trained model weights in combination with transfer learning, so as to avoid overfitting of the model on the enhanced samples.

[0070] All of the above enhancement strategies can be automatically executed through the data pipeline during the data preprocessing stage, and can also be used as offline simulation tools during the data collection stage to facilitate real-time enhancement in scenarios where multi-source data streams are input.

[0071] By incorporating data augmentation and training set expansion mechanisms, the multimodal model in this invention not only performs well on standard samples but also adapts to the input requirements of extreme or unconventional documents in audit scenarios. This module, serving as the training support unit in the method, provides a foundation for building an intelligent audit system with semantic robustness and structural generalization capabilities.

[0072] Step S6: Parallel computing and hardware acceleration technology In this embodiment, to further optimize the computational efficiency of the intelligent auditing method based on a multimodal large model when processing high-resolution images, large document batches, and multi-task semantic analysis, the system incorporates parallel computing mechanisms and hardware acceleration solutions. This step is designed to improve the response speed and resource utilization of the overall processing flow, ensuring the system's engineering feasibility and scalability in actual deployment.

[0073] Specifically, in processes such as OCR recognition, image analysis, and deep model inference, there are a large number of subtask units that can be parallelized, including image block preprocessing, independent feature extraction of different image regions by the model, and sequence label prediction. The present invention fully utilizes the independence of these subtasks, preferably adopting a task partitioning strategy in a multi-core processor environment to process the input image or image group in slices, thereby achieving parallel execution of the processing flow.

[0074] In its implementation, the system first logically partitions the input image based on its layout. This partitioning can be based on features such as table boundaries, text density, and image content areas. Subsequently, each image subregion is assigned to a different processing thread, utilizing a multi-core scheduling mechanism to perform OCR recognition and local feature extraction in parallel.

[0075] Based on image partitioning, the scheduling of OCR recognition subtasks can be prioritized based on regional complexity estimation, rationally allocating computing resources and improving processing response in key areas. This mechanism also supports dynamic thread pool adjustment to adapt to changes in input size and system resources.

[0076] To further reduce the time required for model inference, this embodiment preferably introduces a hardware acceleration module, specifically a graphics processing unit (GPU), a tensor processing unit (TPU), or a dedicated neural network inference chip. These acceleration modules are primarily used to execute computationally intensive tasks such as convolutional neural networks (CNNs), Transformer language models, and self-attention mechanism calculations.

[0077] Taking the convolution operation as an example, when processing image modality feature extraction, the system uses GPU acceleration to perform the following feature convolution operations: Among them, F conv (x, y) is the image feature obtained after convolution, W(i, j) is the weight of the convolution kernel, I(x+i, y+j) is the pixel value of the image at position (x+i, y+j), k is the radius of the convolution window, which indicates the size of the convolution window, (x, y) is the coordinate position in the output feature map of the convolution operation, (i, j) is the offset coordinate of the convolution kernel relative to the center point, and the value range is (-k, k), and k is the "radius" of the convolution kernel.

[0078] This formula requires a large number of pixel multiplication and addition operations under high-resolution images, and the execution efficiency is limited when using general-purpose processors. However, with the help of the parallel matrix operation capabilities of the GPU, the execution time can be significantly reduced under the tensor calculation framework.

[0079] Similarly, the self-attention module in the Transformer architecture involves multiple vector inner products, softmax normalization, and linear mapping operations. The system can use GPUs to parallelize and accelerate vector calculations and weight matrix multiplications in this process, reducing inference latency during the semantic modeling phase.

[0080] In this embodiment, the hardware acceleration module's scheduling mechanism is implemented using a computation graph. This divides the model's forward computation process into several dependent nodes, and the system schedules them based on their dependencies and execution priorities. Computational bottleneck modules, such as fully connected layers and high-dimensional matrix multiplication layers, are prioritized for GPU resources. Lightweight tasks, such as character post-processing and rule correction, remain in parallel execution within CPU threads.

[0081] To further optimize overall execution efficiency, the system also deploys a unified resource scheduling and control module. This module monitors the current CPU and GPU resource utilization and adjusts the scheduling strategies for the OCR, feature extraction, and language understanding modules based on the input task load, achieving a dynamic balance between performance and resources.

[0082] It's important to note that the parallelization and acceleration strategies in this embodiment are not limited to specific hardware platforms and possess strong cross-platform adaptability. During system design, the appropriate parallel granularity and hardware call strategy can be dynamically selected based on the computing power of the actual deployment environment to support a variety of deployment scenarios, from lightweight terminal devices to large server clusters.

[0083] For step S7: structured output and financial analysis interface In this embodiment, after completing image processing, feature extraction, OCR recognition, semantic understanding and model reasoning, the system will unify the recognition results output by each processing module and further perform structured conversion to achieve automatic archiving, field extraction and interface docking of key content in the audit documents, ensuring that the final output form meets the application requirements of subsequent financial analysis, intelligent retrieval and business integration.

[0084] To achieve structured conversion of audit working paper information, this embodiment uses a combination of hierarchical structure extraction and standard data format mapping. Specifically, the system abstractly models the semantic units identified by the previous module (such as "date," "amount," "account," "project name," etc.) and generates a logically hierarchical structured data representation based on field attributes and their contextual positional relationships.

[0085] In this conversion process, the recognition results are first classified and labeled at the field level. Preferably, the system uses a field classification function Φ based on the contextual self-attention mechanism. f By aggregating semantic context weight information, the attribution of each text segment is judged. The core formula is as follows: Among them, T i represents the text segment to be classified, α ijRepresents the attention between the i-th fragment and the j-th context Token, weight represents the semantic contribution vector of the jth Token to the field type c, Φ f is a field classification function that outputs the category label to which the field belongs. c is the candidate field category label (such as "amount", "invoice number", "date", etc.). The one with the highest score is selected from all predefined field categories. n is the number of tokens involved in the classification calculation in the context, that is, the total number of tokens contained in the current context window. The total score of the aggregated context semantic contribution of field type c is taken as the field type c that maximizes the aggregate score, argm c ax is used to determine the field category to which the text fragment is most likely to belong.

[0086] Through this mechanism, the system can accurately classify text recognition results into predefined structural fields, such as "invoicing unit", "invoice amount", "approval number", etc., laying a semantic foundation for subsequent structural reconstruction.

[0087] In order to further realize the sequence output of structured data, this embodiment designs a hierarchical structure representation generation function Γ s , used to map annotation fields to nested JSON, CSV, or XML structures. This function considers the logical hierarchy and order dependencies between fields, and is expressed as follows: Among them, F k Represents the set of structural fields in the k-th document sample, K m Indicates the field key name, V m Indicates the value corresponding to the field, L m Denotes the depth label of the field in the structure hierarchy, Γ s Represents the final data structured mapping function, and M represents the total number of structure fields.

[0088] Based on the above mechanism, the system can automatically map the recognition results into nested structure objects, further supporting downstream business system calls, data interface output or file storage.

[0089] To ensure the semantic consistency and compliance of structured output, this embodiment also introduces a standardized mapping rule library and field matching templates. In this rule system, each structure field corresponds to a set of standard names, unit forms, and numerical formats. Before output, the system will perform rule verification and format unification operations on each field value. For example, "One thousand yuan" will be standardized as "1000.00", or "January 5, 2023" will be uniformly formatted as "2023-01-05".

[0090] In addition, to enhance the adaptability and scalability of the system, the structured data in this embodiment also supports multi-format export, including standard JSON format, CSV format for spreadsheet applications, and data XML structure compatible with reporting systems. Each format is parameterized and generated through a template engine to ensure that the data can be directly transferred to the financial management platform, audit analysis tools or database interface.

[0091] During the output phase, the system also features a universal field mapping middleware, responsible for mapping neutral field names generated internally to the field definitions required by the target system. For example, "invoice_amount" can be mapped to "invoice amount," "kpje," or other business system field names to achieve semantic compatibility across systems.

[0092] In summary, this embodiment implements a complete process from OCR recognition results to structured semantic expression through field classification modeling, structural hierarchical mapping, semantic verification, and multi-format export mechanisms. This step not only ensures data formatting and structural integrity but also provides interface-level support for the automated audit data analysis and business integration described in this invention.

[0093] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent audit method based on a multimodal large model, characterized by: include: S1. Image preprocessing, including grayscale conversion, binarization, contrast enhancement and denoising of the input audit working paper image; S2. Deep feature extraction based on a multimodal large model. In the image modality, a convolutional neural network (CNN) is used to extract the spatial features of the image, and in the text modality, a Transformer model is used to extract the linguistic features of the text. The image and text information are then fused through a self-attention mechanism. S3. Dynamically adjust OCR recognition parameters, including dynamically adjusting OCR recognition thresholds based on the complexity of the document area and automatically adapting to different text types; S4, using the semantic understanding module to perform contextual analysis on the OCR recognition results and correct misrecognition content based on domain knowledge; S5. Use data enhancement technology to expand the training data set; S6. Utilize parallel computing and hardware acceleration technology; S7. Convert the OCR recognition result into a structured data format.

2. The intelligent audit method based on a multimodal large model according to claim 1 is characterized in that: The image preprocessing step further comprises: Grayscale the input audit working paper image to reduce unnecessary color information and highlight text features; The Otsu method is used for automatic binarization, and the optimal threshold is determined by maximizing the inter-class variance. Enhance the contrast of the image by using histogram equalization to improve the contrast between text and background; Use median filtering method to denoise and reduce noise interference in the image; Otsu binarization formula: in, represents the inter-class variance, T is the optimal threshold for binarization, τ is the candidate threshold, The optimal threshold is determined by calculating the inter-class variance, argm τ ax represents the value of τ that maximizes a certain objective function among all possible values ​​of τ.

3. The intelligent audit method based on a multimodal large model according to claim 1 is characterized in that: The deep feature extraction step is performed in the following way: Use convolutional neural network (CNN) to extract the spatial features of the image modality and generate the feature vector of the image; Use the Transformer model to extract the language features of the text modality and generate the feature vector of the text; Fuse image and text features and jointly represent image and text information through the self-attention mechanism; Convolutional neural network operation formula: Among them, F conv (x, y) is the image feature obtained after convolution, W(i, j) is the weight of the convolution kernel, I(x+i, y+j) is the pixel value of the image at position (x+i, y+j), k is the radius of the convolution window, which indicates the size of the convolution window, (x, y) is the coordinate position in the output feature map of the convolution operation, (i, j) is the offset coordinate of the convolution kernel relative to the center point, and the value range is (-k, k), and k is the "radius" of the convolution kernel.

4. The intelligent audit method based on a multimodal large model according to claim 1 is characterized in that: The steps for dynamically adjusting OCR recognition parameters include: Dynamically calculate the complexity of the text region based on the complexity of the document region; Adjust the OCR recognition threshold through adaptive algorithms and automatically adjust parameters according to regional complexity to improve the recognition accuracy of different regions; Adaptive adjustment of OCR threshold formula: T adjusted =f(C complexity ); Among them, T adjusted is the adjusted OCR threshold, C complexity represents the complexity of the text area, f(·) is an adaptive algorithm, according to the complexity C complexity Automatically adjust the OCR recognition threshold.

5. The intelligent audit method based on multimodal large model according to claim 1 is characterized in that: The semantic understanding module corrects the OCR recognition results in the following ways: Perform contextual analysis on OCR recognition results and use deep learning models to analyze the semantic relevance of text; Combined with expertise in the financial auditing field, knowledge graphs are used to correct common recognition errors, including misidentification of numbers or handwriting errors.

6. The intelligent audit method based on a multimodal large model according to claim 1 is characterized in that: Data augmentation techniques include the following steps: Rotate, scale, and crop the training dataset to simulate different document formats and blurry conditions; Noise injection technology is used to improve the robustness of the model to noise interference and achieve diversity in training data.

7. The intelligent audit method based on a multimodal large model according to claim 1 is characterized in that: Parallel computing and hardware acceleration technologies are implemented in the following ways: Parallel execution of OCR recognition tasks on multi-core processors; Use GPU or TPU to accelerate the OCR calculation process and shorten processing time.

8. The intelligent audit method based on multimodal large model according to claim 1 is characterized in that: The steps for converting structured data include: Convert OCR-recognized text and table data into JSON or CSV format for subsequent analysis and processing; Transmit structured data to financial analysis software through standardized data output interface.

9. The intelligent audit method based on a multimodal large model according to claim 1 is characterized in that: Image preprocessing steps include: Automatically identify and enhance areas of audit working papers containing handwritten annotations and complex table structures through adaptive algorithms; Automatically adjust the OCR recognition threshold in handwritten areas to accommodate irregular fonts or blurred text.

10. The intelligent audit method based on multimodal large model according to claim 1 is characterized in that: The model training process further includes the following methods: Adopting reinforcement learning strategy and adjusting model parameters through feedback mechanism to achieve recognition capability of complex documents; Use transfer learning methods to transfer trained models to different types of audit working paper documents.

Images