Multi-modal auditing method based on large model

Abstract

The invention relates to the technical field of auditing, and provides a multi-modal auditing method based on a large model, which ensures the quality and consistency of data through comprehensive data acquisition and preprocessing, performs feature extraction on image, text and numerical data by using CNN, NLP and other technologies, realizes deep fusion and analysis of multi-modal data, and improves the auditing efficiency. Through doubtful point mining, knowledge graph correlation analysis and sentiment analysis, abnormal modes and potential relationships in data and application of a causal traceability model are deeply revealed, causal relationships and influence factors in a specific scene are further clarified, and finally, the generated audit report is detailed in content, clear in structure and high in practicability. According to the technical scheme, the persuasion of the report is enhanced through data visualization, the auditing accuracy and efficiency are improved through the method, powerful support is provided for auditing decision making, and the problems of insufficient accuracy and low efficiency in the prior art are solved through the technical scheme.

Description

Technical Field

[0001] The present invention relates to the field of audit technology, and in particular to a multimodal audit method based on a large model. Background Art

[0002] At present, the multimodal auditing method based on large models refers to a method that uses large language models and multimodal data models to conduct comprehensive analysis and auditing of data in various forms including text, images, voice, etc.

[0003] In traditional auditing methods, manual review and analysis of a single type of data (such as financial statements, transaction records, etc.) is often relied on, which is not only time-consuming and labor-intensive, but also difficult to fully capture the complex relationships and abnormal patterns in the data. With the rapid development of big data and artificial intelligence technologies, multimodal auditing methods based on big models have gradually emerged, but existing methods are still limited in some aspects. Although the existing multimodal auditing methods based on big models can handle various types of data, they may still be insufficient in data collection, preprocessing, feature extraction, and deep fusion and analysis. For example, data collection may not be comprehensive enough, resulting in missing or biased data; the preprocessing process may not be sophisticated enough, affecting the quality and consistency of the data; the feature extraction method may not be efficient or accurate enough, making it difficult to fully mine the valuable information in the data; the fusion and analysis process may lack depth and fail to reveal the complex relationships and abnormal patterns in the data. Summary of the invention

[0004] The present invention proposes a multimodal auditing method based on a large model, which solves the problems of insufficient accuracy and low efficiency in related technologies.

[0005] The technical solution of the present invention is as follows: A multimodal audit method based on a large model, comprising:

[0006] S1: Data collection;

[0007] S2: data preprocessing;

[0008] S3: data is decomposed into images, texts and numerical values;

[0009] S4: Perform doubt mining, knowledge graph and sentiment analysis on images, texts and values ​​respectively;

[0010] S5: The causal tracing model analyzes the causes of specific scenarios;

[0011] S6: Generate audit report.

[0012] As a preferred solution of the present invention, the specific steps of step S1: data collection are:

[0013] S11: Data collection channels and methods;

[0014] S111: A program that automatically obtains web page content using crawler technology, grabs required information from the Internet and stores it locally or in a database;

[0015] The collection steps are as follows: 1. Send a request: send a request to the target website to obtain the web page content;

[0016] 2. Parse HTML: parse the obtained HTML format code to extract the required information;

[0017] 3. Data extraction: Use regular expressions or other methods to extract required information from HTML code;

[0018] 4. Store data: store the extracted data locally or in a database;

[0019] S112: The database log records the history of database operations, including user access and data change information;

[0020] Its collection method:

[0021] 1. Log file collection: By monitoring the log files on the server, they are collected in real time into a centralized storage system for analysis and processing. This is achieved using file monitoring tools such as logstash and fluentd.

[0022] 2. Database and API collection: Some systems and applications store log data in the database, and can directly collect data through database connection, API call, etc.;

[0023] S12: Establish a data security management system

[0024] S121: Organizational structure: Establish a dedicated data security management organization to be responsible for the overall planning, strategy formulation and execution supervision of data security;

[0025] S121: System and process: Develop a comprehensive data security management system and process to clarify the security requirements for each link of data collection, storage, processing, sharing and destruction;

[0026] S122: Technical tools: Use data encryption, access control, and security auditing techniques to ensure data security during collection, transmission, and storage;

[0027] S13. Data classification and grading;

[0028] S131, Classification criteria: Classify data according to its sensitivity and importance;

[0029] S132, hierarchical protection: take corresponding protection measures for data of different levels;

[0030] S14. Data anonymization and de-identification;

[0031] S141, anonymization processing: anonymize the collected data so that it is impossible to restore the original data through reverse operations, thereby effectively protecting user privacy;

[0032] S142. De-identification: Remove personal identification information from the data as much as possible without affecting the effectiveness of data analysis;

[0033] S15. Strengthen access control;

[0034] S151. Permission management: Implement role-based access control to ensure that only authorized personnel can access sensitive data; this includes data access account and user permission management, access permission management during data use, and data sharing permission management;

[0035] S152. Audit tracking: Audit and track data access and usage to promptly detect and handle unauthorized access;

[0036] S16. Data encryption and desensitization: Use encryption technology during data transmission and storage to ensure that data will not be abused in the event of unauthorized access or leakage, and desensitize sensitive data to reduce the risk of data leakage.

[0037] As a preferred solution of the present invention, the specific steps of step S2: data preprocessing are:

[0038] S21, image data preprocessing;

[0039] S211, Denoising: Use filter techniques such as Gaussian filter and median filter to smooth the image and remove high-frequency noise while retaining the edge and detail information of the image;

[0040] S212, scaling: using image scaling algorithms, bilinear interpolation, and bicubic interpolation, to scale the image, maintain image clarity and detail information, and avoid image distortion;

[0041] S213, normalization: performing linear transformation on the image pixel values ​​so that they fall within a specified range. For an image with pixel values ​​in the range of [0, 255], it can be normalized to the range of [0, 1] by dividing by 255;

[0042] S22, text data preprocessing;

[0043] S221. Word segmentation: Split the text into words or tokens to facilitate subsequent processing and analysis. Use word segmentation tools or algorithms to segment the text. For Chinese text, commonly used word segmentation tools include Jieba word segmentation and StanfordNLP word segmentation.

[0044] S222, stop word removal: remove words that appear frequently in the text but do not contribute much to the meaning of the text, reduce data dimensions, improve processing efficiency, use predefined stop word lists, customize stop word lists according to specific tasks, and remove stop word processing from the text;

[0045] S223, stem extraction: restore words to their basic form (root), reduce vocabulary diversity, and improve the generalization ability of the model. Use the stem extraction algorithm to perform stem extraction on the text. The stem extraction algorithm restores the word to its basic form by removing the suffix or prefix of the word;

[0046] S23, numerical data preprocessing;

[0047] S231, Missing value processing: Process missing values ​​in numerical data to improve data integrity and accuracy;

[0048] method:

[0049] Deletion method: When the proportion of missing values ​​is small, you can choose to delete samples or features containing missing values;

[0050] Filling method: Use mean, median, mode and other methods to fill missing values. For normally distributed data, the mean can be used for filling; for skewed distributed data, the median can be used for filling. In addition, interpolation and regression model methods can be used for prediction and filling;

[0051] Model method: Use machine learning models to predict missing data points. This method requires building a suitable prediction model and training the model to predict missing values.

[0052] S232, Normalization: Map data of different scales to the same range (usually [0,1] or [-1,1]), eliminate the dimensional differences between features, improve the convergence speed and accuracy of the model, and use linear transformation to scale the data to the specified range. For data with minimum and maximum values ​​of xmin and xmax respectively, the formula x′= xmax -xmin x-xmin Normalize it to the range [0,1];

[0053] S233, Standardization: Make all features in the data set have zero mean and unit variance to facilitate subsequent processing and analysis. Use the standardized formula to process the data. For feature x, its mean and standard deviation are μ and σ respectively, then the standardized data is x′=σx-μ. The standardized data has the same scale, reducing the impact of inconsistent dimensions between features.

[0054] As a preferred solution of the present invention, the specific steps of step S3: decomposing data into images, texts and numerical values ​​are as follows:

[0055] S31, Image feature extraction: using convolutional neural network (CNN);

[0056] S311, Overview of Convolutional Neural Network (CNN);

[0057] By constructing a multi-level neural network structure, the human brain's processing of image and voice information is simulated to achieve feature extraction and classification of input data. CNN is mainly composed of convolutional layers, pooling layers, and fully connected layers.

[0058] S312, image feature extraction process:

[0059] 1. Convolution layer: The input image is convolved by sliding the convolution kernel to extract the local features of the image. The size and number of convolution kernels determine the type and number of extracted features.

[0060] 2. Pooling layer: Downsample the feature map output by the convolutional layer to reduce the dimension of the data, avoid overfitting, and retain important feature information. Commonly used pooling operations include Max Pooling and Average Pooling.

[0061] 3. Fully connected layer: Flatten the feature map output by the pooling layer into a one-dimensional vector, and perform further feature extraction and classification through the fully connected layer.

[0062] S313, Application of CNN in image feature extraction:

[0063] CNN performs well in image feature extraction and is widely used in image recognition, target detection, and image segmentation. In image classification tasks, CNN can extract image features through automatic learning and use these features for classification.

[0064] S32, text feature extraction: using natural language processing (NLP) technology;

[0065] S321. Overview of Natural Language Processing (NLP) Technology:

[0066] In NLP, it can help us convert raw, irregular text data into structured features that can be understood and processed by computers;

[0067] S322. Text feature extraction method:

[0068] BERT model: BERT (Bidirectional Encoder Representations from Transformers) learns rich language knowledge and semantic information through unsupervised learning on massive text data. In terms of text feature extraction, BERT can convert text into high-dimensional vector representations. These vectors can capture the semantic information of the text and facilitate subsequent processing and analysis.

[0069] Transformer model: It captures long-distance dependencies in text through the self-attention mechanism and extracts the global features of the text. In terms of text feature extraction, the Transformer model can convert text into a series of vector representations that can capture the semantic and structural information of the text.

[0070] S323. Application of NLP technology in text feature extraction:

[0071] NLP uses models such as BERT and Transformer to convert text into high-dimensional vector representations. These vectors can capture the semantic information of the text and facilitate subsequent processing and analysis.

[0072] S33, numerical feature extraction: statistical analysis of numerical data;

[0073] S331. Overview of numerical feature extraction:

[0074] In the process of numerical feature extraction, we mainly focus on the statistical characteristics and distribution patterns of data;

[0075] S332, numerical feature extraction method:

[0076] Statistical characteristics: By calculating the mean, standard deviation, maximum, minimum, median, and mode statistics of numerical data, the distribution law and statistical characteristics of the data can be extracted. These statistics can reflect the central trend, degree of dispersion, and distribution form of the data;

[0077] Binning: Divide numerical data into several intervals (or "boxes") and count the number or proportion of data in each interval. Binning helps capture the distribution pattern and outliers of data.

[0078] Feature Engineering: Improve the performance and accuracy of the model by creating new features or transforming existing features. Nonlinear transformations such as logarithmic transformation and square root transformation can be performed on numerical data to improve the distribution characteristics of the data.

[0079] S333, Application of numerical feature extraction

[0080] Numerical feature extraction in data analysis, by extracting the statistical characteristics and distribution patterns of numerical data, we can better understand the nature and internal laws of the data, and provide strong support for subsequent model training and prediction.

[0081] As a preferred solution of the present invention, step S4: the specific steps of performing doubt mining, knowledge graph and sentiment analysis on images, texts and values ​​are as follows:

[0082] S41. Use multimodal features to mine doubts and discover anomalies and doubts in the data;

[0083] S411, Multimodal feature fusion: Fusion of data features from different modalities to form a unified feature table. This fusion can be based on simple feature concatenation, feature selection or more complex feature mapping methods. By fusing multimodal features, the complementarity between different types of data can be fully utilized to improve the accuracy and reliability of suspicious point mining.

[0084] S412, doubt mining method;

[0085] S4121. Anomaly detection algorithm: Anomaly detection algorithms are used to identify anomalies or suspicious points in data, and can automatically learn the normal pattern of data and detect data points that deviate significantly from the normal pattern;

[0086] S4122. Multimodal feature analysis: Combine multimodal features for comprehensive analysis to identify abnormal patterns and trends in data;

[0087] S42. Combine knowledge graphs to perform association analysis and find out the potential relationships between data;

[0088] S421. Knowledge graph construction: Knowledge graph is a semi-structured way of expressing knowledge, which uses graph theory, natural language processing and other technologies to construct a network of concepts, entities, attributes and relationships.

[0089] Its construction process includes entity recognition, relationship extraction, and knowledge representation. Entity recognition is to automatically identify specific entities from text; relationship extraction is to automatically identify the semantic relationship between two or more entities from text; knowledge representation is to convert entities and relationships into a language that can be recognized and processed by computers.

[0090] S422, association analysis method;

[0091] S4221. Reasoning based on knowledge graph: Using the entity and relationship information in the knowledge graph, perform logical reasoning and path query to discover the potential relationship between data;

[0092] S4222. Graph Algorithm Application: Apply graph algorithms (such as community discovery algorithms, shortest path algorithms, etc.) to identify association patterns and group characteristics in data;

[0093] S43, perform sentiment analysis to evaluate the sentiment tendency and attitude in text data;

[0094] S431, Sentiment analysis technology;

[0095] S4311. Dictionary-based method: Use sentiment dictionary to match keywords to evaluate the sentiment tendency of the text;

[0096] S4312, machine learning-based methods: using the SVM algorithm to train a classifier to identify the sentiment tendency of text;

[0097] S4313, Deep learning-based method: Use recurrent neural network (RNN) model to extract text features and perform sentiment classification;

[0098] S432. Multimodal sentiment analysis: Use multimodal learning models (such as Transformer-based models) for training and analysis; these models can automatically learn the associations and complementarities between multimodal features, improving the accuracy and robustness of sentiment analysis.

[0099] As a preferred solution of the present invention, the specific steps of step S5: the causal tracing model performs model analysis of the cause of a specific scenario are:

[0100] S51, construction of causal tracing model;

[0101] S511, technical selection:

[0102] Machine learning techniques: Choose appropriate machine learning algorithms, such as decision trees, random forests, support vector machines, etc. These algorithms can process structured data and reveal the causal relationship between variables;

[0103] Deep learning technology: For complex or unstructured data, such as text, images, etc., you can consider using deep learning technologies such as neural networks, convolutional neural networks (CNNs), recurrent neural networks (RNNs), etc. to capture deeper features and relationships;

[0104] S512, Model Design:

[0105] Feature selection: Select relevant feature variables based on the needs of the audit scenario. These variables should be able to reflect the key information and potential factors in the scenario.

[0106] Model architecture: Design the architecture of the model, including the input layer, hidden layer, and output layer, as well as the connection method and activation function between layers;

[0107] Loss function and optimization algorithm: Choose a suitable loss function to measure the prediction error of the model, and use an optimization algorithm (such as gradient descent) to minimize the loss function to train the model;

[0108] S513, Model training:

[0109] Data preparation: Collect and preprocess data, including data cleaning, missing value processing, outlier detection, etc., to ensure data quality and consistency;

[0110] Training process: Use the training data set to train the model, and optimize the model parameters iteratively so that the model gradually converges to the optimal solution;

[0111] Verification and testing: Use the validation data set to verify the model and evaluate the performance of the model; and use the test data set to test the model to verify the generalization ability of the model;

[0112] S52, scene setting;

[0113] S521. Audit needs analysis: clarify the audit objectives and scope, determine the specific scenarios that need to be analyzed, analyze the key elements and potential risk points in the scenarios, and provide direction for model analysis;

[0114] S522, Scenario description: Describe the background, environment, participants and behaviors of a specific scenario in detail to better understand the cause and effect relationship in the scenario, determine the dependent variables (results) and independent variables (causes or influencing factors) in the scenario, and provide a basis for model construction;

[0115] S53, model analysis;

[0116] S531. Data input: Input the preprocessed data into the causal traceability model, including the characteristic variables and dependent variables, and ensure that the format and scope of the data match the requirements of the model;

[0117] S532, model operation: run the model, analyze and process the input data, obtain the causal relationship and influencing factors between the dependent variable and the independent variable, and record the analysis results of the model;

[0118] S533, Result Analysis: Conduct in-depth analysis of the model's analysis results to reveal the key causal relationships and influencing factors in the scenario, compare the differences between the model's prediction results and the actual data, and evaluate the accuracy and reliability of the model;

[0119] S54, interpretation of results;

[0120] S541. Result verification: Use independent data sets or expert knowledge to verify the analysis results of the model to ensure the accuracy and reliability of the results, analyze outliers or inconsistencies in the results, and identify possible causes;

[0121] S542. Interpretation of results: Explain and elaborate the analysis results of the model, including the strength, directionality and significance of the causal relationship, and use charts, reports and other forms to intuitively display the analysis results for easy understanding and communication;

[0122] S543. Application of results: Apply the analysis results to audit practice, provide basis and support for audit decisions, and propose improvement suggestions or measures based on the analysis results to prevent or reduce the occurrence of similar problems.

[0123] As a preferred solution of the present invention, the specific steps of step S6: generating an audit report are:

[0124] S61. Determination of report contents;

[0125] S611, Audit Objective Review: Clarify the main objectives of this audit, such as evaluating the accuracy of financial statements, checking business compliance, and identifying potential risks, ensure that the audit objectives are consistent with the objectives in the audit plan, and serve as the cornerstone of the report writing;

[0126] S612, Audit Results Analysis: Carefully analyze the data, evidence, and findings collected during the audit process to determine which results are critical and need to be emphasized, and which results are secondary and can be briefly mentioned;

[0127] S613, Content structure planning:

[0128] According to the audit objectives and results, plan the content structure of the audit report, including the introduction, audit purpose, audit scope, audit methods, audit results, issues and suggestions, and conclusions, determine the key points and presentation order of each part, and ensure that the report is logically clear and well-organized;

[0129] S614, Focus determination: highlighting the important findings, key issues and potential risks in the audit process, and emphasizing the audit results that have a significant impact on the organization's operations, financial status or compliance;

[0130] S62. Data Visualization

[0131] S621. Choose appropriate visualization tools: Consider using professional data visualization software tools to improve visualization effects based on the type and characteristics of audit data;

[0132] S622, Data preparation and processing: Clean, organize and analyze audit data to ensure data accuracy and consistency, and appropriately convert and format data according to visualization requirements;

[0133] S623, Chart design and production: Design the style, color and layout of charts to make them consistent with the overall style and tone of the audit report, produce charts, and add necessary titles, axis labels and legends so that readers can understand the content of the charts;

[0134] S624, Chart integration and presentation: Integrate the prepared charts into the audit report, ensure that the charts and text content echo and complement each other, and adjust the size and position of the charts to make them coordinated with the report page layout;

[0135] S63. Report writing;

[0136] S631. Introduction: briefly introduce the background, purpose and importance of the audit, explain the scope and limitations of the audit, and summarize the structure and contents of the report;

[0137] S632, Audit Purpose and Scope: Describe in detail the specific purpose of the audit, such as assessing the compliance of specific business processes, reviewing the accuracy of financial statements, and clarify the scope of the audit, including the audit period, departments or business units involved, and key areas of the audit;

[0138] S633, Audit Methods: Introduce the methods, techniques and tools used in the audit process, such as data sampling, interviews, document review, data analysis, and explain the reasons for selecting these methods and their applicability;

[0139] S634, Audit results: present key issues, potential risks and compliance deficiencies found in the audit, use visualization tools to display audit data and analysis results, and enhance the persuasiveness and readability of the report;

[0140] S635, Problems and Suggestions: Propose specific suggestions or measures for improvement in response to the problems found in the audit, emphasizing the severity and urgency of the problems, as well as the feasibility and effectiveness of the suggestions;

[0141] S636, Conclusion section: Summarizes the main findings and conclusions of the audit, emphasizing the significant impact of the audit on the organization's operations, financial status or compliance;

[0142] S64, report review and release;

[0143] S641, Internal Audit: Submit the audit report to the internal audit team or relevant department for review, including the accuracy, completeness, logic and readability of the report;

[0144] S642, Revision and Improvement: Based on the audit feedback, make necessary revisions and improvements to the report to ensure that the information in the report is accurate and the expression is clear;

[0145] S643, External review (if necessary): If the report requires review by external institutions or experts, such as external auditors or legal advisors, it will be submitted to them for review, and the report will be further revised and improved based on the external review opinions;

[0146] S644, Report Release: Release the final audit report to relevant stakeholders, such as management, board of directors, and regulatory authorities, ensuring that the release method and channel of the report complies with the organization's regulations and requirements;

[0147] S645. Follow-up: Track the implementation of the recommendations or measures proposed in the audit report, and review and update the audit report regularly to reflect the organization’s latest status and audit results.

[0148] The working principle and beneficial effects of the present invention are:

[0149] The present invention ensures the quality and consistency of data through comprehensive data collection and preprocessing, and uses CNN, NLP and other technologies to extract features from images, texts and numerical data, realizing deep fusion and analysis of multimodal data. It deeply reveals abnormal patterns and potential relationships in the data through doubt mining, knowledge graph association analysis and sentiment analysis. The application of causal tracing models further clarifies the causal relationships and influencing factors in specific scenarios. Ultimately, the generated audit report is detailed and clearly structured, and the persuasiveness of the report is enhanced through data visualization. This method improves the accuracy and efficiency of audits and provides strong support for audit decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0150] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0151] Figure 1 This is a schematic diagram of the framework of the present invention. DETAILED DESCRIPTION

[0152] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0153] In this embodiment, it includes:

[0154] S1: Data collection;

[0155] S2: data preprocessing;

[0156] S3: data is decomposed into images, texts and numerical values;

[0157] S4: Perform doubt mining, knowledge graph and sentiment analysis on images, texts and values ​​respectively;

[0158] S5: The causal tracing model analyzes the causes of specific scenarios;

[0159] S6: Generate audit report.

[0160] Specifically, the specific steps of step S1: data collection are:

[0161] S11: Data collection channels and methods;

[0162] S111: A program that automatically obtains web page content using crawler technology, grabs required information from the Internet and stores it locally or in a database;

[0163] The collection steps are as follows: 1. Send a request: send a request to the target website to obtain the web page content;

[0164] 2. Parse HTML: parse the obtained HTML format code to extract the required information;

[0165] 3. Data extraction: Use regular expressions or other methods to extract required information from HTML code;

[0166] 4. Store data: store the extracted data locally or in a database;

[0167] S112: The database log records the history of database operations, including user access and data change information;

[0168] Its collection method:

[0169] 1. Log file collection: By monitoring the log files on the server, they are collected in real time into a centralized storage system for analysis and processing. This is achieved using file monitoring tools such as logstash and fluentd.

[0170] 2. Database and API collection: Some systems and applications store log data in the database, and can directly collect data through database connection, API call, etc.;

[0171] S12: Establish a data security management system

[0172] S121: Organizational structure: Establish a dedicated data security management organization to be responsible for the overall planning, strategy formulation and execution supervision of data security;

[0173] S121: System and process: Develop a comprehensive data security management system and process to clarify the security requirements for each link of data collection, storage, processing, sharing and destruction;

[0174] S122: Technical tools: Use data encryption, access control, and security auditing techniques to ensure data security during collection, transmission, and storage;

[0175] S13. Data classification and grading;

[0176] S131, Classification criteria: Classify data according to its sensitivity and importance;

[0177] S132, hierarchical protection: take corresponding protection measures for data of different levels;

[0178] S14. Data anonymization and de-identification;

[0179] S141, anonymization processing: anonymize the collected data so that it is impossible to restore the original data through reverse operations, thereby effectively protecting user privacy;

[0180] S142. De-identification: Remove personal identification information from the data as much as possible without affecting the effectiveness of data analysis;

[0181] S15. Strengthen access control;

[0182] S151. Permission management: Implement role-based access control to ensure that only authorized personnel can access sensitive data; this includes data access account and user permission management, access permission management during data use, and data sharing permission management;

[0183] S152. Audit tracking: Audit and track data access and usage to promptly detect and handle unauthorized access;

[0184] S16. Data encryption and desensitization: Use encryption technology during data transmission and storage to ensure that data will not be abused in the event of unauthorized access or leakage, and desensitize sensitive data to reduce the risk of data leakage.

[0185] In this embodiment, the specific steps of step S2: data preprocessing are:

[0186] S21, image data preprocessing;

[0187] S211, Denoising: Use filter techniques such as Gaussian filter and median filter to smooth the image and remove high-frequency noise while retaining the edge and detail information of the image;

[0188] S212, scaling: using image scaling algorithms, bilinear interpolation, and bicubic interpolation, to scale the image, maintain image clarity and detail information, and avoid image distortion;

[0189] S213, normalization: performing linear transformation on the image pixel values ​​so that they fall within a specified range. For an image with pixel values ​​in the range of [0, 255], it can be normalized to the range of [0, 1] by dividing by 255;

[0190] S22, text data preprocessing;

[0191] S221. Word segmentation: Split the text into words or tokens to facilitate subsequent processing and analysis. Use word segmentation tools or algorithms to segment the text. For Chinese text, commonly used word segmentation tools include Jieba word segmentation and StanfordNLP word segmentation.

[0192] S222, stop word removal: remove words that appear frequently in the text but do not contribute much to the meaning of the text, reduce data dimensions, improve processing efficiency, use predefined stop word lists, customize stop word lists according to specific tasks, and remove stop word processing from the text;

[0193] S223, stem extraction: restore words to their basic form (root), reduce vocabulary diversity, and improve the generalization ability of the model. Use the stem extraction algorithm to perform stem extraction on the text. The stem extraction algorithm restores the word to its basic form by removing the suffix or prefix of the word;

[0194] S23, numerical data preprocessing;

[0195] S231, Missing value processing: Process missing values ​​in numerical data to improve data integrity and accuracy;

[0196] method:

[0197] Deletion method: When the proportion of missing values ​​is small, you can choose to delete samples or features containing missing values;

[0198] Filling method: Use mean, median, mode and other methods to fill missing values. For normally distributed data, the mean can be used for filling; for skewed distributed data, the median can be used for filling. In addition, interpolation and regression model methods can be used for prediction and filling;

[0199] Model method: Use machine learning models to predict missing data points. This method requires building a suitable prediction model and training the model to predict missing values.

[0200] S232, Normalization: Map data of different scales to the same range (usually [0,1] or [-1,1]), eliminate the dimensional differences between features, improve the convergence speed and accuracy of the model, and use linear transformation to scale the data to the specified range. For data with minimum and maximum values ​​of xmin and xmax respectively, the formula x′= xmax -xmin x-xmin Normalize it to the range [0,1];

[0201] S233, Standardization: Make all features in the data set have zero mean and unit variance to facilitate subsequent processing and analysis. Use the standardized formula to process the data. For feature x, its mean and standard deviation are μ and σ respectively, then the standardized data is x′=σx-μ. The standardized data has the same scale, reducing the impact of inconsistent dimensions between features.

[0202] Specifically, the specific steps of step S3: decomposing data into images, texts and numerical values ​​are as follows:

[0203] S31, Image feature extraction: using convolutional neural network (CNN);

[0204] S311, Overview of Convolutional Neural Network (CNN);

[0205] By constructing a multi-level neural network structure, the human brain's processing of image and voice information is simulated to achieve feature extraction and classification of input data. CNN is mainly composed of convolutional layers, pooling layers, and fully connected layers.

[0206] S312, image feature extraction process:

[0207] 1. Convolution layer: The input image is convolved by sliding the convolution kernel to extract the local features of the image. The size and number of convolution kernels determine the type and number of extracted features.

[0208] 2. Pooling layer: Downsample the feature map output by the convolutional layer to reduce the dimension of the data, avoid overfitting, and retain important feature information. Commonly used pooling operations include Max Pooling and Average Pooling.

[0209] 3. Fully connected layer: Flatten the feature map output by the pooling layer into a one-dimensional vector, and perform further feature extraction and classification through the fully connected layer.

[0210] S313, Application of CNN in image feature extraction:

[0211] CNN performs well in image feature extraction and is widely used in image recognition, target detection, and image segmentation. In image classification tasks, CNN can extract image features through automatic learning and use these features for classification.

[0212] S32, text feature extraction: using natural language processing (NLP) technology;

[0213] S321. Overview of Natural Language Processing (NLP) Technology:

[0214] In NLP, it can help us convert raw, irregular text data into structured features that can be understood and processed by computers;

[0215] S322. Text feature extraction method:

[0216] BERT model: BERT (Bidirectional Encoder Representations from Transformers) learns rich language knowledge and semantic information through unsupervised learning on massive text data. In terms of text feature extraction, BERT can convert text into high-dimensional vector representations. These vectors can capture the semantic information of the text and facilitate subsequent processing and analysis.

[0217] Transformer model: It captures long-distance dependencies in text through the self-attention mechanism and extracts the global features of the text. In terms of text feature extraction, the Transformer model can convert text into a series of vector representations that can capture the semantic and structural information of the text.

[0218] S323. Application of NLP technology in text feature extraction:

[0219] NLP uses models such as BERT and Transformer to convert text into high-dimensional vector representations. These vectors can capture the semantic information of the text and facilitate subsequent processing and analysis.

[0220] S33, numerical feature extraction: statistical analysis of numerical data;

[0221] S331. Overview of numerical feature extraction:

[0222] In the process of numerical feature extraction, we mainly focus on the statistical characteristics and distribution patterns of data;

[0223] S332, numerical feature extraction method:

[0224] Statistical characteristics: By calculating the mean, standard deviation, maximum, minimum, median, and mode statistics of numerical data, the distribution law and statistical characteristics of the data can be extracted. These statistics can reflect the central trend, degree of dispersion, and distribution form of the data;

[0225] Binning: Divide numerical data into several intervals (or "boxes") and count the number or proportion of data in each interval. Binning helps capture the distribution pattern and outliers of data.

[0226] Feature Engineering: Improve the performance and accuracy of the model by creating new features or transforming existing features. Nonlinear transformations such as logarithmic transformation and square root transformation can be performed on numerical data to improve the distribution characteristics of the data.

[0227] S333, Application of numerical feature extraction

[0228] Numerical feature extraction in data analysis, by extracting the statistical characteristics and distribution patterns of numerical data, we can better understand the nature and internal laws of the data, and provide strong support for subsequent model training and prediction.

[0229] In this embodiment, the specific steps of step S4: performing doubt mining, knowledge graph analysis and sentiment analysis on images, texts and values ​​are as follows:

[0230] S41. Use multimodal features to mine doubts and discover anomalies and doubts in the data;

[0231] S411, Multimodal feature fusion: Fusion of data features from different modalities to form a unified feature table. This fusion can be based on simple feature concatenation, feature selection or more complex feature mapping methods. By fusing multimodal features, the complementarity between different types of data can be fully utilized to improve the accuracy and reliability of suspicious point mining.

[0232] S412, doubt mining method;

[0233] S4121. Anomaly detection algorithm: Anomaly detection algorithms are used to identify anomalies or suspicious points in data, and can automatically learn the normal pattern of data and detect data points that deviate significantly from the normal pattern;

[0234] S4122. Multimodal feature analysis: Combine multimodal features for comprehensive analysis to identify abnormal patterns and trends in data;

[0235] S42. Combine knowledge graphs to perform association analysis and find out the potential relationships between data;

[0236] S421. Knowledge graph construction: Knowledge graph is a semi-structured way of expressing knowledge, which uses graph theory, natural language processing and other technologies to construct a network of concepts, entities, attributes and relationships.

[0237] Its construction process includes entity recognition, relationship extraction, and knowledge representation. Entity recognition is to automatically identify specific entities from text; relationship extraction is to automatically identify the semantic relationship between two or more entities from text; knowledge representation is to convert entities and relationships into a language that can be recognized and processed by computers.

[0238] S422, association analysis method;

[0239] S4221. Reasoning based on knowledge graph: Using the entity and relationship information in the knowledge graph, perform logical reasoning and path query to discover the potential relationship between data;

[0240] S4222. Graph Algorithm Application: Apply graph algorithms (such as community discovery algorithms, shortest path algorithms, etc.) to identify association patterns and group characteristics in data;

[0241] S43, perform sentiment analysis to evaluate the sentiment tendency and attitude in text data;

[0242] S431, Sentiment analysis technology;

[0243] S4311. Dictionary-based method: Use sentiment dictionary to match keywords to evaluate the sentiment tendency of the text;

[0244] S4312, machine learning-based methods: using the SVM algorithm to train a classifier to identify the sentiment tendency of text;

[0245] S4313, Deep learning-based method: Use recurrent neural network (RNN) model to extract text features and perform sentiment classification;

[0246] S432. Multimodal sentiment analysis: Use multimodal learning models (such as Transformer-based models) for training and analysis; these models can automatically learn the associations and complementarities between multimodal features, improving the accuracy and robustness of sentiment analysis.

[0247] Specifically, the specific steps of step S5: the causal tracing model performs model analysis of the cause of a specific scenario are:

[0248] S51, construction of causal tracing model;

[0249] S511, technical selection:

[0250] Machine learning techniques: Choose appropriate machine learning algorithms, such as decision trees, random forests, support vector machines, etc. These algorithms can process structured data and reveal the causal relationship between variables;

[0251] Deep learning technology: For complex or unstructured data, such as text, images, etc., you can consider using deep learning technologies such as neural networks, convolutional neural networks (CNNs), recurrent neural networks (RNNs), etc. to capture deeper features and relationships;

[0252] S512, Model Design:

[0253] Feature selection: Select relevant feature variables based on the needs of the audit scenario. These variables should be able to reflect the key information and potential factors in the scenario.

[0254] Model architecture: Design the architecture of the model, including the input layer, hidden layer, and output layer, as well as the connection method and activation function between layers;

[0255] Loss function and optimization algorithm: Choose a suitable loss function to measure the prediction error of the model, and use an optimization algorithm (such as gradient descent) to minimize the loss function to train the model;

[0256] S513, Model training:

[0257] Data preparation: Collect and preprocess data, including data cleaning, missing value processing, outlier detection, etc., to ensure data quality and consistency;

[0258] Training process: Use the training data set to train the model, and optimize the model parameters iteratively so that the model gradually converges to the optimal solution;

[0259] Verification and testing: Use the validation data set to verify the model and evaluate the performance of the model; and use the test data set to test the model to verify the generalization ability of the model;

[0260] S52, scene setting;

[0261] S521. Audit needs analysis: clarify the audit objectives and scope, determine the specific scenarios that need to be analyzed, analyze the key elements and potential risk points in the scenarios, and provide direction for model analysis;

[0262] S522, Scenario description: Describe the background, environment, participants and behaviors of a specific scenario in detail to better understand the cause and effect relationship in the scenario, determine the dependent variables (results) and independent variables (causes or influencing factors) in the scenario, and provide a basis for model construction;

[0263] S53, model analysis;

[0264] S531. Data input: Input the preprocessed data into the causal traceability model, including the characteristic variables and dependent variables, and ensure that the format and scope of the data match the requirements of the model;

[0265] S532, model operation: run the model, analyze and process the input data, obtain the causal relationship and influencing factors between the dependent variable and the independent variable, and record the analysis results of the model;

[0266] S533, Result Analysis: Conduct in-depth analysis of the model's analysis results to reveal the key causal relationships and influencing factors in the scenario, compare the differences between the model's prediction results and the actual data, and evaluate the accuracy and reliability of the model;

[0267] S54, interpretation of results;

[0268] S541. Result verification: Use independent data sets or expert knowledge to verify the analysis results of the model to ensure the accuracy and reliability of the results, analyze outliers or inconsistencies in the results, and identify possible causes;

[0269] S542. Interpretation of results: Explain and elaborate the analysis results of the model, including the strength, directionality and significance of the causal relationship, and use charts, reports and other forms to intuitively display the analysis results for easy understanding and communication;

[0270] S543. Application of results: Apply the analysis results to audit practice, provide basis and support for audit decisions, and propose improvement suggestions or measures based on the analysis results to prevent or reduce the occurrence of similar problems.

[0271] In this embodiment, the specific steps of step S6: generating an audit report are:

[0272] S61. Determination of report contents;

[0273] S611, Audit Objective Review: Clarify the main objectives of this audit, such as evaluating the accuracy of financial statements, checking business compliance, and identifying potential risks, ensure that the audit objectives are consistent with the objectives in the audit plan, and serve as the cornerstone of the report writing;

[0274] S612, Audit Results Analysis: Carefully analyze the data, evidence, and findings collected during the audit process to determine which results are critical and need to be emphasized, and which results are secondary and can be briefly mentioned;

[0275] S613, Content structure planning:

[0276] According to the audit objectives and results, plan the content structure of the audit report, including the introduction, audit purpose, audit scope, audit methods, audit results, issues and suggestions, and conclusions, determine the key points and presentation order of each part, and ensure that the report is logically clear and well-organized;

[0277] S614, Focus determination: highlighting the important findings, key issues and potential risks in the audit process, and emphasizing the audit results that have a significant impact on the organization's operations, financial status or compliance;

[0278] S62. Data Visualization

[0279] S621. Choose appropriate visualization tools: Consider using professional data visualization software tools to improve visualization effects based on the type and characteristics of audit data;

[0280] S622, Data preparation and processing: Clean, organize and analyze audit data to ensure data accuracy and consistency, and appropriately convert and format data according to visualization requirements;

[0281] S623, Chart design and production: Design the style, color and layout of charts to make them consistent with the overall style and tone of the audit report, produce charts, and add necessary titles, axis labels and legends so that readers can understand the content of the charts;

[0282] S624, Chart integration and presentation: Integrate the prepared charts into the audit report, ensure that the charts and text content echo and complement each other, and adjust the size and position of the charts to make them coordinated with the report page layout;

[0283] S63. Report writing;

[0284] S631. Introduction: briefly introduce the background, purpose and importance of the audit, explain the scope and limitations of the audit, and summarize the structure and contents of the report;

[0285] S632, Audit Purpose and Scope: Describe in detail the specific purpose of the audit, such as assessing the compliance of specific business processes, reviewing the accuracy of financial statements, and clarify the scope of the audit, including the audit period, departments or business units involved, and key areas of the audit;

[0286] S633, Audit Methods: Introduce the methods, techniques and tools used in the audit process, such as data sampling, interviews, document review, data analysis, and explain the reasons for selecting these methods and their applicability;

[0287] S634, Audit results: present key issues, potential risks and compliance deficiencies found in the audit, use visualization tools to display audit data and analysis results, and enhance the persuasiveness and readability of the report;

[0288] S635, Problems and Suggestions: Propose specific suggestions or measures for improvement in response to the problems found in the audit, emphasizing the severity and urgency of the problems, as well as the feasibility and effectiveness of the suggestions;

[0289] S636, Conclusion section: Summarizes the main findings and conclusions of the audit, emphasizing the significant impact of the audit on the organization's operations, financial status or compliance;

[0290] S64, report review and release;

[0291] S641, Internal Audit: Submit the audit report to the internal audit team or relevant department for review, including the accuracy, completeness, logic and readability of the report;

[0292] S642, Revision and Improvement: Based on the audit feedback, make necessary revisions and improvements to the report to ensure that the information in the report is accurate and the expression is clear;

[0293] S643, External review (if necessary): If the report requires review by external institutions or experts, such as external auditors or legal advisors, it will be submitted to them for review, and the report will be further revised and improved based on the external review opinions;

[0294] S644, Report Release: Release the final audit report to relevant stakeholders, such as management, board of directors, and regulatory authorities, ensuring that the release method and channel of the report complies with the organization's regulations and requirements;

[0295] S645. Follow-up: Track the implementation of the recommendations or measures put forward in the audit report, and review and update the audit report regularly to reflect the latest status of the organization and the audit results. The above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multimodal audit method based on a large model, characterized by: include: S1: Data collection; S2: data preprocessing; S3: data is decomposed into images, texts and numerical values; S4: Perform doubt mining, knowledge graph and sentiment analysis on images, texts and values ​​respectively; S5: The causal tracing model analyzes the causes of specific scenarios; S6: Generate audit report.

2. The multimodal audit method based on a large model according to claim 1 is characterized in that: The specific steps of step S1: data collection are: S11: Data collection channels and methods, including crawler technology to capture web content and database log collection; S12: Establish a data security management system, including organizational structure, institutional processes, and technical tools; S13: Data classification and classification, classified protection according to sensitivity and importance; S14: Data anonymization and de-identification to protect user privacy; S15: Strengthen access control, including permission management and audit tracking. S16: Data encryption and desensitization to ensure data transmission and storage security.

3. The multimodal audit method based on a large model according to claim 2 is characterized in that: The specific steps of step S2: data preprocessing are: S21: image data preprocessing, including denoising, scaling, and normalization; S22: Text data preprocessing, including word segmentation, stop word removal, and stemming; S23: Numerical data preprocessing, including missing value processing, normalization, and standardization.

4. The multimodal audit method based on a large model according to claim 3 is characterized in that: Step S3: The specific steps of decomposing data into images, texts and numerical values ​​are as follows: S31: Image feature extraction using convolutional neural network (CNN); S32: Text feature extraction, using natural language processing (NLP) technology, BERT, Transformer model; S33: Numerical feature extraction and statistical analysis, including statistical feature extraction, binning, and feature engineering.

5. The multimodal audit method based on a large model according to claim 4 is characterized in that: Step S4: The specific steps of performing doubt mining, knowledge graph and sentiment analysis on images, texts and values ​​are as follows: S41: Using multimodal features to mine doubts, including multimodal feature fusion and doubt mining methods; S42: Combining knowledge graphs for association analysis, including knowledge graph construction and association analysis methods; S43: Perform sentiment analysis to assess sentiment in text data, including lexicon-based, machine learning and deep learning-based methods, and multimodal sentiment analysis.

6. The multimodal audit method based on a large model according to claim 5 is characterized in that: The specific steps of step S5: the causal tracing model performs model analysis of the cause of a specific scenario are: S51: Causal tracing model construction, including technology selection, model design, and model training; S52: Scenario setting, including audit needs analysis and scenario description; S53: Model analysis, including data input, model operation, and result analysis; S54: Result interpretation, including result verification, result interpretation, and result application.

7. The multimodal audit method based on a large model according to claim 6 is characterized in that: The specific steps of step S6: generating an audit report are: S61: Determination of report content, including review of audit objectives, analysis of audit results, content structure planning, and determination of key points; S62: Data visualization, choosing appropriate tools, preparing and processing data, designing and making charts, integrating and presenting charts; S63: Report writing, including introduction, audit objectives and scope, audit methods, audit results, issues and recommendations, and conclusions. S64: Report review and release, including internal review, revision and improvement, external review (if necessary), report release, and subsequent follow-up.

Images