Intelligent document compliance auditing system and method based on multi-modal deep learning

Abstract

The invention discloses an intelligent document compliance auditing system and method based on multi-modal deep learning, and belongs to the technical field of information processing, and the system comprises a data recognition module which is used for recognizing multi-modal data in a target document, the multi-modal data comprises a text, an image, a table and a scanning copy, carrying out information integration on the multi-modal data, and carrying out information processing on the integrated information; the information integration comprises the step of carrying out association analysis on data in images and tables and character description in texts; the compliance auditing module responds to information integration and is used for conducting compliance auditing on the target document after information integration, and compliance auditing comprises illegal content and format errors. According to the method, efficient and accurate compliance auditing is realized by identifying and integrating the multi-modal data in the document, illegal contents and format errors can be accurately identified, the risk level is evaluated, targeted countermeasures are formulated, and the auditing efficiency and accuracy are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information processing, in particular to an intelligent document compliance auditing system and method based on multi-modal deep learning. BACKGROUND

[0002] In the current wave of digital transformation, enterprise operations increasingly rely on efficient and accurate document management. From contracts, financial statements to various business reports, documents as an important carrier of information run through all business links of the enterprise. Therefore, auditing documents is a key step to ensure the information quality, compliance and smooth business progress.

[0003] For this research, the application number CN201910902257.9 application file provides an intelligent contract information extraction method based on deep learning and statistical extraction model, which reads the fields in various types of contract documents, and uses different information extraction methods for different types of information to divide the contract information into two formats: written standard information format and non-written semantic understanding format; two information formats are established respectively, and according to the training model, the information contained in the text is extracted by using the written standard information extraction model. This technical solution realizes the conversion of non-structured data to structured data of contract, which improves the management level of the company to a certain extent.

[0004] Another application number CN201911025558.4 application file provides a document automatic classification and optical character recognition method and system based on deep learning, which includes: first, scanning the document to be scanned to obtain a scanned picture, and preprocessing the scanned picture to obtain a preprocessed scanned picture; then using convolutional neural network to extract features and classify the preprocessed scanned picture, obtaining the picture features of the scanned picture, and classifying the scanned picture according to the picture features to realize the automatic classification of the scanned document. This technical solution realizes the classification and arrangement of scanned document pictures, and the automatic and intelligent processing of folder naming.

[0005] However, for documents containing multi-modal data such as text, images, tables and scans, the above technical solutions have great difficulty in feature extraction and alignment of different modalities. For example, the text in the scan may be misrecognized by OCR due to font blurring and background interference, which reduces the relevance analysis of the charts and text in the image, and further causes missed detection in the audit. SUMMARY

[0006] In view of the above problems existing in the prior art of information processing technology field, the present application is proposed.

[0007] Therefore, one of the purposes of the present application is to provide an intelligent document compliance auditing system and method based on multi-modal deep learning, which realizes efficient and accurate compliance auditing by automatically identifying and integrating multi-modal data (such as text, images, tables, etc.) in the document, can accurately identify illegal content and format errors, scientifically assess the risk level and formulate targeted countermeasures, and at the same time, use semantic similarity calculation and context analysis to enhance document consistency, improve auditing efficiency and accuracy.

[0008] To solve the above technical problems, the present application provides the following technical solutions:

[0009] In one aspect, the present application provides an intelligent document compliance auditing system based on multi-modal deep learning, comprising:

[0010] A data recognition module for recognizing multi-modal data in a target document, the multi-modal data including text, images, tables and scanned copies, and performing information integration on the multi-modal data, the information integration including correlating and analyzing data in images and tables with textual descriptions in text;

[0011] A compliance auditing module, which responds to the information integration, for performing compliance auditing on the target document after information integration, the compliance auditing including illegal content and format errors;

[0012] A feature extraction module for extracting features of the illegal content and format errors of the compliance auditing, the extraction method including constructing an extraction architecture, and setting a reading layer in the extraction architecture, the reading layer including reading the textual content of the target document, identifying risk information in the illegal content based on the textual content, the risk information including enterprise secrets and transaction amounts;

[0013] A data fusion processing module for analyzing and processing the extracted features, the data fusion processing module including an analysis unit, a comparison unit and a calculation unit;

[0014] The analysis unit is used for content analysis according to the textual content, the content analysis including content division of the target document, including division into market analysis, competition analysis, financial analysis and strategic suggestions;

[0015] The comparison unit responds to the divided content, and is used for comparing the illegal content with the divided content to obtain content corresponding to the risk information;

[0016] The calculation unit responds to the comparison unit, and is used for calculating the risk level of the risk information according to the content corresponding to the risk information, and formulating risk response measures according to the calculation result.

[0017] As a preferred scheme of the present application, wherein: in the calculation unit, the risk level of the risk information is calculated according to the content corresponding to the risk information, and the step includes risk information classification, determination of evaluation dimensions, establishment of scoring standards, risk level calculation and risk response measure establishment.

[0018] As a preferred scheme of the present application, wherein: the risk information classification includes dividing the risk information into legal risk, compliance risk and reputation risk according to different natures;

[0019] The determination of evaluation dimensions includes determining the evaluation dimensions according to the division of the nature of the risk information and the influence range, and the evaluation dimensions include severity, the severity includes the degree of damage caused, and the influence range includes the audience or business scope;

[0020] The scoring standard establishment includes establishing a scoring standard for each evaluation dimension, including dividing the severity into four levels of slight, general, severe and extremely severe, and assigning corresponding scores; and the influence range includes division according to the number of audiences or the importance of business, and assigning corresponding scores;

[0021] The risk level calculation includes single item scoring, comprehensive score calculation and risk level division;

[0022] The single item scoring is to score each evaluation dimension according to the risk information;

[0023] The comprehensive score calculation includes weighted summation or product operation of single item scores to obtain the comprehensive score of the risk information;

[0024] The risk level division includes dividing the risk information into different risk levels according to the comprehensive score, including low risk, medium risk, high risk and extremely high risk;

[0025] According to the divided risk level, corresponding risk response measures are established, including modifying document content, strengthening audit process and conducting legal consultation.

[0026] As a preferred scheme of the present application, wherein: in the data recognition module, the data in the image and table are associated and analyzed with the textual description in the text, and the step includes:

[0027] Denoising the image;

[0028] Different regions in the image are segmented by image segmentation, and the text in the image is recognized by OCR (Optical Character Recognition);

[0029] Recognize the row and column structure of the table;

[0030] Extract data values from the table;

[0031] Cleaning the extracted data to remove duplicates, errors, or invalid data;

[0032] Segmenting the text and annotating the part of speech of each word;

[0033] Extracting keywords or phrases from the text;

[0034] Analyzing the location of images or tables in the target document and the relevant location information mentioned in the textual description to establish a correlation relationship in the location;

[0035] Calculating the semantic similarity between the textual description and the data in the image / table through natural language processing; the semantic similarity includes converting the text and the textual description in the image / table into vector representations and calculating the cosine similarity between the vectors;

[0036] Matching the keywords in the text with the textual description in the image / table, and if there are identical keywords or synonyms, establishing a correlation relationship;

[0037] Analyzing the context information of the text and the context environment of the image / table in the target document to determine whether the context information and the context environment describe the same topic or concept.

[0038] As a preferred scheme of the present application, if the context information and the context environment do not belong to the description of the same topic or concept, the textual description is checked for consistency with the data in the image / table, and whether there are contradictions or omissions, while collecting features of the textual description corresponding to the data in the image / table, the features including semantic features and word meaning features; when the document is audited for compliance in the future, if the semantic features and word meaning features of the textual description corresponding to the data in the image / table are the same as the features, the system determines that the document is not compliant; otherwise, it is not determined.

[0039] As a preferred scheme of the present application, the cosine similarity between the vectors is calculated according to the following formula:

[0040] ;

[0041] In the formula, denotes the cosine similarity, denotes the dot product of vectors and , and the calculation formula is:

[0042] ;

[0043] wherein, and are vectors and the components in the first dimension, the dimension of a vector;

[0044] and respectively represent the modulus of vectors and , and the calculation formula is:

[0045] ; .

[0046] As a preferred scheme of the present application, wherein: further comprising calculating according to Jaccard similarity coefficient:

[0047] ; wherein, Jaccard similarity coefficient;

[0048] In the formula, represents the first set;

[0049] represents the second set;

[0050] represents the number of elements in the set ;

[0051] represents the number of elements in the set ;

[0052] represents the size of the intersection of the set and the set , the size is the number of keywords appearing in and ;

[0053] represents the size of the union of the set and the set , the size is the number of all non-repeated keywords appearing in or or both.

[0054] As a preferred scheme of the present application, wherein: according to the calculation result, a critical value of semantic similarity is preset, the critical value is marked as a first critical value, and the textual description corresponding to the data in the image / table is divided into paragraphs, a second critical value is preset according to the divided paragraphs, when the semantic similarity of the document is calculated at a future time, if the second critical value corresponding to the first divided paragraph is lower than the first critical value, the system determines that the semantic similarity of the textual description corresponding to the data in the image / table is low, and determines that the document is an illegal document; otherwise, it is not determined.

[0055] In another aspect, the present application provides a method applied to the intelligent document compliance auditing system based on multi-modal deep learning as described above, comprising the following steps:

[0056] The present application provides to identify multi-modal data in the target document, the multi-modal data including text, image, table and scan, and to integrate information of the multi-modal data, the information integration including correlation analysis of data in the image and table and textual description in the text;

[0057] Compliance auditing is performed on the target document after information integration, the compliance auditing including illegal content and format error;

[0058] Feature extraction is performed on the illegal content and format error of the compliance auditing, the extraction mode including constructing an extraction architecture, setting a reading layer in the extraction architecture, the reading layer including reading the textual content of the target document, identifying risk information in the illegal content based on the textual content, the risk information including enterprise secrets and transaction amounts;

[0059] Content analysis is performed according to the textual content, the content analysis including content division of the target document, including division into market analysis, competition analysis, financial analysis and strategic suggestions;

[0060] The illegal content is compared with the divided content to obtain the content corresponding to the risk information;

[0061] The risk level of the risk information is calculated according to the content corresponding to the risk information, and risk response measures are developed according to the calculation result.

[0062] Advantages:

[0063] 1、The present application automatically identifies multi-modal data (text, images, tables and scanned documents) in the target document through the data recognition module and integrates the information, improving the efficiency and accuracy of data processing, and conducting a comprehensive compliance check on the integrated document, including identifying illegal content and format errors, which ensures that the document complies with relevant regulations and standards, reducing legal risks caused by non-compliance;

[0064] 2、The feature extraction module extracts features of illegal content and format errors by constructing an extraction architecture, which can identify risk information such as enterprise secrets and transaction amounts, and the data fusion processing module further analyzes and processes these features to calculate the risk level and develop appropriate risk response measures, which helps enterprises respond to potential risks in a timely manner;

[0065] 3、The system formulates scoring standards and calculates risk levels according to the nature, severity and impact range of risk information, and according to the risk levels (low risk, medium risk, high risk and extremely high risk), the system can develop appropriate risk response measures such as modifying document content, strengthening audit procedures and seeking legal advice, which helps enterprises allocate resources reasonably and effectively respond to various risks;

[0066] 4、The system uses a series of efficient data processing techniques such as denoising, image segmentation, OCR recognition, table row and column structure recognition, and data cleaning in the data recognition module, and at the same time, through natural language processing technology, the semantic similarity between the text description and the data in the image / table is calculated, further improving the accuracy and efficiency of data processing;

[0067] 5、The system flexibly evaluates the semantic similarity between the text description and the data in the image / table according to the cosine similarity and Jaccard similarity coefficient calculation methods, and at the same time, the system allows the critical value of semantic similarity to be preset, so that the document can be quickly determined whether it is compliant when conducting compliance audit in the future, and the flexible similarity calculation and critical value setting ability enhances the adaptability and practicality of the system;

[0068] 6、By analyzing the consistency of the text context information and the image / table context environment, the system can check whether the text description is consistent with the data in the image / table, and collect the corresponding semantic and lexical features, which can be used by the system to quickly determine whether the document is compliant when conducting compliance audit in the future, thereby enhancing the consistency and compliance of the document. BRIEF DESCRIPTION OF DRAWINGS

[0069] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. Among them: Figure 1 A modular structure schematic diagram of the intelligent document compliance auditing system based on multi-modal deep learning of the embodiments of the present application; Figure 2 A method flow schematic diagram of the embodiments of the present application;

[0070] The figure label: 110-data recognition module; 120-compliance auditing module; 130-feature extraction module; 140-data fusion processing module; 1401-analysis unit; 1402-comparison unit; 1403-computing unit. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be described clearly and completely below in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by those skilled in the art belong to the scope of protection of the present application.

[0072] Because the existing technology has great difficulty in feature extraction and alignment of documents containing multi-modal data such as text, image, table and scan, the relevance analysis of charts and text in the image is weakened, which further causes missed detection in auditing.

[0073] Based on this, the present application provides an intelligent document compliance auditing system and method based on multi-modal deep learning, which realizes efficient and accurate compliance auditing by automatically recognizing and integrating multi-modal data (such as text, image, table, etc.) in the document, can accurately identify illegal content and format errors, scientifically assess the risk level and develop targeted countermeasures, at the same time, uses semantic similarity calculation and context analysis to enhance document consistency, and improves the auditing efficiency and accuracy.

[0074] The present application will be further described in detail below through embodiments and in combination with the drawings.

[0075] Reference Figures 1-2 For an embodiment of the present application, the embodiment provides an intelligent document compliance auditing system based on multi-modal deep learning, comprising:

[0076] The data recognition module 110 is configured to recognize multi-modal data in the target document, the multi-modal data including text, images, tables and scanned documents, and to perform information integration on the multi-modal data, the information integration including correlation analysis of data in the images and the tables and textual descriptions in the text;

[0077] In the intelligent document compliance auditing system, the correlation analysis of data in the images and the tables and textual descriptions in the text is a key step to improve the accuracy of document understanding and the efficiency of compliance auditing;

[0078] In this embodiment, the data recognition module is configured to recognize multi-modal data (text, images, tables and scanned documents) in the target document and perform information integration;

[0079] The information integration includes correlation analysis of data in the images and the tables and textual descriptions in the text.

[0080] The specific steps include image denoising, image segmentation, OCR recognition, table row and column structure recognition, data cleaning, text segmentation processing, keyword extraction, position correlation analysis, semantic similarity calculation and the like.

[0081] This correlation analysis ensures the consistency of the image, table and text data, reduces the error rate, and enables the present application to process complex documents containing multiple data types and improve the comprehensiveness of the audit.

[0082] The compliance auditing module 120 is configured to perform compliance auditing on the target document after information integration, the compliance auditing including illegal content and format errors.

[0083] In this embodiment, the compliance auditing module performs compliance auditing on the target document after information integration, including identifying illegal content and format errors.

[0084] This improves the auditing efficiency while checking the document content and format at the same time, ensures the overall compliance of the document, and timely discovers and corrects illegal content to avoid legal risks caused by non-compliance;

[0085] The feature extraction module 130 is configured to extract features of the illegal content and the format errors in the compliance auditing, the extraction including constructing an extraction architecture and setting a reading layer in the extraction architecture, the reading layer including reading the textual content of the target document, identifying risk information in the illegal content based on the textual content, the risk information including enterprise secrets and transaction amounts.

[0086] In this embodiment, the feature extraction module extracts features from the identified violations and format errors in the compliance audit, including constructing an extraction architecture, setting a reading layer, reading the text content of the target document, and identifying risk information (such as enterprise secrets and transaction amounts) in the violation content based on the text content.

[0087] This can accurately identify sensitive information such as enterprise secrets and transaction amounts in the document, providing a basis for risk assessment. Through feature extraction, focus on key violation content, improve the relevance and effectiveness of the audit;

[0088] The data fusion processing module 140 is configured to analyze and process the extracted features. The data fusion processing module 140 includes an analysis unit 1401, a comparison unit 1402, and a calculation unit 1403.

[0089] The analysis unit 1401 is configured to perform content analysis based on the text content. The content analysis includes dividing the target document into content, including market analysis, competition analysis, financial analysis, and strategic recommendations.

[0090] The comparison unit 1402 is configured to compare the violation content with the divided content to obtain content corresponding to the risk information.

[0091] The calculation unit 1403 is configured to calculate the risk level of the risk information based on the content corresponding to the risk information, and to develop risk response measures based on the calculation results.

[0092] In the compliance audit of the document, calculating the risk level of the risk information is a key step that helps identify potential compliance risks in the document and take appropriate management measures accordingly.

[0093] However, there is no uniform, fixed formula suitable for all scenarios, as the severity assessment often depends on specific circumstances, industry standards, laws and regulations, and the organization's internal risk management policies. However, an evaluation framework based on key factors can be constructed, and the severity of risk information can be quantified or described through weighted scoring or qualitative description.

[0094] It should be noted that through content division and comparison, the problem can be more accurately located, providing a basis for developing effective measures, and based on the nature and impact of the risk information, the scoring standard is developed and the risk level is calculated to ensure the scientificity of the evaluation.

[0095] In the calculation unit, the risk level of the risk information is calculated based on the content corresponding to the risk information, including risk information classification, determination of evaluation dimensions, development of scoring standards, risk level calculation, and development of risk response measures.

[0096] On this basis, the risk information is classified, including dividing the risk information into legal risk, compliance risk and reputation risk according to the nature of the risk information;

[0097] The evaluation dimensions are determined, including determining the evaluation dimensions according to the division of the nature of the risk information and the influence range, the evaluation dimensions including severity, the severity including the degree of damage caused (such as legal penalties, financial losses, reputation damage, etc.), and the influence range including the audience or business scope;

[0098] This step helps to more accurately assess and manage the risk in the subsequent steps;

[0099] The scoring criteria are developed, including developing scoring criteria for each evaluation dimension, including dividing the severity into four levels of slight, general, severe and extremely severe, and assigning corresponding scores; and dividing the influence range according to the number of audiences or the importance of business and assigning corresponding scores;

[0100] The risk level calculation includes single-item scoring, comprehensive scoring calculation and risk level division;

[0101] Single-item scoring, which is to score each evaluation dimension according to the risk information;

[0102] For example, for a piece of information involving legal risk, the severity may be rated as "severe" (score 4), the likelihood as "medium" (score 3), and the influence range as "extensive" (score 4)

[0103] Comprehensive scoring calculation, which includes weighted sum or product operation of single-item scores to obtain the comprehensive score of the risk information;

[0104] Weighted sum: according to the importance of each evaluation dimension, different weights are assigned, and then the weighted sum is calculated. For example, the severity weight is 0.5, the likelihood weight is 0.3, and the influence range weight is 0.2, then the comprehensive score is 40.5 + 30.3 + 4*0.2 = 3.7;

[0105] Product operation: multiply each single-item score to obtain the comprehensive score. This method is suitable for cases where the evaluation dimensions have a significant impact on each other;

[0106] Risk level division, which is to divide the risk information into different risk levels according to the comprehensive score, including low risk, medium risk, high risk and extremely high risk;

[0107] The division criteria can be developed according to actual conditions, for example, the comprehensive score between 1 and 2 is low risk, between 2 and 3 is medium risk, between 3 and 4 is high risk, and above 4 is extremely high risk

[0108] According to the classified risk level, corresponding risk response measures are developed, including modifying the document content, strengthening the audit process and seeking legal advice;

[0109] In this embodiment, by classifying and evaluating the dimensions, a systematic risk management framework is established to improve management efficiency;

[0110] And considering the various factors of risk, ensuring the comprehensiveness and accuracy of risk assessment;

[0111] And can develop targeted response measures according to different risk types and evaluation dimensions to improve the effectiveness of measures;

[0112] In the data recognition module, the data in the image and table are associated and analyzed with the textual description in the text, including:

[0113] Denoising the image, such as removing noise points, blurred areas, etc., and enhancing the clarity of the image to more accurately identify the image content

[0114] Different regions in the image (such as text area, chart area, graphic area, etc.) are segmented by image segmentation, and the text in the image is recognized by OCR (Optical Character Recognition);

[0115] Identify the row and column structure of the table, such as determining the table header, table body, etc.

[0116] Extract data values such as numbers, text, etc. from the table;

[0117] Clean the extracted data to remove duplicate, incorrect or invalid data;

[0118] Tokenize the text and label the part of speech of each word (such as noun, verb, adjective, etc.)

[0119] Extract keywords or phrases from the text that may describe the content of the image or table;

[0120] Analyze the location of the image or table in the target document and the relevant location information mentioned in the textual description (such as "as shown", "data in Table 2", etc.), and establish a correlation relationship in the location; Figure 1

[0121] Calculate the semantic similarity between the textual description and the data in the image / table by natural language processing, for example, using word embedding models (such as Word2Vec, BERT, etc.) Semantic similarity includes converting text and image / table text descriptions into vector representations, calculating the cosine similarity between vectors;

[0122] ​Match the keywords in the text with the textual descriptions in the image / table. If there are identical keywords or synonyms, establish a correlation relationship;

[0123] Analyze the context information of the text and the context environment of the image / table in the target document to determine whether the context information and the context environment describe the same topic or concept.

[0124] If the context information and the context environment do not belong to the description of the same topic or concept, check whether the textual description is consistent with the data in the image / table and whether there are contradictions or omissions. At the same time, collect the features of the textual description corresponding to the data in the image / table, including semantic features and word meaning features. When the document is audited for compliance at a future time, if the semantic features and word meaning features of the textual description corresponding to the data in the image / table are the same as the features, the system determines that the document is not compliant; otherwise, it is not determined.

[0125] In this embodiment, the consistency and logicality of the content of each part of the document are ensured through context analysis;

[0126] And in terms of feature collection, semantic and word meaning features are automatically collected and stored to provide reference for future audits;

[0127] And through context analysis and feature collection, the future audit process is simplified and the audit efficiency is improved.

[0128] The cosine similarity between the vectors is calculated according to the following formula:

[0129] ;

[0130] In the formula, denotes the cosine similarity, denotes the dot product of vectors and , and the calculation formula is:

[0131] ;

[0132] where, and are the components of vectors and in the first dimension, is the dimension of the vector;

[0133] and denote the norms of vectors and , respectively, and the calculation formula is:

[0134] ; ;

[0135] Need to explain, the vector dot product is the sum of the projection length of two vectors in the same direction of the product;

[0136] If the components of two vectors in the same direction are large, their dot product will also be large, indicating that they are similar in direction;

[0137] And the vector module is the length of the vector, which is the square root of the sum of the square of the components in each dimension;

[0138] The vector module is used to normalize the dot product result, so that the cosine similarity is in the range of [−1, 1];

[0139] The cosine similarity is calculated by dividing the dot product by the product of the two vector modules. When two vectors are exactly the same, the cosine similarity is 1, indicating complete similarity;

[0140] When two vectors are completely opposite, the cosine similarity is -1, indicating complete dissimilarity;

[0141] When two vectors are perpendicular, the cosine similarity is 0, indicating no similarity;

[0142] In the compliance audit system of the document, the text description and the text description in the image / table can be converted into vector representation (such as using word embedding model), and then the cosine similarity between them is calculated. If the cosine similarity is high, it means that the text description and the data in the image / table are similar in semantics, and there may be a correlation. On the contrary, if the cosine similarity is low, it means that they are different in semantics, and there may be no direct correlation;

[0143] In this way, the system can automatically identify and associate multi-modal data in the document, improve the accuracy of document understanding and the efficiency of compliance audit;

[0144] Also includes according to Jaccard similarity coefficient calculation:

[0145] ; Wherein, Jaccard similarity coefficient is represented;

[0146] In the formula, The first set is represented, for example, the keyword set extracted from the text description

[0147] The second set is represented, for example, the keyword set extracted from the image annotation or table content extraction

[0148] The set is represented The number of elements in the set, i.e., the number of keywords

[0149] denotes the set The number of elements in the set

[0150] denotes the set The size of the intersection of the sets The number of keywords that appear in both and ;

[0151] denotes the set The size of the union of the sets The number of all unique keywords that appear in or or both

[0152] It is noted that for the intersection part ( ), the number of elements common to both sets is reflected, which is a direct manifestation of similarity. The more common keywords, the more similar the two sets in content are;

[0153] For the union part ( ), as the denominator, it is used to normalize the size of the intersection, so that the value range of the similarity coefficient is between [0, 1]. The union considers all elements that appear in both sets, avoiding the deviation caused by different set sizes;

[0154] For the similarity coefficient , it is calculated by the ratio of the size of the intersection to the size of the union. The closer the value is to 1, the more similar the two sets are; the closer the value is to 0, the less similar the two sets are;

[0155] In document compliance audits, the Jaccard similarity coefficient can be used to compare the similarity of text descriptions and image / table content at the keyword level;

[0156] For example, the keyword set = { "sales", "growth", "2025"} is extracted from a piece of text description, and the keyword set = { "sales", "2025", "quarterly data"} is extracted from the corresponding table.

[0157] The intersection = { "sales", "2025"} is calculated, with a size of 2;

[0158] The union = { "sales", "growth", "2025", "quarterly data"}, size 4.

[0159] Therefore, the Jaccard similarity coefficient is = 2 / 4 = 0.5, indicating that the text description has a certain similarity with the table content at the keyword level;

[0160] On the basis of the above, according to the preset critical value of the semantic similarity according to the calculation result, the critical value is marked as the first critical value, and the text description corresponding to the data in the image / table is divided into paragraphs, and according to the divided paragraphs, the second critical value is preset, when the semantic similarity of the document is calculated at the future time, if the second critical value corresponding to the first divided paragraph is lower than the first critical value, the system determines that the semantic similarity of the text description corresponding to the data in the image / table is low, and determines that the document is an illegal document; otherwise, it is not determined;

[0161] Through the vector representation and the similarity calculation method, the consistency of the text description and the image / table data is quickly evaluated;

[0162] And the critical value can be preset according to the actual demand, improving the adaptability and flexibility of the system;

[0163] And through the similarity evaluation and critical value determination, the consistency and compliance of the document content are ensured.

[0164] Based on the above, the present application realizes efficient and accurate compliance audit by automatically identifying and integrating multi-modal data (such as text, image, table, etc.) in the document, can accurately identify illegal content and format errors, scientifically assess risk level and formulate targeted response measures, at the same time, the document consistency is enhanced by using semantic similarity calculation and context analysis, which improves the audit efficiency and accuracy.

[0165] The embodiment combines the above-mentioned intelligent document compliance audit system based on multi-modal deep learning, and further proposes a working method applied to the system, the steps are as follows:

[0166] S10: identifying multi-modal data in the target document, the multi-modal data including text, image, table and scan, and integrating information of the multi-modal data, the information integration including correlating and analyzing data in the image and table with the text description in the text;

[0167] S20: performing compliance audit on the target document after information integration, the compliance audit including illegal content and format error;

[0168] S30: Feature extraction is performed on the violation content and format errors of the compliance audit, and the extraction includes constructing an extraction architecture, setting a reading layer in the extraction architecture, the reading layer including reading the text content of the target document, identifying risk information in the violation content based on the text content, the risk information including enterprise secrets and transaction amounts;

[0169] S40: Content analysis is performed according to the text content, and the content analysis includes content division of the target document, including division into market analysis, competition analysis, financial analysis and strategic suggestions;

[0170] S50: The violation content is compared with the divided content to obtain the content corresponding to the risk information;

[0171] S60: The risk level of the risk information is calculated according to the content corresponding to the risk information, and risk response measures are formulated according to the calculation result.

[0172] In summary, the present application realizes efficient and accurate compliance audit by automatically identifying and integrating multi-modal data (such as text, image, table, etc.) in the document, can accurately identify violation content and format errors, scientifically assess risk level and formulate targeted response measures, at the same time, uses semantic similarity calculation and context analysis to enhance document consistency, significantly improves audit efficiency and accuracy, reduces compliance risk and labor cost, and provides comprehensive and intelligent document compliance management solution for enterprises.

[0173] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalent, without departing from the spirit and scope of the technical solutions of the present application, which should be covered in the scope of the claims of the present application.

Claims

1. An intelligent document compliance auditing system based on multi-modal deep learning, characterized in that, The method comprises the following steps: A data recognition module is used to recognize multi-modal data in a target document, the multi-modal data including text, images, tables and scanned documents, and to integrate information of the multi-modal data, the information integration including correlating and analyzing data in images and tables with textual descriptions in the text; A compliance audit module is used to audit the target document after information integration for compliance, the compliance audit including illegal content and format errors; A feature extraction module is used to extract features of the illegal content and format errors in the compliance audit, the extraction including constructing an extraction architecture, and setting a reading layer in the extraction architecture, the reading layer including reading textual content of the target document, and identifying risk information in the illegal content based on the textual content, the risk information including enterprise secrets and transaction amounts; A data fusion processing module is used to analyze and process the extracted features, the data fusion processing module including an analysis unit, a comparison unit and a calculation unit; The analysis unit is used to analyze content according to the textual content, the content analysis including dividing the target document into content, including market analysis, competition analysis, financial analysis and strategic suggestions; The comparison unit is used to compare the illegal content with the divided content to obtain content corresponding to the risk information in response to the divided content; The calculation unit is used to calculate a risk level of the risk information according to the content corresponding to the risk information in response to the comparison unit, and to develop risk response measures according to the calculation result.

2. The multimodal deep learning based intelligent document compliance auditing system as claimed in claim 1, wherein, In the calculation unit, the risk level of the risk information is calculated according to the content corresponding to the risk information, including risk information classification, determination of evaluation dimensions, development of scoring standards, risk level calculation and development of risk response measures.

3. The multi-modal deep learning based intelligent document compliance auditing system as claimed in claim 2, wherein, The risk information classification includes dividing the risk information into legal risk, compliance risk and reputation risk according to different natures; The determination of evaluation dimensions includes determining evaluation dimensions according to the division of the nature of the risk information and the influence range, the evaluation dimensions including severity, the severity including the degree of damage, and the influence range including the audience or the business scope; The development of scoring standards includes developing scoring standards for each evaluation dimension, including dividing the severity into four levels of slight, general, severe and extremely severe, and assigning corresponding scores; and the influence range includes dividing according to the number of audiences or the importance of business, and assigning corresponding scores; The risk level calculation includes single-item scoring, comprehensive scoring calculation and risk level division; The single-item scoring includes scoring each evaluation dimension according to the risk information; The comprehensive scoring calculation includes weighting and summing or multiplying the single-item scores to obtain the comprehensive score of the risk information; The risk level division includes dividing the risk information into different risk levels according to the comprehensive score, including low risk, medium risk, high risk and extremely high risk. According to the classified risk level, corresponding risk response measures are formulated, including modifying the document content, strengthening the audit process and seeking legal advice.

4. The multi-modal deep learning based intelligent document compliance auditing system as claimed in claim 1, wherein, In the data identification module, the data in the image and table are associated and analyzed with the textual description in the text, and the steps include: Denoising the image; Different regions in the image are segmented by image segmentation, and the text in the image is recognized by OCR (Optical Character Recognition); The row and column structure of the table is recognized; Data values are extracted from the table; The extracted data is cleaned to remove duplicate, incorrect or invalid data; The text is segmented, and the part of speech of each word is labeled; Key words or phrases are extracted from the text; The position of the image or table in the target document and the related position information mentioned in the textual description are analyzed to establish a correlation relationship in the position; The semantic similarity between the textual description and the data in the image / table is calculated by natural language processing, including converting the text and the textual description in the image / table into vector representation and calculating the cosine similarity between the vectors; The key words in the text are matched with the textual description in the image / table, and if there are the same key words or synonyms, a correlation relationship is established; The context information of the text and the context environment of the image / table in the target document are analyzed to determine whether the context information and the context environment describe the same theme or concept.

5. The multimodal deep learning based intelligent document compliance auditing system as claimed in claim 4, wherein, If the context information and the context environment do not belong to the description of the same theme or concept, check whether the textual description is consistent with the data in the image / table, and whether there are contradictions or omissions, while collecting the features of the textual description corresponding to the data in the image / table, including semantic features and word meaning features; When the document is audited for compliance in the future, if the semantic features and word meaning features of the textual description corresponding to the data in the image / table are the same as the features, the system determines that the document is not compliant; otherwise, it is not determined.

6. The multi-modal deep learning based intelligent document compliance auditing system as claimed in claim 4, wherein, The cosine similarity between the vectors is calculated according to the following formula: ； wherein denotes the cosine similarity, denotes the dot product of the vectors and with the formula ； wherein and are vectors and are components of the vectors in the th dimension, is the dimension of the vector. and denote the norm of the vector and the norm of the vector ； 。 7. The multimodal deep learning based intelligent document compliance auditing system as claimed in claim 6, wherein, Also includes according to Jaccard similarity coefficient calculation: ; wherein, represents the Jaccard similarity coefficient; In the formulae, denotes the first set; denotes the second set; representing a set the number of elements in the set; representing a set the number of elements in the set; representing a set and a set the size of the intersection of and the number of keywords that appear in both representing a set and a set the size of the union of the two sets, which is the number of all non-duplicate keywords that appear in or or both.

8. The intelligent document compliance auditing system based on multi-modal deep learning of any one of claims 6-7, wherein, According to the calculation result, a critical value of semantic similarity is preset, which is marked as the first critical value, and the textual description corresponding to the data in the image / table is divided into paragraphs, and a second critical value is preset according to the divided paragraphs. When the document is calculated for semantic similarity in the future, if the second critical value corresponding to the first divided paragraph is lower than the first critical value, the system determines that the semantic similarity of the textual description corresponding to the data in the image / table is low, and determines that the document is an uncompliant document; otherwise, it is not determined.

9. A method applied to the multi-modal deep learning based intelligent document compliance auditing system as claimed in claim 1, wherein, The steps include: Identify the multi-modal data in the target document, including text, image, table and scan, and integrate the information of the multi-modal data, including associating and analyzing the data in the image and table with the textual description in the text; Conduct a compliance audit on the target document after information integration, including violation of content and format error; The feature extraction of the violation content and format error of the compliance audit includes constructing an extraction architecture, setting a reading layer in the extraction architecture, the reading layer including reading the text content of the target document, identifying risk information in the violation content based on the text content, the risk information including enterprise secrets and transaction amounts; Content analysis is performed according to the text content, which includes content division of the target document, including division into market analysis, competition analysis, financial analysis and strategic suggestions; The violation content is compared with the divided content to obtain the content corresponding to the risk information; The risk level of the risk information is calculated according to the content corresponding to the risk information, and risk response measures are developed according to the calculation results.

Images