Intelligent document compliance auditing system and method based on multi-modal deep learning

Abstract

The application discloses an intelligent document compliance auditing system and method based on multi-modal deep learning, belongs to the technical field of information processing, and comprises a data recognition module, a compliance auditing module and the like.The data recognition module is used for recognizing multi-modal data in a target document, the multi-modal data including text, images, tables and scanned copies, and performing information integration on the multi-modal data, the information integration including correlating and analyzing data in the images and the tables with textual descriptions in the text.The compliance auditing module is responsive to the information integration and is used for performing compliance auditing on the target document after information integration, the compliance auditing including illegal content and format errors.The application realizes efficient and accurate compliance auditing by recognizing and integrating multi-modal data in a document, can accurately identify illegal content and format errors, simultaneously assesses a risk level and formulates targeted countermeasures, and improves auditing efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, and in particular to an intelligent document compliance review system and method based on multimodal deep learning. Background Technology

[0002] In today's wave of digital transformation, business operations are increasingly reliant on efficient and accurate document management. From contracts and financial statements to various business reports, documents, as vital carriers of information, permeate every aspect of a company's operations. Therefore, document review is a crucial step in ensuring information quality, compliance, and the smooth progress of business operations.

[0003] Regarding this research, application CN201910902257.9 provides an intelligent contract information extraction method based on deep learning and statistical extraction models. This technical solution reads fields from various types of contract documents and uses different information extraction methods to divide the contract information into two formats: a specified standard information format and an unspecified semantically understandable format. Extraction models are established for each of the two information formats. Based on the trained models, the specified standard information extraction model is used to extract the information contained in the text. This technical solution transforms irregular contract data into structured data, improving the company's management level to some extent.

[0004] Another application, CN201911025558.4, provides a method and system for automatic document classification and optical character recognition based on deep learning. This technical solution includes: first, scanning the document to be scanned to obtain a scanned image; then, preprocessing the scanned image to obtain a preprocessed scanned image; next, using a convolutional neural network to extract and classify features from the preprocessed scanned image to obtain image features; and finally, classifying the scanned image based on these image features to achieve automatic classification of the document to be scanned. This technical solution automates and intelligently processes the classification, organization, and folder naming of scanned document images.

[0005] However, for documents containing multimodal data such as text, images, tables, and scanned documents, the above-mentioned technical solutions face significant challenges in feature extraction and alignment across different modalities. For example, text in scanned documents may suffer from OCR recognition errors due to blurry fonts or background interference, which weakens the analysis of the correlation between charts and text in images, leading to missed detections during the review process. Summary of the Invention

[0006] In view of the problems existing in the field of information processing technology, the present invention is proposed.

[0007] Therefore, one of the objectives of this invention is to provide an intelligent document compliance review system and method based on multimodal deep learning. By automatically identifying and integrating multimodal data (such as text, images, tables, etc.) in documents, it achieves efficient and accurate compliance review. It can accurately identify illegal content and format errors, scientifically assess risk levels and formulate targeted countermeasures. At the same time, it uses semantic similarity calculation and context analysis to enhance document consistency, thereby improving review efficiency and accuracy.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0009] On the one hand, this invention provides an intelligent document compliance review system based on multimodal deep learning, comprising:

[0010] A data recognition module is used to recognize multimodal data in a target document, including text, images, tables, and scanned documents, and to integrate information from the multimodal data. The information integration includes correlation analysis between data in images and tables and textual descriptions in the text.

[0011] A compliance review module, which responds to the information integration, is used to conduct a compliance review on the target document after information integration. The compliance review includes violations and formatting errors.

[0012] The feature extraction module is used to extract features from the non-compliant content and format errors in the compliance review. The extraction method includes constructing an extraction architecture and setting a reading layer in the extraction architecture. The reading layer includes reading the text content of the target document and identifying risk information in the non-compliant content based on the text content. The risk information includes corporate secrets and transaction amounts.

[0013] The data fusion processing module is used to analyze and process the extracted features. The data fusion processing module includes an analysis unit, a comparison unit, and a calculation unit.

[0014] The analysis unit is used to perform content analysis based on the text content. The content analysis includes dividing the target document into content categories, including market analysis, competitive analysis, financial analysis, and strategic recommendations.

[0015] The comparison unit responds to the divided content and is used to compare the illegal content with the divided content to obtain the content corresponding to the risk information;

[0016] The calculation unit responds to the comparison unit by calculating the risk level of the risk information based on the content corresponding to the risk information, and formulates risk response measures based on the calculation results.

[0017] In a preferred embodiment of the present invention, the risk level of the risk information is calculated in the calculation unit according to the content corresponding to the risk information. The steps include risk information classification, determination of assessment dimensions, formulation of scoring standards, calculation of risk level, and formulation of risk response measures.

[0018] In a preferred embodiment of the present invention, the risk information classification includes dividing the risk information into legal risks, compliance risks, and reputational risks according to their different natures.

[0019] The determination of assessment dimensions includes determining assessment dimensions based on the nature and scope of impact of risk information. The assessment dimensions include severity, which includes the degree of damage caused, and the scope of impact includes the audience or business scope.

[0020] The aforementioned scoring criteria are developed for each evaluation dimension, including dividing severity into four levels: minor, moderate, severe, and extremely severe, and assigning corresponding scores; the scope of impact is divided based on the number of audience members or business importance, and assigned corresponding scores.

[0021] The risk level calculation includes individual scoring, comprehensive scoring calculation, and risk level classification.

[0022] The individual scoring refers to scoring each assessment dimension individually based on the risk information.

[0023] The comprehensive score calculation includes weighted summation or multiplication of individual scores to obtain a comprehensive score of risk information.

[0024] The risk level classification is based on a comprehensive score, dividing risk information into different risk levels, including low risk, medium risk, high risk and extremely high risk.

[0025] Based on the risk levels identified, corresponding risk response measures were developed, including revising document content, strengthening the review process, and seeking legal advice.

[0026] In a preferred embodiment of the present invention, the data recognition module performs correlation analysis between the data in the image and table and the text description in the text, including the following steps:

[0027] Denoise the image;

[0028] The image is segmented into different regions using image segmentation, and the text in the image is recognized using OCR (Optical Character Recognition).

[0029] Identify the row and column structure of a table;

[0030] Extract data values ​​from the table;

[0031] The extracted data is cleaned to remove duplicate, erroneous, or invalid data;

[0032] Perform word segmentation on the text and label the part of speech of each word;

[0033] Extract keywords or phrases from text;

[0034] Analyze the location of images or tables in the target document, as well as the relevant location information mentioned in the text description, and establish locational relationships;

[0035] The semantic similarity between text descriptions and data in images / tables is calculated using natural language processing. The semantic similarity includes converting the text and text descriptions in images / tables into vector representations and calculating the cosine similarity between the vectors.

[0036] Match keywords in the text with text descriptions in images / tables; if the same keywords or synonyms are found, establish an association.

[0037] Analyze the contextual information of the text, as well as the context of images / tables in the target document, to determine whether the contextual information and context describe the same topic or concept.

[0038] In a preferred embodiment of the present invention, if the context information and context environment do not belong to the description of the same topic or concept, then it is checked whether the text description is consistent with the data in the image / table, and whether there are any contradictions or omissions. At the same time, the features of the text description corresponding to the data in the image / table are collected, and the features include semantic features and word meaning features. When the document is reviewed for compliance in the future, if the semantic features and word meaning features of the text description corresponding to the data in the image / table are the same as the features, then the system determines that the document is non-compliant; otherwise, no determination is made.

[0039] In a preferred embodiment of the present invention, the cosine similarity between vectors is calculated according to the following formula:

[0040] ;

[0041] In the formula, Represents cosine similarity. Representing vectors and The dot product is calculated using the following formula:

[0042] ;

[0043] in, and They are vectors and In the Components in each dimension It is the dimension of the vector;

[0044] and Representing vectors respectively and The modulus is calculated using the following formula:

[0045] ; .

[0046] In a preferred embodiment of the present invention, the method further includes calculating based on the Jaccard similarity coefficient:

[0047] ;in, Represents the Jaccard similarity coefficient;

[0048] In the formula, This represents the first set;

[0049] This represents the second set;

[0050] Represents a set The number of elements in the middle;

[0051] Represents a set The number of elements in the middle;

[0052] Represents a set and set The size of the intersection, where the size is the sum of the sizes of the intersections of the two objects appearing simultaneously. and The number of keywords in;

[0053] Represents a set and set The size of the union of the sets, the size of which appears in or Or the number of all non-repeating keywords in both.

[0054] In a preferred embodiment of the present invention: a threshold value for semantic similarity is preset based on the calculation result, the threshold value is marked as a first threshold value, and the text description corresponding to the data in the image / table is divided into paragraphs. A second threshold value is preset based on the divided paragraphs. When the semantic similarity of the document is calculated in the future, if the second threshold value corresponding to the first divided paragraph is lower than the first threshold value, the system determines that the semantic similarity of the text description corresponding to the data in the image / table is low and determines that the document is a non-compliant document; otherwise, no determination is made.

[0055] On the other hand, the present invention provides a method for applying to the intelligent document compliance review system based on multimodal deep learning as described above, comprising the following steps:

[0056] This invention provides a method for identifying multimodal data in target documents, the multimodal data including text, images, tables, and scanned documents, and for integrating information from the multimodal data, the information integration including correlation analysis between data in images and tables and textual descriptions in the text;

[0057] The integrated target document undergoes a compliance review, which includes checking for non-compliant content and formatting errors.

[0058] The non-compliant content and format errors in the compliance review are feature extracted. The extraction method includes constructing an extraction architecture, setting a reading layer in the extraction architecture, the reading layer including reading the text content of the target document, and identifying risk information in the non-compliant content based on the text content. The risk information includes corporate secrets and transaction amounts.

[0059] Content analysis is performed based on the text content, including dividing the target document into content categories such as market analysis, competitive analysis, financial analysis, and strategic recommendations.

[0060] The violation content is compared with the classified content to obtain the content corresponding to the risk information;

[0061] The risk level of the risk information is calculated based on the content corresponding to the risk information, and risk response measures are formulated based on the calculation results.

[0062] Beneficial effects:

[0063] 1. This invention automatically identifies multimodal data (text, images, tables, and scanned documents) in target documents through a data recognition module and integrates the information, thereby improving the efficiency and accuracy of data processing. Furthermore, it performs a comprehensive compliance check on the integrated documents, including identifying illegal content and formatting errors. This comprehensive auditing capability ensures that the documents comply with relevant laws and standards, reducing legal risks caused by non-compliance.

[0064] 2. The feature extraction module extracts features of illegal content and format errors by constructing an extraction architecture. It can identify risk information such as corporate secrets and transaction amounts. The data fusion and processing module further analyzes and processes these features, calculates the risk level, and formulates corresponding risk response measures. This accurate risk identification and assessment capability helps enterprises respond to potential risks in a timely manner.

[0065] 3. The system formulates scoring criteria and calculates risk levels based on factors such as the nature, severity, and scope of impact of risk information. According to the risk level (low risk, medium risk, high risk, and extremely high risk), the system can formulate corresponding risk response measures, such as modifying document content, strengthening the review process, and providing legal consultation. This risk level classification and response measures help enterprises to allocate resources rationally and effectively deal with various risks.

[0066] 4. The system employs a series of efficient data processing technologies in the data recognition module, including noise reduction, image segmentation, OCR recognition, table row and column structure recognition, and data cleaning. At the same time, it uses natural language processing technology to calculate the semantic similarity between text descriptions and data in images / tables, further improving the accuracy and efficiency of data processing.

[0067] 5. The system flexibly evaluates the semantic similarity between text descriptions and data in images / tables based on calculation methods such as cosine similarity and Jaccard similarity coefficient. At the same time, the system allows preset thresholds for semantic similarity so that it can quickly determine whether a document is compliant when conducting compliance reviews in the future. The system's adaptability and practicality are enhanced by its flexible similarity calculation and threshold setting capabilities.

[0068] 6. By analyzing the consistency between the text context information and the image / table context, the system can check whether the text description is consistent with the data in the image / table, and collect the corresponding semantic and word meaning features. When conducting compliance audits on documents in the future, the system can use these features to quickly determine whether the document is compliant, thereby enhancing the consistency and compliance of the document. Attached Figure Description

[0069] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a schematic diagram of the modular structure of the intelligent document compliance review system based on multimodal deep learning, according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0070] The diagram is labeled as follows: 110 - Data recognition module; 120 - Compliance review module; 130 - Feature extraction module; 140 - Data fusion processing module; 1401 - Analysis unit; 1402 - Comparison unit; 1403 - Calculation unit. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0072] Because existing technologies have difficulty in extracting and aligning features from documents containing multimodal data such as text, images, tables, and scanned documents, they weaken the analysis of the correlation between charts and text in images, thus causing missed detections during the review process.

[0073] Based on this, the present invention proposes an intelligent document compliance review system and method based on multimodal deep learning. It achieves efficient and accurate compliance review by automatically identifying and integrating multimodal data (such as text, images, tables, etc.) in documents. It can accurately identify illegal content and format errors, scientifically assess risk levels and formulate targeted countermeasures. At the same time, it enhances document consistency by using semantic similarity calculation and context analysis, thereby improving the efficiency and accuracy of the review.

[0074] The present solution will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0075] Reference Figures 1 to 2 This is one embodiment of the present invention, which provides an intelligent document compliance review system based on multimodal deep learning, comprising:

[0076] The data recognition module 110 is used to recognize multimodal data in the target document, including text, images, tables and scanned documents, and to integrate information from the multimodal data. The information integration includes correlation analysis between the data in the images and tables and the text descriptions in the text.

[0077] In intelligent document compliance review systems, linking and analyzing data in images and tables with textual descriptions in text is a key step in improving the accuracy of document comprehension and the efficiency of compliance review.

[0078] In this embodiment, the data recognition module is used to identify multimodal data (text, images, tables, and scanned documents) in the target document and to integrate the information.

[0079] Information integration includes correlation analysis between data in images and tables and textual descriptions in text.

[0080] The specific steps include image denoising, image segmentation, OCR recognition, table row and column structure recognition, data cleaning, text segmentation, keyword extraction, positional association analysis, and semantic similarity calculation.

[0081] This invention ensures consistency between image, table, and text data through correlation analysis, reducing error rates and enabling it to handle complex documents containing multiple data types, thus improving the comprehensiveness of the review process.

[0082] The compliance audit module 120 integrates response information and is used to conduct compliance audits on the target documents after information integration. The compliance audit includes violations and formatting errors.

[0083] In this embodiment, the compliance review module responds to the integrated information results and conducts a compliance review on the integrated target document, including identifying illegal content and formatting errors;

[0084] This improves review efficiency while simultaneously checking document content and format, ensuring overall document compliance, and promptly identifying and correcting violations to avoid legal risks arising from non-compliance.

[0085] The feature extraction module 130 is used to extract features of non-compliant content and format errors in compliance audits. The extraction method includes building an extraction architecture, setting a reading layer in the extraction architecture, the reading layer including reading the text content of the target document, and identifying risk information in the non-compliant content based on the text content. The risk information includes corporate secrets and transaction amounts.

[0086] In this embodiment, the feature extraction module extracts features of the non-compliant content and format errors identified in the compliance audit. The extraction method includes building an extraction architecture, setting a reading layer, reading the text content of the target document, and identifying risk information (such as corporate secrets and transaction amounts) in the non-compliant content based on the text content.

[0087] This can accurately identify sensitive information in documents, such as trade secrets and transaction amounts, providing a basis for risk assessment. By extracting features, it can focus on key violations, improving the targeting and effectiveness of the review.

[0088] The data fusion processing module 140 is used to analyze and process the extracted features. The data fusion processing module 140 includes an analysis unit 1401, a comparison unit 1402, and a calculation unit 1403.

[0089] Analysis unit 1401 is used to perform content analysis based on text content. The content analysis includes dividing the target document into content categories, such as market analysis, competitive analysis, financial analysis, and strategic recommendations.

[0090] The comparison unit 1402 responds to the divided content and is used to compare the illegal content with the divided content to obtain the content corresponding to the risk information;

[0091] The calculation unit 1403 is a response comparison unit, used to calculate the risk level of the risk information based on the content corresponding to the risk information, and to formulate risk response measures based on the calculation results;

[0092] Calculating the risk level of risk information is a key step in conducting compliance audits of documents. It helps identify potential compliance risks in the documents and take appropriate management measures accordingly.

[0093] There is usually no single formula for calculating risk levels that is applicable to all scenarios, because severity assessment often depends on specific circumstances, industry standards, laws and regulations, and the organization's internal risk management policies. However, an assessment framework based on key factors can be constructed, and the severity of risk information can be quantified or described through weighted scoring or qualitative description.

[0094] It should be noted that by dividing and comparing the content, the problem can be located more accurately, providing a basis for formulating effective measures. Furthermore, based on the nature and scope of the risk information, scoring criteria are developed and risk levels are calculated to ensure the scientific nature of the assessment.

[0095] In the calculation unit, the risk level of the risk information is calculated based on the content corresponding to the risk information. The steps include risk information classification, determination of assessment dimensions, development of scoring standards, risk level calculation, and development of risk response measures.

[0096] Based on the above, risk information is classified, including dividing risk information into legal risks, compliance risks, and reputational risks according to their different natures.

[0097] Determine the assessment dimensions, including the classification of risk information based on its nature and scope of impact. Assessment dimensions include severity, which includes the degree of damage caused (such as legal penalties, financial losses, reputational damage, etc.), and scope of impact, which includes the audience or business scope.

[0098] This step helps to conduct more accurate risk assessment and management in the future;

[0099] Develop scoring criteria for each assessment dimension, including classifying severity into four levels: minor, moderate, severe, and extremely severe, and assigning corresponding scores; and classifying the scope of impact based on the number of audience members or business importance, and assigning corresponding scores accordingly.

[0100] Risk level calculation includes individual score calculation, comprehensive score calculation, and risk level classification;

[0101] Individual scoring involves assigning a score to each assessment dimension based on risk information.

[0102] For example, a piece of information involving legal risks might be rated as "serious" (4 points) in severity, "moderate" (3 points) in likelihood, and "widespread" (4 points) in scope of impact.

[0103] The comprehensive score calculation includes weighted summation or multiplication of individual scores to obtain a comprehensive risk information score.

[0104] Weighted summation: Different weights are assigned to each assessment dimension based on their importance, and then a weighted sum is calculated. For example, if the severity weight is 0.5, the likelihood weight is 0.3, and the scope of influence weight is 0.2, then the overall score is 40.5 + 30.3 + 4*0.2 = 3.7;

[0105] Product operation: Multiply the individual scores to obtain the comprehensive score. This method is suitable for situations where the evaluation dimensions have significant interrelationships.

[0106] Risk level classification involves dividing risk information into different risk levels based on a comprehensive score, including low risk, medium risk, high risk, and extremely high risk.

[0107] The classification criteria can be formulated according to the actual situation. For example, a comprehensive score between 1 and 2 indicates low risk, between 2 and 3 indicates medium risk, between 3 and 4 indicates high risk, and above 4 indicates extremely high risk.

[0108] Based on the risk levels identified, develop corresponding risk response measures, including revising document content, strengthening the review process, and seeking legal advice.

[0109] In this embodiment, a systematic risk management framework is established by defining classification and evaluation dimensions to improve management efficiency;

[0110] Furthermore, it takes into account various risk factors to ensure the comprehensiveness and accuracy of risk assessment;

[0111] It can also formulate targeted response measures based on different risk types and assessment dimensions to improve the effectiveness of the measures;

[0112] In the data recognition module, the data in images and tables are correlated with the text descriptions in the text. The steps include:

[0113] Image denoising involves removing noise and blurred areas, and enhancing image clarity to more accurately identify image content.

[0114] The image is segmented into different regions (such as text regions, chart regions, graphic regions, etc.) using image segmentation, and the text in the image is recognized by OCR (Optical Character Recognition).

[0115] Identify the row and column structure of a table, such as determining the table header, table body, and other parts;

[0116] Extract data values, such as numbers and text, from a table;

[0117] The extracted data is cleaned to remove duplicate, erroneous, or invalid data;

[0118] Perform word segmentation on the text and label the part of speech (e.g., noun, verb, adjective, etc.) of each word.

[0119] Extract keywords or phrases from text that may describe the content of an image or table;

[0120] Analyze the location of images or tables within the target document, as well as relevant location information mentioned in the text description (such as "..."). Figure 1 Establish positional relationships using data such as "as shown" and "data in Table 2";

[0121] Calculate the semantic similarity between text descriptions and data in images / tables using natural language processing methods; for example, semantic similarity using word embedding models (such as Word2Vec, BERT, etc.) involves converting text and text descriptions in images / tables into vector representations and calculating the cosine similarity between the vectors;

[0122] Match keywords in the text with text descriptions in images / tables; if the same keywords or synonyms are found, establish an association.

[0123] Analyze the contextual information of the text, as well as the context of images / tables in the target document, to determine whether the contextual information and context describe the same topic or concept.

[0124] If the context information and context environment do not belong to the description of the same topic or concept, then check whether the text description is consistent with the data in the image / table, and whether there are any contradictions or omissions. At the same time, collect the features of the text description corresponding to the data in the image / table. The features include semantic features and word meaning features. When the document is reviewed for compliance in the future, if the semantic features and word meaning features of the text description corresponding to the data in the image / table are the same as the features, then the system determines that the document is non-compliant; otherwise, it does not make a determination.

[0125] In this embodiment, context analysis is used to ensure the consistency and logic of the content in each part of the document;

[0126] In terms of feature collection, semantic and word meaning features are automatically collected and stored to provide a reference for future review.

[0127] Furthermore, by using contextual analysis and feature collection, the future review process can be simplified and review efficiency improved.

[0128] The cosine similarity between vectors is calculated using the following formula:

[0129] ;

[0130] In the formula, Represents cosine similarity. Representing vectors and The dot product is calculated using the following formula:

[0131] ;

[0132] in, and They are vectors and In the Components in each dimension It is the dimension of the vector;

[0133] and Representing vectors respectively and The modulus is calculated using the following formula:

[0134] ; ;

[0135] It should be noted that the vector dot product is the sum of the products of the projected lengths of two vectors in the same direction;

[0136] If two vectors have large components in the same direction, their dot product will also be large, indicating that they are similar in direction.

[0137] The vector modulus is a measure of the length of a vector, representing the square root of the sum of the squares of the vector's components in each dimension.

[0138] The vector modulus is used to normalize the dot product result, so that the cosine similarity value ranges between [−1, 1].

[0139] Cosine similarity is calculated by dividing the dot product by the product of the magnitudes of the two vectors. When two vectors are completely identical, the cosine similarity is 1, indicating that they are completely similar.

[0140] When two vectors are completely opposite, the cosine similarity is -1, indicating that they are completely dissimilar.

[0141] When two vectors are perpendicular, the cosine similarity is 0, indicating that they have no similarity.

[0142] In a document compliance review system, text descriptions and textual descriptions in images / tables can be converted into vector representations (e.g., using word embedding models), and then the cosine similarity between them can be calculated. A high cosine similarity indicates that the text descriptions and the data in the images / tables are semantically similar and may be related. Conversely, a low cosine similarity indicates that they are semantically different and may not be directly related.

[0143] This method allows the system to automatically identify and associate multimodal data in documents, improving the accuracy of document understanding and the efficiency of compliance review.

[0144] It also includes the results calculated based on the Jaccard similarity coefficient:

[0145] ;in, Represents the Jaccard similarity coefficient;

[0146] In the formula, This refers to the first set, such as the set of keywords extracted from the text description.

[0147] This refers to the second set, such as the set of keywords extracted from image annotations or table content.

[0148] Represents a set The number of elements (i.e., the number of keywords).

[0149] Represents a set The number of elements in the middle;

[0150] Represents a set and set The size of the intersection, which is the sum of the sizes of the elements that appear simultaneously. and The number of keywords in;

[0151] Represents a set and set The size of the union of the sets is the size of the set that appears in the union of the sets. or Or the number of all non-repeating keywords in both;

[0152] It should be noted that for the intersection part ( The number of elements shared between two sets reflects their similarity. The more common keywords, the more similar the two sets are in content.

[0153] For the union part ( The value of , as the denominator, is used to normalize the size of the intersection, ensuring that the similarity coefficient ranges between [0, 1]. The union considers all elements appearing in both sets, avoiding bias caused by differences in set sizes;

[0154] For similarity coefficient The value is calculated by the ratio of the size of the intersection to the size of the union. The closer the value is to 1, the more similar the two sets are; the closer the value is to 0, the less similar the two sets are.

[0155] In document compliance auditing, the Jaccard similarity coefficient can be used to compare the similarity between text descriptions and image / table content at the keyword level.

[0156] For example, extracting a set of keywords from a text description. = {"Sales," "Growth," "2025"}, extract the keyword set from the corresponding table. = {"Sales Revenue", "2025", "Quarterly Data"}.

[0157] Calculate the intersection = {"Sales Revenue", "2025"}, size 2;

[0158] Calculate the union = {"Sales", "Growth", "2025", "Quarterly Data"}, size 4.

[0159] Therefore, the Jaccard similarity coefficient is =2 / 4=0.5, indicating that the text description and the table content have a certain degree of similarity at the keyword level;

[0160] Based on the above, a threshold value for semantic similarity is preset according to the calculation results. This threshold value is marked as the first threshold value. The text descriptions corresponding to the data in the images / tables are divided into paragraphs. A second threshold value is preset based on the paragraphs. When the semantic similarity of the document is calculated in the future, if the second threshold value corresponding to the first paragraph is lower than the first threshold value, the system determines that the semantic similarity with the text descriptions corresponding to the data in the images / tables is low and the document is deemed non-compliant. Otherwise, no determination is made.

[0161] Quickly assess the consistency between text descriptions and image / table data using vector representation and similarity calculation methods;

[0162] It allows for the preset of critical values ​​based on actual needs, thereby improving the system's adaptability and flexibility;

[0163] Furthermore, similarity assessment and threshold determination ensure the consistency and compliance of document content.

[0164] As can be seen from the above, this application achieves efficient and accurate compliance review by automatically identifying and integrating multimodal data (such as text, images, tables, etc.) in documents. It can accurately identify illegal content and format errors, scientifically assess risk levels and formulate targeted countermeasures. At the same time, it enhances document consistency by using semantic similarity calculation and context analysis, thereby improving review efficiency and accuracy.

[0165] This embodiment, in conjunction with the aforementioned intelligent document compliance review system based on multimodal deep learning, also proposes a working method for applying this system, the steps of which are as follows:

[0166] S10: Identify multimodal data in the target document, including text, images, tables, and scanned documents, and integrate the information from the multimodal data. Information integration includes correlation analysis between the data in the images and tables and the text descriptions in the text.

[0167] S20: Conduct a compliance review of the target document after information integration. The compliance review includes violations and formatting errors.

[0168] S30: Feature extraction of non-compliant content and format errors in compliance audits. Extraction methods include building an extraction architecture, setting a reading layer in the extraction architecture, the reading layer including reading the text content of the target document, and identifying risk information in the non-compliant content based on the text content. Risk information includes corporate secrets and transaction amounts.

[0169] S40: Conduct content analysis based on the text content. Content analysis includes dividing the target document into content categories, such as market analysis, competitive analysis, financial analysis, and strategic recommendations.

[0170] S50: Compare the violation content with the classified content to obtain the content corresponding to the risk information;

[0171] S60: Calculate the risk level of the risk information based on the content corresponding to the risk information, and formulate risk response measures based on the calculation results.

[0172] In summary, this invention achieves efficient and accurate compliance review by automatically identifying and integrating multimodal data (such as text, images, tables, etc.) in documents. It can accurately identify non-compliant content and formatting errors, scientifically assess risk levels and formulate targeted countermeasures, and enhance document consistency by utilizing semantic similarity calculation and context analysis. This significantly improves review efficiency and accuracy, reduces compliance risks and labor costs, and provides enterprises with a comprehensive and intelligent document compliance management solution.

[0173] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An intelligent document compliance review system based on multimodal deep learning, characterized in that, include: A data recognition module is used to recognize multimodal data in a target document, including text, images, tables, and scanned documents, and to integrate information from the multimodal data. The information integration includes correlation analysis between data in images and tables and textual descriptions in the text. A compliance review module, which responds to the information integration, is used to conduct a compliance review on the target document after information integration. The compliance review includes violations and formatting errors. The feature extraction module is used to extract features from the non-compliant content and format errors in the compliance review. The extraction method includes constructing an extraction architecture and setting a reading layer in the extraction architecture. The reading layer includes reading the text content of the target document and identifying risk information in the non-compliant content based on the text content. The risk information includes corporate secrets and transaction amounts. The data fusion processing module is used to analyze and process the extracted features. The data fusion processing module includes an analysis unit, a comparison unit, and a calculation unit. The analysis unit is used to perform content analysis based on the text content. The content analysis includes dividing the target document into content categories, including market analysis, competitive analysis, financial analysis, and strategic recommendations. The comparison unit responds to the divided content, and is used to compare the violation content with the divided content to obtain the content corresponding to the risk information; The calculation unit responds to the comparison unit to calculate the risk level of the risk information based on the content corresponding to the risk information, and formulates risk response measures based on the calculation results; In the data recognition module, the data in the images and tables are correlated with the text descriptions in the text. The steps include: Denoise the image; The image is segmented into different regions using image segmentation, and the text in the image is recognized using OCR (Optical Character Recognition). Identify the row and column structure of a table; Extract data values ​​from the table; The extracted data is cleaned to remove duplicate, erroneous, or invalid data; Perform word segmentation on the text and label the part of speech of each word; Extract keywords or phrases from text; Analyze the location of images or tables in the target document, as well as the relevant location information mentioned in the text description, and establish locational relationships; The semantic similarity between text descriptions and data in images / tables is calculated using natural language processing. The semantic similarity includes converting the text and text descriptions in images / tables into vector representations and calculating the cosine similarity between the vectors. Match keywords in the text with text descriptions in images / tables; if the same keywords or synonyms are found, establish an association. Analyze the contextual information of the text, as well as the contextual environment of the images / tables in the target document, to determine whether the contextual information and contextual environment describe the same topic or concept; If the context information and context environment do not belong to the description of the same topic or concept, then check whether the text description is consistent with the data in the image / table, and whether there are any contradictions or omissions. At the same time, collect the features of the text description corresponding to the data in the image / table. The features include semantic features and word meaning features. When the document is reviewed for compliance in the future, if the semantic features and word meaning features of the text description corresponding to the data in the image / table are the same as the features, then the system determines that the document is non-compliant; otherwise, it does not make a determination. The cosine similarity between vectors is calculated based on the Jaccard similarity coefficient. A threshold value for semantic similarity is preset based on the calculation result and marked as the first threshold value. The text descriptions corresponding to the data in the image / table are divided into paragraphs, and a second threshold value is preset based on the divided paragraphs. When the semantic similarity of the document is calculated in the future, if the second threshold value corresponding to the first divided paragraph is lower than the first threshold value, the system determines that the semantic similarity with the text descriptions corresponding to the data in the image / table is low and determines that the document is a non-compliant document; otherwise, no determination is made.

2. The intelligent document compliance review system based on multimodal deep learning as described in claim 1, characterized in that, In the calculation unit, the risk level of the risk information is calculated based on the content corresponding to the risk information. The steps include risk information classification, determination of assessment dimensions, formulation of scoring standards, calculation of risk level, and formulation of risk response measures.

3. The intelligent document compliance review system based on multimodal deep learning as described in claim 2, characterized in that, The risk information classification includes dividing the risk information into legal risks, compliance risks, and reputational risks according to their different natures; The determination of assessment dimensions includes determining assessment dimensions based on the nature and scope of impact of risk information. The assessment dimensions include severity, which includes the degree of damage caused, and the scope of impact includes the audience or business scope. The aforementioned scoring criteria are developed for each evaluation dimension, including dividing severity into four levels: minor, moderate, severe, and extremely severe, and assigning corresponding scores; the scope of impact is divided based on the number of audience members or business importance, and assigned corresponding scores. The risk level calculation includes individual scoring, comprehensive scoring calculation, and risk level classification. The individual scoring refers to scoring each assessment dimension individually based on the risk information. The comprehensive score calculation includes weighted summation or multiplication of individual scores to obtain a comprehensive score of risk information. The risk level classification is based on a comprehensive score, dividing risk information into different risk levels, including low risk, medium risk, high risk, and extremely high risk. Based on the risk levels identified, corresponding risk response measures were developed, including revising document content, strengthening the review process, and seeking legal advice.

4. The intelligent document compliance review system based on multimodal deep learning as described in claim 1, characterized in that, The cosine similarity between vectors is calculated using the following formula: ； In the formula, Represents cosine similarity. Representing vectors and The dot product is calculated using the following formula: ； in, and They are vectors and In the Components in each dimension It is the dimension of the vector; and Representing vectors respectively and The modulus is calculated using the following formula: ； 。 5. The intelligent document compliance review system based on multimodal deep learning as described in claim 1, characterized in that, The Jaccard similarity coefficient is calculated as follows: ;in, Represents the Jaccard similarity coefficient; In the formula, This represents the first set; This represents the second set; Represents a set The number of elements in the middle; Represents a set The number of elements in the middle; Represents a set and set The size of the intersection, where the size is the sum of the sizes of the intersections of the two objects appearing simultaneously. and The number of keywords in; Represents a set and set The size of the union of the sets, the size of which appears in or Or the number of all non-repeating keywords in both.

6. The method applied to the intelligent document compliance review system based on multimodal deep learning as described in claim 1, characterized in that, Includes the following steps: Identify multimodal data in a target document, the multimodal data including text, images, tables, and scanned documents, and perform information integration on the multimodal data, the information integration including correlation analysis between data in images and tables and textual descriptions in the text; The integrated target document undergoes a compliance review, which includes checking for non-compliant content and formatting errors. The non-compliant content and format errors in the compliance review are feature extracted. The extraction method includes constructing an extraction architecture, setting a reading layer in the extraction architecture, the reading layer including reading the text content of the target document, and identifying risk information in the non-compliant content based on the text content. The risk information includes corporate secrets and transaction amounts. Content analysis is performed based on the text content, including dividing the target document into content categories such as market analysis, competitive analysis, financial analysis, and strategic recommendations. The violation content is compared with the classified content to obtain the content corresponding to the risk information; The risk level of the risk information is calculated based on the content corresponding to the risk information, and risk response measures are formulated based on the calculation results.

Images