Multi-modal content compliance auditing method and system

Abstract

The invention provides a compliance auditing method and system for multi-modal content. The method comprises the following steps: performing feature extraction on unstructured to-be-audited multi-modal content to obtain a structured feature vector; performing image-text semantic association on the text semantic feature vector and the image visual feature vector to obtain a fusion feature vector involving image-text semantic contradiction; constructing a domain knowledge graph based on the compliance guidance data of the domain to which the to-be-audited multi-modal content belongs; inputting the fusion feature vector into a domain knowledge graph, and performing compliance rule retrieval by adopting a sub-graph matching algorithm to determine a violation type corresponding to the fusion feature vector and a violated compliance term; and generating an interactive compliance audit report. The system comprises functional modules for realizing the steps in a one-to-one correspondence manner. According to the technical scheme, the problem that cross-modal semantic analysis of an existing multi-modal content compliance auditing method is not accurate can be solved.

Description

Technical Field

[0001] The present invention belongs to the field of information processing and compliance review, and more specifically, relates to a compliance review method and system for multimodal content. Background Art

[0002] In the field of multimodal content compliance review, existing technologies have significant deficiencies for promotional materials and product descriptions containing text and images. In terms of cross-modal semantic association analysis, existing multimodal content compliance review methods usually process text and image data independently, lacking an effective cross-modal information fusion mechanism. This makes it difficult to accurately capture the deep connection between text semantics and image content, resulting in insufficient accuracy in image-text consistency detection. For example, in the compliance review of advertising content, the implicit product function hints in the image may conflict with the text description, and existing multimodal content compliance review methods have difficulty accurately identifying such semantic conflicts. Summary of the Invention

[0003] The purpose of the present invention is to solve the problem of inaccurate cross-modal semantic analysis in existing multimodal content compliance review methods.

[0004] According to a first aspect of the present invention, a method for compliance review of multimodal content is provided, the method comprising the following steps: Extracting features from unstructured multimodal content to be reviewed to obtain a structured feature vector, wherein the structured feature vector includes a text semantic feature vector and an image visual feature vector; Performing image-text semantic association on the text semantic feature vector and the image visual feature vector to obtain a fusion feature vector involving image-text semantic contradiction; Constructing a domain knowledge graph based on compliance guidance data of the domain to which the multimodal content to be reviewed belongs; Inputting the fused feature vector into the domain knowledge graph and performing compliance rule retrieval using a subgraph matching algorithm to determine the violation type and the violated compliance clause corresponding to the fused feature vector; Generate interactive compliance audit reports.

[0005] Optionally, the step of extracting features from the unstructured multimodal content to be reviewed to obtain a structured feature vector includes: Performing visual feature extraction on the image portion of the multimodal content to be reviewed to obtain an image visual feature vector, wherein the image visual feature vector includes image region features, and the image region features include text, graphics, color, and layout; Natural language processing is performed on the text portion of the multimodal content to be reviewed to obtain a text semantic feature vector.

[0006] Optionally, the step of performing text-image semantic association on the text semantic feature vector and the image visual feature vector to obtain a fused feature vector involving a text-image semantic contradiction includes: Arrange all the extracted text semantic feature vectors into a text semantic feature vector sequence; Arrange all the extracted image visual feature vectors into an image visual feature vector sequence; Dimensionally aligning the text semantic feature vector sequence and the image visual feature vector sequence; Calculate the cosine similarity between each text semantic feature vector and each image visual feature vector to generate the first attention matrix; Calculate the cosine similarity between each image visual feature vector and each text semantic feature vector to generate the second attention matrix; According to the first attention matrix and the second attention matrix, cross-modal attention information is fused by weighted summation to determine whether there is a fused feature vector involving semantic contradictions between the image and text. If so, the fused feature vector is output.

[0007] Optionally, the step of constructing a domain knowledge graph based on the compliance guidance data of the domain to which the multimodal content to be reviewed belongs includes: Obtaining compliance guidance data for the field to which the multimodal content to be reviewed belongs, wherein the compliance guidance data includes industry regulatory policies, business rules, and compliance cases; Information extraction and entity relationship modeling are performed on the compliance guidance data to obtain a domain knowledge graph.

[0008] Optionally, the step of inputting the fused feature vector into the domain knowledge graph and using a subgraph matching algorithm to perform compliance rule retrieval to determine the violation type corresponding to the fused feature vector and the violated compliance clause includes: Locating corresponding nodes in the domain knowledge graph according to entities in the multimodal content to be reviewed; Based on the fused feature vector, the violation type nodes in the domain knowledge graph are matched and associated with specific compliance clauses and penalty basis.

[0009] According to a second aspect of the present invention, a compliance review system for multimodal content is provided, the system comprising the following functional modules: A data preprocessing module is used to extract features from unstructured multimodal content to be reviewed to obtain a structured feature vector, wherein the structured feature vector includes a text semantic feature vector and an image visual feature vector; a cross-modal semantic association module, configured to perform image-text semantic association on the text semantic feature vector and the image visual feature vector to obtain a fused feature vector involving image-text semantic contradictions; A knowledge graph construction module, configured to construct a domain knowledge graph based on compliance guidance data of the domain to which the multimodal content to be reviewed belongs; A knowledge graph matching module, configured to input the fused feature vector into the domain knowledge graph and perform compliance rule retrieval using a subgraph matching algorithm to determine the violation type and the violated compliance clause corresponding to the fused feature vector; Report generation module, used to generate interactive compliance audit reports.

[0010] Optionally, the data preprocessing module is further configured to: Performing visual feature extraction on the image portion of the multimodal content to be reviewed to obtain an image visual feature vector, wherein the image visual feature vector includes image region features, and the image region features include text, graphics, color, and layout; Natural language processing is performed on the text portion of the multimodal content to be reviewed to obtain a text semantic feature vector.

[0011] Optionally, the cross-modal semantic association module is further configured to: Arrange all the extracted text semantic feature vectors into a text semantic feature vector sequence; Arrange all the extracted image visual feature vectors into an image visual feature vector sequence; Dimensionally aligning the text semantic feature vector sequence and the image visual feature vector sequence; Calculate the cosine similarity between each text semantic feature vector and each image visual feature vector to generate the first attention matrix; Calculate the cosine similarity between each image visual feature vector and each text semantic feature vector to generate the second attention matrix; According to the first attention matrix and the second attention matrix, cross-modal attention information is fused by weighted summation to determine whether there is a fused feature vector involving semantic contradictions between the image and text. If so, the fused feature vector is output.

[0012] Optionally, the knowledge graph construction module is further configured to: Obtaining compliance guidance data for the field to which the multimodal content to be reviewed belongs, wherein the compliance guidance data includes industry regulatory policies, business rules, and compliance cases; Information extraction and entity relationship modeling are performed on the compliance guidance data to obtain a domain knowledge graph.

[0013] Optionally, the knowledge graph matching module is further configured to: Locating corresponding nodes in the domain knowledge graph according to entities in the multimodal content to be reviewed; Based on the fused feature vector, the violation type nodes in the domain knowledge graph are matched and associated with specific compliance clauses and penalty basis.

[0014] The beneficial effects of the present invention are: The compliance audit method for multimodal content of the present invention performs text-image semantic association on text semantic feature vectors and image visual feature vectors extracted based on the multimodal content to be audited, and obtains a fused feature vector involving semantic contradictions between text and image, and then determines the violation type and the violated compliance clause corresponding to the fused feature vector based on the fused feature vector and in combination with the domain knowledge graph, thereby achieving compliance audit of multimodal content. The compliance audit method for multimodal content of the present invention, by adopting a cross-modal information fusion mechanism, can accurately capture the deep association between text semantics and image content, so that the accuracy of text-image consistency detection is significantly improved, and effectively solves the problem of inaccurate cross-modal semantic analysis in existing multimodal content compliance audit methods.

[0015] The compliance review system for multimodal content of the present invention and the compliance review method for multimodal content described above belong to a general inventive concept, and have at least the same beneficial effects as the compliance review method for multimodal content described above, and the beneficial effects will not be repeated here.

[0016] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present invention may be better understood by referring to the following description taken in conjunction with the accompanying drawings, in which the same or similar reference numerals are used throughout the drawings to designate the same or similar parts.

[0018] Figure 1 A flowchart illustrating an implementation method of a compliance review method for multimodal content according to an embodiment of the present invention is shown; Figure 2 A principle block diagram of a compliance review system for multimodal content according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to more fully understand the technical solution of the present invention, the exemplary embodiments of the present invention will be described in more comprehensive and detailed in conjunction with the accompanying drawings below. Obviously, the one or more embodiments of the present invention described below are only one or more of the specific ways of implementing the technical solution of the present invention, and are not exhaustive. It should be understood that other ways belonging to a general inventive concept can be used to implement the technical solution of the present invention, and should not be limited by the exemplary embodiments described. Based on one or more embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present invention.

[0020] Example: Figure 1 The following is a flowchart showing the implementation of the compliance audit method for multimodal content according to an embodiment of the present invention. Figure 1 The compliance review method for multimodal content in an embodiment of the present invention includes the following steps: Step S100: extracting features from the unstructured multimodal content to be reviewed to obtain a structured feature vector, where the structured feature vector includes a text semantic feature vector and an image visual feature vector; Step S200: Performing image-text semantic association on the text semantic feature vector and the image visual feature vector to obtain a fusion feature vector involving image-text semantic contradictions; Step S300: constructing a domain knowledge graph based on compliance guidance data of the domain to which the multimodal content to be reviewed belongs; Step S400: Input the fused feature vector into the domain knowledge graph and use a subgraph matching algorithm to perform compliance rule retrieval to determine the violation type and the violated compliance clause corresponding to the fused feature vector; Step S500: Generate an interactive compliance audit report.

[0021] Furthermore, in the embodiment of the present invention, the step S100 of extracting features from the unstructured multimodal content to be reviewed to obtain a structured feature vector includes: Performing visual feature extraction on the image portion of the multimodal content to be reviewed to obtain an image visual feature vector, wherein the image visual feature vector includes image region features, and the image region features include text, graphics, color, and layout; Perform natural language processing on the text portion of the multimodal content to be reviewed to obtain a text semantic feature vector.

[0022] In an embodiment of the present invention, the content to be reviewed is multimodal content containing an image portion and a text portion, such as a promotional poster containing text and an image, and a product manual containing content. In an embodiment of the present invention, step S100 is used to achieve standardized processing of the multimodal content to be reviewed, so as to convert the unstructured multimodal data into a structured feature vector, thereby providing a unified input format for subsequent cross-modal analysis. Specifically, for image data, computer vision technology (such as convolutional neural network CNN) is used to extract visual features, including regional features such as text, graphics, color, and layout in the image; for text data, natural language processing (NLP) technology is used to perform operations such as word segmentation, part-of-speech tagging, and entity recognition to extract semantic features (such as keywords, topic vectors, semantic relationships, etc.).

[0023] Furthermore, in an embodiment of the present invention, the text semantic feature vector and the image visual feature vector are associated with each other to obtain a fused feature vector involving a semantic contradiction between the text and the image in step S200. Arrange all the extracted text semantic feature vectors into a text semantic feature vector sequence; Arrange all the extracted image visual feature vectors into an image visual feature vector sequence; Align the text semantic feature vector sequence and the image visual feature vector sequence in dimension; Calculate the cosine similarity between each text semantic feature vector and each image visual feature vector to generate the first attention matrix; Calculate the cosine similarity between each image visual feature vector and each text semantic feature vector to generate the second attention matrix; According to the first attention matrix and the second attention matrix, the cross-modal attention information is fused by weighted summation to determine whether there is a fused feature vector involving semantic contradictions between the image and text. If so, the fused feature vector is output.

[0024] In step S200 of the embodiment of the present invention, a cross-modal attention model is constructed, which takes the text semantic feature vector and the image visual feature vector as input and calculates the semantic association between modalities through a bidirectional attention mechanism. Specifically, the model includes the following steps: Feature alignment: Dimensionally align text feature sequences (such as word vector sequences) with image region feature sequences (such as image block features based on object detection); Attention calculation: For each word vector in the text, calculate its cosine similarity with all image region features, generate a word-region attention matrix, and locate the key image region corresponding to the word; at the same time, for each region feature in the image, calculate its similarity with all word vectors, generate a region-word attention matrix, and locate the key text semantics corresponding to the region; Correlation Fusion: This method fuses cross-modal attention information through a weighted summation approach to generate a fused feature vector that contains semantic associations between the image and text. For example, in an ad review, if the text mentions "safe and non-toxic," the cross-modal attention mechanism can locate the corresponding safety certification icon in the image and verify the semantic consistency between the image and text. Its purpose: It enables bidirectional mapping between text and image semantics, accurately capturing potential associations in multimodal data and addressing the shortcomings of traditional independent analysis.

[0025] Specifically, in an embodiment of the present invention, the cross-modal attention model can be replaced by other models that can achieve the same function. For example, it can be replaced by a Transformer-based cross-modal pre-training model (such as ViLT, FLAVA) for feature fusion to improve the semantic association analysis capabilities in complex scenarios.

[0026] Furthermore, in an embodiment of the present invention, the step S300 of constructing a domain knowledge graph based on compliance guidance data of the domain to which the multimodal content to be reviewed belongs includes: Obtain compliance guidance data for the field to which the multimodal content to be reviewed belongs, including industry regulatory policies, business rules, and compliance cases; Information extraction and entity relationship modeling are performed on compliance guidance data to obtain a domain knowledge graph.

[0027] Specifically, in the existing technology, in terms of compliance rule adaptation capabilities, most systems rely on static rule bases, lack dynamic updates and intelligent matching mechanisms based on domain knowledge systems, and are unable to adapt to the rapid changes in compliance requirements in different industries and scenarios. When regulatory policies, industry specifications, or business rules are updated, static rule bases are difficult to synchronize in real time, resulting in lags and limitations in compliance audits. To this end, embodiments of the present invention dynamically update domain knowledge graphs and use dynamic knowledge graphs to achieve intelligent matching of compliance rules.

[0028] Specifically, in an embodiment of the present invention, when constructing a domain knowledge graph, automated knowledge extraction technology (such as remote supervision and weakly supervised learning) can be introduced to automatically extract entity relationships from unstructured documents (such as industry specifications in PDF format), thereby reducing manual construction costs.

[0029] Furthermore, in an embodiment of the present invention, the step S400 of inputting the fused feature vector into the domain knowledge graph and using a subgraph matching algorithm to perform compliance rule retrieval to determine the violation type corresponding to the fused feature vector and the violated compliance clause includes: Locate the corresponding nodes in the domain knowledge graph based on the entities in the multimodal content to be reviewed; Based on the fused feature vector, the violation type nodes in the domain knowledge graph are matched and associated with specific compliance clauses and penalty basis.

[0030] Specifically, in an embodiment of the present invention, cross-modal fusion features are input into the knowledge graph, and a subgraph matching algorithm (such as path search and graph embedding matching) is used to retrieve compliance rules: the corresponding nodes in the graph are located according to the entities in the multimodal data (such as product names and promotional keywords), and then the violation type nodes in the graph are matched based on the violation risk features obtained based on the cross-modal analysis (such as contradictions between images and texts), and are associated with specific compliance clauses and penalty basis.

[0031] Specifically, when it comes to presenting audit results, reports generated by existing technologies often only provide a conclusion on compliance or non-compliance, lacking visual positioning and detailed explanation of evidence of violations. For example, when pointing out violations in a product manual, it is impossible to visually display which part of the text and which area of ​​the image are associated with the violation, and it is also difficult to explain the specific rules and clauses on which the violations are based. This makes it difficult for users to understand the audit logic and make targeted corrections. In step S500 of the embodiment of the present invention, visual evidence, rule explanations, and audit conclusions are integrated into an interactive compliance audit report. The compliance audit report allows users to click to locate the violation point and view the detailed rule basis; based on the rule matching results and the audit report, a multi-dimensional decision is generated, including compliance decisions (approved, rejected, recommended modification), risk levels (low, medium, and high risk labels), and rectification suggestions (based on similar cases in the knowledge graph, providing specific modification directions, such as supplementing text descriptions and replacing image elements). The decision results and reports are stored synchronously in the database to support historical record queries and statistical analysis. The compliance audit report of the embodiment of the present invention supports 3D image annotation (such as locating illegal areas on a product 3D model) or dynamic video frame annotation (such as locating illegal scenes in an advertising video) to expand compatibility with multimodal data.

[0032] The following is a more detailed description of the multimodal content compliance review method according to an embodiment of the present invention based on specific examples: Specific example 1 (advertising poster compliance review): S1 data preprocessing: Image: Object detection is used to extract regional features such as product images, certification marks, and background patterns from posters, and OCR is used to recognize text content in images. Text: Segment the poster text to extract keywords such as "efficiency," "safety," and "limited-time offer," along with their corresponding sentiment and semantic categories.

[0033] S2 cross-modal semantic association analysis: Calculate the attention weight of the "safety" keyword and the "quality inspection certification icon" area in the image. If the weight value is lower than the threshold (indicating a weak association between the image and text) and there is a risk warning color (such as a red exclamation mark) in the image, it is determined that the image and text are semantically contradictory.

[0034] S3 compliance rule dynamic matching: By inputting features such as "inconsistencies between images and text" and "false propaganda" into the advertising supervision knowledge graph, we can match them with the clause in the Advertising Law that "it shall not contain false or misleading content", triggering a violation warning.

[0035] S4 interpretability report generation: The keyword "safety" is highlighted in the text, and the risk icon area is marked in the image. The report states "The image risk element is inconsistent with the text safety description, violating XX clause" and provides modification suggestions for similar violation cases (such as replacing the risk icon and supplementing the safety certification instructions).

[0036] Specific example 2 (e-commerce product detail page review): S1 data preprocessing: Image: Extract regional features such as parameter tables, material schematics, and user review screenshots from product detail pages; Text: Parse product titles, function descriptions, and specifications to extract key attributes such as "wear-resistant" and "IP68 waterproof rating."

[0037] S2 cross-modal semantic association analysis: Check whether the "Waterproof Grade IP68" text is consistent with the parameter annotations on the waterproof test diagram in the image. Use the attention mechanism to locate the mapping relationship between text keywords and image parameter areas. If there is a difference in the values, it is determined that the image and text are inconsistent.

[0038] S3 compliance rule dynamic matching: In the e-commerce compliance knowledge graph, "inconsistent product parameters" corresponds to the "false product information" violation type, which is related to the provision in the "E-Commerce Law" that "product information should be true and accurate."

[0039] S4 interpretability report generation: Marks the conflicting values ​​of "IP68" in the text and the image parameter area, generates a visual comparison chart, explains the violation basis in the report, and recommends correcting the image or text parameters to ensure consistency between the text and the image.

[0040] The compliance review method for multimodal content in an embodiment of the present invention realizes deep semantic correlation analysis of images and texts through a cross-modal attention mechanism, uses a dynamic knowledge graph to realize intelligent matching of compliance rules, and combines visualization technology to generate explainable review reports, thereby improving the accuracy, flexibility and transparency of multimodal content compliance review, and is suitable for compliance review needs in multiple industries and scenarios.

[0041] Accordingly, based on the compliance review method for multimodal content in an embodiment of the present invention, an embodiment of the present invention also proposes a compliance review system for multimodal content.

[0042] Figure 2 The following is a block diagram showing the principle of a compliance review system for multimodal content according to an embodiment of the present invention. Figure 2 The multimodal content compliance review system of the embodiment of the present invention includes the following functional modules: The data preprocessing module is used to extract features from unstructured multimodal content to be reviewed to obtain structured feature vectors. The structured feature vectors include text semantic feature vectors and image visual feature vectors. The cross-modal semantic association module is used to perform image-text semantic association on the text semantic feature vector and the image visual feature vector to obtain a fused feature vector involving semantic contradictions between the image and text; A knowledge graph construction module is used to construct a domain knowledge graph based on the compliance guidance data of the domain to which the multimodal content to be reviewed belongs; The knowledge graph matching module is used to input the fused feature vector into the domain knowledge graph and use the subgraph matching algorithm to perform compliance rule retrieval to determine the violation type corresponding to the fused feature vector and the violated compliance clause; Report generation module, used to generate interactive compliance audit reports.

[0043] Furthermore, in an embodiment of the present invention, the data preprocessing module is further configured to: Performing visual feature extraction on the image portion of the multimodal content to be reviewed to obtain an image visual feature vector, wherein the image visual feature vector includes image region features, and the image region features include text, graphics, color, and layout; Perform natural language processing on the text portion of the multimodal content to be reviewed to obtain a text semantic feature vector.

[0044] Furthermore, in an embodiment of the present invention, the cross-modal semantic association module is further configured to: Arrange all the extracted text semantic feature vectors into a text semantic feature vector sequence; Arrange all the extracted image visual feature vectors into an image visual feature vector sequence; Align the text semantic feature vector sequence and the image visual feature vector sequence in dimension; Calculate the cosine similarity between each text semantic feature vector and each image visual feature vector to generate the first attention matrix; Calculate the cosine similarity between each image visual feature vector and each text semantic feature vector to generate the second attention matrix; According to the first attention matrix and the second attention matrix, the cross-modal attention information is fused by weighted summation to determine whether there is a fused feature vector involving semantic contradictions between the image and text. If so, the fused feature vector is output.

[0045] Furthermore, in an embodiment of the present invention, the knowledge graph construction module is further used to: Obtain compliance guidance data for the field to which the multimodal content to be reviewed belongs, including industry regulatory policies, business rules, and compliance cases; Information extraction and entity relationship modeling are performed on compliance guidance data to obtain a domain knowledge graph.

[0046] Furthermore, in an embodiment of the present invention, the knowledge graph matching module is further used to: Locate the corresponding nodes in the domain knowledge graph based on the entities in the multimodal content to be reviewed; Based on the fusion feature vector, the violation type nodes in the domain knowledge graph are matched and associated with specific compliance clauses and penalty basis.

[0047] The multimodal content compliance review system of this embodiment proposes a closed-loop multimodal data processing architecture, creatively integrating a text-image cross-modal attention mechanism, dynamic knowledge graph matching, and visual report generation technology to form an end-to-end processing flow of "feature extraction - semantic association - rule matching - evidence presentation." Each module operates independently, but rather achieves data interaction and functional synergy through cross-modal fusion of feature vectors, rule matching results, and evidence location information. For example, cross-modal analysis results directly serve as input features for knowledge graph matching, and rule matching results drive the presentation logic of evidence visualization.

[0048] The multimodal content compliance review system of the embodiment of the present invention proposes a three-layer mapping mechanism of semantics, rules, and evidence: Semantic layer: A bidirectional cross-modal attention mechanism is used to establish fine-grained semantic associations between text and images (word-level to pixel-level mapping), solving the problem of modality separation in traditional methods. Rule layer: Dynamically adapt compliance rules based on domain knowledge graphs, automatically mapping semantic contradictions (such as inconsistencies between images and text) identified through cross-modal analysis to specific regulatory clauses, overcoming the limitations of static rule bases. Evidence layer: Use cross-modal attention weights to directly locate violation evidence (text fragments + image areas) and associate them with rule explanations in the knowledge graph to form a traceable evidence chain loop.

[0049] The multimodal content compliance review system of the embodiment of the present invention adopts an industry generalization capability design: through modular architecture design (such as replaceable domain knowledge graphs and extensible cross-modal model interfaces), the system can quickly adapt to different industries such as finance, e-commerce, advertising, and content platforms without major modifications, realizing the generalization capability of "universalization of technical core + plug-in of industry rules".

[0050] Although one or more embodiments of the present invention have been described above, it should be understood by those skilled in the art that the present invention can be implemented in any other form without departing from its spirit and scope. Therefore, the embodiments described above are illustrative and not restrictive, and many modifications and substitutions will be apparent to those skilled in the art without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A compliance review method for multimodal content, characterized in that: include: Extracting features from unstructured multimodal content to be reviewed to obtain a structured feature vector, wherein the structured feature vector includes a text semantic feature vector and an image visual feature vector; Performing image-text semantic association on the text semantic feature vector and the image visual feature vector to obtain a fusion feature vector involving image-text semantic contradiction; Constructing a domain knowledge graph based on compliance guidance data of the domain to which the multimodal content to be reviewed belongs; Inputting the fused feature vector into the domain knowledge graph and performing compliance rule retrieval using a subgraph matching algorithm to determine the violation type and the violated compliance clause corresponding to the fused feature vector; Generate interactive compliance audit reports.

2. The compliance review method for multimodal content according to claim 1, characterized in that: The step of extracting features from the unstructured multimodal content to be reviewed to obtain a structured feature vector includes: Performing visual feature extraction on the image portion of the multimodal content to be reviewed to obtain an image visual feature vector, wherein the image visual feature vector includes image region features, and the image region features include text, graphics, color, and layout; Natural language processing is performed on the text portion of the multimodal content to be reviewed to obtain a text semantic feature vector.

3. The compliance review method for multimodal content according to claim 2, characterized in that: The step of performing text-image semantic association on the text semantic feature vector and the image visual feature vector to obtain a fusion feature vector involving text-image semantic contradiction comprises: Arrange all the extracted text semantic feature vectors into a text semantic feature vector sequence; Arrange all the extracted image visual feature vectors into an image visual feature vector sequence; Dimensionally aligning the text semantic feature vector sequence and the image visual feature vector sequence; Calculate the cosine similarity between each text semantic feature vector and each image visual feature vector to generate the first attention matrix; Calculate the cosine similarity between each image visual feature vector and each text semantic feature vector to generate the second attention matrix; According to the first attention matrix and the second attention matrix, cross-modal attention information is fused by weighted summation to determine whether there is a fused feature vector involving semantic contradictions between the image and text. If so, the fused feature vector is output.

4. The compliance review method for multimodal content according to claim 3, characterized in that: The step of constructing a domain knowledge graph based on the compliance guidance data of the domain to which the multimodal content to be reviewed belongs includes: Obtaining compliance guidance data for the field to which the multimodal content to be reviewed belongs, wherein the compliance guidance data includes industry regulatory policies, business rules, and compliance cases; Information extraction and entity relationship modeling are performed on the compliance guidance data to obtain a domain knowledge graph.

5. The compliance review method for multimodal content according to claim 4, characterized in that: The step of inputting the fused feature vector into the domain knowledge graph and using a subgraph matching algorithm to perform compliance rule retrieval to determine the violation type corresponding to the fused feature vector and the violated compliance clause includes: Locating corresponding nodes in the domain knowledge graph according to entities in the multimodal content to be reviewed; Based on the fused feature vector, the violation type nodes in the domain knowledge graph are matched and associated with specific compliance clauses and penalty basis.

6. A compliance review system for multimodal content, characterized by: include: A data preprocessing module is used to extract features from unstructured multimodal content to be reviewed to obtain a structured feature vector, wherein the structured feature vector includes a text semantic feature vector and an image visual feature vector; a cross-modal semantic association module, configured to perform image-text semantic association on the text semantic feature vector and the image visual feature vector to obtain a fused feature vector involving image-text semantic contradictions; A knowledge graph construction module, configured to construct a domain knowledge graph based on compliance guidance data of the domain to which the multimodal content to be reviewed belongs; A knowledge graph matching module, configured to input the fused feature vector into the domain knowledge graph and perform compliance rule retrieval using a subgraph matching algorithm to determine the violation type and the violated compliance clause corresponding to the fused feature vector; Report generation module, used to generate interactive compliance audit reports.

7. The multimodal content compliance review system according to claim 6, characterized in that: The data preprocessing module is further used for: Performing visual feature extraction on the image portion of the multimodal content to be reviewed to obtain an image visual feature vector, wherein the image visual feature vector includes image region features, and the image region features include text, graphics, color, and layout; Natural language processing is performed on the text portion of the multimodal content to be reviewed to obtain a text semantic feature vector.

8. The multimodal content compliance review system according to claim 7, characterized in that: The cross-modal semantic association module is further used to: Arrange all the extracted text semantic feature vectors into a text semantic feature vector sequence; Arrange all the extracted image visual feature vectors into an image visual feature vector sequence; Dimensionally aligning the text semantic feature vector sequence and the image visual feature vector sequence; Calculate the cosine similarity between each text semantic feature vector and each image visual feature vector to generate the first attention matrix; Calculate the cosine similarity between each image visual feature vector and each text semantic feature vector to generate the second attention matrix; According to the first attention matrix and the second attention matrix, cross-modal attention information is fused by weighted summation to determine whether there is a fused feature vector involving semantic contradictions between the image and text. If so, the fused feature vector is output.

9. The multimodal content compliance review system according to claim 8, characterized in that: The knowledge graph construction module is further used to: Obtaining compliance guidance data for the field to which the multimodal content to be reviewed belongs, wherein the compliance guidance data includes industry regulatory policies, business rules, and compliance cases; Information extraction and entity relationship modeling are performed on the compliance guidance data to obtain a domain knowledge graph.

10. The multimodal content compliance review system according to claim 9, characterized in that: The knowledge graph matching module is further used to: Locating corresponding nodes in the domain knowledge graph according to entities in the multimodal content to be reviewed; Based on the fused feature vector, the violation type nodes in the domain knowledge graph are matched and associated with specific compliance clauses and penalty basis.

Images