Image content auditing method and system based on multilayer scene graph structure

Abstract

The invention discloses an image content auditing method and system based on a multilayer scene graph structure, and belongs to the technical field of image content auditing, and the method comprises the steps: segmenting an image, and obtaining an entity and attribute description thereof; clustering the entities into a plurality of entity semantic clusters based on the image visual features of the entities, the text semantic features described by the attributes and the normalized entity center point spatial position coordinates, and analyzing the incidence relation between the entity pairs in the clusters; based on the overall semantic summary of each entity semantic cluster, identifying a high-order relationship between the cluster pairs, and generating an overall narrative intention of the image; performing role recognition and anaphora analysis on characters in the image to obtain a text-image association relationship; and integrating the generated data into a hierarchical graph structure, establishing a thinking chain prompt, and inputting the thinking chain prompt into a large language model for comprehensive reasoning. According to the method, the image content can be deeply understood layer by layer from the local entity to the global intention, and the complex violation semantics of the to-be-audited image can be accurately identified.

Description

Technical Field

[0001] This invention belongs to the field of image content moderation technology, specifically relating to an image content moderation method and system based on a multi-layer scene graph structure. Background Technology

[0002] With the rapid development of the internet and social media, the production and dissemination of digital image content have experienced explosive growth. While this has greatly enriched the public's access to information, it has also raised numerous challenges in content moderation. Failure to promptly and effectively review and filter inaccurate content may lead to serious social and ethical problems and negative impacts. Therefore, fast, accurate, and automated image content moderation methods are gradually becoming an important issue that urgently needs to be addressed in the field of artificial intelligence.

[0003] Traditional image content moderation methods largely rely on manual annotation and static rule matching, which are inefficient and difficult to scale to massive data processing scenarios. In recent years, the emergence and development of deep learning methods, especially Visual Language Models (VLMs), have brought new solutions to image content moderation. These models can simultaneously understand the visual features of an image and its corresponding textual semantics, achieving more accurate content analysis through cross-modal feature interaction. For example, models such as CLIP, BLIP-2, and Flamingo, through pre-training on massive image-text pairs, have learned powerful visual and linguistic alignment capabilities, enabling them to describe image content and answer questions.

[0004] To address deeper visual understanding challenges, the academic community has proposed Scene Graphs as a structured image representation method. Scene Graphs, by resolving images into entity nodes and relational edges, enable machines to understand complex interactions within images. From 2024 to 2025, research on Scene Graph Generation (SGG) made significant progress, with increasingly diverse research directions. For example, researchers began exploring general scene graph representation methods, aiming to construct comprehensive semantic scenes from arbitrary modal combinations. Simultaneously, for dynamic scenes such as videos, online video scene graph generation methods based on diffusion models emerged, along with 4D panoramic scene graphs and spatiotemporal scene graphs capable of handling spatiotemporal information. Furthermore, to improve the generalization ability of SGG in downstream tasks, unbiased and zero-shot scene graph generation has also become a research hotspot.

[0005] These structured representations are often combined with graph neural networks (GNNs) for complex reasoning. GNNs can effectively propagate and aggregate information across graph structures, thereby learning high-level semantic representations of entities and relationships. Research in 2024 further explored the oversmoothing problem of GNNs and proposed a more interpretable graph additive network model.

[0006] Ultimately, both the flattened descriptions generated by VLM and the structured information provided by scene graphs require a powerful inference engine to make the final judgment. Scene graph-based visual reasoning tasks have emerged to address this need. Recent research is exploring visual chain-of-thought prompting and spatial and mathematical reasoning using large language models, offering new avenues for achieving more reliable and in-depth visual understanding and review.

[0007] Despite significant advancements in image content understanding, existing technologies still suffer from the following drawbacks when handling obscure and complex image review tasks: 1. Mainstream visual language models lack structured representation of image content: VLMs, such as CLIP, typically encode the entire image or generate a descriptive text. This "flattened" approach ignores the rich structured relationships between entities within the image. For example, information such as spatial layout, relative position, and semantic associations between entities is largely lost during the conversion process, making it difficult for the model to understand complex semantics based on these relationships, such as irony, metaphor, and group attacks.

[0008] 2. Scene graph generation methods struggle to identify abstract and implicit semantic relationships: Even the latest scene graph generation methods primarily focus on identifying objective physical and spatiotemporal relationships. Their ability to model the abstract semantic relationships required for review tasks (such as A satirizing B, group C excluding individual D) remains insufficient. While zero-shot scene graph generation offers some insights, it still struggles to capture temporary, context-dependent implicit intentions.

[0009] 3. Visual Relationship Detection Methods Struggle to Handle Group Referentiality: Visual Relationship Detection (VRD) methods typically focus on identifying relationships between pairs of entities. However, many violations are not expressed through a single entity or entity pair, but rather through semantic clusters of multiple entities that collectively refer to a group or concept. Existing VRD methods lack the ability to semantically cluster entities and analyze cluster relationships, making it difficult to detect such implicit attack patterns composed of multiple dispersed entities.

[0010] 4. Insufficient modeling of the association between text and image entities: In multimodal hate speech detection tasks, text in images (such as titles, tags, and dialogue) is often key to understanding their true intent. Existing technologies typically treat text as independent supplementary information or simply associate it broadly with the entire image. This approach fails to accurately bind the semantics of text to specific entities or entity relationships in the image, resulting in insufficient ability to recognize satirical content that combines text and images with veiled criticism.

[0011] 5. The interpretability and reliability of visual reasoning models need improvement: While existing visual reasoning methods can improve performance through methods such as thought chains, their reasoning process often remains a black box. When the model makes an incorrect judgment, it is difficult to trace the cause. Although interpretable graph neural networks offer some ideas, combining interpretability with complex review tasks remains a challenge. Summary of the Invention

[0012] In view of the above, the purpose of this invention is to provide an image content review method and system based on a multi-layer scene graph structure. Through an innovative and modular technical process, unstructured image information is systematically transformed into a multi-layered, semantically rich structured scene graph, and deep reasoning is performed based on the structured scene graph, thereby achieving accurate and interpretable identification of obscure and complex illegal content.

[0013] To achieve the above-mentioned objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides an image content review method based on a multi-layer scene graph structure, comprising the following steps: The image to be reviewed is segmented to identify all entities, and fine-grained attribute descriptions are generated for each entity. Based on the visual features of the entity image, the semantic features of the attribute description text, and the normalized spatial coordinates of the entity center point, the entities are clustered to obtain several entity semantic clusters, and the relationship between entity pairs within the clusters is analyzed. Based on the overall semantic summary of each entity semantic cluster, the higher-order relationships between cluster pairs are identified, and the overall narrative intent is generated based on the entities and their relationships, entity semantic clusters and their higher-order relationships. Role recognition and referential analysis are performed on the text in the image to be reviewed to obtain the text-image association relationship between the text content and the corresponding entity in the image; The generated entities and their attribute descriptions and relationships, entity semantic clusters and their higher-order relationships, overall narrative intent and text-image relationships are integrated into a hierarchical graph structure, which is then used to construct a structured thinking chain prompt. This prompt is then input into a large language model for comprehensive reasoning to obtain the image content review results.

[0014] Preferably, the segmentation process of the image to be reviewed to identify all entities and generate fine-grained attribute descriptions for each entity includes: The image segmentation model is used to generate initial segmentation results for multiple regions in the image to be reviewed. A visual language model is used to perform semantic-level region merging on the initial segmentation results to determine the complete segmentation region corresponding to each entity; The visual language model is used to analyze each merged entity region to generate fine-grained attribute descriptions that include entity category, entity name, and entity status.

[0015] Preferably, the entities are clustered based on the entity's image visual features, attribute description text semantic features, and normalized entity center point spatial coordinates to obtain several entity semantic clusters, including: The visual feature vector of each entity region is extracted using a visual coding model, and the entity attribute description text is encoded into a text semantic feature vector using a text encoder. At the same time, the spatial coordinates of the center point of each entity are normalized. The high-dimensional feature vector of each entity is obtained by multimodal fusion of the image visual feature vector, the text semantic feature vector, and the normalized spatial coordinates of the entity center point. Based on high-dimensional feature vectors, a clustering algorithm is used to aggregate entities that are spatially adjacent, visually similar in images, or semantically similar in text into entity semantic clusters.

[0016] Preferably, the association relationships between entity pairs within a cluster include: Within each formed entity semantic cluster, entity pairs are traversed to determine the direct relationships between each entity pair. Direct relationships include part-whole spatial relationships, semantic scene relationships, and subordinate relationships.

[0017] Preferably, the step of identifying higher-order relationships between cluster pairs based on the overall semantic summary of each entity semantic cluster includes: Generate an overall semantic summary for each entity semantic cluster. Use the overall semantic summaries of any two entity semantic clusters and representative entity information as context, input them into the large language model and guide its reasoning to identify the higher-order relations between the two clusters. Higher-order relations include metaphor, contrast, objectification and irony.

[0018] Preferably, generating the overall narrative intent based on entities and their relationships, entity semantic clusters and their higher-order relationships includes: After integrating all entities, entity semantic clusters, inter-entity relationships, and higher-order relationships between clusters into structured information, the information is input into a visual language model to guide the model to generate a comprehensive expression as the overall narrative intent, thereby achieving a top-level narrative interpretation of the overall expressive intent of the image.

[0019] Preferably, the step of performing role recognition and referential analysis on the text in the image to be reviewed, to obtain the text-image association relationship between the text content and the corresponding entity in the image, includes: Identify text regions in an image and extract the text content; Sentence component analysis is performed on the text content, and the semantic role of the text in the image is determined based on the text's position in the image, font, and contextual information. Syntactic analysis is performed on the text content. When fuzzy pronouns, lack of overall sentence structure, or target-oriented features are detected, the visual saliency score and entity-text semantic role similarity score of each non-text entity in the image are calculated and weighted by the pronoun resolution algorithm. The fuzzy pronoun relationship is associated with one or more candidate entities with the highest weighted score to establish the text-image relationship between the text content and the corresponding entity in the image.

[0020] Secondly, embodiments of the present invention also provide an image content review system based on a multi-layer scene graph structure, used to implement the above-mentioned image content review method based on a multi-layer scene graph structure, including: an entity recognition and attribute extraction module, a semantic clustering and relationship analysis module, a high-order semantic and narrative intent analysis module, a text-image fusion and reference resolution module, and a graph-based LLM integrated reasoning module; The entity recognition and attribute extraction module is used to segment the image to be reviewed in order to identify all entities and generate fine-grained attribute descriptions for each entity. The semantic clustering and relationship analysis module is used to cluster entities based on the image visual features of entities, the text semantic features of attribute descriptions, and the normalized spatial coordinates of the entity center point to obtain several entity semantic clusters, and analyze the association relationship between entity pairs within the clusters. The higher-order semantics and narrative intent analysis module is used to identify higher-order relationships between cluster pairs based on the overall semantic summary of each entity semantic cluster, and to generate an overall narrative intent based on the entity and its association, entity semantic cluster and its higher-order relationships. The text-image fusion and referential resolution module is used to perform role recognition and referential analysis on the text in the image to be reviewed, and to obtain the text-image association relationship between the text content and the corresponding entity in the image; The graph-based LLM integrated reasoning module is used to integrate the generated entities and their attribute descriptions and relationships, entity semantic clusters and their higher-order relationships, overall narrative intent and text-image relationships into a hierarchical graph structure, and then construct a structured thinking chain prompt before inputting it into a large language model for integrated reasoning to obtain the image content review result.

[0021] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described image content review method based on a multi-layer scene graph structure when the computer program is executed.

[0022] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described image content review method based on a multi-layer scene graph structure.

[0023] Compared with the prior art, the beneficial effects of the present invention include at least the following: (1) This invention forms a multi-layer scene graph that can structurally represent image content by constructing from the bottom up, from visual atoms, entities, entity semantic clusters, high-order relationships between clusters to the overall narrative intent. This structure realizes hierarchical and structured modeling of image content from low-level local details to high-level global intent, fully preserving the spatial and semantic relationships between entities, effectively overcoming the defects of flat and low-structured input information in existing visual language models, and providing a solid foundation for deep semantic understanding.

[0024] (2) This invention performs semantic clustering of entities by fusing multimodal features, aggregating semantically related and spatially proximate entities into entity semantic clusters, and analyzing direct associations within clusters and higher-order relationships between clusters based on this. This method can reveal implicit semantic groups composed of multiple entities, thereby accurately identifying complex semantics and attack intentions such as common references, metaphors, contrasts, and irony targeting specific groups or concepts. It solves the problem that existing technologies are difficult to handle group references and implicit relationships, and achieves effective identification of implicit relationships and group semantics between entities in images.

[0025] (3) This invention proposes a text role recognition and semantic completion strategy. By analyzing the role of text in an image and its syntactic semantics, and combining visual saliency and entity-text similarity for cross-modal matching and referential resolution, it can accurately associate text with ambiguous referentials with one or more specific entities in the image. This strategy effectively deciphers the true intent of the combined text and image content, overcomes the problems of coarse text-image association and unclear referentials in existing technologies, and establishes a precise and deep semantic association between text and visual entities in the image.

[0026] (4) This invention serializes the multi-layered scene graph structure generated in the aforementioned steps into a standardized hierarchical graph structure, and uses this to construct a guided thought chain prompt input large language model (LLM) for comprehensive reasoning. This solves the problem of the single level of semantic understanding in the model and can effectively enhance the high-dimensional semantic reasoning ability of the large language model based on structured information. This method utilizes the rich relational information in the graph structure to guide the LLM to make deeper and more reliable logical judgments, thereby accurately identifying the overall narrative structure and final intent of the image and completing the review task. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating the image content review method based on a multi-layer scene graph structure provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the framework of the image content review method based on a multi-layer scene graph structure provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of the structure and process of an image content review system based on a multi-layer scene graph structure provided in an embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0030] The inventive concept of this invention is as follows: In view of the shortcomings of existing methods such as single semantic understanding level, difficulty in handling group reference and abstract relationship, insufficient modeling of image-text association and unreliable reasoning process, the embodiments of this invention provide an image content review method and system based on multi-layer scene graph structure.

[0031] First, a multi-layer scene graph is constructed to achieve structured representation: In response to the problem that existing visual language models lack structured representation, this invention designs a multi-layer graph structure from the bottom-level visual features to the top-level narrative intent, ensuring that the spatial and semantic relationships between entities are not lost during the analysis process.

[0032] Secondly, semantic clustering is introduced to identify implicit group relationships: In order to solve the problem that existing scene graph generation and visual relationship detection methods are difficult to identify group references and abstract relationships, this invention innovatively introduces an entity semantic clustering strategy to discover semantic clusters composed of multiple scattered entities and analyze their internal and external relationships.

[0033] Furthermore, the design of a referential resolution mechanism to accurately associate text and image information is proposed: To address the problem of insufficient text and image association modeling, this invention designs a saliency-based referential resolution mechanism to accurately anchor the text in the image to the specific visual entity it refers to.

[0034] Finally, a graph-structured input is used to enhance the interpretability of reasoning: To address the shortcomings in reliability and interpretability of existing visual reasoning models, this invention uses a complete, structured scene graph as input to a large language model and guides it to perform thought chain reasoning, making its judgment process traceable.

[0035] By organically combining the above-mentioned hierarchical modeling, group relationship mining, deep image-text fusion and structured guided reasoning, this invention can achieve a layer-by-layer in-depth understanding of image content from local entities to global intent, accurately identify complex and illegal semantics including irony, metaphor, and group attacks, and is suitable for scenarios such as social media and content platforms that require high-intensity and high-accuracy image content security review.

[0036] like Figure 1 and Figure 2 As shown, the embodiment provides an image content review method based on a multi-layer scene graph structure, including the following steps: S1: Segment the image to be reviewed to identify all entities and generate fine-grained attribute descriptions for each entity.

[0037] In this embodiment, this step aims to address the fundamental problem of the lack of structured representation, namely, how to extract high-quality, information-rich basic units—entities—from the original image to be reviewed. Its design goal is to perform a comprehensive preliminary analysis of the input image, accurately identify and locate all key entities with latent semantics in the image, and generate rich, fine-grained attribute descriptions for each entity, providing a foundation for all subsequent structured analyses.

[0038] Specifically, firstly, the input image to be reviewed is processed using an image segmentation model (SAM) to generate a series of segmentation masks and corresponding bounding boxes as the initial segmentation results. This step ensures comprehensive entity recognition, avoiding the omission of unconventional regions or areas outside the region of interest that still affect the semantic content of the image.

[0039] Subsequently, due to the detailed nature of the segmented regions (e.g., segmenting a face into facial skin, features, and hair), a lightweight visual language model (VLM) is invoked to determine the entity range within each segmented region (e.g., facial skin, features, and hair are identified as a face). By sorting the segmented regions according to their spatial location and feeding them sequentially into the VLM based on the principle of region center distance, the VLM determines whether they belong to the same entity and merges them to obtain the complete segmented region corresponding to each entity. The purpose of this part is to obtain the entity semantics in the image; these entities serve as the basis for understanding and composing the image content.

[0040] After obtaining the merged entity regions, they are input into the VLM (Visual Model), which outputs an attribute description containing detailed features of the entities. These features include entity category (e.g., object, person, background), entity name (e.g., specific person name, specific object name), and entity state (e.g., person's expression, object's state). The purpose of this part is to mine the deep semantics of individual entities. This fine-grained information is the key basis for subsequent semantic clustering and relational reasoning.

[0041] This step transforms the original pixel image into a structured array of entity objects. These entities not only have precise boundaries but also come with categories, natural language descriptions, and dynamically expandable detailed labels. This directly addresses the problem of flattened input information in existing VLM models, providing high-quality node information for constructing structured scene graphs.

[0042] S2, based on the visual features of the entity's image, the semantic features of the attribute description, and the normalized spatial coordinates of the entity's center point, clusters the entities to obtain several entity semantic clusters, and analyzes the relationship between entity pairs within the clusters.

[0043] In this embodiment, this step aims to address the difficulties in identifying relationships between entities and handling group references. Its design goal is to discover implicit "semantic groups" composed of multiple independent entities and analyze the structural relationships within these groups, thereby identifying complex semantics such as collective irony and group references that cannot be expressed by a single entity.

[0044] Specifically, firstly, multimodal feature fusion is performed to construct a high-dimensional feature vector for each entity. This vector is composed of three parts: 1) the image visual feature vector of each entity region extracted from the visual encoding model; 2) the text semantic feature vector converted from the entity attribute description text by the text encoder; and 3) the normalized spatial coordinates of the entity's center point. This part ensures that the clustering is based on comprehensive and multi-dimensional factors, taking into account factors such as "similar appearance," "describing the same type of thing," and "proximity."

[0045] Next, entity density clustering is performed. Based on the spatial relationship between entities, entities that are visually close are clustered into several semantic clusters using the K-nearest neighbor (K-means) algorithm. These clusters serve as the core of image semantic representation.

[0046] Finally, intra-cluster relationship analysis is performed. Within each formed entity semantic cluster, entity pairs are traversed, and direct relationships between them are determined using LLM or specific rules (such as spatial proximity and text similarity), such as part-whole spatial relationships, semantic scene relationships, and subordinate relationships. The role of this part is to reveal the organizational structure within the group.

[0047] This step organizes the originally flat list of entities into semantic clusters with inherent connections, which enables the system to recognize advanced semantic patterns such as "multiple similar symbols pointing to a single target" or "a group of props and costumes forming a satirical image".

[0048] S3 identifies higher-order relationships between cluster pairs based on the overall semantic summary of each entity semantic cluster, and generates an overall narrative intent based on the entity and its association, entity semantic cluster and its higher-order relationships.

[0049] In this embodiment, this step further addresses the problem of difficulty in identifying abstract relationships. Its design goal is to go beyond the direct physical relationships between entities, infer more abstract and macroscopic social and intentional relationships between different semantic clusters, and form a top-level understanding of the entire image.

[0050] Specifically, first, inter-cluster relationship reasoning is performed. A holistic semantic summary of the cluster is generated for each entity semantic cluster.

[0051] Then, the LLM (Limited Language Management) is used to summarize the overall semantics of any two clusters and use all entity information as contextual input for the final judgment. Guiding questions are then posed (e.g., "From a social or cultural perspective, are there ironic, contrasting, or power-unequal relationships between cluster A and cluster B?") to identify higher-order relationships such as visual metaphor, contrast, objectification, and irony. This technique elevates the granularity of analysis from entities to conceptual groups, thereby enabling the discovery of more macroscopic narrative structures.

[0052] Finally, a holistic narrative is generated. All the structured information generated in the preceding steps (entities, entity semantic clusters, inter-entity relationships, and higher-order relationships between clusters) is input into a VLM (Visual Model). It is required to synthesize all the information and generate a fluent and coherent comprehensive statement as the overall narrative intent, achieving a top-level narrative interpretation of the image's overall expressive intention. The purpose of this part is to form a final summary of the image's intent, providing a global perspective for review and judgment.

[0053] This module elevates the understanding of images from "what is there" to "what intention is being expressed." It can clearly reveal the complex relationships such as antagonism and irony between different groups or concepts in an image, providing the most crucial top-level semantic basis for the final review decision.

[0054] S4 performs role recognition and referential analysis on the text in the image to be reviewed, and obtains the text-image association relationship between the text content and the corresponding entity in the image.

[0055] In this embodiment, this step aims to address the problem of insufficient modeling of the association between text and image entities. The meaning of some emojis or related images is closely related to the text accompanying them, because the text itself is within the image area, and its function includes supplementing the image content, summarizing the image structure, or adding the author's opinion. Therefore, the key meaning of some combined text and image images lies in understanding the text within the image. Thus, the design goal of this module is to accurately understand the true referent of the text in the image and the object it describes, in order to decipher indirect or veiled criticisms in combined text and image content.

[0056] Specifically, the first step is to perform semantic role recognition of the text. This involves analyzing sentence components, considering the overall image description, its position, font, context, and the semantic content the text intends to convey, to determine the text's role within the image. This step provides a preliminary basis for understanding the text's function.

[0057] Next, cross-modal referential resolution is performed. Syntactic analysis is conducted on the text content, and the referential resolution algorithm is activated when ambiguous pronouns (such as "he" or "this"), a lack of overall sentence structure, or a need for target-specific referentiality are detected. Referential resolution calculates the visual saliency scores of all non-textual entities in the image (ranked based on entity size and spatial relationship between entity center and text center) and the similarity scores between the semantic roles of image entities and text semantic roles, and associates ambiguous referentials with the candidate entities with the highest weighted scores. This part utilizes visual cues to help resolve ambiguities in the text, achieving precise "anchoring" between text and visual entities to establish a text-image association between text content and corresponding entities in the image.

[0058] This step explicitly associates each meaningful piece of text with one or more specific entities in the image. This enables the system to understand complex situations, such as "an offensive piece of text, though geographically distant from the target, actually points to that target," directly solving the problems of insufficient depth and inaccurate association in existing image-text fusion technologies.

[0059] S5 integrates the generated entities and their attribute descriptions and relationships, entity semantic clusters and their higher-order relationships, overall narrative intent and text-image relationships into a hierarchical graph structure, and then constructs a structured thinking chain prompt before inputting it into a large language model for comprehensive reasoning to obtain the image content review result.

[0060] In this embodiment, this step aims to address the interpretability and reliability issues of the visual reasoning model. Its design goal is to integrate the complex structured information generated by all preceding modules, perform comprehensive and in-depth logical reasoning, and output a reliable and interpretable audit conclusion.

[0061] Specifically, first, all the previous image information is integrated to serialize the graph structure. The complete data structure generated by all the aforementioned modules (including entities and their attribute descriptions and relationships, entity semantic clusters and their higher-order relationships, overall narrative intent, and text-image relationships) is serialized into a complete JSON text.

[0062] Next, structured thought chain reasoning is performed. Using the JSON text as the core context, along with the review rules, a detailed thought chain prompt is constructed and sent to the LLM. This prompt guides the LLM to analyze the data step by step: 1) Understanding Entities and Clusters: Identifying key entities and groups; 2) Analyze relationships: Examine the relationships between entities and clusters; 3) Examine the relationship between text and images: understand the meaning behind the text; 4) Assess the overall narrative: Make a comprehensive judgment on the image's intent; 5) Make a final judgment.

[0063] The purpose of this section is to force the LLM reasoning process to follow the logical framework constructed in this invention, so that its reasoning is no longer based on wild associations, but on logical deduction based on structured evidence. Compared with VLM, LLM has a much stronger reasoning ability, and its logical relationships and comprehension capabilities are significantly improved. Moreover, the aforementioned multi-layer graph structure has already converted the knowledge in the image into formalized natural language, so using LLM in this method can provide more accurate conclusions than VLM.

[0064] This step ultimately outputs a comprehensive conclusion that includes the type of violation, confidence level, detailed reasoning logic chain, and key evidence ID. This approach directly solves the problems of existing reasoning models being "black box" operations, having unreliable results, and being difficult to trace, greatly enhancing the reliability and interpretability of the audit results.

[0065] Finally, to further verify the effectiveness of the present invention, the following comparative experiments were conducted. The harmful dataset is introduced first, followed by the comparison method and the base VLM, and finally, the experimental results of the present invention are presented.

[0066] 1. Harmful datasets This invention selected the Facebook Hateful emoji dataset (containing approximately 12,000 emoji images with accompanying text, and manually labeled to indicate whether they contain hateful content) for experimental verification.

[0067] 2. Comparison Methods This invention compares and analyzes two academic methods: Pro-Cap (reference title: Pro-cap: Leveraging a frozen vision-language model for hateful meme detection) and ExplainHM (reference title: Towards explainable harmful meme detection through multimodal debate between large language models). Pro-Cap utilizes visual and textual features extracted from a frozen model to identify hate speech in image-text combinations. ExplainHM is a state-of-the-art method that aims to achieve explainable detection of harmful memes by introducing large language models and employing a multimodal debate mechanism.

[0068] 3. Base VLM and LLM This invention evaluates the system based on existing VLMs. VLMs themselves possess a certain image understanding capability. To demonstrate the effectiveness of this paper, Qwen2.5-vl-32B and Qwen2.5-32B are selected as the base VLM and LLM for the implementation of this method, respectively. Furthermore, to ensure the effectiveness of this method, the method also uses Qwen2.5-vl-32B as a comparison, directly inputting the image for its judgment.

[0069] 4. Experimental Results Table 1 Experimental Results

[0070] As shown in Table 1, the image content moderation method based on a multi-layer scene graph structure proposed in this invention significantly outperforms existing comparative methods in key performance indicators such as accuracy, precision, recall, and F1 score. Specifically, the accuracy of this invention reaches 0.736, which is significantly improved compared to 0.704 of the base VLM (Qwen2.5-vl-32B) and other comparative methods (0.610 of ExplainHM and 0.611 of Pro-Cap). In terms of F1 score, this invention achieves a score of 0.698, also higher than all comparative methods, indicating that this invention achieves a better balance between precision and recall. Especially in terms of recall, this invention reaches 0.765, far exceeding other methods, proving that this invention can more effectively identify harmful content. In summary, the experimental data above fully demonstrate the effectiveness of the present invention in constructing multi-layer scene graphs and combining them with large language models for structured reasoning. Compared with existing technologies, it shows significant superiority in terms of the accuracy and comprehensiveness of the review.

[0071] Based on the same inventive concept, such as Figure 3 As shown, this embodiment of the invention also provides an image content review system 300 based on a multi-layer scene graph structure, including: an entity recognition and attribute extraction module 310, a semantic clustering and relationship analysis module 320, a high-order semantic and narrative intent analysis module 330, a text-image fusion and reference resolution module 340, and a graph-based LLM comprehensive reasoning module 350.

[0072] The entity recognition and attribute extraction module 310 is used to segment the image to be reviewed in order to identify all entities and generate fine-grained attribute descriptions for each entity.

[0073] The semantic clustering and relation analysis module 320 is used to cluster entities based on the image visual features of entities, the text semantic features of attribute descriptions, and the normalized spatial coordinates of entity center points to obtain several entity semantic clusters, and analyze the association relationship between entity pairs within the clusters.

[0074] The higher-order semantics and narrative intent analysis module 330 is used to summarize the overall semantics of each entity semantic cluster, identify the higher-order relationships between cluster pairs, and generate the overall narrative intent based on the entity and its association, entity semantic clusters and their higher-order relationships.

[0075] The text-image fusion and referential resolution module 340 is used to perform role recognition and referential analysis on the text in the image to be reviewed, and to obtain the text-image association relationship between the text content and the corresponding entity in the image.

[0076] The LLM integrated reasoning module 350 based on graph structure is used to integrate the generated entities and their attribute descriptions and relationships, entity semantic clusters and their higher-order relationships, overall narrative intent and text-image relationship into a hierarchical graph structure, and then construct a structured thinking chain prompt before inputting it into a large language model for integrated reasoning to obtain the image content review result.

[0077] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described image content review method based on a multi-layer scene graph structure when executing the computer program.

[0078] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described image content review method based on a multi-layer scene graph structure.

[0079] It should be noted that the image content review system, electronic device, and computer-readable storage medium based on the multi-layer scene graph structure provided in the above embodiments all belong to the same inventive concept as the image content review method based on the multi-layer scene graph structure. For details of their specific implementation process, please refer to the embodiment of the image content review method based on the multi-layer scene graph structure, which will not be repeated here.

[0080] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image content review method based on a multi-layer scene graph structure, characterized in that, Includes the following steps: The image to be reviewed is segmented to identify all entities, and fine-grained attribute descriptions are generated for each entity. Based on the visual features of the entity image, the semantic features of the attribute description text, and the normalized spatial coordinates of the entity center point, the entities are clustered to obtain several entity semantic clusters, and the relationship between entity pairs within the clusters is analyzed. Based on the overall semantic summary of each entity semantic cluster, the higher-order relationships between cluster pairs are identified, and the overall narrative intent is generated based on the entities and their relationships, entity semantic clusters and their higher-order relationships. Role recognition and referential analysis are performed on the text in the image to be reviewed to obtain the text-image association relationship between the text content and the corresponding entity in the image; The generated entities and their attribute descriptions and relationships, entity semantic clusters and their higher-order relationships, overall narrative intent and text-image relationships are integrated into a hierarchical graph structure, which is then used to construct a structured thinking chain prompt. This prompt is then input into a large language model for comprehensive reasoning to obtain the image content review results.

2. The image content review method based on a multi-layer scene graph structure according to claim 1, characterized in that, The process of segmenting the image to be reviewed to identify all entities and generating fine-grained attribute descriptions for each entity includes: The image segmentation model is used to generate initial segmentation results for multiple regions in the image to be reviewed. A visual language model is used to perform semantic-level region merging on the initial segmentation results to determine the complete segmentation region corresponding to each entity; The visual language model is used to analyze each merged entity region to generate fine-grained attribute descriptions that include entity category, entity name, and entity status.

3. The image content review method based on a multi-layer scene graph structure according to claim 1, characterized in that, The entity-based image visual features, attribute-descriptive text semantic features, and normalized entity center point spatial coordinates are used to cluster entities to obtain several entity semantic clusters, including: The visual feature vector of each entity region is extracted using a visual coding model, and the entity attribute description text is encoded into a text semantic feature vector using a text encoder. At the same time, the spatial coordinates of the center point of each entity are normalized. The high-dimensional feature vector of each entity is obtained by multimodal fusion of the image visual feature vector, the text semantic feature vector, and the normalized spatial coordinates of the entity center point. Based on high-dimensional feature vectors, a clustering algorithm is used to aggregate entities that are spatially adjacent, visually similar in images, or semantically similar in text into entity semantic clusters.

4. The image content review method based on a multi-layer scene graph structure according to claim 1 or 3, characterized in that, The relationships between entity pairs within a cluster include: Within each formed entity semantic cluster, entity pairs are traversed to determine the direct relationships between each entity pair. Direct relationships include part-whole spatial relationships, semantic scene relationships, and subordinate relationships.

5. The image content review method based on a multi-layer scene graph structure according to claim 1, characterized in that, The overall semantic summary based on each entity semantic cluster identifies higher-order relationships between cluster pairs, including: Generate an overall semantic summary for each entity semantic cluster. Use the overall semantic summaries of any two entity semantic clusters and representative entity information as context, input them into the large language model and guide its reasoning to identify the higher-order relations between the two clusters. Higher-order relations include metaphor, contrast, objectification and irony.

6. The image content review method based on a multi-layer scene graph structure according to claim 1 or 5, characterized in that, The generation of overall narrative intent based on entities and their relationships, entity semantic clusters and their higher-order relationships includes: After integrating all entities, entity semantic clusters, inter-entity relationships, and higher-order relationships between clusters into structured information, the information is input into a visual language model to guide the model to generate a comprehensive expression as the overall narrative intent, thereby achieving a top-level narrative interpretation of the overall expressive intent of the image.

7. The image content review method based on a multi-layer scene graph structure according to claim 1, characterized in that, The process of performing role recognition and referential analysis on the text in the image to be reviewed, and obtaining the text-image association relationship between the text content and the corresponding entities in the image, includes: Identify text regions in an image and extract the text content; Sentence component analysis is performed on the text content, and the semantic role of the text in the image is determined based on the text's position in the image, font, and contextual information. Syntactic analysis is performed on the text content. When fuzzy pronouns, lack of overall sentence structure, or target-oriented features are detected, the visual saliency score and entity-text semantic role similarity score of each non-text entity in the image are calculated and weighted by the pronoun resolution algorithm. The fuzzy pronoun relationship is associated with one or more candidate entities with the highest weighted score to establish the text-image relationship between the text content and the corresponding entity in the image.

8. An image content review system based on a multi-layer scene graph structure, used to implement the image content review method based on a multi-layer scene graph structure as described in any one of claims 1 to 7, characterized in that, include: The module includes entity recognition and attribute extraction, semantic clustering and relation analysis, high-order semantics and narrative intent analysis, text-image fusion and reference resolution, and graph-based LLM integrated reasoning module. The entity recognition and attribute extraction module is used to segment the image to be reviewed in order to identify all entities and generate fine-grained attribute descriptions for each entity. The semantic clustering and relationship analysis module is used to cluster entities based on the image visual features of entities, the text semantic features of attribute descriptions, and the normalized spatial coordinates of the entity center point to obtain several entity semantic clusters, and analyze the association relationship between entity pairs within the clusters. The higher-order semantics and narrative intent analysis module is used to identify higher-order relationships between cluster pairs based on the overall semantic summary of each entity semantic cluster, and to generate an overall narrative intent based on the entity and its association, entity semantic cluster and its higher-order relationships. The text-image fusion and referential resolution module is used to perform role recognition and referential analysis on the text in the image to be reviewed, and to obtain the text-image association relationship between the text content and the corresponding entity in the image; The graph-based LLM integrated reasoning module is used to integrate the generated entities and their attribute descriptions and relationships, entity semantic clusters and their higher-order relationships, overall narrative intent and text-image relationships into a hierarchical graph structure, and then construct a structured thinking chain prompt before inputting it into a large language model for integrated reasoning to obtain the image content review result.

9. An electronic device comprising a memory and one or more processors, the memory being used to store a computer program, characterized in that, The processor is used to implement the image content review method based on a multi-layer scene graph structure as described in any one of claims 1 to 7 when executing a computer program.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the image content review method based on a multi-layer scene graph structure as described in any one of claims 1 to 7.

Images