Image sensitive content review method and system

Abstract

This invention discloses a method and system for reviewing sensitive content in images, comprising: obtaining the original description of the image to be reviewed, the original description including a textual summary of the visual content of the image and the text content present in the image; extracting entity regions from the image to be reviewed, retrieving relevant external background information based on the image to be reviewed and the entity regions, and using the labels of the entity regions to exclude interference information to generate enhanced search results; performing semantic relevance filtering on the enhanced search results based on sensitive topics, and iteratively fusing the filtered enhanced search results into the original description to generate a final fused description; constructing a training dataset containing positive and negative judgment sample pairs based on the fused description, fine-tuning the visual language model, and using the fine-tuned visual language model to review sensitive content in new images. This invention can effectively improve the accuracy and reliability of sensitive content recognition in images.

Description

Technical Field

[0001] This invention belongs to the field of image content review technology, specifically relating to a method and system for reviewing sensitive image content. Background Technology

[0002] In recent years, with the rapid development of the internet and social media, the production and dissemination of digital image content have experienced explosive growth. While this has greatly enriched the public's access to information, it has also raised numerous challenges in content moderation. In particular, image content involving sensitive topics, if not reviewed and filtered in a timely and effective manner, may lead to serious social and ethical problems and negative impacts. Therefore, fast, accurate, and automated methods for image content moderation are gradually becoming an important issue that urgently needs to be addressed in the field of artificial intelligence.

[0003] Traditional image content moderation methods rely heavily on manual annotation and static rule matching, which are inefficient and difficult to scale to massive data processing scenarios. In recent years, the emergence and development of deep learning methods, especially Vision-Language Models (VLMs), have brought new solutions to image content moderation. These models can simultaneously understand the visual features of an image and its corresponding textual semantics, achieving more accurate content analysis through cross-modal feature interaction.

[0004] However, despite the superior performance of large models in routine semantic understanding and generation tasks, existing VLMs still have significant shortcomings in practical sensitive content moderation, which limits the practical application effectiveness and reliability of visual models in sensitive and harmful content moderation tasks. Specifically, these shortcomings are as follows:

[0005] First, there is a lack of effective integration of external knowledge, and the model is susceptible to problems such as knowledge obsolescence or information illusion in real-world tasks. Large models typically draw their knowledge base from static pre-trained datasets, meaning they often lack effective mastery of real-time updates or domain-specific background knowledge. This is especially true in sensitive content recognition tasks, where accurately understanding the context behind images and text often relies on external background knowledge such as history, culture, and even immediate social events. If the model cannot effectively retrieve, absorb, and integrate this external knowledge, its judgments may result in misjudgments, omissions, or even information illusions, thus reducing the effectiveness and reliability of content moderation.

[0006] Secondly, it struggles to effectively understand the high-dimensional semantics of sensitive content. Visual Learning Models (VLMs) are trained on massive datasets of visual question-and-answer pairs, enabling them to understand image content and interpret images based on user input. However, human society's mapping of harmful visual content is diverse. It can be a direct representation, or it can add high-dimensional semantic meanings that are understandable to human language, based on rhetorical devices and symbolic implications derived from social hot topics. Although large models demonstrate superior performance in conventional semantic understanding and generation tasks, they often lack sufficient high-dimensional semantic understanding capabilities to accurately identify aggressive components in the complex contexts and metaphorical expressions of sensitive content.

[0007] Furthermore, the values ​​of language models differ from those of the real world. Generally, existing large language models undergo safety alignment training to correct their values ​​and avoid outputting harmful visual content. However, this can lead to a bias in the model's output towards the positive. Some sensitive content images may target specific groups or events. From the perspective of other viewers, this may not be offensive. This results in VLMs (Voice over Model) or Large Language Models (LLMs) being unwilling or unaware of the potential offensiveness to certain groups during sensitive content moderation, and their output tends to express neutral or even positive ideas. This leads to a lack of objectivity and thorough consideration in the values ​​of large models during sensitive content moderation.

[0008] To address the aforementioned issues, how to endow large models with stronger high-dimensional semantic perception capabilities, and how to effectively introduce and utilize relevant external background knowledge, have become one of the major challenges in current research. Summary of the Invention

[0009] In view of the above, the purpose of this invention is to provide a method and system for reviewing sensitive image content. This method involves acquiring rich external background knowledge related to the image through retrieval enhancement and integrating it into the image description of a Visual Model (VLM) to obtain a fused description. Based on this, a scenario simulation judgment method guides the model to generate both positive and negative explanations regarding whether an image violates regulations. Training sample pairs containing opposing judgment reasons are constructed and used to fine-tune the VLM, effectively improving the accuracy of sensitive content identification and providing a more robust, efficient, and reliable solution for automated review of sensitive content.

[0010] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0011] In a first aspect, an embodiment of the present invention provides a method for reviewing image-sensitive content, comprising the following steps:

[0012] Obtain the original description of the image to be reviewed. The original description includes a textual summary of the visual content of the image and the text content present in the image.

[0013] Extract entity regions from the image to be reviewed, retrieve relevant external background information based on the image to be reviewed and entity regions, and use the labels of entity regions to eliminate interference information and generate enhanced search results.

[0014] The enhanced search results are semantically relevant based on sensitive topics, and the enhanced search results are iteratively integrated into the original description to generate the final integrated description.

[0015] A training dataset containing positive and negative decision pairs is constructed based on the fusion description. The visual language model is then fine-tuned, and the fine-tuned visual language model is used to perform sensitive content review on new images.

[0016] Preferably, the step of retrieving relevant external background information based on the image to be reviewed and the entity region, and then using the labels of the entity region to exclude interfering information and generate enhanced search results, includes:

[0017] The object detection model identifies entities in the image and extracts multiple entity regions and their labels from the image to be reviewed;

[0018] Use an image search engine to retrieve relevant external background information text from the image to be reviewed and its multiple entity regions.

[0019] Based on entity region labels, a large language model is used to aggregate the retrieved external background information text, remove interfering information, and generate enhanced search results.

[0020] Preferably, the generated prompts for retrieval enhancement performed by the auxiliary large language model are configured as follows:

[0021] The large language model is guided to semantically aggregate the text attributes of the input external background information and generate a coherent structured description, filter redundant information that is unrelated to the labels of entity regions, and output a concise description that focuses on the attributes of entities and does not involve entity ontology.

[0022] Preferably, the step of performing semantic relevance filtering on the enhanced search results based on sensitive topics includes:

[0023] Based on a predefined list of sensitive topics, a large language model is used to evaluate the relevance of the search enhancement results, and only the search enhancement results with a relevance that reaches a preset threshold are selected and retained.

[0024] Preferably, the evaluation prompts used to assist the large language model in performing semantic relevance filtering are configured as follows:

[0025] The large language model is required to output a relevance level and a brief reason for the judgment based on the enhanced retrieval results and a predefined list of sensitive topics. The relevance level should include at least one of the following: irrelevant, weakly relevant, moderately relevant, strongly relevant, and extremely strongly relevant.

[0026] Preferably, the step of iteratively integrating the filtered search enhancement results into the original description to generate the final integrated description includes:

[0027] The process employs an iterative approach, sequentially fusing each filtered enhanced search result with the current image description into the large language model to output an updated image description. In the first iteration, the original description is used as the current image description.

[0028] By iterating multiple times, external knowledge is gradually integrated into the image description to obtain the final fused description.

[0029] Preferably, the step of constructing a training dataset containing positive and negative decision sample pairs based on the fusion description and fine-tuning the visual language model includes:

[0030] By using a scenario simulation review method based on authoritative assumptions, the visual language model is guided to generate judgment results on whether the image to be reviewed is in violation or not, along with the reasons for the judgment.

[0031] Based on the actual review label of the image to be reviewed, the judgment results and reasons that match the actual review label are marked as accepted responses and correspond to positive samples, while the judgment results and reasons that contradict the actual review label are marked as rejected responses and correspond to negative samples.

[0032] Construct structured training samples that include system prompts, user queries, accepted responses, and rejected responses, where user queries include original descriptions, fused descriptions, and guided analysis prompts;

[0033] Based on the training dataset consisting of structured training samples, the visual language model is fine-tuned using the direct preference optimization method.

[0034] Secondly, embodiments of the present invention also provide an image-sensitive content review system, implemented using the above-described image-sensitive content review method, comprising: an image original description module, a retrieval enhancement query module, a content description fusion module, and a model fine-tuning module;

[0035] The original image description module is used to obtain the original description of the image to be reviewed. The original description includes a textual summary of the visual content of the image and the text content present in the image.

[0036] The search enhancement query module is used to extract entity regions in the image to be reviewed, retrieve relevant external background information based on the image to be reviewed and entity regions, and use the labels of entity regions to eliminate interference information and generate search enhancement results.

[0037] The content description fusion module is used to perform semantic relevance filtering on the search enhancement results based on sensitive topics, and to iteratively fuse the filtered search enhancement results into the original description to generate the final fused description.

[0038] The model fine-tuning module is used to construct a training dataset containing positive and negative decision sample pairs based on the fusion description, fine-tune the visual language model, and use the fine-tuned visual language model to perform sensitive content review on new images.

[0039] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described image-sensitive content review method when executing the computer program.

[0040] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described method for reviewing image-sensitive content.

[0041] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0042] (1) This invention dynamically retrieves relevant external background information from external knowledge bases or the Internet and integrates it into the image description through the retrieval enhancement generation method, which significantly improves the factual accuracy and knowledge comprehensiveness of the generated content. Especially in the image review task, the external background information introduced by cross-modal retrieval can effectively make up for the model's shortcomings in recognizing sensitive contexts, enhance the model's ability to understand image content and potential metaphorical meanings, and show excellent performance in knowledge-intensive tasks.

[0043] (2) The strategy proposed in this invention, which is based on the retrieval enhancement results and the iterative aggregation of image descriptions, can ensure that the large model can understand the image more thoroughly as the background knowledge gradually increases. In the iterative process, it can combine some new understandings with the original image description, so that the VLM can understand the high-dimensional semantics of the image more deeply.

[0044] (3) This invention first guides a large model to generate judgment data in two directions—violation and non-violation—for images through scenario simulation review. Then, judgment results that do not match the real review labels are used as rejection data, and judgment results that match the real review labels are used as acceptance data. A training dataset containing structured positive and negative samples is constructed, and the visual language model is fine-tuned using direct preference optimization. This innovative training paradigm effectively promotes the alignment of the model with human values, corrects the model's value judgment criteria, and significantly improves the model's recognition accuracy and robustness for high-dimensional and complex semantic content such as sensitive content. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating an image-sensitive content review method provided in an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of the framework of an image-sensitive content review method provided in an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the structure and process of an image-sensitive content review system provided in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0050] Existing large-scale models often struggle to accurately identify offensive elements in sensitive content due to a lack of sufficient high-dimensional semantic understanding when faced with complex contexts and metaphorical expressions. This is particularly true in the area of ​​sensitive content moderation, where such content typically involves multiple sensitive social issues and is expressed through complex rhetorical devices such as metaphor, allusion, irony, or symbolism. This high-dimensional semantic complexity poses a significant challenge to the content moderation capabilities of existing large-scale models.

[0051] In view of this, the inventive concept of the present invention is as follows: In view of the problems of insufficient high-dimensional semantic understanding of image-sensitive content by large models in the prior art, lack of background knowledge support, and inaccurate value alignment, the embodiments of the present invention provide an image-sensitive content review method and system. By introducing retrieval enhancement generation technology to construct external knowledge background, adopting an iterative fusion mechanism to deepen semantic understanding, and achieving accurate alignment of model values ​​based on scenario simulation review and direct preference optimization, the invention effectively improves the recognition accuracy and judgment reliability of complex sensitive content.

[0052] like Figure 1 As shown in the embodiment, an image-sensitive content review method is provided, including the following steps:

[0053] S1, Obtain the original description of the image to be reviewed. The original description includes a textual summary of the visual content of the image and the text content present in the image.

[0054] The core purpose of this step is to first grasp the overall content of the image, including the people, objects, and overall structure of the scene, so that VLM can have a certain foundation when understanding the image.

[0055] In this embodiment, VLM is used as a base model to extract the original description of the image to be reviewed, including a textual summary of the visual content of the image and the text content present in the image.

[0056] S2, extract entity regions from the image to be reviewed, retrieve relevant external background information based on the image to be reviewed and entity regions, use the labels of entity regions to exclude interference information, and generate enhanced search results.

[0057] This step takes into account situations where the base model has not encountered certain special images, including figures, locations, symbols, flags, etc., from recent events or the development of human society, as well as objects that embody the style of specific ethnic groups (such as totems). The meanings behind these images and objects are unknown to the larger model. Therefore, the core objective is to address the issue that VLM may have an incomplete or biased understanding of the content when interpreting image semantics.

[0058] The embodiment specifically includes the following sub-steps:

[0059] S2.1, Image entity extraction.

[0060] The RAM-Grounding joint model is used to automatically detect image regions. First, the RAM model is used to label all objects, people, scenes and other elements in the image. Then, the GroundingDino model is used to filter and frame the image objects that best match these labels.

[0061] To improve efficiency, the Non-Maximum Suppression (NMS) algorithm, a post-processing algorithm for object detection, was also used to remove redundant detection boxes.

[0062] Finally, the block-shaped entity region images, their coordinates in the original image, and the RAM-Grounding detection labels (i.e., the entity region labels) were obtained.

[0063] S2.2, Image Content Retrieval.

[0064] After entity extraction, the original image and a series of region images were obtained. To acquire background knowledge of the entity region images, an image search engine was used to perform image-to-image searches on these entity region images to obtain background information or related reports about the images.

[0065] At the same time, the original overall image is also input into the image search engine, because the overall image may be a famous portrait or scene, or it may be from some news reports.

[0066] Then, the image search engine's output includes images that have appeared in or related to that image. Figure 1 The number of web pages and their titles that can be obtained varies depending on the extent of image dissemination. Since image search engines return varying numbers of results, and these results are sorted by relevance, later pages often contain duplicate or less relevant results. Therefore, the results are truncated to the Top K results (maximum 60) to obtain the relevant external background information text.

[0067] S2.3, Image Content Enhancement.

[0068] Titles obtained through image search engines may contain redundant items or irrelevant search results. To integrate all content into an effective image description, LLM is used to fuse all title content and RAM-Grounding tags, eliminating distracting information and identifying commonalities among most titles as the final image description.

[0069] LLMs should have the following features:

[0070] (1) Aggregating information: LLM should be able to extract the same or related information from the input and describe their purpose in a well-structured sentence. If there are multiple perspectives, they should be smoothly integrated into a natural and coherent description based on their commonalities.

[0071] (2) Eliminate interference information: Ignore any irrelevant details that are not directly related to the object represented by the image name.

[0072] (3) Concise description: The description should be clear, accurate and not violate the input data, while generating concise and well-formatted output.

[0073] Therefore, the following is an example of the generated prompt words for the LLM:

[0074] Task Description: Your task is to process a given object label and its corresponding list of text attributes. Use the following input: {}.

[0075] Aggregate information: Extract relevant attributes from the input paragraphs and explain their uses in a clear and well-organized sentence; if there are multiple aspects, integrate them naturally and fluently into a coherent description.

[0076] Eliminate distracting information: Ignore any attributes that are not directly related to the object represented by the object label.

[0077] Output format: Returns a well-structured, concise sentence that seamlessly integrates all relevant attributes and their descriptions without adding any new details not explicitly mentioned in the input.

[0078] Example output: This is an 8K resolution display for computer use; and a self-cleaning mop and vacuum cleaner combo from a certain brand, designed specifically for home cleaning.

[0079] Require:

[0080] 1. Do not mention specific object tags in the description; focus directly on the attributes of the object or person.

[0081] 2. Attributes and their descriptions should be concise, relevant, and strictly derived from the input text.

[0082] 3. Avoid including any additional or irrelevant information not mentioned in the input text.

[0083] 4. This task aims to ensure that the generated content is clear, accurate, and strictly follows the input data, while maintaining concise and standardized output format.

[0084] In the example of generating prompts above, the content within {} is the actual input. Let's assume the original image to be reviewed... The first in Image of a physical region The search results are Each time a user requests aggregated search results, their input should be in the following format:

[0085] ,

[0086] in, Enter the label of the RAM-Grounding result in the field; if it is the entire image, leave this field blank. Enter all the results found in the query. The title of each result. From this, we can obtain... Corresponding enhanced search results .

[0087] S3 performs semantic relevance filtering on the search enhancement results based on sensitive topics, and iteratively integrates the filtered search enhancement results into the original description to generate the final fused description.

[0088] The embodiment specifically includes the following sub-steps:

[0089] S3.1, semantically relevant filtering.

[0090] In the semantic relevance screening stage, external background knowledge is introduced into the image content review process. The purpose of this stage is to uncover the context of potentially sensitive information from the original image description, enriching the understanding of the image's implicit meaning by retrieving relevant materials and filtering valuable content. In the aforementioned search enhancement query step, preliminary search results aggregated for all entities are returned. However, not all entity background knowledge is relevant to sensitive content review; many are irrelevant. Therefore, further screening of the semantic relevance of these candidate texts is necessary.

[0091] Specifically, based on the definitions of sensitive content on relevant social media platforms, a series of sensitive topics that require special attention during content review were pre-determined.

[0092] Then, the design of evaluation prompts enables the LLM to evaluate whether candidate search keywords and text have or may reflect the above topics or are related to one or more of the above topics.

[0093] The goal of LLM is to determine the relevance level of each text to the aforementioned sensitive topics for evaluation. In this example, the levels are defined as irrelevant, weakly relevant, moderately relevant, strongly relevant, or extremely strongly relevant as evaluation thresholds. This requires LLM to provide a brief rationale along with the evaluation results. Only text information judged by the model to be moderately relevant or higher is retained, while results with weak or irrelevant relevance are discarded.

[0094] This multi-dimensional definition is more credible than factual judgment because some content related to sensitive topics may not be closely related to the core meaning of the entire image. Setting different levels helps the model analyze in subsequent judgments, rather than becoming overly sensitive. In short, this semantically relevant screening process ensures that subsequent fusion only focuses on external information highly relevant to the potentially illegal content of the image, avoiding irrelevant background information from interfering with the review judgment.

[0095] Therefore, the following is an example of the evaluation prompt words designed for LLM:

[0096] "This is a sentence: {RAG result}. There are currently some situations that require special review, specifically including: Category 1, Category 2, ..., Category N, etc."

[0097] Please explain the degree of relevance between this sentence and these scenarios. First, describe the degree of relevance using [extremely strong / strongly relevant / moderately relevant / weakly relevant / irrelevant], then briefly explain your reasoning.

[0098] Standard output template:

[0099] Example output 1: "Person A is sitting in a wheelchair, and his appearance is extremely similar to an emoji" - Correlation level: Very strong;

[0100] Reasoning: Person A is a patient with a certain disease, and saying that their appearance resembles an emoji is a serious offense to Person A and the public.

[0101] Example output 2: "A portrait of person B is placed on the desk." - Relevance: Extremely strong;

[0102] Reason for judgment: Person B is a symbol of a special meaning and is highly related to the aforementioned sensitive characteristics.

[0103] Example output 3: "A woman is walking on the beach." - Relevance level: weak;

[0104] Reasoning: Although the word "woman" is related to gender, it is difficult to determine its connection with the hate scene based on this sentence alone.

[0105] Output requirements: Please output according to the format given in the example. Do not output any information other than the standard output.

[0106] In the example of the evaluation prompts above, the content from the search enhancement results should be filled in the parentheses {RAG results}.

[0107] S3.2, Integration of text and image information.

[0108] The purpose of image-text information fusion is to combine the filtered search-related information with the visual content of the image itself, further enriching and refining the previous image description. In this step, the LLM system prompts and the initial description come from the original image description step. After obtaining the description, the model is further prompted to fuse the search information and the description.

[0109] Specifically, an iterative strategy is employed to sequentially introduce each relevant retrieval text that is retained. In each iteration, the model receives three inputs: 1) the retrieval enhancement content to be fused; 2) local image regions or visual features related to this content; and 3) the image description generated in the previous round (the original description is used in the first round). This step is designed to guide the model to objectively integrate new textual information into the existing description, maintaining the accuracy of the original description while incorporating background or explanations provided by external knowledge. After processing these multimodal inputs, the model outputs an updated image description, integrating the information provided by the retrieval text into the representation of the image. As iteration progresses, this fusion process is performed sequentially for each relevant text, and the image description is gradually supplemented to become more complete and semantically deeper. The final image description integrates the explicit content of the image and implicit information related to sensitive topics, more comprehensively reflecting the possible meanings of the image and providing richer evidence for the final judgment of content review.

[0110] Let the original description of the image be... And there are If related search results need to be merged, then the final iteratively obtained fused description... for:

[0111] ,

[0112] in, This represents the aggregation process of LLM.

[0113] Therefore, the following is an example of the aggregated prompt words designed for LLM:

[0114] "Complete the description obtained in the previous step using the given sentence ({RAG result}). Output only the new description after completion."

[0115] In the example of the aggregated suggestion above, the {RAG result} field should be filled with the corresponding search enhancement content. The initial result is the original image description, but for each iteration, a previously fused description must be provided. This iterative process continues until the final fused description is generated.

[0116] S4 constructs a training dataset containing positive and negative decision sample pairs based on the fusion description, fine-tunes the visual language model, and uses the fine-tuned visual language model to perform sensitive content review on new images.

[0117] The embodiment specifically includes the following sub-steps:

[0118] S4.1, Scenario-based audit.

[0119] To address the issue of inaccurate values ​​in Visual Semantic Analytical Models (VLMs), this invention employs Direct Preference Optimization (DPO) to fine-tune VLMs, thereby improving their objectivity in visual semantic risk governance tasks. The DPO method requires the training dataset to simultaneously include both accept and reject responses from the model, achieving alignment with human values ​​through contrastive learning.

[0120] In image review scenarios, the core of the review is a binary classification problem, namely, determining whether an image violates regulations. Therefore, the dataset construction should provide two interpretations: the reason for an image's violation and the reason for its non-violation. If the image's label is "harmful," it indicates that the original image violates regulations, and the output "violation and reason for violation" represents the model's acceptance response. Conversely, if the image is deemed non-violation and the output "non-violation and reason for non-violation," it represents the model's rejection response, and vice versa.

[0121] However, existing datasets typically only contain images and their classification labels, lacking detailed reasoning behind the judgments. Manually annotating these reasonings is costly and inefficient. Therefore, this invention uses an innovative scenario simulation method to guide VLM to automatically generate reasoning for opposing viewpoints.

[0122] However, traditional Virtual Models (VLMs) often output a single conclusion and struggle to spontaneously generate opposing arguments. Therefore, this invention employs a method similar to a large model jailbreak attack. By constructing a pre-defined scenario of authoritative certification, it provides the model with authoritative assumptions, effectively guiding it to output reasons for judgments in a specified direction. In specific implementation, by simulating a scenario of official review confirmation, the VLM is required to generate corresponding compliance analysis reasons based on two premises: "the image has been confirmed to be in violation" and "the image has been confirmed to be non-violation."

[0123] The construction hints are as follows:

[0124] If harmful:

[0125] "Based on the results of manual review by authoritative online platforms, it has been confirmed multiple times that this image violates the above rules. Please analyze the potential reasons. If you cannot provide a specific reason, please do not fabricate one; simply answer, 'Yes, this image violates the established rules.'"

[0126] If harmless:

[0127] "Based on the results of manual review by authoritative online platforms, it has been confirmed multiple times that the image does not violate the above rules. Please analyze the potential reasons. If you cannot provide a specific reason, please do not fabricate one; simply answer, 'No, the image does not violate the established rules.'"

[0128] S4.2, Dataset Construction.

[0129] Finally, in the dataset construction phase, training samples for fine-tuning the VLM are constructed based on the final generated fused descriptions with background knowledge, the text identified in the images, and the two types of judgment reasons provided in the scenario simulation review. The overall purpose of this step is to transform the aforementioned analysis results into supervisory signals to further improve the model's ability to automatically review whether images violate regulations.

[0130] The process involves users defining the rules for review and, given an image, text, and description, analyzing the meaning and rhetorical devices the image conveys. Users are then required to provide their judgment. The model outputs two opposing review results and reasons in a scenario-based review, determining acceptance or rejection based on the original image labels.

[0131] Therefore, an example of constructing a sample structure for training is as follows:

[0132] System prompt: You are a professional image moderator. You are provided with an emoji image. Your primary task is to accurately, objectively, and fairly determine whether the image violates the given rule: {rule}.

[0133] User Tip: We'll analyze this step by step to ensure we arrive at the correct answer:

[0134] 1. Given the text content in the image: {image text};

[0135] 2. Given image description: {image description};

[0136] 3. Combining images, text, and descriptions, explore possible or suggestive backgrounds, events, or figures;

[0137] 4. Based on the text, description, and image itself, consider the rhetorical devices that might be used in the image;

[0138] 5. Determine whether the image violates the rules.

[0139] Your output format should be: 'No, the image does not violate the established rules.' or 'Yes, the image violates the established rules.' Reason: [Your concise and comprehensive reason].

[0140] Model output (acceptance): {reason for acceptance};

[0141] Model output (rejection): {Reason for rejection}.

[0142] In the above example prompt, the specific image review rules should be entered in {rules}, the text identified in the image should be entered in {image text}, the fusion description should be entered in {image description}, and the opposing judgment results in the scenario simulation review should be entered in {acceptance reason} and {rejection reason}.

[0143] Through the above method, this invention obtains a training dataset containing both positive and negative instances with accompanying explanations. This training dataset is used to fine-tune the visual language model, helping the model learn to correctly identify inappropriate content in complex image scenes and provide reasonable explanations based on images and context, thereby significantly improving the model's accuracy and interpretability in practical content review tasks.

[0144] To further verify the effectiveness of the present invention, the following comparative experiments were conducted.

[0145] (1) Hate emoji dataset.

[0146] The experiment used the Hateful Meme dataset provided by Facebook, which defines sensitive content (hate speech) and contains approximately 12,000 meme images with accompanying text. These images were manually labeled to indicate whether they contained hate speech. Additionally, the Harmeme COVID-19 dataset was used. This dataset has a similar moderation objective to the Facebook Hateful Meme dataset, aiming to detect harmful content in internet memes. The data primarily comes from Google search and social media platforms.

[0147] (2) Comparison method.

[0148] The method of this invention uses VLMLlama3.2-V-11B and Qwen2.5-VL-7B-Instruct models as base models and performs fine-tuning training. In the comparison method, the detection method of VLMLlama3.2-V-11B and Qwen2.5-VL-7B-Instruct models is to directly judge the image (reference title: UnsafeBench: Benchmarking ImageSafety Classifiers on Real-World and AI-Generated Images). Furthermore, due to the lack of mature commercial methods for detecting high-dimensional semantics, comparative methods for hate memes include two academic approaches: Pro-Cap (reference title: Pro-cap: Leveraging a frozen vision-language model for hateful memedetection) and ExplainHM (reference title: Towards explainable harmful meme detection through multimodal debate between large language models). Pro-Cap utilizes visual and textual features extracted by a frozen model to identify hate speech in image-text combinations. ExplainHM is a state-of-the-art method that aims to achieve explainable detection of harmful memes by introducing a large language model and employing a multimodal debate mechanism.

[0149] Pro-Cap provides versions of the model trained on two datasets, while ExplainHM provides a version trained on the Harmeme COVID-19 dataset. This invention labels the model trained on the Facebook Hateful Meme dataset as Pro-Cap_F, and the models trained on the Harmeme dataset as Pro-Cap_H and ExplainHM_H, respectively.

[0150] (3) Experimental results.

[0151] (a) Results of the effectiveness of hate speech detection

[0152] Table 1 presents the comparison methods and the detection results after training using our proposed method. The results of the pedestal detection methods using Llama3.2-V-11B and Qwen2.5-VL-7B-Instruct show that Qwen2.5-VL-7B-Instruct has better basic performance than Llama3.2-V-11B. However, even using the latest and more capable VLM, improvements are still needed in the detection of hate speech emojis. Our proposed method performs retrieval enhancement queries and fine-tuning on the original, poorly performing VLM model. The results show better performance compared to the original pedestal model and existing hate speech emoji detection methods, demonstrating the effectiveness of our proposed method in detecting hate speech emojis.

[0153] Table 1 Results of Hate Speech Detection Effectiveness

[0154]

[0155] (b) Robustness of hate speech detection results

[0156] In the experiments, a single dataset was used for training, and different datasets were used for testing to demonstrate the robustness of this method across datasets. The results are shown in Table 2. Methods ending in _F in the table indicate training on data from the Facebook HatefulMeme dataset, and methods ending in _H indicate training on data from the Harmeme COVID-19 dataset. The results show that, compared to ExplainHM and Pro-Cap, the detection performance of this method after fine-tuning on a single dataset does not decrease significantly across datasets, especially in terms of precision, recall, and F1 score. Furthermore, compared to the two methods mentioned above that use the entire dataset for fine-tuning, this method only extracts 500 data points for fine-tuning. This demonstrates that this method achieves better results with a smaller amount of data and has stronger generalization ability for different content types. This further proves the feasibility of practical deployment of this method. In practice, if new categories or expressions are encountered, only a small number of samples need to be collected and fine-tuned using this method's workflow to achieve efficient detection results.

[0157] Table 2. Robustness test results

[0158]

[0159] (c) Design effectiveness results

[0160] This experiment demonstrates the effectiveness of the steps designed in this method: 1) the improvement in understanding high-dimensional semantics based on the original image description; 2) the improvement in accuracy by incorporating retrieval augmentation data; and 3) the improvement in performance by fine-tuning training. The results are presented in Table 3. The table lists various combinations to demonstrate the effectiveness of the design steps in this method.

[0161] Among them, the row containing only "model name" indicates the result of directly using two models for detection; "+Prompt" indicates that a high-dimensional semantic prompt was constructed based on the original image description; "+RAG" indicates that the query result was fused into the original description after using retrieval enhancement query and then detection was performed; "+Train" indicates that fine-tuning training was performed.

[0162] The experiment was conducted on manually labeled data.

[0163] Table 3. Results of Design Validity

[0164]

[0165] Experimental results show that each design step has an improving effect. Retrieval enhancement queries can effectively increase background knowledge in images, improving the visual model's ability to understand images. Simultaneously, training can fine-tune the model's values; after fine-tuning, the model's generalization ability is improved, and it can make judgments using more uniform standards. Combining the above methods and steps of this invention yields better final detection accuracy.

[0166] In summary, this invention provides an image-sensitive content moderation method. To address the shortcomings of Visual Language Models (VLMs) in effectively integrating external knowledge and their susceptibility to knowledge obsolescence or generation illusions in real-world tasks, it introduces a retrieval-enhanced generation method, enriching image descriptions by searching for background knowledge through an image search engine. To solve the problem of VLMs' difficulty in effectively understanding the high-dimensional semantics of human society regarding sensitive content, a content description fusion strategy is proposed, combining novel understandings of human society with the original image description. To address the discrepancy between the values ​​of language models and the real world, a scenario simulation strategy is used to provide reasons for opposing judgments, and a DPO fine-tuning strategy is employed to fine-tune VLM. Experimental results on a hate speech meme dataset demonstrate that this method effectively improves VLM's understanding and efficiency in moderizing sensitive content.

[0167] Based on the same inventive concept, such as Figure 3 As shown, this embodiment of the invention also provides an image-sensitive content system 300, including: an image original description module 310, a retrieval enhancement query module 320, a content description fusion module 330, and a model fine-tuning module 340.

[0168] The image original description module 310 is used to obtain the original description of the image to be reviewed. The original description includes a text summary of the visual content of the image and the text content present in the image.

[0169] The enhanced search module 320 is used to extract entity regions in the image to be reviewed. After retrieving relevant external background information based on the image to be reviewed and entity regions, it uses the labels of entity regions to exclude interference information and generates enhanced search results.

[0170] The content description fusion module 330 is used to perform semantic relevance filtering on the search enhancement results based on sensitive topics, and to iteratively fuse the filtered search enhancement results into the original description to generate the final fused description.

[0171] The model fine-tuning module 340 is used to construct a training dataset containing positive and negative decision sample pairs based on the fusion description, fine-tune the visual language model, and use the fine-tuned visual language model to perform sensitive content review on new images.

[0172] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described image-sensitive content method when executing the computer program.

[0173] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described image-sensitive content method.

[0174] It should be noted that the image-sensitive content review system, electronic device, and computer-readable storage medium provided in the above embodiments all belong to the same inventive concept as the image-sensitive content review method. For details of their specific implementation process, please refer to the embodiment of the image-sensitive content review method, which will not be repeated here.

[0175] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for reviewing image-sensitive content, characterized in that, Includes the following steps: Obtain the original description of the image to be reviewed. The original description includes a textual summary of the visual content of the image and the text content present in the image. The process involves extracting entity regions from the image to be reviewed, retrieving relevant external background information based on the image and entity regions, eliminating interference information using entity region labels, and generating enhanced search results. This includes: identifying entities in the image using a target detection model and segmenting and extracting multiple entity regions and their labels from the image to be reviewed; using an image search engine to retrieve relevant external background information text from the image and its multiple entity regions; and using a large language model to aggregate and eliminate interference information based on the entity region labels, generating enhanced search results. The labels are obtained by annotating elements in the image using a model. The enhanced search results are semantically relevant based on sensitive topics, and the enhanced search results are iteratively integrated into the original description to generate the final integrated description. A training dataset containing positive and negative decision pairs is constructed based on the fusion description. The visual language model is then fine-tuned, and the fine-tuned visual language model is used to perform sensitive content review on new images.

2. The image-sensitive content review method according to claim 1, characterized in that, The generated suggestion words for assisting the large language model in performing retrieval enhancement are configured as follows: The large language model is guided to perform semantic aggregation on the text attributes of the input external background information and generate a structured description, filter information that is irrelevant to the labels of entity regions, and output a description that focuses on the attributes of entities and does not involve entity ontology.

3. The image-sensitive content review method according to claim 1, characterized in that, The step of performing semantic relevance filtering on the enhanced search results based on sensitive topics includes: Based on a predefined list of sensitive topics, a large language model is used to evaluate the relevance of the search enhancement results, and only the search enhancement results with a relevance that reaches a preset threshold are selected and retained.

4. The image-sensitive content review method according to claim 3, characterized in that, The evaluation prompts used to assist the large language model in performing semantic relevance filtering are configured as follows: The large language model is required to output a relevance level and the reason for the judgment based on the search enhancement results and the predefined list of sensitive topics. The relevance level should include at least one of the following: irrelevant, weakly relevant, moderately relevant, strongly relevant, and extremely strongly relevant.

5. The image-sensitive content review method according to claim 1, characterized in that, The step of iteratively integrating the filtered enhanced search results into the original description to generate the final integrated description includes: The process employs an iterative approach, sequentially fusing each filtered enhanced search result with the current image description into the large language model to output an updated image description. In the first iteration, the original description is used as the current image description. By iterating multiple times, external knowledge is gradually integrated into the image description to obtain the final fused description.

6. The image-sensitive content review method according to claim 1, characterized in that, The step of constructing a training dataset containing positive and negative decision sample pairs based on fusion description, and fine-tuning the visual language model, includes: The visual language model is guided to generate judgments on whether an image is in violation or not, along with the reasons for the judgments, through scenario simulation review. Based on the actual review label of the image to be reviewed, the judgment results and reasons that match the actual review label are marked as accepted responses and correspond to positive samples, while the judgment results and reasons that contradict the actual review label are marked as rejected responses and correspond to negative samples. Construct structured training samples that include system prompts, user queries, accepted responses, and rejected responses, where user queries include original descriptions, fused descriptions, and guided analysis prompts; Based on the training dataset consisting of structured training samples, the visual language model is fine-tuned using the direct preference optimization method.

7. An image-sensitive content review system, implemented using the image-sensitive content review method according to any one of claims 1 to 6, characterized in that, include: Image original description module, retrieval enhancement query module, content description fusion module, model fine-tuning module; The original image description module is used to obtain the original description of the image to be reviewed. The original description includes a textual summary of the visual content of the image and the text content present in the image. The search enhancement query module is used to extract entity regions in the image to be reviewed, retrieve relevant external background information based on the image to be reviewed and entity regions, and use the labels of entity regions to eliminate interference information and generate search enhancement results. The content description fusion module is used to perform semantic relevance filtering on the search enhancement results based on sensitive topics, and to iteratively fuse the filtered search enhancement results into the original description to generate the final fused description. The model fine-tuning module is used to construct a training dataset containing positive and negative decision sample pairs based on the fusion description, fine-tune the visual language model, and use the fine-tuned visual language model to perform sensitive content review on new images.

8. An electronic device comprising a memory and one or more processors, the memory for storing a computer program, characterized in that, The processor is used to implement the image-sensitive content review method according to any one of claims 1 to 6 when executing a computer program.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the image-sensitive content review method according to any one of claims 1 to 6.

Images