Multimodal content review method and apparatus, computer device, and storage medium

Abstract

The application provides a multi-modal content review method and device, computer equipment and a storage medium, comprising: in the case that the to-be-reviewed image meets a preset risk condition, performing an adversarial purification process on the to-be-reviewed image to obtain a purified image; performing prompt attack detection on input prompt information, and determining purified prompt information corresponding to the input prompt information based on a detection result; constructing a plurality of review paths according to at least one of the to-be-reviewed image and the purified image, the purified prompt information and a preset prompt word template set; reviewing multi-modal content through the plurality of review paths to obtain path review results; performing cross-path consistency verification on the path review results to obtain a consistency verification result; and performing fusion processing on the path review results according to the consistency verification result to obtain a target review result corresponding to the multi-modal content. In this way, the accuracy, stability and reliability of the multi-modal content review result can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically to a multimodal content review method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the development of artificial intelligence technology, multimodal models such as visual language models are widely used in content moderation scenarios. Multimodal content moderation can comprehensively assess whether the content to be reviewed involves violations, sensitivity, or inappropriate display by combining image content and text prompts. Compared to single image or single text moderation methods, multimodal content moderation can better understand the relationship between image content and text information, and therefore has high application value in scenarios such as internet content security, social platform governance, advertising moderation, and intelligent customer service.

[0003] In related technologies, visual language models can be used to jointly understand the image to be reviewed and the corresponding prompts, and output the content review result. However, in practical applications, the image to be reviewed may be subject to adversarial perturbations, or the input prompts may contain malicious content. Such adversarial perturbations or malicious content are usually not easily detected by humans, but may affect the reasoning process of the review model, causing the review model to output a review result that is inconsistent with the actual content of the image. Summary of the Invention

[0004] In view of this, this application provides a multimodal content moderation method, apparatus, computer equipment, and storage medium.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] In a first aspect, embodiments of this application provide a multimodal content review method, the method comprising: Obtain multimodal content to be reviewed, wherein the multimodal content includes an image to be reviewed and input prompt information corresponding to the image to be reviewed; If the image to be reviewed meets the preset risk conditions, the image to be reviewed is subjected to adversarial purification processing to obtain a purified image; and, the input prompt information is subjected to prompt attack detection, and the purified prompt information corresponding to the input prompt information is determined based on the detection result, wherein the prompt attack detection is used to detect whether there is malicious prompt content in the input prompt information; Based on at least one of the image to be reviewed and the cleaned image, the cleaned prompt information and the set of preset prompt word templates, multiple review paths are constructed, wherein the input content and / or review model configurations corresponding to different review paths are different, and the input content includes the input image and the review prompt information; The multimodal content is reviewed through multiple review paths to obtain the review results for each path. Cross-path consistency verification is performed on the audit results of each path to obtain consistency verification results, wherein the consistency verification results are used to characterize the degree of difference between the audit results of different audit paths; Based on the consistency verification results, the review results of each path are merged to obtain the target review result corresponding to the multimodal content.

[0007] In one possible implementation, the image to be reviewed satisfies a preset risk condition, including: performing adversarial perturbation detection on the image to be reviewed to obtain adversarial risk information corresponding to the image to be reviewed; and determining that the image to be reviewed satisfies the preset risk condition if the adversarial risk information satisfies the preset risk condition; wherein the adversarial risk information is determined based on at least one of the frequency domain features and gradient sensitivity features of the image to be reviewed.

[0008] In one possible implementation, the step of performing adversarial cleansing on the image to be reviewed to obtain a cleaned image includes: determining at least one target image transformation operator from a preset set of image transformation operators; performing image transformation processing on the image to be reviewed based on the at least one target image transformation operator to obtain a candidate cleaned image; determining the structural similarity between the candidate cleaned image and the image to be reviewed; if the structural similarity meets a preset similarity condition, determining the candidate cleaned image as the cleaned image; if the structural similarity does not meet the preset similarity condition, adjusting the transformation parameters of the target image transformation operator, and re-performing the image to be reviewed based on the adjusted transformation parameters.

[0009] In one possible implementation, the preset prompt word template set includes standard prompt words, which are prompt information used to instruct the review model to review the content of the image; the construction of multiple review paths includes: determining variant prompt words based on the standard prompt words, wherein the variant prompt words have the same review purpose as the standard prompt words but are expressed differently; and constructing the multiple review paths according to at least one of the image to be reviewed and the cleaned image, at least one of the standard prompt words and the variant prompt words, and the cleaned prompt information.

[0010] In one possible implementation, the review model configuration includes at least one of a visual language model instance, an inference temperature parameter, an attention discard rate, and a cue word configuration; wherein, the review model configuration is different for different review paths, including: different review paths correspond to different visual language model instances, different inference temperature parameters, different attention discard rates, and different cue word configurations.

[0011] In one possible implementation, the path audit result includes at least one of audit category, confidence level, and output feature; the step of performing cross-path consistency verification on each of the path audit results to obtain a consistency verification result includes: determining feature difference information between different audit paths based on the output features in each of the path audit results; and determining the consistency verification result based on the feature difference information.

[0012] In one possible implementation, the output features include semantic embedding vectors; determining the feature difference information between different review paths based on the output features in the review results of each path includes: calculating the distance between the semantic embedding vectors corresponding to different review paths to obtain semantic distance information; determining the feature difference information between the different review paths based on the semantic distance information; and performing fusion processing on the review results of each path based on the consistency verification result to obtain the target review result corresponding to the multimodal content includes: determining the fusion weight corresponding to each review path based on the consistency verification result; and performing weighted fusion processing on the review results of each path based on the fusion weight corresponding to each review path to obtain the target review result.

[0013] Secondly, embodiments of this application provide a multimodal content review device, the device comprising: The content acquisition module is used to acquire multimodal content to be reviewed, wherein the multimodal content includes an image to be reviewed and input prompt information corresponding to the image to be reviewed; The purification detection module is used to perform adversarial purification processing on the image to be reviewed when the image to be reviewed meets the preset risk conditions, so as to obtain a purified image; and to perform prompt attack detection on the input prompt information, and determine the purification prompt information corresponding to the input prompt information based on the detection result. The path construction module is used to construct multiple review paths based on at least one of the image to be reviewed and the cleaned image, the cleaned prompt information, and a set of preset prompt word templates; The path review module is used to review the multimodal content through multiple review paths and obtain the path review results corresponding to each review path. The consistency verification module is used to perform cross-path consistency verification on the audit results of each path and obtain the consistency verification result. The fusion processing module is used to perform fusion processing on the review results of each path based on the consistency verification results, so as to obtain the target review result corresponding to the multimodal content.

[0014] Thirdly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method as described in the first aspect or any possible implementation of the first aspect.

[0015] Fourthly, this application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method as described in the first aspect or any possible implementation of the first aspect.

[0016] The multimodal content moderation method provided in this application first acquires multimodal content including an image to be reviewed and input prompts. If the image to be reviewed meets preset risk conditions, it undergoes adversarial purification processing to obtain a purified image. The input prompts are then subjected to prompt attack detection to determine the purified prompts. Subsequently, multiple review paths are constructed based on at least one of the image to be reviewed and the purified image, the purified prompts, and a preset set of prompt word templates. The multimodal content is then reviewed through these multiple review paths. Afterward, cross-path consistency verification is performed on the review results of each path, and the review results of each path are fused based on the consistency verification results to obtain the target review result. Through this method, on the one hand, adversarial disturbances and malicious prompts that may affect the review results can be processed before the review model performs content review, reducing the impact of abnormal input on the review model. On the other hand, differentiated review of the same multimodal content can be performed through multiple review paths, and abnormal differences between different review paths can be detected based on cross-path consistency verification. This reduces the probability of incorrect review results when a single review path is attacked or misled, thereby improving the accuracy, stability, and reliability of the multimodal content review results. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a multimodal content moderation method according to an exemplary embodiment of this application; Figure 2 This is a schematic diagram of the structure of an anti-disturbance detection module shown in an exemplary embodiment of this application; Figure 3 This is a diagram illustrating a multi-path parallel review architecture in an exemplary embodiment of this application; Figure 4This is a flowchart illustrating a semantic consistency verification process in an exemplary embodiment of this application; Figure 5 This is an overall flowchart illustrating a multimodal content moderation method according to an exemplary embodiment of this application; Figure 6 This is a schematic diagram of the architecture of a multimodal content moderation device shown in an exemplary embodiment of this application; Figure 7 This is a schematic diagram of the structure of a computer device shown in an exemplary embodiment of this application. Detailed Implementation

[0018] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0019] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0020] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0021] All user-related data involved in the technical solution of this application are obtained with the user's authorization, such as through pop-up windows or by inviting individuals to upload their own data.

[0022] In multimodal content moderation scenarios, the content to be reviewed can include images, text, video frames, and combinations of images and text. For ease of explanation, the following mainly uses the example of content to be reviewed including an image and the corresponding input prompts. The input prompts can be user-inputted prompts, prompts generated by the system based on the review task, or prompts resulting from a combination of user input and system review instructions.

[0023] In practical applications, images to be reviewed may contain adversarial perturbations. For example, attackers can add small but aggressive perturbations to an image, making the changes difficult for the human eye to perceive, but causing a shift in the output of review models such as visual language models. Input prompts may also contain malicious content. For example, attackers can use role-playing inducements, security constraint bypassing, output format manipulation, and abnormal character injection to make the review model ignore the original security review requirements. Based on this, this application provides a multimodal content review method that improves the reliability of multimodal content review results through image risk processing, malicious perturbation detection, multi-path review, cross-path consistency verification, and fusion decision-making.

[0024] See Figure 1 The diagram shown is a flowchart of a multimodal content moderation method provided in this application. This method can be applied to servers, terminal devices, content moderation platforms, or other computer devices with data processing capabilities. The multimodal content moderation method may include the following steps S101 to S106.

[0025] S101. Obtain the multimodal content to be reviewed.

[0026] The multimodal content includes the image to be reviewed and the input prompt information corresponding to the image to be reviewed.

[0027] In one possible implementation, the image to be reviewed can be a user-uploaded image, a video frame extracted from a video, an image from a webpage, an advertising material image, or other images that require content review. Input prompts can be natural language prompts entered by the user.

[0028] Optionally, the multimodal content to be reviewed may include both the image to be reviewed and the input prompt information, or it may include the image to be reviewed, the input prompt information, and other contextual information related to the image to be reviewed, such as the image source, publication time, user-submitted descriptive text, content tags, etc. This type of contextual information can be used as supplementary information in subsequent review processes.

[0029] S102. If the image to be reviewed meets the preset risk conditions, perform adversarial purification processing on the image to be reviewed to obtain a purified image; and perform prompt attack detection on the input prompt information, and determine the purification prompt information corresponding to the input prompt information based on the detection result.

[0030] In one possible implementation, preset risk conditions can be used to characterize the risk of adversarial disturbances or other risks that may affect the stability of the review results in the image to be reviewed. If the image to be reviewed meets the preset risk conditions, adversarial cleansing can be performed on the image to obtain a cleaned image. The cleaned image can be used in conjunction with the image to be reviewed in subsequent review paths, thereby reducing the impact of abnormal image-side disturbances on the review results.

[0031] Simultaneously, the system can perform attack detection on input prompts. This detection can be used to check for malicious content within the input prompts. If malicious content is detected, the input prompts can be cleaned up to obtain cleaned prompts; if no malicious content is detected, the input prompts themselves, or the information obtained by combining the input prompts with preset system verification instructions, can be used as cleaned prompts.

[0032] S103. Construct multiple review paths based on at least one of the image to be reviewed and the cleaned image, the cleaned prompt information, and the preset prompt word template set.

[0033] The input content and / or review model configurations for different review paths are different, and the input content includes input images and review prompt information.

[0034] In one possible implementation, the preset prompt template set may include standard prompts, which can be prompts used to instruct the review model to perform content review on images. The standard prompts can be pre-set by the system to specify the review task, review category, output format, etc. Furthermore, variant prompts can be determined based on the standard prompts, wherein the variant prompts have the same review purpose as the standard prompts but are expressed differently.

[0035] When constructing multiple review paths, the input content for each review path can be determined based on at least one of the image to be reviewed and the cleaned image, as well as at least one of the standard prompt words, variant prompt words, and cleaned prompt information. Simultaneously, different review model configurations can be configured for different review paths, such as different visual language model instances, different inference temperature parameters, and different attention dropout rates. This allows for differences in input images, review prompt information, and / or review model configurations between different review paths.

[0036] S104. The multimodal content is reviewed through multiple review paths to obtain the path review results corresponding to each review path.

[0037] In one possible implementation, each review path can invoke a corresponding review model to review the input content and obtain a path review result. The path review result can include at least one of review category, confidence level, and output features. The review category can be used to characterize the category corresponding to the content to be reviewed, the confidence level can be used to characterize the degree of confidence of the review model in the review category, and the output features can include semantic embedding vectors, hidden layer features, text output features, or other features that can be used to measure the differences in results between different review paths.

[0038] In practice, multiple audit paths can be executed in parallel or in a preset order. After multiple audit paths have completed their audits, the corresponding audit results can be sent to the subsequent cross-path consistency verification process.

[0039] S105. Perform cross-path consistency verification on the audit results of each path to obtain the consistency verification results.

[0040] The consistency verification result is used to characterize the degree of difference between the audit results of different audit paths.

[0041] In one possible implementation, feature differences between different review paths can be determined based on the output characteristics in the review results of each path, and the consistency verification result can be determined based on the feature differences. If the differences between different review paths are small, the review results of multiple review paths can be considered relatively consistent; if the differences between different review paths are large, consistency anomalies can be considered to exist between multiple review paths, and these consistency anomalies may be related to factors such as adversarial perturbations, hint attacks, or boundary samples.

[0042] By performing cross-path consistency checks, potential risks can be identified based on the degree of difference between multiple audit paths without relying entirely on the output of a single audit model, thereby reducing the impact of anomalies in a single audit path on the final audit result.

[0043] S106. Based on the consistency verification result, the audit results of each path are fused to obtain the target audit result corresponding to the multimodal content.

[0044] In one possible implementation, the fusion weight corresponding to each audit path can be determined based on the consistency verification results. Then, based on the fusion weight corresponding to each audit path, the audit results of each path are weighted and fused to obtain the target audit result. The target audit result may include one or more of the following: final audit category, final confidence level, risk label, review recommendation, and interpretability report.

[0045] If the consistency check result indicates normal consistency among the audit paths and the fused result has a high confidence level, a deterministic target audit result can be output. If the consistency check result indicates abnormal consistency among the audit paths, a high-risk flag can be output, and a manual review is required. This approach ensures that the final audit result considers both the audit results of each audit path itself and the consistency between them, thereby improving the reliability of the target audit result.

[0046] In the above embodiments, the overall processing flow of the multimodal content review method is first described through steps S101 to S106. For ease of understanding, the following description, in conjunction with specific implementation methods, further explains the adversarial disturbance detection, adversarial purification processing, prompt attack detection, review path construction, cross-path consistency verification, and fusion decision-making process involved in the above steps. It should be noted that the specific implementation methods described below can be used individually or in combination according to actual application scenarios and are not intended to limit the scope of protection of this application.

[0047] In one possible implementation, whether an image to be reviewed meets preset risk conditions can be determined by performing adversarial perturbation detection on the image. Specifically, adversarial perturbation detection can be performed on the image to be reviewed first to obtain adversarial risk information corresponding to the image; then, it can be determined whether the adversarial risk information meets preset conditions; if the adversarial risk information meets the preset conditions, it can be determined that the image to be reviewed meets the preset risk conditions. The adversarial risk information can be determined based on at least one of the frequency domain features and gradient sensitivity features of the image to be reviewed.

[0048] For example, see Figure 2 The diagram shown is a structural schematic of an adversarial perturbation detection module provided in this application. The adversarial perturbation detection module may include a frequency domain analysis submodule, a gradient sensitivity evaluation submodule, and a joint determination submodule. The frequency domain analysis submodule is used to perform frequency domain analysis on the image to be reviewed; the gradient sensitivity evaluation submodule is used to evaluate the gradient sensitivity corresponding to the image to be reviewed; and the joint determination submodule can determine the adversarial risk information (i.e., adversarial suspicion score) corresponding to the image to be reviewed based on the frequency domain analysis results and the gradient sensitivity evaluation results.

[0049] In one possible implementation, the process of performing adversarial disturbance detection on the image to be reviewed may include the following steps.

[0050] Step A1: Perform frequency domain analysis on the image to be reviewed.

[0051] Specifically, images that can be reviewed I Perform a two-dimensional discrete cosine transform to obtain the frequency domain representation. F :

[0052] in, I This indicates an image pending review. DCT Represents the two-dimensional discrete cosine transform. F This represents the frequency domain representation of the image to be reviewed. Then, based on the frequency domain representation... F Determine the energy proportion in the high-frequency region. For example, the energy proportion in the high-frequency region can be determined using the following formula:

[0053] in, This indicates the proportion of energy in the high-frequency region. Frequency domain representation Mid-frequency position Frequency domain coefficients at that location, This represents the frequency threshold. Countermeasures against disturbances typically manifest as an abnormal enhancement of high-frequency components; therefore, when... When the input image exceeds a preset threshold, it can be marked as a suspicious sample.

[0054] Step A2: Perform gradient sensitivity assessment on the image to be reviewed.

[0055] Specifically, the gradient sensitivity of the input image can be calculated using a pre-trained lightweight classification network:

[0056] Then, the gradient norm can be calculated:

[0057] in, Indicates the input image The corresponding input gradient, Indicates the input image Find the gradient. Represents the loss function. This represents a pre-trained lightweight classification network. Indicates a reference label. This represents the gradient norm. Adversarial examples are typically located near the decision boundary and exhibit high gradient sensitivity; therefore, when... When the input image exceeds a preset threshold, it can be marked as a suspicious sample.

[0058] Step A3: Based on the frequency domain analysis results and gradient sensitivity assessment results, determine the adversarial risk information corresponding to the image to be reviewed.

[0059] Specifically, adversarial risk information can be characterized by an adversarial suspicion score. The adversarial suspicion score can be calculated by combining frequency domain analysis and gradient sensitivity results.

[0060] in, The score indicates the degree of suspicion of the confrontation. and Indicates the weighting coefficient. This represents the activation function. This indicates the proportion of energy in the high-frequency region. This represents the high-frequency energy threshold in the frequency domain. Denotes the gradient norm. This represents the gradient sensitivity threshold. When... When the image exceeds a preset suspiciousness threshold, it can be determined that the image under review is a suspicious adversarial sample and that it meets preset risk conditions. This method combines frequency domain anomaly features and gradient-sensitive features for joint judgment of the image under review, thereby improving the reliability of adversarial perturbation detection.

[0061] In one possible implementation, if the image to be reviewed meets a preset risk condition, adversarial descaling can be performed on the image to be reviewed to obtain a cleaned image. The adversarial descaling process may include the following steps: First, determining at least one target image transformation operator from a preset set of image transformation operators; second, performing image transformation processing on the image to be reviewed based on the at least one target image transformation operator to obtain a candidate cleaned image; then, determining the structural similarity between the candidate cleaned image and the image to be reviewed; finally, determining whether to use the candidate cleaned image as the cleaned image, or whether it is necessary to adjust the transformation parameters of the target image transformation operator and perform image transformation processing again, based on whether the structural similarity meets a preset similarity condition.

[0062] Optionally, a set of transformation operators can be defined:

[0063] in, Represents the set of transformation operators. These represent different image transformation operators. For example, the set of transformation operators may include one or more of the following: JPEG compression operator, Gaussian blur operator, random crop rescaling operator, lightweight Gaussian noise injection operator, and color quantization operator.

[0064] In practice, one can start from the set of transformation operators. Random sampling k Several operators are executed in a random order:

[0065] in, This indicates an image pending review. Indicates a cleaned image or a candidate cleaned image. Indicates random arrangement. to Indicates determined by random arrangement A target image transformation operator. By using random sampling and random permutation, the purification process can be prevented from being predicted in advance by attackers, thereby improving the ability to handle unknown adversarial disturbances.

[0066] Furthermore, the structural similarity between the images before and after purification can be calculated:

[0067] in, Image pending review With purified images Structural similarity between them. If SSIM is below a threshold This indicates that the purification process may have excessively damaged the image content, requiring adjustment of the transformation parameters and re-execution. If SSIM meets the preset similarity condition, the candidate purified image can be determined as the purified image. In this way, adversarial perturbations can be eliminated or weakened while reducing the excessive destruction of the image's semantic content.

[0068] In one possible implementation, the input prompt information can be subjected to prompt attack detection, and the corresponding cleaned-up prompt information can be determined based on the detection results. Prompt attack detection can be used to detect whether there is malicious prompt content in the input prompt information. Malicious prompt content may include one or more of the following: role-playing inducement content, security constraint bypass content, output format manipulation content, abnormal character injection content, encoding obfuscation content, abnormal repeating sequences, and delimiter abuse content.

[0069] In one possible implementation, the process of detecting prompt attacks on input prompt information may include the following steps.

[0070] Step B1: Obtain the malicious prompt template library.

[0071] A malicious prompt template library can be represented as:

[0072] in, This indicates a malicious prompt template library. , ,…, This indicates multiple malicious message templates in the malicious message template library. This indicates the number of malicious prompt templates. Malicious prompt templates can include semantic representations of known jailbreak attack patterns, such as role-playing inducement, security constraint bypass, and output format manipulation.

[0073] Step B2: Perform semantic matching on the input prompt information.

[0074] Specifically, semantic embeddings can be extracted from the input prompt information, and the maximum similarity with a malicious prompt template library can be calculated:

[0075] in, This represents the maximum similarity between the input prompt message and the malicious prompt template library. This indicates a malicious prompt template library. This refers to any malicious prompt template in the malicious prompt template library. This indicates the semantic embedding corresponding to the input prompt information. Malicious warning template The corresponding semantic embedding, cosine similarity. When... If the similarity exceeds a preset threshold, a jailbreak attack alert can be triggered.

[0076] Step B3: Identify abnormal token patterns in the input prompt information.

[0077] Specifically, it can detect abnormal token patterns in the input prompt information, such as abnormal characters, abnormal length sequences, encoded obfuscated fields, and abnormal separators. If an abnormal token pattern is detected, it can be marked.

[0078] Step B4: Determine the purification prompt information based on the detection results.

[0079] If the input prompt contains malicious content or an abnormal token pattern, the input prompt can be cleaned up to obtain a cleaned prompt:

[0080] in, This indicates input prompt information; "Sanitize" indicates the purification of prompt words. This indicates a cleanup prompt. Cleanup operations can include removing abnormal characters, truncating excessively long sequences, and standardizing formats. If the input prompt does not contain malicious content or abnormal token patterns, the input prompt itself can be used as the cleanup prompt, or the input prompt can be combined with the system's preset security audit instructions to serve as the cleanup prompt.

[0081] It should be noted that the specific methods of removal, replacement, truncation, or format standardization in this application can be flexibly set according to the actual application scenario, and this application does not limit them. By performing attack detection and purification processing on input prompt information, the impact of malicious prompt content on the subsequent review model reasoning process can be reduced, thereby improving the security and stability of multimodal content review results.

[0082] In one possible implementation, the preset prompt word template set may include standard prompt words, which are prompts used to instruct the review model to perform content review on images. The standard prompt words can be pre-set by the system to specify the review task, review category, output format, etc. For example, standard prompt words can be used to instruct the review model to determine whether the image to be reviewed contains a target violation category and output the review category and confidence level.

[0083] Optionally, variant prompts can be determined based on standard prompts. These variant prompts share the same review purpose as the standard prompts but differ in expression. In other words, variant prompts are not review rules regenerated independently of the standard prompts, but rather prompts obtained by adjusting the expression of the standard prompts. Variant prompts can maintain consistency with the standard prompts in semantic review objectives, but differ in sentence structure, description order, expression of focus points, or expression of output requirements. Constructing different review paths using both standard and variant prompts can reduce the impact of a single prompt expression on the review results.

[0084] In one possible implementation, the process of constructing multiple review paths may include: first, determining an input image for constructing the review path, the input image may include at least one of an image to be reviewed and a cleaned image; second, determining review prompt information for constructing the review path, the review prompt information may include prompt information obtained based on standard prompt words, variant prompt words, and cleaned prompt information; then, determining the review model configuration corresponding to each of the multiple review paths; and finally, constructing multiple review paths according to the combination relationship between the input image, the review prompt information, and the review model configuration.

[0085] For example, multiple review paths may include: a review path using the image to be reviewed and standard prompts as input; a review path using a cleaned image and standard prompts as input; a review path using the image to be reviewed and variant prompts as input; and a review path using a cleaned image and variant prompts as input. By setting multiple review paths, multimodal content can be reviewed from the perspectives of different input images, different prompt expressions, and different model configurations, reducing the impact of abnormal review results from a single path on the final review result.

[0086] In some embodiments, the review model configuration may include at least one of a visual language model instance, an inference temperature parameter, and an attention dropout rate. Different review paths correspond to different review model configurations, which may include at least one of: different visual language model instances, different inference temperature parameters, and different attention dropout rates for different review paths.

[0087] For example, see Figure 3 The diagram shown illustrates a multi-path parallel review architecture provided in this application. Multiple differentiated Visual-Language Model (VLM) instances can form a VLM instance pool. The input allocator can distribute the image to be reviewed (i.e., the original image), the cleaned image, the standard prompt, the variant prompt, and the cleaned prompt information to different review paths according to the configuration of different review paths. The parallel execution engine coordinates the parallel inference of multiple review paths.

[0088] By constructing multiple review paths with different input content and / or review models, the redundancy and diversity of the review process can be improved. For adversarial or cue-based attacks, attackers need to influence multiple differentiated review paths simultaneously to cause errors in the final review result, thus increasing the difficulty of the attack and enhancing the robustness of the multimodal content review system.

[0089] In one possible implementation, after constructing multiple review paths, multimodal content can be reviewed separately through these paths, yielding review results for each path. After parallel inference execution along each path, a set of review results can be obtained.

[0090] in, Indicates the number of review paths. Indicates the first Each review path corresponds to a visual language model instance or review model instance. Indicates the first The input content corresponding to each audit path Indicates the first Each audit path corresponds to an audit result. Each audit result may include one or more of the following: audit category, confidence level, and semantic embedding vector.

[0091] For example, in a multi-path moderation configuration, four moderation paths can be constructed. The first moderation path can take the image to be moderated and the standard prompt as input and perform moderation using a first visual language model instance; the second moderation path can take the cleaned-up image and the standard prompt as input and perform moderation using a second visual language model instance; the third moderation path can take the image to be moderated and the variant prompt as input and perform moderation using an inference temperature parameter different from that of the first moderation path; the fourth moderation path can take the cleaned-up image and the variant prompt as input and perform moderation using an attention dropout rate different from that of the second moderation path.

[0092] Specifically, assuming the image to be reviewed is image A, the cleaned image is image A', the standard prompt word is P1, and the variant prompt word determined based on the standard prompt word is P2, the following review paths can be constructed: Path 1's input is "image A + standard prompt word P1"; Path 2's input is "image A' + standard prompt word P1"; Path 3's input is "image A + variant prompt word P2"; and Path 4's input is "image A' + variant prompt word P2". Each review path can call the corresponding visual language model instance for inference and output the review category, confidence level, and semantic embedding vector, respectively.

[0093] For example, path 1 might output "Violation, confidence 0.82", path 2 "Violation, confidence 0.88", path 3 "Suspicious, confidence 0.76", and path 4 "Violation, confidence 0.85". Since the outputs of each review path generally point to a higher risk category, and the distance between their semantic embedding vectors is small, subsequent cross-path consistency checks can determine that the consistency between the review paths is high, and the target review result is output after fusion processing. For another example, if path 1 outputs "Safe, confidence 0.91", while paths 2 through 4 all output "Violation" or "Suspicious", and the distance between the semantic embedding vector corresponding to path 1 and the semantic embedding vectors corresponding to other paths is large, then path 1 may be affected by adversarial perturbations or hint attacks. In subsequent fusion processing, the fusion weight corresponding to path 1 can be reduced, or a high-risk marker can be output with a prompt for manual review.

[0094] In one possible implementation, cross-path consistency verification can be achieved based on the output features in the review results of each path. These output features may include semantic embedding vectors. The distance between the semantic embedding vectors corresponding to different review paths can be calculated to obtain semantic distance information, and the feature difference information between different review paths can be determined based on this semantic distance information.

[0095] For example, see Figure 4 The diagram shown is a flowchart of a semantic consistency verification process provided in this application. Cross-path consistency verification may include processes such as semantic embedding extraction, distance matrix calculation, consistency score calculation, and anomaly detection.

[0096] In one possible implementation, cross-path consistency verification may include the following steps.

[0097] Step C1: Extract semantic embedding vectors from the review results of each path.

[0098] Specifically, the review result for each path can include the text content, category score, hidden layer features, or other output features output by the review model. The embedding extractor can extract the semantic embedding vector corresponding to each review path based on the text content or hidden layer features output by the review model.

[0099] Step C2: Calculate the semantic distance between different review paths.

[0100] Specifically, the semantic distance between any two review paths can be calculated using a distance calculator to obtain a semantic distance matrix. The semantic distance matrix can contain the semantic distance between any two review paths, and can be calculated using the following formula as an example:

[0101] in, Indicates the first The first review path and the first Semantic distance between each review path Indicates the first The semantic embedding vector corresponding to each review path Indicates the first Each review path corresponds to a semantic embedding vector. By calculating the semantic distance between different review paths, the degree of difference between the review results of different review paths can be measured.

[0102] Step C3: Determine the consistency score based on semantic distance.

[0103] The average consistency score can be calculated:

[0104] in, This represents the average consistency score. Indicates the number of review paths. Indicates the first The first review path and the first The semantic distance between each review path. If If the results are high, it can be assumed that the overall review results from multiple review paths are relatively consistent; if If the value is low, it can be assumed that there are significant differences between different review paths, which may indicate abnormal review situations caused by adversarial disturbances, alert attacks, or boundary samples.

[0105] Step C4: Determine anomalies based on consistency scores and / or semantic distance.

[0106] Specifically, anomaly detection can be performed using an anomaly checker. If the semantic distance between any pair of paths exceeds a preset distance threshold, or the average consistency score is lower than a preset consistency threshold, the consistency check result can be determined as an anomaly. If the semantic distance between any audited path does not exceed the preset distance threshold, and the average consistency score is not lower than the preset consistency threshold, the consistency check result can be determined as normal.

[0107] Cross-path consistency verification can identify potential risks based on the degree of difference between multiple review paths without relying entirely on the output of a single review model. For example, when adversarial perturbations in the image to be reviewed only affect the review path corresponding to the original image, while the output of the review path corresponding to the cleaned image is relatively stable, cross-path consistency verification can identify differences between paths, thereby triggering high-risk marking or adjusting subsequent fusion weights.

[0108] In one possible implementation, when merging the audit results of each path based on the consistency verification results, the fusion weight corresponding to each audit path can be determined according to the consistency verification results, and the audit results of each path can be weighted and merged based on the fusion weight corresponding to each audit path to obtain the target audit result.

[0109] In practice, the weights of each path can be dynamically adjusted based on the consistency score:

[0110] in, Indicates the first The fusion weight corresponding to each review path Indicates the first The confidence level or path credibility of each audit path Indicates the first The average semantic distance between each review path and other review paths Indicates the first The confidence level or path credibility of each audit path Indicates the first The average semantic distance between each review path and other review paths. This formula allows review paths that differ significantly from other paths to receive lower weights, while those that are highly consistent with other paths receive higher weights.

[0111] Furthermore, the review results from each path can be weighted and merged:

[0112] in, This indicates the target review result after integration. Indicates the number of review paths. Indicates the first The fusion weight corresponding to each review path Indicates the first The audit results for each audit path are then used to generate a final decision based on the merged target audit results and consistency status.

[0113] In one possible implementation, if the consistency check result indicates normal consistency among the audit paths and the confidence level of the fused result is high, a deterministic target audit result can be output. If the consistency check result indicates abnormal consistency among the audit paths, a high-risk marker can be output, and a manual review is required. Furthermore, an interpretability report can be generated, which may include one or more of the following: the audit result for each audit path, the semantic distance between paths, the average consistency score, the fusion weight, and risk assessment information.

[0114] By employing the aforementioned fusion processing method, the final review result can consider not only the confidence level of each review path's output but also the consistency between the review paths. When a path is affected by adversarial perturbations or malicious prompts and outputs abnormal results, the semantic differences between this path and other paths are usually significant. Therefore, its fusion weight can be reduced, thereby minimizing the impact of abnormal paths on the target review result.

[0115] The following is a general overview of the above methods, using the overall flowchart as an example. (See also...) Figure 5 The diagram shown is a flowchart of a multimodal content review method provided in this application. Figure 5In this process, the multimodal content to be reviewed can include images to be reviewed and input prompts. For images to be reviewed, image risk detection, such as adversarial perturbation detection, can be performed first. If the detection results indicate that the image to be reviewed has adversarial perturbation risks or meets preset risk conditions, adversarial cleansing can be performed on the image to be reviewed to obtain a cleaned image. If the detection results indicate that the image to be reviewed is normal, it can be directly used to construct subsequent review paths. For input prompts, prompt attack detection can be performed. If the detection results indicate that the input prompts contain malicious prompt content or have prompt attack risks, prompt words can be cleaned on the input prompts to obtain cleaned prompts. If the detection results indicate that the input prompts are normal, the input prompts themselves or information determined based on the input prompts can be used as cleaned prompts.

[0116] Furthermore, multiple review paths can be constructed based on at least one of the image to be reviewed and the cleaned image, cleanup prompts, and a set of preset prompt word templates. These multiple review paths can include path P1, path P2, ..., path PN, and different review paths can correspond to different input images, different review prompts, and / or different review model configurations. Each review path can review multimodal content separately and output corresponding path review results. The path review results can include at least one of review category, confidence level, and output features, wherein the output features can include semantic embedding vectors.

[0117] After obtaining the audit results for each of the multiple audit paths, cross-path consistency checks can be performed on these results to obtain a consistency check result. The consistency check result can characterize the degree of difference between the audit results of different audit paths; for example, it can characterize whether the output results of each audit path are consistent, or whether there is a situation where the output result of one path differs significantly from the output results of other paths. Then, based on the consistency check result, the audit results of each path can be fused to obtain the target audit result. The target audit result may include at least one of the following: a security audit result, a high-risk marker, a review recommendation, and an interpretability report. In this way, risk processing can be performed separately on the image side and the prompt word side, and the reliability of the final audit result can be improved through multi-path auditing, consistency checks, and fusion processing.

[0118] Corresponding to the embodiments of the aforementioned multimodal content moderation method, this application also provides embodiments of a multimodal content moderation device.

[0119] See Figure 6 The diagram shown is an architectural schematic of a multimodal content moderation device provided in this application. The multimodal content moderation device may include: The module includes a content acquisition module 601, a purification and detection module 602, a path construction module 603, a path review module 604, a consistency verification module 605, and a fusion processing module 606.

[0120] The content acquisition module 601 is used to acquire multimodal content to be reviewed, wherein the multimodal content includes an image to be reviewed and input prompt information corresponding to the image to be reviewed.

[0121] The purification detection module 602 is used to perform anti-purification processing on the image to be reviewed to obtain a purified image when the image to be reviewed meets the preset risk conditions; and to perform prompt attack detection on the input prompt information and determine the purification prompt information corresponding to the input prompt information based on the detection result, wherein the prompt attack detection is used to detect whether there is malicious prompt content in the input prompt information.

[0122] The path construction module 603 is used to construct multiple review paths based on at least one of the image to be reviewed and the cleaned image, the cleaned prompt information and the preset prompt word template set. The input content and / or review model configuration corresponding to different review paths are different. The input content includes the input image and the review prompt information.

[0123] The path review module 604 is used to review the multimodal content through multiple review paths and obtain the path review results corresponding to each review path.

[0124] The consistency verification module 605 is used to perform cross-path consistency verification on the audit results of each path to obtain a consistency verification result, wherein the consistency verification result is used to characterize the degree of difference between the audit results of different audit paths.

[0125] The fusion processing module 606 is used to perform fusion processing on the review results of each path according to the consistency verification result, so as to obtain the target review result corresponding to the multimodal content.

[0126] In one possible implementation, the purification detection module 602 is further configured to perform adversarial perturbation detection on the image to be reviewed to obtain adversarial risk information corresponding to the image to be reviewed; and determine that the image to be reviewed meets the preset risk conditions if the adversarial risk information meets the preset conditions; wherein the adversarial risk information is determined based on at least one of the frequency domain features and gradient sensitivity features of the image to be reviewed.

[0127] In one possible implementation, the purification detection module 602 is further configured to: determine at least one target image transformation operator from a preset set of image transformation operators; perform image transformation processing on the image to be reviewed based on the at least one target image transformation operator to obtain a candidate purified image; determine the structural similarity between the candidate purified image and the image to be reviewed; if the structural similarity meets a preset similarity condition, determine the candidate purified image as the purified image; if the structural similarity does not meet the preset similarity condition, adjust the transformation parameters of the target image transformation operator, and perform image transformation processing on the image to be reviewed again based on the adjusted transformation parameters.

[0128] In one possible implementation, the preset prompt word template set includes standard prompt words, which are prompt information used to instruct the review model to conduct content review of the image; the path construction module 603 is further configured to determine variant prompt words based on the standard prompt words, wherein the variant prompt words have the same review purpose as the standard prompt words but are expressed differently; and construct the multiple review paths according to at least one of the image to be reviewed and the cleaned image, at least one of the standard prompt words and the variant prompt words, and the cleaned prompt information.

[0129] In one possible implementation, the review model configuration includes at least one of a visual language model instance, an inference temperature parameter, an attention discard rate, and a cue word configuration; wherein, the review model configuration is different for different review paths, including: different review paths correspond to different visual language model instances, different inference temperature parameters, different attention discard rates, and different cue word configurations.

[0130] In one possible implementation, the path audit result includes at least one of audit category, confidence level, and output feature; the consistency verification module 605 is further configured to determine feature difference information between different audit paths based on the output features in each of the path audit results; and determine the consistency verification result based on the feature difference information.

[0131] In one possible implementation, the output features include semantic embedding vectors; the consistency verification module 605 is further configured to calculate the distance between the semantic embedding vectors corresponding to different review paths to obtain semantic distance information; determine the feature difference information between the different review paths based on the semantic distance information; the fusion processing module 606 is further configured to determine the fusion weight corresponding to each review path based on the consistency verification result; and perform weighted fusion processing on the review results of each path based on the fusion weight corresponding to each review path to obtain the target review result.

[0132] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0133] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0134] This application also provides a computer-readable storage medium storing a computer program that can be used to execute the multimodal content moderation method described in the above embodiments.

[0135] This application also provides a computer device, see [link to relevant documentation] Figure 7 The diagram shown illustrates the structure of the computer device provided in this application. At the hardware level, the computer device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for various business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the multimodal content review method described in the above embodiments. Of course, besides software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution entity of the above processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0136] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0137] The processing and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flows can also be executed by dedicated logic circuits—such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), and the device can also be implemented as dedicated logic circuits.

[0138] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0139] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0140] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0141] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0142] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0143] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A multimodal content review method, characterized in that, The method includes: Obtain multimodal content to be reviewed, wherein the multimodal content includes an image to be reviewed and input prompt information corresponding to the image to be reviewed; If the image to be reviewed meets the preset risk conditions, the image to be reviewed is subjected to adversarial purification processing to obtain a purified image; and, the input prompt information is subjected to prompt attack detection, and the purified prompt information corresponding to the input prompt information is determined based on the detection result, wherein the prompt attack detection is used to detect whether there is malicious prompt content in the input prompt information; Based on at least one of the image to be reviewed and the cleaned image, the cleaned prompt information and the set of preset prompt word templates, multiple review paths are constructed, wherein the input content and / or review model configurations corresponding to different review paths are different, and the input content includes the input image and the review prompt information; The multimodal content is reviewed through multiple review paths to obtain the review results for each path. Cross-path consistency verification is performed on the audit results of each path to obtain consistency verification results, wherein the consistency verification results are used to characterize the degree of difference between the audit results of different audit paths; Based on the consistency verification results, the review results of each path are merged to obtain the target review result corresponding to the multimodal content.

2. The method according to claim 1, characterized in that, The image to be reviewed meets preset risk conditions, including: Perform adversarial perturbation detection on the image to be reviewed to obtain adversarial risk information corresponding to the image to be reviewed; If the risk information meets the preset conditions, it is determined that the image to be reviewed meets the preset risk conditions; The adversarial risk information is determined based on at least one of the frequency domain features and gradient sensitivity features of the image to be reviewed.

3. The method according to claim 1, characterized in that, The process of performing anti-decontamination processing on the image to be reviewed to obtain a decontamination image includes: Determine at least one target image transformation operator from a preset set of image transformation operators; Based on the at least one target image transformation operator, the image to be reviewed is subjected to image transformation processing to obtain a candidate cleaned image; Determine the structural similarity between the candidate cleaned image and the image to be reviewed; If the structural similarity meets the preset similarity conditions, the candidate cleaned image is determined as the cleaned image; If the structural similarity does not meet the preset similarity condition, the transformation parameters of the target image transformation operator are adjusted, and the image to be reviewed is reprocessed based on the adjusted transformation parameters.

4. The method according to claim 1, characterized in that, The preset prompt word template set includes standard prompt words, which are prompt information used to instruct the review model to review the content of the image; The construction of multiple review paths includes: Variant prompts are determined based on the standard prompts, wherein the variant prompts have the same review purpose as the standard prompts but are expressed differently; The multiple review paths are constructed based on at least one of the image to be reviewed and the cleaned image, at least one of the standard prompt words and the variant prompt words, and the cleaned prompt information.

5. The method according to claim 1, characterized in that, The audit model configuration includes at least one of the following: visual language model instance, inference temperature parameter, attention discard rate, and cue word configuration; The different review paths correspond to different review model configurations, including at least one of the following: different visual language model instances, different inference temperature parameters, different attention discard rates, and different prompt word configurations.

6. The method according to claim 1, characterized in that, The path review results include at least one of the following: review category, confidence level, and output feature; The process of performing cross-path consistency verification on the audit results of each path to obtain consistency verification results includes: Based on the output characteristics in the review results of each path, determine the characteristic difference information between different review paths; The consistency verification result is determined based on the feature difference information.

7. The method according to claim 6, characterized in that, The output features include semantic embedding vectors; The step of determining the feature difference information between different review paths based on the output features in the review results of each path includes: Calculate the distance between the semantic embedding vectors corresponding to different review paths to obtain semantic distance information; Based on the semantic distance information, determine the feature difference information between the different review paths; The step of fusing the review results of each path based on the consistency verification result to obtain the target review result corresponding to the multimodal content includes: Based on the consistency verification results, determine the fusion weight corresponding to each of the audit paths; Based on the fusion weights corresponding to each of the aforementioned review paths, the review results of each path are weighted and fused to obtain the target review result.

8. A multimodal content review device, characterized in that, The device includes: The content acquisition module is used to acquire multimodal content to be reviewed, wherein the multimodal content includes an image to be reviewed and input prompt information corresponding to the image to be reviewed; The purification detection module is used to perform adversarial purification processing on the image to be reviewed when the image to be reviewed meets the preset risk conditions, so as to obtain a purified image; and to perform prompt attack detection on the input prompt information, and determine the purification prompt information corresponding to the input prompt information based on the detection result, wherein the prompt attack detection is used to detect whether there is malicious prompt content in the input prompt information; The path construction module is used to construct multiple review paths based on at least one of the image to be reviewed and the cleaned image, the cleaned prompt information and a set of preset prompt word templates. The input content and / or review model configurations corresponding to different review paths are different. The input content includes the input image and the review prompt information. The path review module is used to review the multimodal content through multiple review paths and obtain the path review results corresponding to each review path. The consistency verification module is used to perform cross-path consistency verification on the audit results of each path to obtain a consistency verification result, wherein the consistency verification result is used to characterize the degree of difference between the audit results of different audit paths. The fusion processing module is used to perform fusion processing on the review results of each path based on the consistency verification results, so as to obtain the target review result corresponding to the multimodal content.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the multimodal content review method according to any one of claims 1 to 7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the multimodal content moderation method as described in any one of claims 1 to 7.

Images