Picture auditing method, computer equipment, readable storage medium and program product

Abstract

The invention relates to a picture auditing method, computer equipment, a readable storage medium and a program product. The method comprises the following steps: inputting a to-be-audited picture into a first model of a picture auditing system, and extracting image features of the to-be-audited picture; the method comprises the following steps: matching image features of a to-be-audited picture with a plurality of harmful text features, and recalling the to-be-audited picture as a suspected violation picture; the semantic similarity between the image feature of the suspected violation picture and the target text feature meets a preset condition; and obtaining a target harmful text corresponding to the target text feature in a harmful text library, inputting the suspected violation picture into a second model of the picture auditing system, and identifying the suspected violation picture by adopting an identification mode matched with the target harmful text through the second model, and obtaining the violation category to which the suspected violation picture belongs and picture violation description information thereof. By adopting the method, the picture auditing efficiency and accuracy can be improved, and both the recognition accuracy and the auditing cost are considered.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to an image review method, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the growth of UGC (user-generated content) on the internet and the rapid development of AIGC (artificial intelligence-generated content) technology, the barrier to creating user-generated images has been significantly lowered, leading to a proliferation of inappropriate images. To meet internet regulatory requirements, platforms need to rigorously review image content and quickly respond to newly emerging types of inappropriate or malicious images in order to achieve accurate identification and crackdown.

[0003] In related technologies, to achieve better review results, current image review systems typically input massive amounts of images into multiple review models trained separately for various types of inappropriate images, such as pornography review models, vulgarity review models, gore review models, and terrorism review models. This means that each image needs to be processed by multiple models sequentially. For example, for a massive amount of image data N, if there are n review models, N×n model processing operations are required, resulting in huge computational resource consumption and high review costs. Furthermore, the large number of models to be trained and the long iteration cycle lead to high model training and maintenance costs and slow response speed to new types of violations. Summary of the Invention

[0004] Therefore, it is necessary to provide an improved image review method, computer device, computer-readable storage medium, and computer program product to address the aforementioned technical problems.

[0005] Firstly, this application provides an image review method. The method includes:

[0006] The image to be reviewed is input into the first model of the image review system, and the image features of the image to be reviewed are extracted.

[0007] By matching the image features of the image to be reviewed with multiple harmful text features, the image to be reviewed is recalled as a suspected violation image; the semantic similarity between the image features of the suspected violation image and the target text features meets a preset condition, the target text feature is at least one of the multiple harmful text features, and the harmful text corresponding to each harmful text feature is stored in a preset harmful text library;

[0008] Obtain the target harmful text corresponding to the target text feature in the harmful text database, input the suspected violation image into the second model of the image review system, and use the second model to identify the suspected violation image using a recognition pattern that matches the target harmful text to obtain the violation category to which the suspected violation image belongs and its image violation description information.

[0009] In one embodiment, the method further includes:

[0010] Compare the semantic similarity between the harmful text features corresponding to each harmful text in the harmful text database and the image features of the image to be reviewed;

[0011] The harmful text features with the highest semantic similarity are used as the target text features.

[0012] In one embodiment, prior to the step of inputting the suspected inappropriate image into the second model of the image review system, the method further includes:

[0013] Obtain the recognition mode configuration information of the second model; the recognition mode configuration information includes similarity conditions corresponding to different recognition modes of the model.

[0014] The recognition pattern for the second model is determined based on the similarity conditions met by the maximum semantic similarity.

[0015] In one embodiment, the identification mode includes a first mode, wherein the suspected violation image is input into the image review system via a second model, and the second model identifies the suspected violation image using an identification mode that matches the target harmful text, thereby obtaining the violation category to which the suspected violation image belongs and its image violation description information, including:

[0016] In the first mode, a comprehensive recognition model is used as the second model; the comprehensive recognition model is obtained by fine-tuning and training a preset multimodal large model based on security audit scenario data.

[0017] The suspected illegal image is input into the comprehensive recognition model. The comprehensive recognition model is guided by the prompt information constructed based on the target harmful text to identify local information of the suspected illegal image from the direction of suspected violation, and the output result is obtained.

[0018] If the output result indicates a violation, the violation category of the suspected violation image and the image violation description information in the first mode are obtained; the image violation description information in the first mode includes the violation content and violation reason analysis information.

[0019] In one embodiment, the identification mode includes a second mode, wherein the harmful text library has different violation tags for characterizing the violation category to which each harmful text belongs, and the second model of the image review system inputs the suspected violation image into the image review system, and identifies the suspected violation image by using an identification mode that matches the target harmful text, thereby obtaining the violation category to which the suspected violation image belongs and its image violation description information, including:

[0020] In the second mode, a single-class recognition model corresponding to different violation tags in the harmful text library is used as the second model;

[0021] The suspected illegal image is input into the single-class recognition model corresponding to the illegal label of the target harmful text, and the output result is determined based on the confidence analysis;

[0022] If the output result indicates a violation, the violation category of the suspected violation image and the image violation description information in the second mode are obtained; the image violation description information in the second mode includes a violation confidence score.

[0023] In one embodiment, the recognition mode includes a third mode, wherein the suspected violation image is input into the second model of the image review system, and the suspected violation image is identified by the second model using a recognition mode that matches the target harmful text, to obtain the violation category to which the suspected violation image belongs and its image violation description information, including:

[0024] In the third mode, a comprehensive identification model and a single-class identification model corresponding to different violation tags in the harmful text library are used as the second model;

[0025] The suspected illegal image and the prompt information constructed based on the target harmful text are input into the comprehensive recognition model to obtain a first output result, and the suspected illegal image is input into the single-class recognition model corresponding to the illegal label of the target harmful text to obtain a second output result;

[0026] By combining the first output result and the second output result, if a violation is determined, the violation category to which the suspected violation image belongs and the image violation description information in the third mode are obtained; the image violation description information in the third mode includes a violation confidence score or violation cause analysis information.

[0027] In one embodiment, extracting the image features of the image to be reviewed includes:

[0028] Based on the newly added harmful text in the harmful text database, construct a new harmful text prompt message;

[0029] The newly added harmful text prompt information guides the image feature extraction model in the first model to process the image to be reviewed, so as to extract image features that match the semantic similarity of the newly added harmful text, and obtain the image features of the image to be reviewed.

[0030] Secondly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described above.

[0031] Thirdly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0032] Fourthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described method.

[0033] The aforementioned image review method, computer equipment, readable storage medium, and program product, by inputting the image to be reviewed into the first model of the image review system, extracts the image features of the image to be reviewed, matches the image features of the image to be reviewed with multiple harmful text features, recalls the image to be reviewed as a suspected violation image, and the semantic similarity between the image features of the suspected violation image and the target text features meets a preset condition. The target text feature is at least one of multiple harmful text features, and the harmful text corresponding to each harmful text feature is stored in a preset harmful text library. Then, the target harmful text corresponding to the target text feature in the harmful text library is obtained, and the suspected violation image is input into the second model of the image review system. The second model identifies the suspected violation image by using a recognition pattern that matches the target harmful text, and obtains the violation category and image violation description information of the suspected violation image. In this way, by quickly filtering out suspected violation images with semantic similarity to harmful text through the first model, and then accurately identifying their violation category and description information through the second model, the efficiency and accuracy of image review can be improved, the cost of image review can be reduced, and efficient detection and classification of image violation content can be achieved. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating an image review method in one embodiment;

[0035] Figure 2 This is a schematic diagram of the training process of the first model in one embodiment;

[0036] Figure 3This is a schematic diagram of the processing flow in the first mode of the second model in one embodiment;

[0037] Figure 4 This is a schematic diagram of the processing flow in the second mode of the second model in one embodiment;

[0038] Figure 5 This is a schematic diagram of the processing flow under the second model and third mode in one embodiment;

[0039] Figure 6 This is a flowchart illustrating the image review method in another embodiment;

[0040] Figure 7 This is a structural block diagram of an image review device in one embodiment;

[0041] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0043] In one exemplary embodiment, such as Figure 1 As shown, an image review method is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, or to a system including both a terminal and a server, and is implemented through the interaction between the terminal and the server. The server can be a standalone server or a server cluster consisting of multiple servers; the terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices.

[0044] In this embodiment, the method includes the following steps S101 to S103. Wherein:

[0045] In step S101, the image to be reviewed is input into the first model of the image review system to extract the image features of the image to be reviewed.

[0046] The image review system includes a first model and a second model. The image review method of this embodiment can be realized by cascading the first model and the second model. The image review system can have a cascaded architecture of cross-modal recall and multimodal precision judgment. For example, a cascaded review system composed of a cross-modal recall model (i.e., the first model) and an intelligent discrimination model (i.e., the second model) can be constructed.

[0047] The first model may include an image feature extraction model, such as Figure 2 As shown, the image feature extraction model can be an image encoder model; it can extract image features from the input image to obtain the feature representation of the input image, such as using vectors to represent image features.

[0048] In practical applications, a deep neural network can be trained using massive amounts of image data to obtain an image feature extraction model. This model can be used to extract feature vectors that represent the global information of an image; for example... Figure 2 As shown, by inputting the image to be reviewed into the first model of the image review system, the image feature extraction model can perform general image feature extraction, and the image feature extraction model can also be guided to process the image to be reviewed based on the Prompt (prompt information) to extract image features that match the Prompt in addition to general image features, thereby obtaining the image features of the image to be reviewed.

[0049] In step S102, the image to be reviewed is identified as a suspected violation image by matching the image features of the image to be reviewed with multiple harmful text features.

[0050] Among them, the semantic similarity between the image features of the suspected illegal image and the target text features meets the preset conditions, such as... Figure 2 As shown, semantic similarity needs to reach the recall threshold; suspected violation images can be images that are suspected of containing harmful content, such as images of inappropriate behavior, illegal advertisements, sensitive references, or harmful content containing inappropriate / prohibited content.

[0051] As an example, the target text feature can be at least one of multiple harmful text features; the harmful text corresponding to each harmful text feature is stored in a pre-defined harmful text database, such as... Figure 2 A platform for the operation and management of harmful text databases.

[0052] In specific implementations, such as Figure 2 As shown, a text feature extraction model can be used for text feature extraction. This text feature extraction model can be a text encoder model. By extracting text features from each harmful text in the harmful text library, multiple harmful text features can be obtained and stored in the harmful text feature library. Then, during feature vector retrieval and matching, the image features of the image to be reviewed can be matched one by one with each harmful text feature in the harmful text feature library. By calculating the feature vector similarity, the semantic similarity between the image features of the image to be reviewed and each harmful text feature can be obtained. If the semantic similarity is greater than or equal to a preset recall threshold, the image to be reviewed can be recalled as a suspected violation image for further image review processing based on the second model. Optionally, as... Figure 2As shown, if the semantic similarity is less than the preset recall threshold, the image to be reviewed can be determined to be non-violation.

[0053] In one example, the harmful text features of each harmful text in the harmful text database can be generated offline in advance and stored in the harmful text feature database. Feature vector retrieval methods can include, but are not limited to, similar nearest neighbor retrieval methods, such as HNSW or Faiss retrieval techniques. The methods for calculating feature vector similarity can include, but are not limited to, Euclidean distance and cosine distance.

[0054] In another example, during the training phase of the first model, to establish the association between general image features and harmful text features, intermodal training alignment can be performed on the general image feature vector and the harmful text feature vector. For example... Figure 3 As shown, general image feature vectors and harmful text feature vectors can be paired into data pairs. The training convergence condition can be set to maximize the similarity between data pairs formed by the general image feature vector and the harmful text feature vector corresponding to the text description, while minimizing the similarity between data pairs formed by the general image feature vector and the harmful text feature vector without corresponding text description. During training, a loss function can be set such that the similarity of the product of the general image feature vector and the harmful text feature vector corresponding to the text description tends towards 1, and vice versa, the similarity of the product of the feature vectors tends towards 0.

[0055] In one optional embodiment, the harmful text library operation platform can support adding, deleting, and modifying harmful text content in the harmful text library, and can configure violation tags for each harmful text to characterize the violation category to which each harmful text belongs, thereby enabling rapid adjustment of the malicious image recall strategy.

[0056] In step S103, the target harmful text corresponding to the target text features in the harmful text database is obtained, and the suspected illegal image is input into the second model of the image review system. The second model uses a recognition pattern that matches the target harmful text to identify the suspected illegal image, thereby obtaining the violation category to which the suspected illegal image belongs and its image violation description information.

[0057] As an example, the harmful text feature corresponding to the target harmful text can be the harmful text feature with the highest semantic similarity to the image feature of the image to be reviewed.

[0058] The second model can have different recognition modes for calling the model. For example, in the first mode, a comprehensive recognition model (such as a security vertical multimodal large model) can be called; in the second mode, a single-class recognition model corresponding to different violation tags in the harmful text library (such as a special domain recognition small model) can be called; and in the third mode, a comprehensive recognition model and a single-class recognition model can be called. Other intelligent discrimination modes are also possible, but no specific restrictions are imposed in this embodiment.

[0059] Specifically, in the first model processing stage, general image features are extracted from the input image and vector retrieval and matching are performed with the harmful text features extracted from the harmful text library, which can achieve broad recall on global image features; in the second model processing stage, by combining the target harmful text to call the security vertical category multimodal large model or the small recognition model of each special domain to identify local image features, accurate identification of illegal content can be achieved.

[0060] Compared to traditional methods, when reviewing massive amounts of images, the technical solution in this embodiment uses a cascaded first model and a second model to improve the iteration efficiency of the image review model. Based on the first model, a large number of normal images can be quickly filtered, ensuring a higher precision and recall rate and reducing the cost of image review. Then, suspected illegal images are input into the second model for fine identification, which can accurately obtain the violation status of each type of image, achieving the effect of balancing recognition accuracy and review cost.

[0061] In the aforementioned image review method, the image to be reviewed is input into the first model of the image review system to extract its image features. These features are then matched with multiple harmful text features to identify the image as a suspected violation image. The semantic similarity between the image features and the target text features of this suspected violation image meets a preset condition. The target text feature is at least one of multiple harmful text features, and the harmful text corresponding to each feature is stored in a preset harmful text library. The target harmful text corresponding to the target text feature in the harmful text library is then obtained. The suspected violation image is then input into the second model of the image review system. The second model uses a recognition pattern that matches the target harmful text to identify the suspected violation image, obtaining the violation category and description information of the suspected violation image. Thus, by quickly filtering out suspected violation images semantically similar to harmful text using the first model, and then accurately identifying their violation category and description information using the second model, the efficiency and accuracy of image review can be improved, the cost of image review can be reduced, and efficient detection and classification of image violations can be achieved.

[0062] In one exemplary embodiment, the following steps may also be included:

[0063] Compare the semantic similarity between the harmful text features corresponding to each harmful text in the harmful text database and the image features of the image to be reviewed; use the harmful text feature with the highest semantic similarity as the target text feature.

[0064] For example, by extracting the harmful text features corresponding to each harmful text in the harmful text library, the image features of the image to be reviewed are matched one by one with each harmful text feature, and the similarity of the feature vectors is calculated. The harmful text feature with the highest semantic similarity (e.g., the highest matching similarity) can be selected as the target text feature for further use in the second model processing stage.

[0065] In this embodiment, by comparing the semantic similarity between the harmful text features corresponding to each harmful text in the harmful text library and the image features of the image to be reviewed, and then using the harmful text feature with the highest semantic similarity as the target text feature, it is possible to find the harmful text with the highest matching similarity, which helps to accurately identify the harmful content contained in the image to be reviewed.

[0066] In an exemplary embodiment, before the step of inputting the suspected violation image into the second model of the image review system, the following steps may also be included:

[0067] Obtain the recognition mode configuration information of the second model; the recognition mode configuration information includes the similarity conditions corresponding to different recognition modes used to call the model; determine the recognition mode for the second model based on the similarity conditions satisfied by the maximum semantic similarity.

[0068] In one example, a dynamic model selection algorithm based on similarity weights can be implemented using a model scheduler optimizer. For instance, a threshold can be set as a similarity condition. When the maximum semantic similarity obtained from the first model is greater than the first threshold (e.g., 0.85), only a single-class recognition model (e.g., a small model for specific domain recognition) is called; when the maximum semantic similarity is in the range between the first and second thresholds (e.g., 0.7-0.85), a comprehensive recognition model (e.g., a large multimodal model for security verticals) is called; and when the maximum semantic similarity is less than the second threshold (e.g., 0.7), a full model combination of the comprehensive recognition model and the single-class recognition model is enabled. This effectively reduces service costs and manual review costs.

[0069] In this embodiment, by obtaining the recognition mode configuration information of the second model, and then determining the recognition mode for the second model according to the similarity condition met by the maximum semantic similarity, it is possible to achieve accurate adaptation of the recognition mode of the second model and improve the accuracy and efficiency of collaborative recognition between models.

[0070] In an exemplary embodiment, the identification mode may include a first mode, where a suspected violation image is input into a second model of the image review system. The second model uses an identification mode that matches the target harmful text to identify the suspected violation image, thereby obtaining the violation category to which the suspected violation image belongs and its violation description information. This may include the following steps:

[0071] In the first mode, a comprehensive recognition model is used as the second model. The comprehensive recognition model is obtained by fine-tuning and training a preset multimodal large model based on security audit scenario data. Suspected illegal images are input into the comprehensive recognition model, and the model is guided by prompts constructed based on the target harmful text to identify local information in the suspected illegal images from the direction of suspected illegality, and obtain the output result. If the output result is a violation judgment, the violation category to which the suspected illegal image belongs and the image violation description information in the first mode are obtained. The image violation description information in the first mode includes violation content and violation reason analysis information.

[0072] As an example, the comprehensive identification model can be a multimodal large model for the vertical field of security auditing. It can be obtained by fine-tuning the model by using data from the security auditing field to construct instruction data based on a general multimodal large model.

[0073] In practical applications, such as Figure 2 As shown, suspected illegal images can be input into a security-specific multimodal large-scale model (i.e., a comprehensive recognition model). A prompt can be constructed based on the violation label (i.e., the target harmful text) corresponding to the most semantically similar harmful text. This prompt guides the security-specific multimodal large-scale model to identify local information in the image from the direction of suspected violation. Furthermore, when an image is determined to be illegal, the model can output the violation category, violation content, and violation reason analysis, and can be sent to a manual review platform for manual review or direct action. Otherwise, the image is determined to be non-violation-related. This allows the general multimodal large-scale model to quickly adapt to security review scenarios based on text prompt injection technology. Moreover, by using a small amount of security review domain data to fine-tune the general multimodal large-scale model, higher precision and recall performance in security scenarios can be achieved, significantly reducing model training costs.

[0074] Optionally, based on the security vertical multimodal large model, a knowledge base can be introduced for retrieval enhancement processing, thereby more accurately guiding the identification of the multimodal large model and the adaptation of human review standards.

[0075] In this embodiment, by using a comprehensive recognition model as the second model in the first mode, and then inputting the suspected illegal image into the comprehensive recognition model, the comprehensive recognition model is guided by the prompt information constructed based on the target harmful text to identify the local information of the suspected illegal image from the direction of suspected violation, and obtain the output result. Then, if the output result is a violation judgment, the violation category to which the suspected illegal image belongs and the image violation description information in the first mode are obtained. This can achieve efficient identification and location of illegal local content in the image, and improve the accuracy of image violation content identification.

[0076] In an exemplary embodiment, the identification mode may include a second mode. The harmful text library may have different violation labels to characterize the violation category to which each harmful text belongs. Suspected violation images are input into the second model of the image review system. The second model uses an identification mode that matches the target harmful text to identify the suspected violation images, thereby obtaining the violation category to which the suspected violation images belong and their image violation description information. This may include the following steps:

[0077] In the second mode, a single-class recognition model corresponding to different violation tags in the harmful text library is used as the second model; the suspected violation image is input into the single-class recognition model corresponding to the violation tag of the target harmful text, and the output result is determined based on the confidence analysis; if the output result is a violation judgment, the violation category to which the suspected violation image belongs and the image violation description information in the second mode are obtained; the image violation description information in the second mode includes the violation confidence score.

[0078] Each single-class recognition model (such as a domain-specific recognition mini-model) can be obtained by adding a classifier (such as a fully connected layer or a transformer neural network layer) to fine-tune the model based on the general image features in the first model and using a small amount of harmful image samples in the domain. This can greatly reduce the collection and labeling of training data, reduce training costs, improve the iteration speed, and ensure a high recognition accuracy.

[0079] In one example, such as Figure 4 As shown, the system can determine the specific domain recognition model to be invoked by identifying the violation label corresponding to the harmful text (i.e., the target harmful text) with the highest semantic similarity. For example, if the harmful text with the highest semantic similarity to the recalled suspected violation image is in the pornography category, then the pornography recognition model will be invoked in the second model processing stage. The system can then determine whether the image is a violation based on the confidence score output by the specific domain recognition model. If the confidence score exceeds a set violation threshold, the image is considered a violation, and the violation category and confidence score are output. This image can then be sent to a manual review platform for further review or direct action. Otherwise, the image is considered not to be in violation. Optionally, all specific domain recognition models can be invoked directly, and the specific specific domain recognition model can be matched based on image category recognition.

[0080] In this embodiment, by using a single-class recognition model corresponding to different violation tags in the harmful text library as the second model in the second mode, and then inputting the suspected violation image into the single-class recognition model corresponding to the violation tag of the target harmful text, the output result is determined based on confidence analysis. Then, if the output result is a violation judgment, the violation category to which the suspected violation image belongs and the image violation description information in the second mode are obtained. This achieves targeted detection for different violation types and can improve the accuracy of harmful content identification and classification efficiency.

[0081] In an exemplary embodiment, the identification mode may include a third mode, in which a suspected violation image is input into a second model of the image review system. The second model uses an identification mode that matches the target harmful text to identify the suspected violation image, thereby obtaining the violation category to which the suspected violation image belongs and its violation description information. This may include the following steps:

[0082] In the third mode, a comprehensive recognition model and a single-class recognition model corresponding to different violation tags in the harmful text library are used as the second model. The suspected violation image and the prompt information constructed based on the target harmful text are input into the comprehensive recognition model to obtain the first output result, and the suspected violation image is input into the single-class recognition model corresponding to the violation tag of the target harmful text to obtain the second output result. The first and second output results are combined for judgment. If a violation is determined, the violation category to which the suspected violation image belongs and the image violation description information in the third mode are obtained. The image violation description information in the third mode includes violation confidence score or violation cause analysis information.

[0083] In specific implementations, such as Figure 5 As shown, based on the violation label of the harmful text corresponding to the maximum semantic similarity (i.e., the target harmful text), a prompt (alert message) for the input security vertical multimodal large model can be constructed, and the specific domain recognition small model to be called can be determined. Then, based on the confidence score output by the specific domain recognition small model and the output result of the security vertical multimodal large model, it can be comprehensively judged whether the image is a violation. If it exceeds the set violation threshold or the security vertical multimodal large model determines it to be a violation, it is determined to be a violation image, and the violation category, confidence score, or violation reason analysis is output. It can be sent to the manual review platform for manual review or direct action. Otherwise, the image is determined to be not a violation. Optionally, all specific domain recognition small models can be directly called, and the specific specific domain recognition small model can be matched by the image category recognition, while the security vertical multimodal large model is input to identify local information of the image from the direction of suspected violation.

[0084] In an optional embodiment, the security vertical multimodal large model can also be used as a third-level model. By using various specialized domain identification small models for processing in the second model stage, and then adjusting the prompt of the security vertical multimodal large model according to the violation category output by the second model, the multimodal large model can perform fine analysis on suspected violation points and improve the accuracy of identification.

[0085] In this embodiment, by using a comprehensive recognition model and a single-class recognition model corresponding to different violation tags in the harmful text library as the second model in the third mode, the suspected violation image and the prompt information constructed based on the target harmful text are then input into the comprehensive recognition model to obtain the first output result. The suspected violation image is then input into the single-class recognition model corresponding to the violation tag of the target harmful text to obtain the second output result. The first and second output results are then combined for judgment. If a violation is determined, the violation category to which the suspected violation image belongs and the image violation description information in the third mode are obtained. This multi-model collaborative detection mechanism can effectively enhance the comprehensiveness and reliability of violation content recognition.

[0086] In one exemplary embodiment, extracting image features from the image to be reviewed may include the following steps:

[0087] Based on newly added harmful texts in the harmful text database, a new harmful text prompt message is constructed. The new harmful text prompt message guides the image feature extraction model in the first model to process the image to be reviewed, so as to extract image features that match the semantic similarity with the newly added harmful texts, and obtain the image features of the image to be reviewed.

[0088] Specifically, when new malicious content appears, it is only necessary to add the corresponding harmful text to the harmful text database; for example... Figure 2 As shown, the prompt (i.e., adding harmful text prompts) of the input image encoder model (i.e., the image feature extraction model) can also be appropriately adjusted to guide the first model in extracting image features of the image to be reviewed for recall processing. Thus, by covering the recognition of various types of non-compliant images, for newly appearing non-compliant images, only adding text content or adjusting the prompts in the harmful text library is needed to quickly respond and recall new non-compliant images in the first model, avoiding retraining the model. Furthermore, by adjusting the prompts of the first and second models to guide model recognition, it can quickly respond to the recall needs of various image reviews, possessing better generalization ability and effectively improving business response speed.

[0089] Compared to traditional methods, the technical solution in this embodiment can achieve the following through the first model: 1. A cross-modal feature alignment mechanism, which differs from traditional image and text retrieval systems. It can adopt a contrastive learning strategy customized based on the review scenario. By constructing positive sample pairs and random negative sample pairs (such as image features and illegal text), and using a loss function to optimize the feature space, it can improve the cosine similarity between illegal images and corresponding illegal text, effectively solving the semantic bias problem of traditional image and text matching in security scenarios; 2. Dynamic Prompt engineering, which can fine-tune CLIP (contrastive language-image pre-training) models to adapt to security review tasks according to instructions; 3. A feature library hot update mechanism, which can support rapid response to new illegal content by updating only the harmful text feature library while keeping the image encoder frozen.

[0090] The technical solution of this embodiment can be achieved through the second model: 1. A multimodal joint reasoning architecture, which is different from the traditional single-modal review model. By using text instructions and images as input, it guides the visual model to focus on local sensitive areas (such as clothing and text areas) based on text prompts, which can improve the detection rate of local violation features; 2. A model scheduling optimizer, which can realize a dynamic model selection algorithm based on similarity weights, which greatly reduces service costs and manual review costs.

[0091] In this embodiment, by constructing new harmful text prompt information based on newly added harmful text in the harmful text library, the new harmful text prompt information is then used to guide the image feature extraction model in the first model to process the image to be reviewed, so as to extract image features that match the semantic similarity with the newly added harmful text, thereby obtaining the image features of the image to be reviewed. This can improve the dynamic detection capability of the review system for new harmful content and enhance the cross-modal matching of text and image features.

[0092] In one embodiment, such as Figure 6 The diagram illustrates another image review method. In this embodiment, the method includes the following steps:

[0093] In step S601, the image to be reviewed is input into the first model of the image review system. Based on newly added harmful text in the harmful text database, a new harmful text prompt is constructed. In step S602, the new harmful text prompt guides the image feature extraction model in the first model to process the image to be reviewed, extracting image features that match the semantic similarity of the newly added harmful text, thus obtaining the image features of the image to be reviewed. In step S603, by matching the image features of the image to be reviewed with multiple harmful text features, the image to be reviewed is recalled as a suspected violation image. In step S604, the recognition mode configuration information of the second model is obtained. Based on the similarity condition met by the maximum semantic similarity, the recognition mode for the second model is determined. In step S605, the target harmful text corresponding to the target text features in the harmful text database is obtained. Under the first mode, a comprehensive recognition model is used as the second model. In step S606, the suspected violation image is input into the comprehensive recognition model. Guided by the prompt information constructed based on the target harmful text, the comprehensive recognition model identifies local image information in the suspected violation image from the suspected violation direction, obtaining the output result. In step S607, if the output result is a violation judgment, the violation category to which the suspected violation image belongs and the image violation description information in the first mode are obtained.

[0094] It should be noted that the specific limitations of the above steps can be found in the specific limitations of one image review method described above, and will not be repeated here.

[0095] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0096] Based on the same inventive concept, this application also provides an image review device for implementing the image review method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more image review device embodiments provided below can be found in the limitations of the image review method described above, and will not be repeated here.

[0097] In one embodiment, such as Figure 7As shown, an image review device is provided, comprising:

[0098] The first model input module 701 is used to input the image to be reviewed into the first model of the image review system and extract the image features of the image to be reviewed.

[0099] Image recall module 702 is used to recall the image to be reviewed as a suspected violation image by matching the image features of the image to be reviewed with multiple harmful text features; the semantic similarity between the image features of the suspected violation image and the target text features meets a preset condition, the target text feature is at least one of the multiple harmful text features, and the harmful text corresponding to each harmful text feature is stored in a preset harmful text library;

[0100] The second model input module 703 is used to obtain the target harmful text corresponding to the target text feature in the harmful text database, input the suspected violation image into the second model of the image review system, and use the second model to identify the suspected violation image using a recognition pattern that matches the target harmful text to obtain the violation category to which the suspected violation image belongs and its image violation description information.

[0101] In one embodiment, the device further includes:

[0102] The similarity comparison module is used to compare the semantic similarity between the harmful text features corresponding to each harmful text in the harmful text library and the image features of the image to be reviewed.

[0103] The target text feature determination module is used to select the harmful text features with the highest semantic similarity as the target text features.

[0104] In one embodiment, the device further includes:

[0105] The configuration information acquisition module is used to acquire the recognition mode configuration information of the second model; the recognition mode configuration information includes similarity conditions for calling different recognition modes of the model;

[0106] The recognition pattern determination module is used to determine the recognition pattern for the second model based on the similarity conditions satisfied by the maximum semantic similarity.

[0107] In one embodiment, the recognition mode includes a first mode, and the second model input module 703 is specifically used to employ a comprehensive recognition model as the second model in the first mode; the comprehensive recognition model is obtained by fine-tuning and training a preset multimodal large model based on security audit scenario data; the suspected illegal image is input into the comprehensive recognition model, and the comprehensive recognition model is guided by the prompt information constructed based on the target harmful text to identify local information of the image in the suspected illegal image from the direction of suspected illegality, and an output result is obtained; if the output result is a violation judgment, the violation category to which the suspected illegal image belongs and the image violation description information in the first mode are obtained; the image violation description information in the first mode includes violation content and violation reason analysis information.

[0108] In one embodiment, the identification mode includes a second mode, wherein the harmful text library has different violation tags for characterizing the violation category to which each harmful text belongs, and the second model input module 703 is specifically used to, in the second mode, adopt a single-class identification model corresponding to different violation tags in the harmful text library as the second model; input the suspected violation image into the single-class identification model corresponding to the violation tag of the target harmful text, and determine the output result based on confidence analysis; if the output result is a violation determination, obtain the violation category to which the suspected violation image belongs and the image violation description information in the second mode; the image violation description information in the second mode includes a violation confidence score.

[0109] In one embodiment, the identification mode includes a third mode. The harmful text library has different violation tags for characterizing the violation category to which each harmful text belongs. The second model input module 703 is specifically used in the third mode to employ a comprehensive identification model and a single-class identification model corresponding to different violation tags in the harmful text library as the second model; input the suspected violation image and the prompt information constructed based on the target harmful text into the comprehensive identification model to obtain a first output result; and input the suspected violation image into the single-class identification model corresponding to the violation tag of the target harmful text to obtain a second output result; combine the first output result and the second output result for judgment, and if a violation is determined, obtain the violation category to which the suspected violation image belongs and the image violation description information in the third mode; the image violation description information in the third mode includes a violation confidence score or violation cause analysis information.

[0110] In one embodiment, the first model input module 701 is specifically used to construct new harmful text prompt information based on the newly added harmful text in the harmful text library; and to use the newly added harmful text prompt information to guide the image feature extraction model in the first model to process the image to be reviewed, so as to extract image features that match the semantic similarity of the newly added harmful text, and obtain the image features of the image to be reviewed.

[0111] Each module in the aforementioned image review device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0112] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements an image review method.

[0113] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0114] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0115] The image to be reviewed is input into the first model of the image review system, and the image features of the image to be reviewed are extracted.

[0116] By matching the image features of the image to be reviewed with multiple harmful text features, the image to be reviewed is recalled as a suspected violation image; the semantic similarity between the image features of the suspected violation image and the target text features meets a preset condition, the target text feature is at least one of the multiple harmful text features, and the harmful text corresponding to each harmful text feature is stored in a preset harmful text library;

[0117] Obtain the target harmful text corresponding to the target text feature in the harmful text database, input the suspected violation image into the second model of the image review system, and use the second model to identify the suspected violation image using a recognition pattern that matches the target harmful text to obtain the violation category to which the suspected violation image belongs and its image violation description information.

[0118] In one embodiment, the processor also performs the steps described in the other embodiments when executing the computer program.

[0119] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0120] The image to be reviewed is input into the first model of the image review system, and the image features of the image to be reviewed are extracted.

[0121] By matching the image features of the image to be reviewed with multiple harmful text features, the image to be reviewed is recalled as a suspected violation image; the semantic similarity between the image features of the suspected violation image and the target text features meets a preset condition, the target text feature is at least one of the multiple harmful text features, and the harmful text corresponding to each harmful text feature is stored in a preset harmful text library;

[0122] Obtain the target harmful text corresponding to the target text feature in the harmful text database, input the suspected violation image into the second model of the image review system, and use the second model to identify the suspected violation image using a recognition pattern that matches the target harmful text to obtain the violation category to which the suspected violation image belongs and its image violation description information.

[0123] In one embodiment, the computer program, when executed by a processor, also implements the steps described in the other embodiments above.

[0124] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0125] The image to be reviewed is input into the first model of the image review system, and the image features of the image to be reviewed are extracted.

[0126] By matching the image features of the image to be reviewed with multiple harmful text features, the image to be reviewed is recalled as a suspected violation image; the semantic similarity between the image features of the suspected violation image and the target text features meets a preset condition, the target text feature is at least one of the multiple harmful text features, and the harmful text corresponding to each harmful text feature is stored in a preset harmful text library;

[0127] Obtain the target harmful text corresponding to the target text feature in the harmful text database, input the suspected violation image into the second model of the image review system, and use the second model to identify the suspected violation image using a recognition pattern that matches the target harmful text to obtain the violation category to which the suspected violation image belongs and its image violation description information.

[0128] In one embodiment, the computer program, when executed by a processor, also implements the steps described in the other embodiments above.

[0129] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0130] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0131] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for image review, characterized in that, The method includes: The image to be reviewed is input into the first model of the image review system, and the image features of the image to be reviewed are extracted. By matching the image features of the image to be reviewed with multiple harmful text features, the image to be reviewed is recalled as a suspected violation image; the semantic similarity between the image features of the suspected violation image and the target text features meets a preset condition, the target text feature is at least one of the multiple harmful text features, and the harmful text corresponding to each harmful text feature is stored in a preset harmful text library; Obtain the target harmful text corresponding to the target text feature in the harmful text database, input the suspected violation image into the second model of the image review system, and use the second model to identify the suspected violation image using a recognition pattern that matches the target harmful text to obtain the violation category to which the suspected violation image belongs and its image violation description information.

2. The method according to claim 1, characterized in that, The method further includes: Compare the semantic similarity between the harmful text features corresponding to each harmful text in the harmful text database and the image features of the image to be reviewed; The harmful text features with the highest semantic similarity are used as the target text features.

3. The method according to claim 2, characterized in that, Before the step of inputting the suspected inappropriate image into the second model of the image review system, the method further includes: Obtain the recognition mode configuration information of the second model; the recognition mode configuration information includes similarity conditions corresponding to different recognition modes of the model. The recognition pattern for the second model is determined based on the similarity conditions met by the maximum semantic similarity.

4. The method according to any one of claims 1 to 3, characterized in that, The identification mode includes a first mode, and a second model that inputs the suspected illegal image into the image review system. The second model uses an identification mode that matches the target harmful text to identify the suspected illegal image, obtaining the violation category of the suspected illegal image and its violation description information, including: In the first mode, a comprehensive recognition model is used as the second model; the comprehensive recognition model is obtained by fine-tuning and training a preset multimodal large model based on security audit scenario data. The suspected illegal image is input into the comprehensive recognition model. The comprehensive recognition model is guided by the prompt information constructed based on the target harmful text to identify local information of the suspected illegal image from the direction of suspected violation, and the output result is obtained. If the output result indicates a violation, the violation category of the suspected violation image and the image violation description information in the first mode are obtained; the image violation description information in the first mode includes the violation content and violation reason analysis information.

5. The method according to any one of claims 1 to 3, characterized in that, The identification mode includes a second mode. The harmful text database has different violation tags to characterize the violation category to which each harmful text belongs. The suspected violation image is input into the second model of the image review system. The second model uses an identification mode that matches the target harmful text to identify the suspected violation image, obtaining the violation category to which the suspected violation image belongs and its image violation description information, including: In the second mode, a single-class recognition model corresponding to different violation tags in the harmful text library is used as the second model; The suspected illegal image is input into the single-class recognition model corresponding to the illegal label of the target harmful text, and the output result is determined based on the confidence analysis; If the output result indicates a violation, the violation category of the suspected violation image and the image violation description information in the second mode are obtained; the image violation description information in the second mode includes a violation confidence score.

6. The method according to any one of claims 1 to 3, characterized in that, The identification mode includes a third mode, wherein the suspected illegal image is input into the second model of the image review system, and the second model identifies the suspected illegal image using an identification mode that matches the target harmful text, thereby obtaining the violation category to which the suspected illegal image belongs and its image violation description information, including: In the third mode, a comprehensive identification model and a single-class identification model corresponding to different violation tags in the harmful text library are used as the second model; The suspected illegal image and the prompt information constructed based on the target harmful text are input into the comprehensive recognition model to obtain a first output result, and the suspected illegal image is input into the single-class recognition model corresponding to the illegal label of the target harmful text to obtain a second output result; By combining the first output result and the second output result, if a violation is determined, the violation category to which the suspected violation image belongs and the image violation description information in the third mode are obtained; the image violation description information in the third mode includes violation confidence score or violation cause analysis information.

7. The method according to claim 1, characterized in that, The extraction of image features from the image to be reviewed includes: Based on the newly added harmful text in the harmful text database, construct a new harmful text prompt message; The newly added harmful text prompt information guides the image feature extraction model in the first model to process the image to be reviewed, so as to extract image features that match the semantic similarity of the newly added harmful text, and obtain the image features of the image to be reviewed.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images