A system, method, computer device and medium for reviewing image sensitive elements

Abstract

The present invention provides a review system, method, computer equipment and medium for image sensitive elements, and belongs to the technical field of image sensitive element review. The system includes an image preprocessing module, a preliminary judgment module, a triple extraction module, a multi-head expansion module and a screening and denoising module. The image preprocessing module performs preprocessing operations on the image to obtain the image to be tested. The preliminary judgment module uses a multimodal large language model to make a preliminary judgment on the relevance between the image to be tested and the sensitive words. The triple extraction module extracts the triple information in the image to be tested, and the multi-head expansion module performs multi-head expansion on each single word in the triple, and searches for related sensitive words in combination with the sensitive word list. The screening and denoising module screens and denoises the sensitive words, determines whether the sensitive words actually appear in the image to be tested, and determines the safety of the image to be tested accordingly. The present invention improves the accuracy of image sensitive element review.

Description

Technical Field

[0001] The present invention relates to the technical field of image sensitive element review, and in particular to an image sensitive element review system, method, computer equipment and medium. Background Art

[0002] As a type of artificial intelligence technology that can convert text descriptions into images, the text-based image model has shown great application potential in many fields such as intelligent creation, virtual scene generation and product design, bringing unprecedented digital experience to the creative industry, virtual reality and e-commerce. However, this type of technology also has the risk of being abused to generate false and deceptive images, especially those involving pornographic, violent or sensitive content, which may cause serious privacy, legal and social problems.

[0003] With the increasing popularity of large-scale text and image models, the process of creating unsafe content has been greatly simplified, and the possibility of combining and styling illegal elements has also increased significantly. These models can express malicious content in a more obscure and abstract way, bringing unprecedented challenges to review work.

[0004] To address these challenges, many studies have focused on the development of safety guardrails for large models. For example, Meta developed Llama Guard on the Llama2-7b architecture to assist in security risk analysis through multi-label classification and binary decision making. However, although this approach uses a relatively small dataset (14,000 sets of prompts and responses for fine-tuning), the pertinence and sensitivity of the data remain a major challenge. In addition, major companies such as OpenAI, Google, and Microsoft have also launched their own content review APIs, which are also based on multi-label classifiers, but can only be applied to preset security categories. For situations where actual needs do not match the category definition, users need to collect data and fine-tune the model on their own, which is not only time-consuming and labor-intensive, but may also face the problem of catastrophic forgetting.

[0005] There are two core problems when applying existing classifier-based security defense solutions to complex, obscure and sensitive image detection:

[0006] (1) Data collection difficulties: In order to train sensitive element classifiers, a large number of targeted images must be collected. However, due to the insufficiency of existing datasets and the difficulty of building datasets on one’s own (the network censorship mechanism leads to a shortage of sensitive image resources, and the complexity and obscurity of sensitive elements further exacerbate the difficulty of data collection), the feasibility of implementing this solution is low.

[0007] (2) Poor model scalability: As sensitive words continue to expand in the era of large text-based models, security classifiers need to be retrained. However, continuous training may lead to catastrophic forgetting, that is, the model forgets previously learned information while learning new knowledge, thus affecting the overall classification performance. Summary of the invention

[0008] The purpose of the present invention is to provide a system, method, computer equipment and medium for reviewing image sensitive elements, thereby improving the accuracy of reviewing image sensitive elements.

[0009] To achieve the above object, the present invention provides a review system for image sensitive elements, comprising:

[0010] The image preprocessing module is used to preprocess the image to obtain the image to be tested; the preprocessing includes operations such as color space conversion and size scaling;

[0011] A preliminary judgment module is used to receive the image to be tested and the sensitive words, and use the multimodal large language model to make a preliminary judgment on the relevance between the image to be tested and the sensitive words;

[0012] A triplet extraction module is used to extract triplet information from the image to be tested transmitted by the preliminary judgment module;

[0013] The multi-head expansion module is used to perform multi-head expansion on each single word in the extracted triples, and to search for sensitive words related to the single word in the text space, word vector space and image-text vector space respectively in combination with the sensitive word list, and integrate to obtain a set of sensitive words that may be related to the image triples to be tested;

[0014] The screening and denoising module is used to traverse the sensitive word set, screen and denoise each sensitive word, and use the cross-modal understanding ability of the multimodal large language model to determine whether the sensitive word actually appears in the image to be tested, obtain the set of sensitive words that are truly related to the triplet of the image to be tested, and judge the safety of the image to be tested based on this.

[0015] Preferably, the preliminary judgment module uses sensitive words to construct prompt words, allowing the multimodal large language model to make a preliminary judgment on the relevance between the image to be tested and the sensitive words, and obtain a preliminary judgment result, which is any one of related, irrelevant or unknown.

[0016] Preferably, when the preliminary judgment result is related, the image to be tested is determined to be unsafe and the review process is exited; when the preliminary judgment result is irrelevant or unknown, the image to be tested is input into the triple extraction module.

[0017] Preferably, the triple extraction module utilizes the recognition and understanding capabilities of the multimodal large language model to identify scenes, objects, people, actions and contextual relationships in the image to be tested, and then distinguishes different objects in the image to be tested, identifies attributes, extracts relationships, generates descriptive text, and extracts triple information based on the descriptive text.

[0018] Preferably, the multi-head expansion module uses a complete text matching method in the text space to search for sensitive words related to a single word in the sensitive word library, and adds the obtained sensitive words to the acquisition set; in the word vector space and the image-text vector space, the cosine correlation of the word vector embedding or the image-text vector embedding of the single word and all the words in the sensitive word library is calculated, and the sensitive words exceeding the empirical threshold are added to the acquisition set.

[0019] Empirical threshold determination method:

[0020] Different thresholds need to be obtained for different embeddings. The method for obtaining the empirical threshold for each embedding is as follows:

[0021] First, prepare a labeled dataset containing sensitive words and non-sensitive words, and choose cosine similarity to calculate the similarity between words. Secondly, calculate the similarity between non-sensitive words and all sensitive words, and analyze the distribution characteristics of the results. The choice of threshold depends on the similarity distribution, and the cutoff point is manually selected to ensure a balanced ratio of false positives and false negatives. Finally, the threshold selection is continuously optimized during the experiment to ensure its effectiveness and stability in practical applications, so as to achieve the goal of accurately judging whether a word is sensitive.

[0022] Preferably, the screening and denoising module traverses the sensitive word set, constructs a new prompt word for each sensitive word, uses the multimodal large language model to extract the triplet information in the image to be tested again, and checks whether the sensitive word used as a prompt appears in the newly obtained triplet. If so, it is determined that the sensitive word appears in the image to be tested, otherwise, the sensitive word is skipped.

[0023] Preferably, a sensitive word library is also included, which stores sensitive words for detecting whether an image contains sensitive elements.

[0024] The present invention also provides a method for examining image sensitive elements, comprising the following steps:

[0025] Step S1, using an image preprocessing module to perform preprocessing operations on the image to obtain an image to be tested;

[0026] Step S2: The preliminary judgment module receives the image to be tested and the sensitive words, and uses the multimodal large language model to make a preliminary judgment on the relevance between the image to be tested and the sensitive words, and obtains a preliminary judgment result, which is classified into any one of related, irrelevant or unknown;

[0027] Step S3: When the preliminary judgment result is related, the image to be tested is determined to be unsafe, and the review process is exited;

[0028] When the preliminary judgment result is irrelevant or unknown, execute step S4;

[0029] Step S4, using the triplet extraction module to identify scenes, objects, people, actions and contextual relationships in the image to be tested, distinguish different objects in the image to be tested, identify attributes, extract relationships, generate descriptive text, and extract triplet information based on the descriptive text;

[0030] Step S5: Use the multi-head expansion module to perform multi-head expansion on each single word in the extracted triples, and combine with the sensitive word list to search for sensitive words related to the single word in the text space, word vector space and image-text vector space, and integrate to obtain a set of sensitive words that may be related to the image triples to be tested;

[0031] Step S6: Use the screening and denoising module to traverse the sensitive word set, screen and denoise each sensitive word, and use the cross-modal understanding ability of the multimodal large language model to determine whether the sensitive word actually appears in the image to be tested, obtain the sensitive word set that is truly related to the triplet of the image to be tested, and determine the safety of the image to be tested based on this.

[0032] The present invention also provides a computer device, comprising: a memory and a processor; the memory stores a computer program, and the processor implements the steps of the above-mentioned image sensitive element review method when executing the computer program.

[0033] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for reviewing image sensitive elements are implemented.

[0034] Therefore, the present invention adopts the above-mentioned system, method, computer device and medium for reviewing image sensitive elements, and the beneficial technical effects are as follows:

[0035] (1) This invention introduces a multimodal large language model, which first performs preliminary screening and then performs fine sensitive word screening to achieve accurate identification of sensitive elements in images. Compared with traditional classifier-based review methods, this innovation not only bypasses the cumbersome data collection and model training obstacles, but also significantly enhances the accuracy of the review and ensures the accuracy of the review work.

[0036] (2) In view of the rapid development of large-scale text graph model technology, the update and expansion of sensitive word libraries have become a normal demand. The review system and method designed by the present invention can flexibly integrate new sensitive words, and with the help of multi-head expansion modules and screening and denoising modules, it can effectively capture emerging sensitive content, greatly improving the scalability of the model and its ability to adapt to future changes.

[0037] (3) Significantly reduce the risk of misjudgment and missed judgment: By constructing a multi-level review mechanism including preliminary judgment, triple extraction, multi-head expansion and screening denoising, the present invention realizes a comprehensive and in-depth analysis of image sensitive elements. This multi-level and multi-angle review strategy effectively reduces the possibility of misjudgment and missed judgment, and greatly enhances the reliability and accuracy of the review. More importantly, it successfully copes with the challenges of large models in dealing with complex and obscure sensitive element identification, further improving the accuracy of the review. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 The present invention is a flowchart of a method for examining image-sensitive elements. DETAILED DESCRIPTION

[0039] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0040] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0041] Embodiment 1

[0042] like Figure 1 As shown, it is a flow chart of a method for examining image sensitive elements of the present invention. The present invention will be further explained below through specific examples.

[0043] Data acquisition: Given that the currently publicly available datasets involving obscure and sensitive content are relatively scarce, the present invention adopts a method of autonomously collecting and constructing datasets. By using the Python programming language and web crawler technology, image data collection for sensitive words was carried out on Wikimedia Commons, a well-known public domain image resource website. After completing the initial data collection, in order to ensure the accuracy of the dataset and the relevance of the research, the collected image data was further manually annotated and classified to form a set of datasets suitable for experimental purposes and that have been finely processed.

[0044] Data processing: In this experiment, the selected experimental model needs to meet multiple requirements such as multimodal data processing, model open source, and instruction compliance. After in-depth analysis and comparison, it was decided to use the LLaVA-OneVision series of models to carry out experimental work. Specifically, the multimodal capabilities of this series of models can meet the processing requirements of image data types, allowing experiments to be carried out in complex visual language scenes. At the same time, the open source characteristics of this series of models provide great convenience for experiments and lay the foundation for the subsequent construction of an open source framework. In addition, this series of models also performs well in instruction compliance. It can accurately understand and execute complex instructions, ensuring that the experimental process is strictly carried out in accordance with the preset plan, greatly reducing the difficulty and uncertainty of experimental operations. In summary, the advantages of the LLaVA-OneVision model in multimodal processing, open source, and instruction compliance make it an ideal choice for conducting experimental work. In the actual experiment, three model versions with different parameter scales were selected, namely LLaVA-OneVision-Qwen2-0.5b-ov, LLaVA-OneVision-Qwen2-7b-ov and LLaVA-OneVision-Qwen2-72b-ov-chat, to test the improvement effect of the solution proposed in the present invention on models with different parameter scales.

[0045] Experimental results: In this experiment, the performance of large model direct judgment and the method proposed in this invention were compared on three LLaVA-OneVision models with different parameter scales. The following is a description of the specific results (as shown in Table 1):

[0046] For the LLaVA-OneVision-Qwen2-0.5b-ov model, the Precision, Recall and F1 scores directly judged by the large model are all 0, indicating that it failed to correctly identify any positive examples in this case. The method proposed in the present invention shows significant advantages, with Precision reaching 0.907, Recall being 0.621, and F1 score being 0.737, showing higher accuracy and comprehensive performance.

[0047] On the LLaVA-OneVision-Qwen2-7b-ov model, the precision of the large model directly judged was 0.691, the recall was 0.721, and the F1 score was 0.706. The method proposed in the present invention improved the precision to 0.702, the recall to 0.836, and the F1 score to 0.763, which has a relatively better performance in comprehensive performance.

[0048] For the LLaVA-OneVision-Qwen2-72b-ov-chat model with the largest parameter scale, the Precision directly judged by the large model is 0.823, Recall is 0.739, and F1 score is 0.779. The method proposed in this paper has improved all indicators, with Precision reaching 0.831, Recall being 0.854, and the highest F1 score being 0.842, which shows that this method significantly improves the recall rate while maintaining high accuracy, thus achieving the best result in comprehensive performance.

[0049] In summary, the method proposed in the present invention shows better performance than direct judgment of large models on LLaVA-OneVision models with different parameter scales, especially in improving recall rate and F1 score.

[0050] Table 1 Performance

[0051]

[0052] Embodiment 2

[0053] A review system for image sensitive elements, comprising:

[0054] The image preprocessing module is used to preprocess the image to obtain the image to be tested; the preprocessing includes operations such as color space conversion and size scaling;

[0055] A preliminary judgment module is used to receive the image to be tested and the sensitive words, and use the multimodal large language model to make a preliminary judgment on the relevance between the image to be tested and the sensitive words;

[0056] A triplet extraction module is used to extract triplet information from the image to be tested transmitted by the preliminary judgment module;

[0057] The multi-head expansion module is used to perform multi-head expansion on each single word in the extracted triples, and to search for sensitive words related to the single word in the text space, word vector space and image-text vector space respectively in combination with the sensitive word list, and integrate to obtain a set of sensitive words that may be related to the image triples to be tested;

[0058] The screening and denoising module is used to traverse the sensitive word set, screen and denoise each sensitive word, and use the cross-modal understanding ability of the multimodal large language model to determine whether the sensitive word actually appears in the image to be tested, obtain the set of sensitive words that are truly related to the triplet of the image to be tested, and judge the safety of the image to be tested based on this;

[0059] It also includes a sensitive word library, which stores sensitive words used to detect whether an image contains sensitive elements.

[0060] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0061] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0062] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0063] It is worth noting that the contents not elaborated in detail in the present invention are all prior art and are well known to those skilled in the art.

[0064] Therefore, the present invention adopts the above-mentioned image-sensitive element review system, method, computer device and medium to improve the accuracy of image-sensitive element review.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A review system for image sensitive elements, characterized in that: include: An image preprocessing module is used to preprocess the image to obtain an image to be tested; A preliminary judgment module is used to receive the image to be tested and the sensitive words, and use the multimodal large language model to make a preliminary judgment on the relevance between the image to be tested and the sensitive words; A triplet extraction module is used to extract triplet information from the image to be tested transmitted by the preliminary judgment module; The multi-head expansion module is used to perform multi-head expansion on each single word in the extracted triples, and to search for sensitive words related to the single word in the text space, word vector space and image-text vector space respectively in combination with the sensitive word list, and integrate to obtain a set of sensitive words that may be related to the image triples to be tested; The screening and denoising module is used to traverse the sensitive word set, screen and denoise each sensitive word, and use the cross-modal understanding ability of the multimodal large language model to determine whether the sensitive word actually appears in the image to be tested, obtain the set of sensitive words that are truly related to the triplet of the image to be tested, and judge the safety of the image to be tested based on this.

2. The image sensitive element review system according to claim 1, characterized in that: The preliminary judgment module uses sensitive words to construct prompt words, allowing the multimodal large language model to make a preliminary judgment on the relevance between the image to be tested and the sensitive words, and obtain a preliminary judgment result, which is any one of related, irrelevant or unknown.

3. The image sensitive element review system according to claim 2, characterized in that: When the preliminary judgment result is related, the image to be tested is determined to be unsafe and the review process is exited; when the preliminary judgment result is irrelevant or unknown, the image to be tested is input into the triple extraction module.

4. The image sensitive element review system according to claim 3, characterized in that: The triple extraction module uses the recognition and understanding capabilities of the multimodal large language model to identify the scenes, objects, people, actions and contextual relationships in the test image, and then distinguishes different objects in the test image, identifies attributes, extracts relationships, generates descriptive text, and extracts triple information based on the descriptive text.

5. The image sensitive element review system according to claim 4, characterized in that: The multi-head expansion module uses a complete text matching method in the text space to search for sensitive words related to a single word in the sensitive word library, and adds the obtained sensitive words to the acquisition set; in the word vector space and the image-text vector space, it calculates the cosine correlation of the word vector embedding or the image-text vector embedding of a single word with all the words in the sensitive word library, and adds the sensitive words that exceed the empirical threshold to the acquisition set.

6. The image sensitive element review system according to claim 5, characterized in that: The screening and denoising module traverses the sensitive word set, constructs a new prompt word for each sensitive word, and uses the multimodal large language model to extract the triple information in the test image again, and checks whether the sensitive word used as a prompt appears in the newly obtained triple. If so, it is determined that the sensitive word appears in the test image, otherwise, the sensitive word is skipped.

7. The image sensitive element review system according to claim 6, characterized in that: It also includes a sensitive word library, which stores sensitive words used to detect whether an image contains sensitive elements.

8. A method for examining image sensitive elements, characterized in that: The following steps are involved: Step S1, using an image preprocessing module to perform preprocessing operations on the image to obtain an image to be tested; Step S2: The preliminary judgment module receives the image to be tested and the sensitive words, and uses the multimodal large language model to make a preliminary judgment on the relevance between the image to be tested and the sensitive words, and obtains a preliminary judgment result, which is classified into any one of related, irrelevant or unknown; Step S3: When the preliminary judgment result is related, the image to be tested is determined to be unsafe, and the review process is exited; When the preliminary judgment result is irrelevant or unknown, execute step S4; Step S4, using the triplet extraction module to identify scenes, objects, people, actions and contextual relationships in the image to be tested, distinguish different objects in the image to be tested, identify attributes, extract relationships, generate descriptive text, and extract triplet information based on the descriptive text; Step S5: Use the multi-head expansion module to perform multi-head expansion on each single word in the extracted triples, and combine with the sensitive word list to search for sensitive words related to the single word in the text space, word vector space and image-text vector space, and integrate to obtain a set of sensitive words that may be related to the image triples to be tested; Step S6: Use the screening and denoising module to traverse the sensitive word set, screen and denoise each sensitive word, and use the cross-modal understanding ability of the multimodal large language model to determine whether the sensitive word actually appears in the image to be tested, obtain the sensitive word set that is truly related to the triplet of the image to be tested, and determine the safety of the image to be tested based on this.

9. A computer device comprising: Memory and processor; The memory stores a computer program, wherein the processor implements the steps of the method for examining image-sensitive elements as described in claim 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for examining image-sensitive elements described in claim 7 are implemented.

Images