Image sensitive content detection method and device, electronic device and storage medium

Abstract

The invention relates to an image sensitive content detection method and device, an electronic device and a storage medium, and the method comprises the steps: obtaining a to-be-detected image; inputting the to-be-detected image into the visual feature judgment network in the target multi-modal large model for visual detection processing to obtain initial image semantic information, and inputting the initial image semantic information into the text feature judgment network in the target multi-modal large model for text detection processing to obtain target image semantic information; obtaining a preset first vector library; sensitive semantic vectors are stored in the first vector library; and generating a first sensitive content detection result of the to-be-detected image based on a first matching result between the target image semantic information and each sensitive semantic vector in the first vector library. Through the method and the device, the problem of low accuracy of image sensitive content detection is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method, device, electronic device and storage medium for detecting image sensitive content. Background Art

[0002] Every day, hundreds of millions of images are spread on the Internet, which brings rich entertainment experience to people's lives, but also triggers a series of challenges and hidden dangers related to content security. In related technologies, usually based on Convolutional Neural Network (CNN) or self-attention mechanism-based models (such as Transformer model architecture), through training on large-scale annotated data sets, dedicated classification models are built to identify images containing common sensitive content. However, the above methods have the disadvantages of strong data dependence, limited generalization ability, and insufficient ability to recognize obscure features, resulting in low detection accuracy for sensitive content in images.

[0003] Currently, no effective solution has been proposed for the problem of low accuracy in detecting sensitive content in images in related technologies. Summary of the invention

[0004] Embodiments of the present application provide a method, device, electronic device, and storage medium for detecting image-sensitive content, so as to at least solve the problem of low accuracy in detecting image-sensitive content in the related art.

[0005] In a first aspect, an embodiment of the present application provides a method for detecting sensitive content of an image, the method comprising:

[0006] Acquire the image to be detected;

[0007] Inputting the image to be detected into a visual feature judgment network in a target multimodal large model for visual detection processing to obtain initial image semantic information, and inputting the initial image semantic information into a text feature judgment network in the target multimodal large model for text detection processing to obtain target image semantic information;

[0008] Obtaining a preset first vector library; wherein the first vector library stores sensitive semantic vectors;

[0009] Based on a first matching result between the target image semantic information and each of the sensitive semantic vectors in the first vector library, a first sensitive content detection result of the image to be detected is generated.

[0010] In some embodiments, the step of inputting the image to be detected into a visual feature judgment network in a target multimodal large model for visual detection processing to obtain initial image semantic information includes:

[0011] Inputting the image to be detected into the visual feature judgment network in the target multimodal large model, and performing sensitive content detection on the image to be detected via the visual feature judgment network to determine whether the image to be detected contains sensitive content;

[0012] If so, the visual feature judgment network is used to perform visual sensitive feature extraction on the image to be detected to obtain a first sensitive semantic type and behavior information, and the initial image semantic information is obtained based on the first sensitive semantic type and the behavior information; otherwise, the visual feature judgment network is used to perform semantic extraction on the image to be detected to obtain the initial image semantic information.

[0013] In some embodiments, the inputting the initial image semantic information into the text feature judgment network in the target multimodal large model to perform text detection processing to obtain the target image semantic information includes:

[0014] Inputting the initial image semantic information into the text feature judgment network in the target multimodal large model, and performing sensitive content detection on the initial image semantic information via the text feature judgment network to determine whether the text feature network contains sensitive content;

[0015] If so, the text feature judgment network is used to perform text sensitive feature extraction processing on the initial image semantic information to obtain a second sensitive semantic type, and the target image semantic information is obtained based on the second sensitive semantic type; otherwise, the text feature judgment network is used to perform semantic extraction processing on the initial image semantic information to obtain the target image semantic information.

[0016] In some embodiments, obtaining the target image semantic information based on the second sensitive semantic type includes:

[0017] The text feature judgment network is used to perform semantic feature compression on the initial image semantic information to obtain core image semantic information, and the target image semantic information is obtained based on the second sensitive semantic type and the core image semantic information.

[0018] In some embodiments, generating a first sensitive content detection result of the image to be detected based on a first matching result between the target image semantic information and each of the sensitive semantic vectors in the first vector library includes:

[0019] Performing vector conversion processing on the semantic information of the target image to obtain a semantic vector to be detected;

[0020] Traversing the sensitive semantic vectors in the first vector library;

[0021] Calculate the similarity result between the traversed sensitive semantic vector and the semantic vector to be detected; determine whether the sensitive semantic vector matches the target image semantic information based on the similarity result, and generate the first sensitive content detection result.

[0022] In some embodiments, judging whether the sensitive semantic vector matches the semantic vector to be detected based on the similarity result and generating the first sensitive content detection result includes:

[0023] Obtain preset sensitivity level information and category weight information;

[0024] Based on the sensitivity level information and the category weight information, assigning a corresponding similarity threshold to each of the sensitive semantic vectors in the first vector library;

[0025] According to the comparison result between the similarity result and the corresponding similarity threshold, it is determined whether the sensitive semantic vector matches the semantic vector to be detected, and the first sensitive content detection result is generated.

[0026] In some embodiments, the method further comprises:

[0027] Calculating a first semantic similarity between the initial image semantic information and the image to be detected, and calculating a second semantic similarity between the target image semantic information and the initial image semantic information;

[0028] Based on the first semantic similarity and the second semantic similarity, generating a verification result for the target multimodal large model;

[0029] When the verification result indicates that the target multimodal large model has not passed the verification, obtaining a preset second vector library; the second vector library stores sensitive image vectors;

[0030] The image vector to be tested of the image to be tested is calculated, and based on a second matching result between the image vector to be tested and each of the sensitive image vectors in the second vector library, a second sensitive content detection result of the image to be tested is generated.

[0031] In a second aspect, an embodiment of the present application provides a device for detecting image-sensitive content, including:

[0032] An image acquisition module, used for acquiring an image to be detected;

[0033] A semantic detection module, used for inputting the image to be detected into a visual feature judgment network in a target multimodal large model for visual detection processing to obtain initial image semantic information, and inputting the initial image semantic information into a text feature judgment network in the target multimodal large model for text detection processing to obtain target image semantic information;

[0034] A first vector library acquisition module, used to acquire a preset first vector library; the first vector library stores sensitive semantic vectors;

[0035] A generation module is used to generate a first sensitive content detection result of the image to be detected based on a first matching result between the target image semantic information and each of the sensitive semantic vectors in the first vector library.

[0036] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the method for detecting image-sensitive content described in the first aspect above.

[0037] In a fourth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for detecting image-sensitive content as described in the first aspect above.

[0038] Compared with the related art, the method, device, electronic device and storage medium for detecting sensitive content of an image provided by the embodiments of the present application obtain an image to be detected; input the image to be detected into a visual feature judgment network in a target multimodal large model for visual detection processing to obtain initial image semantic information, and input the initial image semantic information into a text feature judgment network in a target multimodal large model for text detection processing to obtain target image semantic information; obtain a preset first vector library; the first vector library stores sensitive semantic vectors; and based on a first matching result between the target image semantic information and each sensitive semantic vector in the first vector library, generate a first sensitive content detection result for the image to be detected.

[0039] Based on this, a multimodal large model that comprehensively utilizes vision and text is adopted to uniformly project the image information to be detected into a high-dimensional semantic space, achieving a deep understanding of image and text features and enhancing the sensitivity analysis of complex image content. At the same time, by constructing a dynamic sensitive semantic vector database, it can effectively support the real-time update of sensitive categories, can quickly adapt to new sensitive content, and has good dynamic expansion capabilities, thereby effectively solving the problem of low accuracy of image sensitive content detection and realizing an accurate and high-speed image sensitive content detection method.

[0040] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 is a hardware structure block diagram of a terminal according to a method for detecting image-sensitive content in an embodiment of the present application;

[0043] Figure 2 is a flow chart of a method for detecting image-sensitive content according to an embodiment of the present application;

[0044] Figure 3 is a flow chart of another method for detecting image-sensitive content according to an embodiment of the present application;

[0045] Figure 4 is a schematic diagram of a method for detecting image-sensitive content according to an embodiment of the present application;

[0046] Figure 5 It is a structural block diagram of a device for detecting image-sensitive content according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means, and should not be understood as insufficient contents disclosed in the present application.

[0048] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0049] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.

[0050] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. Taking running on a terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a terminal according to an image sensitive content detection method of an embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0051] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as a computer program corresponding to a method for detecting image-sensitive content in an embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0052] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0053] With the development of the Internet, the content security of image dissemination has attracted more and more attention. In related technologies, it is usually detected whether an image contains sensitive content through a classifier method based on deep learning or a visual language model.

[0054] However, deep learning methods for detecting sensitive content in images are highly data-dependent, and deep learning models are prone to failure when faced with new types of sensitive content. In particular, when dealing with a small number of samples, new categories, or cross-domain sensitive content, existing technologies are difficult to adapt quickly, resulting in limited generalization capabilities. In addition, there is still a lack of comprehensive understanding of potential implicit features (such as hidden discriminatory symbols) and contextual information in the combination of image and text modal data (such as hate memes), and it performs poorly in identifying biased, obscure, or disguised sensitive content.

[0055] On the other hand, the methods for detecting sensitive content in images based on visual language models lack dynamic adaptability. For emerging new content security issues, frequent model retraining or fine-tuning is required, which is too costly and difficult to adapt quickly. Therefore, the above-mentioned related technologies for detecting sensitive content in images all have problems that easily affect detection accuracy.

[0056] In order to improve the above problems, this embodiment provides a method for detecting image sensitive content. Figure 2 is a flow chart of a method for detecting image sensitive content according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0057] Step S210, obtaining an image to be detected.

[0058] The above-mentioned image to be detected refers to an image to be detected whether it contains sensitive content. The image to be detected may be an image collected and published on the Internet through a website or forum. Among them, the acquired image to be detected may be one or several images. It should be understood that before the image to be detected is input into the target multimodal model in the subsequent step, appropriate data preprocessing, such as image scaling, denoising, enhancement, etc., can also be performed to improve the detection accuracy.

[0059] Step S220, input the image to be detected into the visual feature judgment network in the target multimodal large model for visual detection processing to obtain initial image semantic information, and input the initial image semantic information into the text feature judgment network in the target multimodal large model for text detection processing to obtain target image semantic information.

[0060] In this step, first, the visual feature judgment network in the target multimodal large model is used to extract the visual behavior features in the image to be detected. Among them, through few-sample learning, the model can capture subtle features in the image to be detected, including hidden or disguised sensitive information, such as sensitive symbols hidden in complex backgrounds, etc., so as to use the visual feature judgment network to extract the initial image semantic information in the image to be detected. Specifically, the first question template corresponding to the image sensitive content detection task provided by the user can be obtained first, and the input information to be input into the visual feature judgment network is generated according to the first question template and the image to be detected, and the visual feature judgment network performs visual detection processing on the image to be detected according to the first question template, and outputs the initial image semantic information. Exemplarily, for the implicit sensitive image detection scenario, the currently input image to be detected contains a man wearing glasses who makes a squinting gesture, and the example is explained as follows:

[0061] The user provides a description of the first question template, for example: "I will give you a image below, please return the corresponding content according to the following format: isoffensive:{YES / NO}, if the isoffensive is YES,…Please reply in jsonformat."; the above description and the image to be detected are input into the visual feature judgment network for processing. Then, the visual feature judgment network processes the image to be detected based on the guidance provided by the first question template description and outputs the initial image semantic information: "{"isoffensive":"YES","type":"cultural and racialdiscrimination","action":"squinting eyes","description":"In the image, a man with glasses is making a gesture of squinting his eyes, which can be perceived as a racist stereotype towards Asians."}".

[0062] Next, the initial image semantic information output by the above-mentioned visual feature judgment network is input into the text feature judgment network in the target multimodal large model, and the text feature judgment network is used to further extract and compress the semantic features of the initial image semantic information from the text perspective to obtain the target image semantic information. More specifically, the second question template corresponding to the current semantic recognition task provided by the user can be obtained first, and the input information to be input into the text feature judgment network is generated according to the second question template and the initial image semantic information. The text feature judgment network performs text detection processing on the initial image semantic information according to the second question template, and outputs the target image semantic information. Still taking the implicit sensitive image detection scenario, the input image to be detected contains a man wearing glasses who makes a squinting gesture as an example, and this part of the example is explained as follows:

[0063] The user provides a description of the second question template, for example: "I will give you a description below, please return the corresponding content according to the following format: \nisoffensive:{YES / NO}, if isoffensive is YES, ... Please reply in jsonformat and as briefly as possible"; the above description and the initial image semantic information are input into the text feature judgment network for processing. Then, through the text feature network, based on the guidance provided by the second question template description, the target image semantic information is processed and output as follows: "{"isoffensive":"YES","type":"culturaland racial discrimination","summary":"Stereotypical racist gesture targetingAsians"}".

[0064] It should be noted that the above-mentioned target multimodal large model refers to a large model that includes a dual channel of visual feature judgment network and text feature judgment network, and is used to process multimodal information such as vision and text. Through the above steps, its potential in processing such open tasks can be fully explored, and the ability of small-sample and zero-sample learning of multimodal large models can be used to achieve effective recognition of new sensitive categories, significantly improving the generalization and adaptability of image sensitive content detection.

[0065] Step S230, obtaining a preset first vector library; the first vector library stores sensitive semantic vectors.

[0066] Specifically, the sensitive semantic vectors can be obtained by obtaining batches of sensitive semantic information pre-entered by staff, or by importing online sensitive semantic information from an online system in real time, and converting the sensitive semantic information into sensitive semantic vectors using semantic models or encoding algorithms. The obtained sensitive semantic vectors can be stored in a pre-built database, thereby obtaining the first vector library.

[0067] It should be noted that by constructing the first vector library, sensitive semantic vectors for matching the semantic information of the target image can be provided, thereby supporting the real-time update of the sensitive semantic vectors. The update method for the sensitive semantic vector of the first vector library can be: determine the update trigger conditions, set a mechanism to detect the emergence of new sensitive content, such as monitoring specific data sources (such as social media, news websites, etc.) or using user feedback to achieve it. Alternatively, the frequency of updating the sensitive semantic vector can be determined according to actual needs; for example, it can be set to update once a day, week, or month. Finally, the semantic vector of the new sensitive content and its related information are inserted into the above-mentioned first vector library. In this way, when new sensitive content appears, the semantic vector can be updated in an incremental manner without retraining the overall large model. Compared with the frequent model retraining and fine-tuning in the prior art, the first vector library constructed as above helps to achieve the real-time root of the semantic vector, greatly reducing the time and computational cost of accessing new sensitive content.

[0068] Step S240: generating a first sensitive content detection result of the image to be detected based on a first matching result between the target image semantic information and each sensitive semantic vector in the first vector library.

[0069] Among them, after obtaining the above-mentioned first vector library, the text feature judgment network in the above-mentioned target multimodal large model can be efficiently matched with each sensitive semantic vector in the first vector library, so as to achieve accurate judgment of sensitive content.

[0070] Specifically, the category of the sensitive information contained in the image to be detected can be determined based on the sensitive semantic vector that matches the semantic information of the target image searched in the first vector library; finally, the first sensitive content detection result is comprehensively obtained based on the category of sensitive information in the detected image to be detected and the content of the target image semantic information output by the above-mentioned target multimodal large model.

[0071] In the above-mentioned method for detecting sensitive content in images, a multimodal large model that comprehensively utilizes vision and text is adopted to uniformly project the image information to be detected into a high-dimensional semantic space, thereby achieving a deep understanding of image and text features and enhancing the sensitivity analysis of complex image content. At the same time, by constructing a dynamic sensitive semantic vector database, it can effectively support real-time updates of sensitive categories, can quickly adapt to new sensitive content, and has good dynamic expansion capabilities, thereby effectively solving the problem of low accuracy in image sensitive content detection and realizing an accurate and high-speed image sensitive content detection method.

[0072] In some of the embodiments, a method for detecting image sensitive content is also provided, such as Figure 3As shown, in the above step S220, the image to be detected is input into the visual feature judgment network in the target multimodal large model for visual detection processing to obtain the initial image semantic information, and the following steps may also be included:

[0073] Step S310: input the image to be detected into the visual feature judgment network in the target multimodal large model, and perform sensitive content detection on the image to be detected through the visual feature judgment network to determine whether the image to be detected contains sensitive content.

[0074] Step S320, if yes, the image to be detected is processed by visual sensitive feature extraction through the visual feature judgment network to obtain the first sensitive semantic type and behavior information, and the initial image semantic information is obtained based on the first sensitive semantic type and behavior information; otherwise, the image to be detected is processed by visual feature judgment network for semantic extraction to obtain the initial image semantic information.

[0075] Among them, after the image to be detected is input into the visual feature judgment network, the visual feature judgment network first performs sensitive content detection on the image to be detected. This detection method can be implemented by designing a prompt. If the image to be detected is judged to contain sensitive content, the visual sensitive feature extraction process is further performed; specifically, the visual feature judgment network further gives the first sensitive semantic type obtained by the judgment, and gives the behavioral information in the image based on which this judgment is made, and finally gives the text description obtained from the image, that is, the above-mentioned initial image semantic information is obtained. Since the test found that the initial image semantic information contains sensitive semantic types and behavioral information, in the subsequent steps, it is only necessary to transmit the initial image semantic information to the next level of text feature judgment network. On the contrary, if the image to be detected is not judged to contain a sensitive image, the text description information of the image (that is, the above-mentioned initial image semantic information) is directly given, and the result is passed to the next level of text feature judgment network for processing.

[0076] It should also be noted that the initial image semantic information includes: the text originally contained in the extracted image, the contextual information associated with the image (such as the output results of the visual feature judgment network: "type":"cultural and racial discrimination","action":"squinting eyes"), and the image summary information (such as the output results of the visual feature judgment network: "description":"In the image, a man with glasses is making a gesture of squinting his eyes,which can be perceived as aracist stereotype towards Asians.").

[0077] Through the above steps S310 to S320, a judgment is first made from a visual perspective on whether the image to be detected contains sensitive information based on the target multimodal large model, thereby utilizing the powerful ability of the multimodal large model in processing visual and language information to capture subtle features and semantic information in the image.

[0078] In some embodiments, the above-mentioned inputting the initial image semantic information into the text feature judgment network in the target multimodal large model to perform text detection processing to obtain the target image semantic information may also include the following steps:

[0079] The initial image semantic information is input into the text feature judgment network in the target multimodal large model, and the text feature judgment network performs sensitive content detection on the initial image semantic information to determine whether the text feature network contains sensitive content; if so, the text feature judgment network performs text sensitive feature extraction processing on the initial image semantic information to obtain a second sensitive semantic type, and obtains the target image semantic information based on the second sensitive semantic type; otherwise, the text feature judgment network performs semantic extraction processing on the initial image semantic information to obtain the target image semantic information.

[0080] In this step, after the initial image semantic information output by the visual feature judgment network is input into the text feature judgment network, the text feature judgment network first performs sensitive content detection on the initial image semantic information. Similarly, the detection method can be implemented by designing a prompt. If the initial image semantic information is judged to contain sensitive content, the text sensitive feature extraction process is further performed; specifically, the text feature judgment network can further give a second sensitive semantic type obtained by judgment, and perform semantic detection on the above initial image semantic information, and finally give a text description obtained from the image, that is, obtain the above target image semantic information. On the contrary, if the image to be detected is not judged to contain a sensitive image, the core text description of the above initial image semantic information (that is, the above target image semantic information) is directly given.

[0081] Optionally, the obtaining of the target image semantic information based on the second sensitive semantic type may further include the following steps:

[0082] The text feature judgment network is used to perform semantic feature compression on the initial image semantic information to obtain core image semantic information, and the target image semantic information is obtained based on the second sensitive semantic type and the core image semantic information.

[0083] Specifically, if the text feature judgment network determines from the text perspective that the initial image semantic information contains sensitive content through the above steps, the determined sensitive semantic type is further given, and the original text description is compressed with semantic features; the semantic feature compression method can be: the text feature judgment network finds repeated information or unnecessary details in the initial image semantic information, identifies and deletes modifiers, synonyms or lengthy explanations that do not contribute to the core semantics, and finally determines the core concept, theme or key action of the text, and extracts the core semantics. In addition, if the initial image semantic information of the text feature judgment network Ouan does not contain sensitive content, the semantic viewpoint obtained after semantic compression of the initial image semantic information is given.

[0084] Through the above-mentioned embodiments, by inputting the initial image semantic information into the text feature judgment network in the target multimodal large model, and performing sensitive content detection and text semantic extraction processing, the target image semantic information can be obtained, thereby effectively solving the problem in the related technology that, especially when processing images with complex backgrounds or multiple potential meanings, the extraction and analysis of visual features alone may not be able to fully capture all semantic information in the image, and realizing further semantic compression of text descriptions, in-depth analysis in the semantic space, and a second judgment on whether the image contains sensitive information from a text perspective. The image sensitive content detection method further improves the accuracy and comprehensiveness of image sensitive content detection.

[0085] In some embodiments, the generating of the first sensitive content detection result of the image to be detected based on the first matching result between the semantic information of the target image and each sensitive semantic vector in the first vector library may further include the following steps:

[0086] Perform vector conversion processing on the semantic information of the target image to obtain the semantic vector to be detected; traverse the sensitive semantic vectors in the first vector library; calculate the similarity result between the traversed sensitive semantic vector and the semantic vector to be detected; based on the similarity result, determine whether the sensitive semantic vector matches the semantic information of the target image, and generate a first sensitive content detection result.

[0087] In this step, the semantic information of the target image output by the above-mentioned target multimodal large model is converted into a vector to obtain the above-mentioned semantic vector to be detected. Afterwards, each sensitive semantic vector in the first vector library is traversed, the cosine similarity between the semantic vector to be detected and the currently traversed sensitive semantic vector is calculated, and the input semantic vector to be detected is judged whether it matches the sensitive semantic vector according to the preset similarity threshold, and finally the semantic information of the target image output by the target multimodal large model is combined to comprehensively obtain the sensitive review judgment result. Through the above-mentioned embodiments, based on a unified semantic space, a fast retrieval mechanism and dynamic expansion capabilities, efficient and accurate technical support is provided for complex and changeable sensitive content recognition.

[0088] In some embodiments, the above-mentioned determining whether the sensitive semantic vector matches the semantic vector to be detected based on the similarity result and generating the first sensitive content detection result may also include the following steps:

[0089] Obtain preset sensitivity level information and category weight information; based on the sensitivity level information and category weight information, assign a corresponding similarity threshold to each sensitive semantic vector in the first vector library; based on the comparison result between the similarity result and the corresponding similarity threshold, determine whether the sensitive semantic vector matches the semantic vector to be detected, and generate a first sensitive content detection result.

[0090] It should be noted that the above-mentioned sensitive feature determination process has the following two characteristics: first, the final decision rule; if the similarity between an input semantic vector to be detected and a sensitive category semantic vector exceeds a threshold, the possible sensitive content is marked; second, when multiple categories are matched, the sensitivity level and category weight can be defined according to business needs, thereby realizing customized review for business needs.

[0091] More specifically, with regard to the above-mentioned multi-category matching, different sensitivity levels can be set in advance. For different categories of sensitive content, corresponding sensitivity level labels can be set according to actual business needs. For example, the sensitive semantic vector containing sensitive content such as discrimination is set to the first-level sensitivity level, and the corresponding first sensitivity level label is assigned to this type of sensitive semantic vector; further, the sensitive semantic vector containing sensitive content such as racial discrimination and gender discrimination is set to the second-level sensitivity level, and a second sensitivity level label is added to this type of sensitive semantic vector.

[0092] In addition, for sensitive content belonging to different categories, corresponding category weights can be pre-set for each category in consideration of actual business needs. Next, sensitive semantic vectors of different categories can be quickly retrieved from the above-mentioned first vector library according to the set sensitivity level, and corresponding category weights can be assigned to each sensitive semantic vector. Based on the category weights assigned to each sensitive semantic vector, a corresponding similarity threshold can be assigned to each sensitive semantic vector. For example, based on the category weight information, a similarity threshold of 0.8 can be assigned to a sensitive semantic vector of a certain category, and a similarity threshold of 0.7 can be assigned to a sensitive semantic vector of another category.

[0093] In this way, when matching and identifying sensitive content for the semantic vector to be detected based on the first vector library, the calculated similarity result between the semantic vector to be detected and the currently traversed sensitive semantic vector can be compared with the similarity threshold corresponding to the currently traversed sensitive semantic vector. If the calculated similarity result exceeds the similarity threshold, it means that the semantic vector to be detected belongs to the sensitive content under this category. The above judgment method based on the corresponding similarity threshold is also adopted for the next traversed sensitive semantic vector until all sensitive semantic vectors in the first vector library are traversed, and finally the sensitive semantic vector result closest to the input semantic vector to be detected is provided, which is convenient for explaining and verifying the recognition result.

[0094] Through the above-mentioned embodiments, a corresponding similarity threshold is allocated to each sensitive semantic vector based on the preset sensitivity level information, so that not only sensitive content information can be identified, but also the granularity of detection can be flexibly adjusted according to the review requirements in the actual scenario to achieve personalized customization, thus laying a solid technical foundation for building a reliable content security system.

[0095] In some embodiments, after obtaining the semantic information of the target image, the following steps may also be included:

[0096] A first semantic similarity between the semantic information of the initial image and the image to be detected is calculated, and a second semantic similarity between the semantic information of the target image and the semantic information of the initial image is calculated.

[0097] Specifically, the calculation method of the first semantic similarity can be: using CLIPScore (an open source tool that uses the CLIP (Contrastive Language-Image Pretraining) model for evaluation) to ensure the semantic consistency between the generated text description and the original image, as shown in the following formula:

[0098] CLIPScore(c,v)=w*max(cos(c,v),0)

[0099] In the above formula, c is used to identify the initial image semantic information, v is used to represent the image to be detected, cos() is the cosine similarity calculation function, and max() is the maximum value function; w is a weight coefficient that can be preset, for example, w=2.5 can be set. CLIPScore(c,v) is used to represent the first semantic similarity score between the calculated initial image semantic information and the image to be detected; this score can represent the semantic similarity between the image and the text, that is, their distance in the semantic space. The higher the CLIPScore value, the higher the semantic similarity between the image and the text; conversely, the lower the value, the lower the semantic similarity.

[0100] On the other hand, the second semantic similarity can be calculated by ensuring the semantic consistency between the core semantics after semantic feature compression and the text description of the image to be detected (i.e., the initial image semantic information) through BertScore (a text similarity evaluation tool based on the bidirectional encoder representation BERT (Bidirectional Encoder Representations from Transformers, BERT) model based on the transformer), and tokenizing the initial image semantic information as x= <x1,…,x k >, the semantic information of the target image after semantic feature compression is expressed as Compare each token in x with The tokens in are matched to calculate the recall R BERT ,Will Each token in is converted to a token in x to calculate the precision P BERT , use greedy matching to maximize the matching similarity score, and combine precision and recall to calculate the F1 score F BERT The calculation formula is as follows:

[0101]

[0102] Next, based on the first semantic similarity and the second semantic similarity, a verification result for the target multimodal large model is generated. Wherein, if the first semantic similarity is less than the preset first semantic similarity threshold, it means that the semantic similarity between the text description output by the visual feature judgment network (i.e., the initial image semantic information) and the original image to be detected input to the visual feature judgment network does not meet the requirements; similarly, if the second semantic similarity is less than the preset second semantic similarity threshold, it means that the semantic similarity between the core semantic description output by the text feature judgment network (i.e., the target image semantic information) and the text description input to the text feature judgment network (i.e., the initial image semantic information) does not meet the requirements. Wherein, the first semantic similarity threshold and the second semantic similarity threshold can be set in advance according to actual conditions.

[0103] Optionally, in order to ensure the accuracy of the verification, if the first semantic similarity currently detected is less than the first semantic similarity threshold, and / or the second semantic similarity is less than the second semantic similarity threshold, the above-mentioned target multimodal large model is used to re-detect the image to be detected, and the above process is repeated. If the value of the first semantic similarity and / or the second semantic similarity is still low after repeated N times, a verification result indicating that the target multimodal large model verification has not passed is generated. Otherwise, a verification result indicating that the target multimodal large model verification has passed is generated. Wherein, N is a positive integer.

[0104] Among them, the large model may produce output that does not meet the specifications when processing batch input, or when the image situation is too complex, the large model may not be able to accurately output the text description of the image, resulting in a low semantic similarity between the text description output by the large model and the input information, affecting the accuracy of image sensitive content detection.

[0105] In order to improve the above problem, in this embodiment, when the verification result indicates that the target multimodal large model verification has not passed, a preset second vector library is obtained; the second vector library stores sensitive image vectors; the image vector to be tested of the image to be tested is calculated, and based on the second matching result between the image vector to be tested and each sensitive image vector in the second vector library, a second sensitive content detection result of the image to be tested is generated. Among them, the sensitive image vector can be a batch of sensitive images pre-input by the staff, or a sensitive image imported by the online system, and the obtained sensitive image is obtained by vector calculation.

[0106] Optionally, the second vector library may be a vector database different from the first vector library, that is, the first vector library and the second vector library respectively store two types of vectors, namely, sensitive semantic vectors and image semantic vectors.

[0107] Alternatively, the first vector library and the second vector library can also be constructed as the same vector database. In other words, in this vector database, sensitive image data and sensitive text data can be processed simultaneously by using the CLIP model and other methods, and data of different modalities can be mapped to the vector space of the same latitude, so as to achieve the alignment of images and texts in the semantic vector space; therefore, the vector database constructed at this time stores sensitive semantic vectors and image semantic vectors at the same time. Then, when the large model verification fails, the image vector to be tested of the image to be tested can be matched with all sensitive image vectors and sensitive semantic vectors in the vector database in turn, and the matched vector can be a sensitive image vector or a sensitive semantic vector, and finally the label content corresponding to the matched vector is output.

[0108] It should also be understood that when the above verification result indicates that the target multimodal large model has passed the verification, it means that the current target multimodal large model operates well and the result output by the large model is valid. Therefore, the target image semantic information output by the large model can be directly matched with the sensitive semantic vector in the first vector library to generate the first sensitive content detection result.

[0109] Through the above-mentioned embodiments, in the process of image processing of the target multimodal large model, the semantic information output by the two nodes in the large model is verified based on the semantic similarity to ensure the validity of the target multimodal large model; and, in the case that the large model verification fails, the backup retrieval scheme based on the second vector library is enabled, thereby avoiding the problem of errors in the detection of image sensitive content caused by possible errors in model detection, thereby effectively ensuring the accuracy and reliability of image sensitive content detection.

[0110] The present application is described below in conjunction with specific embodiments. Figure 4 is a schematic diagram of a method for detecting image sensitive content according to an embodiment of the present application, such as Figure 4 As shown, the method comprises the following steps:

[0111] Step S401, obtaining an image to be detected.

[0112] Step S402, visual feature judgment. The image to be detected is input into the target multimodal large model, and the target multimodal large model is used to firstly perform visual feature judgment on the image, and a judgment is made from a visual perspective on whether the image to be detected contains sensitive information. If the judgment result is yes, the sensitive semantic type obtained by the judgment is further given, and the behavioral information based on which this judgment is made is given, and finally the text description obtained from the image (i.e., the initial image semantic information) is given; if the result of the visual behavioral feature judgment is that it does not contain sensitive content, the text description information of the image (i.e., the initial image semantic information) is directly given, and the result is passed to the next node for processing.

[0113] During this process, CLIPScore can also be used to ensure the semantic consistency between the generated text description and the original image to be detected.

[0114] Step S403, text feature judgment: The initial image semantic information output by the visual feature judgment network is input into the text feature judgment network to perform semantic feature judgment and core idea extraction, and output the target image semantic information.

[0115] In this process, BertScore can also be used to ensure the semantic consistency between the generated target text description and the original initial text description.

[0116] Step S404: sensitive viewpoint search: perform similarity search on the target image semantic information and the vector data formed by the sensitive semantic viewpoints, and output the search result.

[0117] Step S405: outputting the judgment result. Based on the above-mentioned first vector library search result and the core summary in the target image semantic information, a judgment result of whether the image to be detected contains sensitive content and a detection result of the core summary content of the image are finally obtained.

[0118] For the scene of obscure sensitive image detection, the example shows the input:

[0119] Input an image to be detected containing a man with glasses making a squinting gesture; the user provides a description for guiding the visual feature judgment network: "I will give you a image below, please return the corresponding content according to the following format: isoffensive:{YES / NO}, if the isoffensive is YES,…Please reply in json format."; the output result of the visual feature judgment network for the image to be detected is: "{"isoffensive":"YES","type":"cultural andracial discrimination","action":"squinting eyes","description":"In the image,a man with glasses is making a gesture of squinting his eyes,which can beperceived as a racist stereotype towards Asians."}".

[0120] In addition, a description for guiding the text feature judgment network is provided: "I will give you a description below, please return the corresponding content according to the following format:\n isoffensive:{YES / NO},if isoffensive is YES,…Please reply in json format and as briefly as possible"; the output result of the text feature judgment network is: "{"isoffensive":"YES","type":"cultural and racial discrimination","summary":"Stereotypical racist gesture targeting Asians"}".

[0121] Next, the target image semantic information output by the text feature judgment network is matched with the sensitive semantic vector in the first vector library. The semantic vector of the retrieved core semantic viewpoint has a similarity of 0.95 with the semantic vector under the category of "cultural and racial discrimination" in the vector database. It is marked as sensitive content with the category of "cultural and racial discrimination", and the sensitive text corresponding to the semantic vector with the highest similarity is output.

[0122] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0123] This embodiment also provides a device for detecting image-sensitive content, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated here. As used below, the terms "module", "unit", "subunit", etc. can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0124] Figure 5 is a structural block diagram of a device for detecting image-sensitive content according to an embodiment of the present application, such as Figure 5 As shown, the device includes an image acquisition module 51, a semantic detection module 52, a first vector library acquisition module 53 and a generation module 54; wherein:

[0125] An image acquisition module 51 is used to acquire an image to be detected;

[0126] The semantic detection module 52 is used to input the image to be detected into the visual feature judgment network in the target multimodal large model for visual detection processing to obtain initial image semantic information, and input the initial image semantic information into the text feature judgment network in the target multimodal large model for text detection processing to obtain target image semantic information;

[0127] A first vector library acquisition module 53 is used to acquire a preset first vector library; the first vector library stores sensitive semantic vectors;

[0128] The generating module 54 is used to generate a first sensitive content detection result of the image to be detected based on a first matching result between the semantic information of the target image and each sensitive semantic vector in the first vector library.

[0129] In some embodiments, the above device further includes a verification module, a second vector library acquisition module and a backup retrieval module, wherein:

[0130] A verification module, used to calculate a first semantic similarity between the semantic information of the initial image and the image to be detected, and to calculate a second semantic similarity between the semantic information of the target image and the semantic information of the initial image;

[0131] The verification module is also used to generate a verification result for the target multimodal large model based on the first semantic similarity and the second semantic similarity;

[0132] A second vector library acquisition module is used to acquire a preset second vector library when the verification result indicates that the target multimodal large model verification has not passed; the second vector library stores sensitive image vectors;

[0133] The backup retrieval module is used to calculate the image vector to be detected of the image to be detected, and generate a second sensitive content detection result of the image to be detected based on the second matching result between the image vector to be detected and each sensitive image vector in the second vector library.

[0134] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination. For specific examples in this embodiment, reference can be made to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0135] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0136] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0137] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0138] S1, obtaining the image to be detected.

[0139] S2, input the image to be detected into the visual feature judgment network in the target multimodal large model for visual detection processing to obtain initial image semantic information, and input the initial image semantic information into the text feature judgment network in the target multimodal large model for text detection processing to obtain target image semantic information.

[0140] S3, obtaining a preset first vector library; the first vector library stores sensitive semantic vectors.

[0141] S4, generating a first sensitive content detection result of the image to be detected based on a first matching result between the target image semantic information and each sensitive semantic vector in the first vector library.

[0142] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0143] In addition, in combination with the image sensitive content detection method in the above embodiment, the present application embodiment can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any image sensitive content detection method in the above embodiment is implemented.

[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0145] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for detecting sensitive content of an image, characterized in that: The method comprises: Acquire the image to be detected; Inputting the image to be detected into a visual feature judgment network in a target multimodal large model for visual detection processing to obtain initial image semantic information, and inputting the initial image semantic information into a text feature judgment network in the target multimodal large model for text detection processing to obtain target image semantic information; Obtaining a preset first vector library; wherein the first vector library stores sensitive semantic vectors; Based on a first matching result between the target image semantic information and each of the sensitive semantic vectors in the first vector library, a first sensitive content detection result of the image to be detected is generated.

2. The detection method according to claim 1, characterized in that: The step of inputting the image to be detected into a visual feature judgment network in a target multimodal large model for visual detection processing to obtain initial image semantic information includes: Inputting the image to be detected into the visual feature judgment network in the target multimodal large model, and performing sensitive content detection on the image to be detected via the visual feature judgment network to determine whether the image to be detected contains sensitive content; If so, the visual feature judgment network is used to perform visual sensitive feature extraction on the image to be detected to obtain a first sensitive semantic type and behavior information, and the initial image semantic information is obtained based on the first sensitive semantic type and the behavior information; otherwise, the visual feature judgment network is used to perform semantic extraction on the image to be detected to obtain the initial image semantic information.

3. The detection method according to claim 1, characterized in that: The step of inputting the initial image semantic information into the text feature judgment network in the target multimodal large model for text detection processing to obtain the target image semantic information includes: Inputting the initial image semantic information into the text feature judgment network in the target multimodal large model, and performing sensitive content detection on the initial image semantic information via the text feature judgment network to determine whether the text feature network contains sensitive content; If so, the text feature judgment network is used to perform text sensitive feature extraction processing on the initial image semantic information to obtain a second sensitive semantic type, and the target image semantic information is obtained based on the second sensitive semantic type; otherwise, the text feature judgment network is used to perform semantic extraction processing on the initial image semantic information to obtain the target image semantic information.

4. The detection method according to claim 3, characterized in that: The obtaining the target image semantic information based on the second sensitive semantic type includes: The text feature judgment network is used to perform semantic feature compression on the initial image semantic information to obtain core image semantic information, and the target image semantic information is obtained based on the second sensitive semantic type and the core image semantic information.

5. The detection method according to claim 1, characterized in that: The generating a first sensitive content detection result of the image to be detected based on a first matching result between the target image semantic information and each of the sensitive semantic vectors in the first vector library includes: Performing vector conversion processing on the semantic information of the target image to obtain a semantic vector to be detected; Traversing the sensitive semantic vectors in the first vector library; Calculate the similarity result between the traversed sensitive semantic vector and the semantic vector to be detected; determine whether the sensitive semantic vector matches the target image semantic information based on the similarity result, and generate the first sensitive content detection result.

6. The detection method according to claim 5, characterized in that: The determining, based on the similarity result, whether the sensitive semantic vector matches the semantic vector to be detected, and generating the first sensitive content detection result includes: Obtain preset sensitivity level information and category weight information; Based on the sensitivity level information and the category weight information, assigning a corresponding similarity threshold to each of the sensitive semantic vectors in the first vector library; According to the comparison result between the similarity result and the corresponding similarity threshold, it is determined whether the sensitive semantic vector matches the semantic vector to be detected, and the first sensitive content detection result is generated.

7. The detection method according to any one of claims 1 to 6, characterized in that: The method further comprises: Calculating a first semantic similarity between the initial image semantic information and the image to be detected, and calculating a second semantic similarity between the target image semantic information and the initial image semantic information; Based on the first semantic similarity and the second semantic similarity, generating a verification result for the target multimodal large model; When the verification result indicates that the target multimodal large model has not passed the verification, obtaining a preset second vector library; the second vector library stores sensitive image vectors; The image vector to be tested of the image to be tested is calculated, and based on a second matching result between the image vector to be tested and each of the sensitive image vectors in the second vector library, a second sensitive content detection result of the image to be tested is generated.

8. A device for detecting image sensitive content, characterized in that: include: An image acquisition module, used for acquiring an image to be detected; A semantic detection module, used for inputting the image to be detected into a visual feature judgment network in a target multimodal large model for visual detection processing to obtain initial image semantic information, and inputting the initial image semantic information into a text feature judgment network in the target multimodal large model for text detection processing to obtain target image semantic information; A first vector library acquisition module, used to acquire a preset first vector library; the first vector library stores sensitive semantic vectors; A generation module is used to generate a first sensitive content detection result of the image to be detected based on a first matching result between the target image semantic information and each of the sensitive semantic vectors in the first vector library.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method for detecting image-sensitive content according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method for detecting image-sensitive content according to any one of claims 1 to 7 when running.

Images