Picture detection method, device, and computer storage medium

Abstract

Embodiments of the present description provide a picture detection method, a device, and a computer storage medium. In a scenario in which it is detected whether a picture is a picture of a target type, a category label library can be pre-constructed on the basis of the characteristic of the picture of the target type, and for each category label in the category label library, a probability threshold of the category label can be determined on the basis of the accuracy of the prediction results obtained by a target detection model for a large number of sample pictures. For each category label in the category label library, the probability threshold matched with the category label is preset, so that when the target detection model detects the picture of the target type, the accuracy is higher, and the phenomenon of erroneous detection or missing detection can be reduced. Moreover, the probability threshold corresponding to each category label can be adjusted on the basis of the accuracy of the prediction results obtained by the target detection model, so that a user can adjust the strictness of a detection criterion on the basis of actual requirements, and thus the picture detection is more flexible.

Description

Image detection method, device and computer storage medium

[0001] This disclosure claims priority to Chinese patent application number 202311396792.4, filed with the China Patent Office on October 25, 2023, and entitled “Image Detection Method, Device, and Computer Storage Medium,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments of this specification relate to the field of image processing technology, and more particularly to an image detection method, device, and computer storage medium. Background Art

[0003] In many scenarios, images need to be detected to determine whether they are of a specific type. Currently, when detecting image content, images are usually detected using some target detection models. The target detection model can output the probability that the image contains multiple preset categories of objects. For example, after inputting an image into the target detection model, the target detection model can output the probability that the image contains objects such as people, houses, vehicles, and trees. If this probability is greater than a pre-set probability threshold, the image is considered to contain the above-mentioned objects. Currently, when setting this probability threshold, a unified probability threshold is usually set for all objects. For example, the probability threshold is set to 80%. If the target detection model outputs a probability greater than 80% that the image contains a person, the image is considered to contain a person. Similarly, for other categories of objects, such as houses and vehicles, if the target detection model outputs a probability greater than 80% that the image contains a house or vehicle, the image is considered to contain a house or vehicle.

[0004] However, for different objects, since the target detection model has different difficulties in identifying the object, if a unified probability threshold is used as the judgment criterion, the target detection model may have inconsistent recognition accuracy for objects of different categories, resulting in the recognition accuracy of certain categories of objects being too low. In addition, the probability thresholds in related technologies are mostly set based on human experience, and it is difficult to control the judgment criteria of the target detection model. When detecting a certain type of picture, it is easy for the judgment criteria to be either too strict, and too many pictures that do not belong to the specific type are also screened out, or too loose, resulting in too many pictures of the specific type being unable to be recognized, causing too many false detections and missed detections.

[0005] Summary of the Invention

[0006] To overcome the problems existing in the related art, the embodiments of this specification provide an image detection method, device, and computer storage medium.

[0007] According to a first aspect of an embodiment of this specification, a method for detecting an image is provided, the method comprising:

[0008] Get the image to be detected;

[0009] Using the target detection model to predict the predicted probability of the object indicated by the category label in the preset category label library in the image to be detected;

[0010] If the predicted probability is greater than the probability threshold corresponding to the category label, determining that the image to be detected is a target type of image;

[0011] Among them, each category label in the category label library corresponds to a probability threshold, and the probability threshold corresponding to each category label is set based on the accuracy of the prediction results of the target detection model for multiple sample images. For each category label, if the target detection model predicts that the probability of including the object indicated by the category label in any sample image is greater than the probability threshold corresponding to the category label, it is determined that the sample image includes the object indicated by the category label, and then the accuracy of the prediction results of the target detection model for the multiple sample images is greater than the preset accuracy threshold.

[0012] According to a second aspect of the embodiments of this specification, a method for detecting an image is provided, the method comprising:

[0013] Get the image to be detected;

[0014] Obtaining a text description of the image to be detected using an image-text conversion model;

[0015] Determining whether the image to be detected includes a target object according to the text description;

[0016] If yes, use the target detection model to predict the predicted probability that the image to be detected includes an object of the first category, and if the predicted probability is greater than a preset probability threshold, determine that the image to be detected is a target type image;

[0017] The proportion of the first category of objects in the image to be detected is less than a preset first proportion, and the inclusion of the target object in the image to be detected is a prerequisite for the inclusion of the first category of objects in the image to be detected.

[0018] According to a third aspect of the embodiments of this specification, an electronic device is provided, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the computer program implements the method mentioned in the first and / or second aspects above when executed.

[0019] According to a fourth aspect of the embodiments of this specification, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method mentioned in the first and / or second aspects above is implemented.

[0020] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method as mentioned in the first aspect and / or the second aspect is implemented.

[0021] The beneficial effects of the embodiments of this specification are as follows: in the scenario of detecting whether a picture is a picture of a target type, a category label library can be constructed in advance based on the characteristics of the picture of the target type. The category label library can include one or more category labels, that is, when the picture includes an object indicated by any category label in the category label library, the picture can be determined to be a picture of the target type. In order to ensure that the target detection model has a relatively accurate recognition rate when identifying objects under each category label, the probability threshold corresponding to the category label can be predetermined for each category label. The probability threshold of each category label can be adjusted based on the accuracy of the prediction results of the target detection model for a large number of sample pictures. For example, if the probability of the sample picture predicted by the target detection model including the object indicated by the category label is greater than the probability threshold corresponding to the category label, it is considered that the sample picture includes the object indicated by the category label as a judgment criterion. Under this judgment criterion, the accuracy of the prediction results of the target detection model for these sample pictures is greater than the preset accuracy threshold. By presetting a probability threshold for each category label in the category label library, the target detection model achieves higher accuracy when detecting target images, reducing false or missed detections. Furthermore, the probability threshold for each category label can be adjusted based on the accuracy of the target detection model's prediction results, allowing users to adjust the strictness of the detection criteria based on actual needs, making image detection more flexible.

[0022] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory and are not restrictive of the embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings herein are incorporated in and constitute a part of the embodiments of this specification, illustrate embodiments consistent with the embodiments of this specification, and together with the description, serve to explain the principles of the embodiments of this specification.

[0024] FIG1 is a schematic diagram of detecting an image using target detection;

[0025] FIG2 is a schematic diagram of an image detection method according to an exemplary embodiment of this specification;

[0026] FIG3 is a schematic diagram of an application scenario shown in an exemplary embodiment of this specification;

[0027] FIG4 is a flow chart of an image detection method according to an exemplary embodiment of this specification;

[0028] FIG5 is a schematic diagram showing an exemplary embodiment of this specification showing detection of an image by combining an image-text conversion model and an object detection model;

[0029] FIG6 is a schematic diagram of constructing a tag category library according to an exemplary embodiment of this specification;

[0030] FIG7 is a logic block diagram of an electronic device according to an exemplary embodiment of this specification. DETAILED DESCRIPTION

[0031] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible implementations consistent with the embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of the embodiments of this specification, as detailed in the appended claims.

[0032] The terms used in the embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit the embodiments of this specification. The singular forms "a," "the," and "the" used in the embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0033] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when..." or "when..." or "in response to determining."

[0034] In many scenarios, images need to be tested to determine whether they are of a specific type. For example, to promote a product or service, advertising images may be displayed on certain apps or websites. To prevent these images from containing illegal content, such as violent or indecent images, these images can be tested before display to determine whether they contain such content. If so, the images are prohibited from being displayed.

[0035] Generally speaking, when determining whether an image is a specific type of image, the image's content is typically tested to determine whether it contains certain categories of objects, and then whether it is a specific type of image. For example, if an image contains guns, powdery objects, or excessively exposed body parts, it may be considered an illegal image.

[0036] At present, when detecting the content of an image, the image is usually detected through some target detection models. The target detection model can output the probability that the image includes multiple preset categories of objects. For example, as shown in Figure 1, after the image is input into the target detection model 11, the target detection model 11 can output the probability that the image includes objects such as people, houses, vehicles, trees, etc. If the probability is greater than a certain pre-set probability threshold, it is considered that the image includes the above-mentioned objects. At present, when setting the probability threshold, a probability threshold is usually set uniformly for all objects. For example, the probability threshold is set to 80%. If the target detection model 11 outputs that the probability of including people in the image is greater than 80%, it is considered that the image includes people. Similarly, for other categories of objects, such as houses, vehicles, etc., if the target detection model 11 outputs that the probability of including houses and vehicles in the image is greater than 80%, it is considered that the image includes houses and vehicles. However, for different objects, since the target detection model 11 has different difficulties in identifying the object, if a unified probability threshold is used as the judgment criterion, the recognition accuracy of the target detection model 11 for objects of different categories may be inconsistent, resulting in the recognition accuracy of certain categories of objects being too low. In addition, the probability thresholds in related technologies are mostly set based on human experience, and it is difficult to control the judgment criteria of the target detection model 11. When detecting a certain type of picture, it is easy for the judgment criteria to be either too strict, and too many pictures that do not belong to the specific type are also screened out, or the judgment criteria are too loose, resulting in too many pictures of the specific type being unable to be recognized, causing too many false detections and missed detections.

[0037] Based on this, an embodiment of the present disclosure provides an image detection method. As shown in FIG2 , in the scenario of detecting whether an image is a target type image, a category label library can be constructed in advance based on the characteristics of the target type image. The category label library can include one or more category labels. That is, when the image includes an object indicated by any category label in the category label library, the image can be determined to be a target type image. In order to ensure that the target detection model 11 has a relatively accurate recognition rate when identifying objects under each category label, a probability threshold corresponding to the category label can be predetermined for each category label. The probability threshold of each category label can be adjusted based on the accuracy of the prediction results of the target detection model 11 on a large number of sample images. For example, if the probability of the sample image predicted by the target detection model 11 including the object indicated by the category label is greater than the probability threshold corresponding to the category label, it is considered that the sample image includes the object indicated by the category label as the judgment criterion. Under this judgment criterion, the accuracy of the prediction results of the target detection model 11 on these sample images is greater than the preset accuracy threshold.

[0038] By presetting a probability threshold that matches each category label in the category label library, the object detection model 11 can achieve higher accuracy when detecting images of target types, reducing false detections or missed detections. Furthermore, the probability threshold corresponding to each category label can be adjusted based on the accuracy of the prediction results of the object detection model 11, allowing users to adjust the strictness of the detection criteria based on actual needs, making image detection more flexible.

[0039] The image detection method of the embodiment of the present disclosure can be executed by various terminal devices such as mobile phones, tablets, and computers, or by a server or server cluster. For example, the terminal device can detect the image before displaying it, or the server can detect the image to be displayed when the terminal device sends a request to display the image to the server, and after determining that the image is not an image of the target type, send it to the terminal device for display. For example, as shown in Figure 3, a schematic diagram of an application scenario of the embodiment of the present disclosure is shown. Taking the overseas business scenario of e-commerce products as an example, merchants usually provide advertising materials for their products, which often include a large number of images. Since overseas advertising has certain requirements for advertising images, for example, the placement of illegal images involving violence, guns, etc. is prohibited. Therefore, before placing these merchants' advertising images, the third-party e-commerce platform can detect these advertising images and filter out some illegal images. For example, merchants can upload these advertising images through the client 21 of the third-party e-commerce platform. The client 21 sends the advertising images to the server 22 of the third-party e-commerce platform. The server 22 can be set with a target detection model 11 to detect whether these advertising images are illegal images through the target detection model 11. If so, the advertising images of this type are rejected. Of course, the above application scenarios are only illustrative examples. The image detection solution provided by the embodiments of the present disclosure is not limited to the above scenarios, but is applicable to all image detection scenarios.

[0040] As shown in FIG4 , the image detection method according to an embodiment of the present disclosure may include the following steps:

[0041] S402, obtaining a picture to be detected;

[0042] In step S402, a picture to be detected may be obtained, wherein the picture to be detected may be various types of pictures that need to be detected, and the embodiment of the present disclosure does not impose any limitation thereto.

[0043] S404, using the object detection model 11 to predict the predicted probability of the object indicated by the category label in the preset category label library in the image to be detected;

[0044] In step S404, after obtaining the image to be detected, the target detection model 11 can be used to detect the image to be detected to predict the predicted probability of the image including the object indicated by the category label in the category label library. Considering that when a picture usually includes objects of certain categories, it is determined to be a picture of a certain target type. Therefore, based on the characteristics of the target type of picture, that is, which category objects it includes, a category label library can be pre-built. The category label library can include one or more category labels. That is, when the picture includes the object indicated by the category label in the category label library, the picture is a picture of the target type.

[0045] For example, consider the target type of images: illegal images. Considering that illegal images include a variety of categories, such as violent images of guns and weapons, illegal images of drugs, and indecent images of excessively exposed body parts, we can construct a category label library based on the content of the illegal images. This category label library may include the following category labels: guns, powder, and specific exposed body parts. We then determine whether the image to be tested contains the objects indicated by these category labels. If so, we can determine that the image to be tested is an illegal image.

[0046] Among them, the target detection model 11 can be various pre-trained open source target detection models 11, such as the blip-2 (Bootstrapping Language-Image Pre-training) model, the Faster-CNN (Faster Region-based Convolutional Neural Network) model, or a target detection model 11 trained by using sample images. As long as it has the function of target detection, the embodiment of the present disclosure does not impose any restrictions.

[0047] S406: If the predicted probability is greater than the probability threshold corresponding to the category label, the image to be detected is determined to be a target type image;

[0048] Among them, each category label in the category label library corresponds to a probability threshold, and the probability threshold corresponding to each category label is set based on the accuracy of the prediction results of the target detection model 11 for multiple sample images. For each category label, if the target detection model 11 predicts that the probability of any sample image including the object indicated by the category label is greater than the probability threshold corresponding to the category label, it is determined that the sample image includes the object indicated by the category label, and then the accuracy of the prediction results of the target detection model 11 for multiple sample images is greater than the preset accuracy.

[0049] In step S406, to ensure the accuracy of the target detection model 11 in identifying objects indicated by various category labels, after constructing a category label library based on target type images, a corresponding probability threshold can be pre-determined for each category label in the category label library. When determining the probability threshold corresponding to each category label, a large number of sample images can be obtained, and then the probability threshold can be set using the accuracy of the target detection model 11's prediction results for these sample images. If the criterion for determining whether a sample image includes the object indicated by the category label is: "When the probability that the target detection model 11 predicts that the sample image includes the object indicated by the category label is greater than the probability threshold corresponding to the category label, then the sample image is determined to include the object indicated by the category label," then when these sample images are inspected according to this criterion, the accuracy of the target detection model 11's prediction results for these sample images is greater than the preset accuracy threshold. This accuracy threshold can be flexibly set based on the user's desired strictness of the criterion. For example, if a lower false detection rate is desired, the accuracy threshold can be set higher.

[0050] For example, taking the category label "person" as an example, multiple sample images that include and exclude people can be obtained, and the target detection model 11 is used to predict the probability that these sample images include "person". If the probability is greater than the probability threshold, it is considered that "person" is included. Among them, the probability threshold must make the accuracy of the prediction results of the target detection model 11 for these sample images higher than the preset accuracy threshold. For example, assuming the preset accuracy threshold is 80%, and the sample images are 100 images, then the probability threshold must make the prediction results of the target detection model 11 for at least 80 of the 100 images accurate.

[0051] In some embodiments, when determining the probability threshold corresponding to each category label in the category label library, multiple sample images can be obtained, and these sample images carry labels, which are used to indicate whether the sample image includes the object indicated by the category label. Then, an initial threshold of the probability threshold can be set, and the sample image can be detected using the target detection model 11. When the probability that the sample image predicted by the target detection model 11 includes the object indicated by the category label is greater than the preset initial threshold, it is determined that the sample image includes the object indicated by the category label. Then, based on the judgment results corresponding to each of the multiple sample images, the accuracy of the prediction results of the target detection model 11 under the initial threshold can be statistically calculated. Then, it can be determined whether the accuracy is lower than the preset accuracy threshold. If it is lower, the initial threshold is adjusted and then, based on the adjusted initial threshold, each sample image is re-determined to determine whether it includes the object indicated by the category label. The above process is repeated until the accuracy of the prediction result of the target detection model 11 is higher than the preset accuracy threshold, and the final adjusted initial threshold is used as the probability threshold.

[0052] For example, taking the category label "person" as an example, you can first set an initial value (for example, 0.6) and an adjustment range (for example, 0.05) of the probability threshold, and then based on the probability of each sample image including a person predicted by the target detection model 11, if it is greater than 0.6, it is determined that the sample image includes a person, and the prediction result of the sample image can be determined based on the label of the sample image. After completing the prediction of all sample images, the accuracy of the prediction results can be counted. If it is less than the preset accuracy threshold (for example, 0.8), the probability threshold can be increased, such as adjusted to 0.65, and then the above steps are repeated until the accuracy of the prediction results of the final target detection model 11 for these sample images is greater than or equal to 0.8, and the adjusted probability threshold at this time is used as the final probability threshold.

[0053] In some embodiments, in order to improve the accuracy of setting the probability threshold corresponding to each category label, the sample images may include positive samples and negative samples, that is, sample images that include the object indicated by the category label and sample images that do not include the object indicated by the category label. In addition, the sample images may include some positive samples or negative samples that are easily confused. For example, taking the category label "person" as an example, the sample images may include some sample images containing sculptures or dresses that are easy to confuse the object detection model 11, causing it to determine that the image contains "person".

[0054] Considering most scenarios, the number of images to be detected may be large. If the target detection model 11 is used to predict the probability of each image to be detected containing the object indicated by the category label library, the detection speed may be slow. To improve detection speed, the image to be detected can be detected in conjunction with a text-to-image conversion model. The text-to-image conversion model can directly convert the content of the image to be detected into a text description, and then directly determine whether the text description contains the category label from the category label library, that is, whether the image to be detected contains the object indicated by the category label. Obviously, text matching is relatively simple. Compared with directly detecting the probability of each type of object in the image to be detected, this method is more efficient and can improve detection speed. However, since the text-to-image conversion model can generally only recognize the content of the image at a larger granularity, that is, it often describes the large-scale subject in the image and cannot recognize and describe small-scale objects. For example, taking the detection of illegal images as an example, assuming there is a picture of a person with a specific body part exposed, the text-to-image conversion model may often only recognize the person in the picture, but not the specific exposed body part, and thus cannot detect the illegal image. The target detection model 11 can overcome the above-mentioned defects, that is, the target detection model 11 can identify more granular objects in the image, but its detection speed is slower.

[0055] In order to improve both the detection speed and the accuracy of the detection results, in some embodiments, as shown in FIG5 , the above two types of models can be combined to detect images. For example, the objects indicated by the category labels in the category label library can be divided into two categories, objects of the first category and objects of the second category. Among them, the proportion of objects of the first category in the image to be detected is usually small, for example, less than the preset first proportion, and such objects may often be local areas of certain objects, or some details. The proportion of objects of the second category in the image to be detected is usually large, for example, greater than the preset second proportion, that is, such objects are often the main part of the image. Among them, the first proportion and the second proportion can be the same or different.

[0056] For objects of the first category, since the image-text conversion model 12 may not be able to recognize them, the image-text conversion model 12 can be used to determine whether the image includes a target object of larger granularity and easier to identify. Among them, the inclusion of the target object in the image to be detected is a prerequisite for it to include the object of the first category. In some embodiments, the object of the first category can be a local area of ​​the target object. For example, the object of the first category can be a naked body part, and the target object can be a person. The prerequisite for the image to be detected to include a naked body part is that the image to be detected includes a person. Before using the target detection model 11 to detect the image to be detected, the image-text conversion model 12 can be used to obtain a text description of the image to be detected, and then the text description can be used to determine whether the image to be detected includes the target object. If it does not include it, it means that the image to be detected cannot include the object of the first category, and there is no need to input it into the target detection model 11 for the next step of finer-grained object detection. If it is determined that the image to be detected includes the target object, the target detection model 11 can be further used to detect the image to be detected to further determine whether it includes the object of the first category.

[0057] By using the image-text conversion model 12 to pre-judge whether a more fine-grained object detection is required for the image to be detected, if not, there is no need to use the target detection model 11 for detection. If it is required, further detection using the target detection model 11 can be performed, which can greatly improve the detection speed. In addition, since the detection results of the target detection model 11 cannot be guaranteed to be 100% accurate and there is a certain amount of false detection, and the image-text conversion model 12 is often more accurate in identifying the subject in the image, performing further detection based on the pre-judgement results of the image-text conversion model can also improve the accuracy of the detection results to a certain extent.

[0058] In some embodiments, considering that the objects of the second category account for a large proportion of the image to be detected and are often the main body of the image, they can be directly identified using the image-text conversion model 12. Therefore, when it is determined that the image to be detected does not include the target object based on the text description output by the image-text conversion model 12, it can be further determined based on the text description whether the image to be detected includes the objects of the second category. If included, the image to be detected is determined to be a picture of the target type. For objects of the second category with large granularity, since the image-text conversion model 12 can obtain more accurate detection results, detection can be performed directly using the image-text conversion model 12, thereby improving the detection speed.

[0059] Among them, the image-text conversion model 12 can be any model that can convert images into text. The model can be a pre-trained open source model or a self-trained model, and the embodiment of the present disclosure does not limit this.

[0060] In some embodiments, considering that if the image to be detected is determined to be a target type image based only on the prediction results of the target detection model 11, there may be misjudgments due to the limited accuracy of the target detection model 11, in order to further improve the accuracy of the image detection results, the image to be detected can be detected in combination with the image-text conversion model 12. For example, before determining that the image to be detected is a target type image, the image-text conversion model 12 can be used to obtain a text description of the image to be detected, and then determine whether the text description includes a category label in the category label library. If the text description of the image to be detected includes a category label in the category label library, and the predicted probability of the image to be detected output by the target detection model 11 including the object indicated by the category label is greater than the probability threshold corresponding to the category label, then the image to be detected is determined to be a target type image.

[0061] By combining the text description output by the image-text conversion model 12 and the prediction probability output by the target detection model 11, it is determined whether the image to be detected is a target type image, which can greatly improve the accuracy of the detection result.

[0062] In some embodiments, the image-text conversion model 12 may be a blip-2 model, and the target detection model 11 may be a clip (Contrastive Language-Image Pretraining) model. The blip-2 model can identify the main content of an image, but cannot sink to a more fine-grained dimension (for example, it can only identify whether there is a person in the image, but cannot identify the specific body part of the person), while the clip model can output the probability of the presence of certain fine-grained objects in the current image, for example, the probability of the presence of a person is 0.95, and the probability of the presence of a specific body part is 0.9, but the clip model cannot 100% identify whether the image includes a certain object. Taking the identification of illegal images that expose specific body parts as an example, the blip-2 model can determine whether the image to be tested includes a person, while the clip model can identify specific body parts at a more fine-grained level and output the probability of including specific body parts. Therefore, the blip-2 model and the clip model can be combined for image recognition. First, blip-2 is used to identify whether there is a person. If so, the clip is used to determine whether there are certain body parts that cannot be exposed to determine whether the image is illegal.

[0063] In some embodiments, the category label library can be constructed manually by the user or automatically. For example, considering that there are various ways to name or describe the same object, that is, the way to express its category label is also various. For example, for the object "house", it can be described as "house", "building", "construction" and so on. Considering that the category labels constructed by the user may be expressed in a different way than the description output by the model, if the category labels constructed by the user are used directly, there may be a problem of inconsistency between the description output by the model and the description of the category labels constructed by the user, resulting in misrecognition.

[0064] For example, taking the image-text conversion model 12 as an example, assuming that the image to be detected is input into the image-text conversion model 12, the text description input by the image-text conversion model 12 is "a car parked in front of the house", and the category label set by the user for the object "house" is "building". Therefore, when determining whether the image to be detected includes "house" based on the text description, the inconsistency between the two expressions may lead to the final determination that the image does not include "house". Similarly, for the target detection model 11, if the target detection model 11 outputs "the probability of the house is 0.9", and the category label set by the user for the object "house" is "building", it may also cause the model to ultimately determine that the image does not include "house", thereby causing misrecognition.

[0065] Therefore, a category label library that is compatible with the model's description can be constructed to avoid the problem of misjudgment of the model due to inconsistent descriptions. For example, as shown in Figure 6, in order to obtain category labels that are compatible with the above-mentioned image-text conversion model 12, when constructing the category label library, a target picture set can be obtained, and the pictures in the target picture set are pictures of the target type. Then, the pictures in the target picture set can be input into the image-text conversion model 12, and the image-text conversion model 12 obtains the text description corresponding to the picture. Then, the category label of the specified type can be extracted from the text description to construct a category label library. By extracting the category label that meets the requirements from the text description output by the image-text conversion model 12, the constructed category label is consistent with the description of the image-text conversion model 12, thereby reducing the phenomenon of false detection due to inconsistent descriptions.

[0066] In some embodiments, a similar approach can also be used for the target detection model 11 to obtain category labels that are compatible with the above-mentioned target detection model 11. When constructing a category label library, a target picture set can be obtained, and the pictures in the target picture set are pictures of the target type. Then, the pictures in the target picture set can be input into the target detection model 11, and the target detection model 11 outputs the probability of objects of each category label included in the picture. Then, the category label of the specified type can be extracted from the category label output by the target detection model 11 to construct a category label library.

[0067] By extracting the category labels that meet the requirements from the category labels output by the target detection model 11, the constructed category labels are consistent with the category labels output by the target detection model 11 during the prediction process, thereby reducing the phenomenon of false detection due to inconsistent expressions.

[0068] In some embodiments, in order to automatically extract category labels that match the target type of pictures, the user can first set a baseline category label based on the characteristics of the target type of pictures. The category label of the specified type can be consistent with the baseline category label set by the user, or it can have the same semantics as the baseline category label set by the user.

[0069] In addition, considering most scenarios, the number of images to be detected may be large. If the target detection model is used one by one to predict the probability of each image to be detected including the object indicated by the category label in the category label library, the detection speed may be slow. In order to improve the detection speed, one way the applicant has thought of is to combine the image-text conversion model to detect the image to be detected. The image-text conversion model can directly convert the content in the image to be detected into a text description, and then directly determine whether the text description includes the category label of the category label library, that is, it can determine whether the image to be detected includes the object indicated by the category label. Obviously, text matching is relatively simple, and this method is more efficient than directly detecting the probability of including various objects in the image to be detected, which can improve the detection speed. However, since the image-text conversion model can usually only recognize the content of the image at a larger granularity, that is, it often describes the large-grained subject in the image, and cannot recognize and describe small-grained objects. For example, in the case of detecting illegal images, if a person's image shows a specific naked body part, the image-to-text conversion model may only recognize the person in the image, but not the exposed body part, and thus fail to detect the illegal image. Object detection models, on the other hand, can overcome this limitation by identifying objects at a more granular level within the image, but at a slower speed.

[0070] Based on this, the embodiment of the present disclosure provides an image detection method, which can combine the target detection model and the image-text detection model to detect the image to be detected, and set the corresponding detection order and method, so as to improve the detection efficiency and ensure that the detection results have high accuracy.

[0071] The image detection method specifically includes the following steps:

[0072] Get the image to be detected;

[0073] Use the image-text conversion model to obtain the text description of the image to be detected;

[0074] Determine whether the image to be detected contains the target object based on the text description;

[0075] If so, the target detection model is used to predict the predicted probability that the image to be detected includes an object of the first category. When the predicted probability is greater than a preset probability threshold, the image to be detected is determined to be a target type image; wherein, the proportion of objects of the first category in the image to be detected is less than a preset first proportion, and the inclusion of the target object in the image to be detected is a prerequisite for the image to be detected to include an object of the first category.

[0076] In some embodiments, the object of the first category is a local area of ​​the target object.

[0077] In some embodiments, the method further comprises:

[0078] If it is determined based on the text description that the image to be detected does not include the target object, then determining based on the text description whether the image to be detected includes an object of a second category, wherein a proportion of the objects of the second category in the image to be detected is greater than a preset second proportion;

[0079] If included, it is determined that the image to be detected is a target type of image.

[0080] The specific implementation details of the above detection method can be referred to the description in the above embodiment and will not be repeated here.

[0081] Corresponding to the picture detection method embodiment provided in the embodiments of this specification, this specification also provides a picture detection device, which includes:

[0082] Acquisition module, used to obtain the image to be detected;

[0083] A prediction module is used to use the target detection model to predict the predicted probability of an object indicated by a category label in a preset category label library in the image to be detected;

[0084] A determination module is used to determine that the image to be detected is a target type of image if the predicted probability is greater than the probability threshold corresponding to the category label;

[0085] Among them, the probability threshold corresponding to the category label in the category label library is set based on the accuracy of the prediction results of the target detection model for multiple sample images. If the target detection model predicts that the probability of any sample image including the object indicated by the category label is greater than the probability threshold corresponding to the category label, it is determined that the sample image includes the object indicated by the category label, and the accuracy of the prediction results of the target detection model for multiple sample images is greater than the preset accuracy threshold.

[0086] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned image detection method, and will not be repeated here.

[0087] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiment scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0088] From a hardware perspective, Figure 7 shows a hardware structure diagram of a device that includes an apparatus for determining the office area to which a device belongs, according to an embodiment of this specification. In addition to the processor 72 and memory 74 shown in Figure 7 , the device typically also includes other hardware, such as a forwarding chip responsible for message processing. From a hardware perspective, the device may also be distributed, potentially including multiple interface cards to facilitate hardware-level message processing expansion. Memory 74 stores computer instructions, and processor 72 executes these instructions to implement the methods described in any of the aforementioned embodiments.

[0089] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0090] Since the portion of the embodiment of this specification that contributes to the prior art or the entire or partial technical solution can be embodied in the form of a software product, the computer software product is stored in a storage medium and includes a number of instructions for causing a terminal device to execute all or part of the steps of each method of the embodiment of this specification. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0091] The embodiments of this specification also provide a computer program product, including a computer program. When executed by a processor, the computer program implements the method of any of the aforementioned embodiments. The computer program is stored in a readable storage medium. At least one processor of an electronic device can read the computer program from the readable storage medium. The at least one processor executes the computer program, causing the electronic device to perform the process of the image detection method of any of the aforementioned method embodiments. The specific functions and technical effects achieved are not further described here.

[0092] The above description is only a preferred embodiment of the embodiments of this specification and is not intended to limit the embodiments of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of this specification should be included in the scope of protection of the embodiments of this specification.

Claims

1. A method for image detection, wherein: The method comprises: Get the image to be detected; Predicting the predicted probability of an object indicated by a category label in a preset category label library in the image to be detected by using the target detection model; If the predicted probability is greater than the probability threshold corresponding to the category label, determining that the image to be detected is a target type of image; Among them, each category label in the category label library corresponds to a probability threshold, and the probability threshold corresponding to each category label is set based on the accuracy of the prediction results of the target detection model for multiple sample images. For each category label, if the target detection model predicts that the probability of including the object indicated by the category label in any sample image is greater than the probability threshold corresponding to the category label, it is determined that the sample image includes the object indicated by the category label, then the accuracy of the prediction results of the target detection model for the multiple sample images is greater than the preset accuracy threshold.

2. The method according to claim 1, wherein: The probability threshold corresponding to each category label is set based on the accuracy of the prediction results of the target detection model for multiple sample images, including: Acquire multiple sample images carrying labels, where the labels are used to indicate whether the sample images include an object indicated by the category label; When the probability that the sample image predicted by the target detection model includes the object indicated by the category label is greater than a preset initial threshold, the sample image is determined to include the object indicated by the category label, and the accuracy of the target detection model prediction result under the initial threshold is calculated based on the determination results corresponding to each of the multiple sample images; When it is determined that the accuracy is lower than the preset accuracy threshold, the initial threshold is continuously adjusted until the accuracy of the prediction result of the target detection model is higher than the preset accuracy threshold, and the adjusted initial threshold is used as the probability threshold.

3. The method according to claim 1, wherein: The objects indicated by the category labels in the category label library include objects of a first category, and the proportion of the objects of the first category in the image to be detected is less than a preset first proportion; The method of using the target detection model to predict the predicted probability of the object indicated by the category label in the category label library in the image to be detected includes: Obtaining a text description of the image to be detected using an image-text conversion model; Determining whether the image to be detected includes a target object according to the text description; If yes, the target detection model is used to predict the predicted probability that the image to be detected includes the object of the first category, wherein the inclusion of the target object in the image to be detected is a prerequisite for the image to be detected to include the object of the first category.

4. The method according to claim 3, wherein: The object of the first category is a local area of ​​the target object.

5. The method according to claim 3, wherein: The objects indicated by the category labels in the category label library include objects of the second category, and the proportion of the objects of the second category in the image to be detected is greater than a preset second proportion; The method further includes: If it is determined based on the text description that the image to be detected does not include the target object, determining based on the text description whether the image to be detected includes an object of the second category; If included, it is determined that the picture to be detected is a picture of the target type.

6. The method according to claim 1, wherein: Before executing the step of determining that the image to be detected is a target type image if the predicted probability is greater than the probability threshold corresponding to the category label, the method further includes: Obtaining a text description of the image to be detected by using an image-text conversion model, and determining whether the text description includes a category label in the category label library; If the predicted probability is greater than the probability threshold corresponding to the category label, determining that the image to be detected is a target type of image includes: If the text description includes a category label in the category label library, and the predicted probability is greater than a probability threshold corresponding to the category label, it is determined that the image to be detected is a target type of image.

7. The method according to any one of claims 1 to 6, wherein: The category label library is constructed based on the following method: Acquire a target picture set, where pictures in the target picture set are pictures of the target type; Input the pictures in the target picture set into the image-text conversion model, extract the category labels of the specified type from the text descriptions of the pictures output by the image-text conversion model, so as to construct the category label library; and / or input the pictures in the target picture set into the target detection model, extract the category labels of the specified type from the category labels output by the target detection model, so as to construct the category label library.

8. The method according to claim 7, wherein: The category label of the specified type has the same or similar semantics as the reference category label set by the user.

9. The method according to claim 3 or 6, wherein: The target detection model is a clip model, and the image-text conversion model is a blip-2 model.

10. A method for image detection, wherein: The method comprises: Get the image to be detected; Obtaining a text description of the image to be detected using an image-text conversion model; Determining whether the image to be detected includes a target object according to the text description; If yes, predicting the predicted probability that the image to be detected includes an object of the first category by using the target detection model, and determining that the image to be detected is a target type image when the predicted probability is greater than a preset probability threshold; The proportion of the objects of the first category in the picture to be detected is less than a preset first proportion, and the inclusion of the target object in the picture to be detected is a prerequisite for the inclusion of the objects of the first category in the picture to be detected.

11. An electronic device, wherein: The electronic device comprises a processor and a memory, wherein a computer program is stored in the memory, and the processor executes the computer program to implement the method according to any one of claims 1 to 10.

12. A computer storage medium, wherein: The computer storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

13. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images