Image detection method, device, electronic device and storage medium
Through the method of combining anomaly object detection and classification model with multimodal large model, question text information is generated for image detection, which solves the problem of insufficient detection accuracy of abnormal content in the image and achieves more efficient and accurate abnormal content recognition.
Patent Information
- Application Number
- CN202411016817.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-26
AI Technical Summary
The prior art is difficult to effectively detect and identify abnormal content in images, especially containing uncivilized content, which leads to the risk of bad guidance to users and insufficient detection accuracy.
An image detection method is adopted to perform object detection through an abnormal object detection model, category prediction is performed based on the abnormal classification model, question text information containing prediction category information, and input the image and text information into the multimodal large model for abnormal detection , improve detection accuracy.
It improves the accuracy and efficiency of image detection, reduces invalid detection, reduces manual labeling costs, and enhances the ability to identify abnormal content.
Smart Images

Figure CN118537666B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image detection technology, and in particular to an image detection method, an image detection device, an electronic device, and a computer-readable storage medium. Background Art
[0002] Internet platforms provide users with convenient online interaction channels. In order to enrich the interaction methods and improve the efficiency of information sharing, most platforms support users to interact or share resources by uploading images. However, some images may contain abnormal content, such as uncivilized content. These abnormal contents have the risk of causing bad guidance to other users. Therefore, the detection of image content has become particularly important. Summary of the invention
[0003] The embodiments of the present disclosure at least provide an image detection method, an image detection device, an electronic device, and a computer-readable storage medium, which can improve the accuracy of image anomaly detection.
[0004] The present disclosure provides an image detection method, including:
[0005] Acquire a candidate abnormal image; the candidate abnormal image is obtained by performing target detection on a target image using an abnormal target detection model;
[0006] Based on the abnormal classification model, predict the abnormal category of the candidate abnormal image to obtain the predicted abnormal category information of the candidate abnormal image;
[0007] Based on the predicted abnormality category information, generating question text information for the candidate abnormal image and including the predicted abnormality category information; the question text information is used to indicate whether the candidate abnormal image contains content corresponding to the predicted abnormality category information;
[0008] The candidate abnormal image and the question text information are input into a multimodal large model to obtain an abnormality detection result of the candidate abnormal image; the abnormality detection result is used to characterize whether the candidate abnormal image is an abnormal image.
[0009] In the disclosed embodiment, based on the predicted abnormality category information, question text information is generated for the candidate abnormal image and contains the predicted abnormality category information, and the question text information is used to indicate whether the candidate abnormal image contains content corresponding to the predicted abnormality category information. Compared with the solution of inputting user description information into the multimodal large model in the related art, the accuracy of the information description of the question text information can be improved. In addition, by inputting the question text information, the method can convert the conventional question and answer for the image into a matching method between the image and the text for solving it. In this way, the detection accuracy of the multimodal large model can be improved, thereby improving the detection accuracy of the candidate abnormal images.
[0010] Furthermore, for target images, the accuracy of image detection can be further improved through three stages of image detection (target detection, image classification, and anomaly detection of multimodal large models).
[0011] In an optional implementation, when the abnormality detection result indicates that the candidate abnormal image contains content corresponding to the predicted abnormality category information and the probability is greater than a preset probability, the candidate abnormal image is determined to be the abnormal image.
[0012] In the embodiment of the present disclosure, if the anomaly detection result indicates that the candidate abnormal image contains content corresponding to the predicted abnormal category information and the probability is greater than the preset probability, it means that the anomaly detection result is consistent with the predicted abnormal category information, and further digital calibration through probability can improve the accuracy of the anomaly detection result.
[0013] In an optional implementation manner, before obtaining the candidate abnormal image, the method further includes:
[0014] Acquire the target image;
[0015] Detect the target image based on the abnormal target detection model to obtain candidate abnormal regions and detected abnormality category information corresponding to the candidate abnormal regions;
[0016] The candidate abnormal region in the target image is cropped, and the cropped image is used as the candidate abnormal image.
[0017] In the disclosed embodiment, the target image is detected based on the abnormal target detection model to obtain candidate abnormal regions. As is well known, the target detection model will mark the candidate abnormal regions through a detection frame, which can improve the intuitiveness of the detection.
[0018] In an optional implementation, the generating, based on the predicted abnormality category information, question text information for the candidate abnormal image and containing the predicted abnormality category information comprises:
[0019] When the predicted abnormality type information is consistent with the detected abnormality type information of the candidate abnormality image, question text information for the candidate abnormality image and including the predicted abnormality type information is generated based on the predicted abnormality type information.
[0020] In the disclosed embodiment, if the predicted abnormal category information is consistent with the detected abnormal category information, it means that the probability of the candidate abnormal image being an abnormal image is relatively high, and subsequent detection of the candidate abnormal image is required. In this way, the candidate abnormal images can be indirectly screened to avoid invalid detection.
[0021] In an optional implementation, the abnormal target detection model, the abnormal classification model and the multimodal large model are trained based on the same training image sample set respectively.
[0022] In the disclosed embodiment, the above model is trained based on the same training image sample set, which can improve the reusability of the training image sample set.
[0023] In an optional implementation, the training image sample set is generated by the following method:
[0024] Acquire a first image and a second image; the first image refers to an image containing abnormal content, the first image carries a sample label annotated by a user, the sample label includes an abnormal area and an abnormal category of the abnormal area, and the second image refers to an image that does not contain abnormal content;
[0025] Cropping the abnormal area in the first image to obtain a sub-image, pasting the sub-image to a preset position of the second image, and using the preset position of the sub-image in the second image and the abnormal category corresponding to the sub-image as a sample label of the second image;
[0026] The training image sample set is generated based on the first image and the second image.
[0027] In the disclosed embodiment, the abnormal content in the first image is cropped and the cropped sub-image is pasted into the second image, without having to search for a large number of other images containing abnormal content as training image samples. Moreover, it usually takes a lot of manpower costs for users to label sample labels. The preset position of the sub-image in the second image and the abnormal category corresponding to the sub-image are used as sample labels for the second image, which is conducive to saving labor costs and improving labeling efficiency.
[0028] In an optional implementation, the multimodal large model is generated by the following method:
[0029] For each of the training image samples in the training image sample set, based on the position information of the abnormal region in the training image sample, perform region cropping on the abnormal region in the training image sample to obtain a sub-image sample corresponding to the abnormal region, and determine the abnormal category of the abnormal region as a sample label of the sub-image sample;
[0030] For each of the sub-image samples, based on the sample label of the sub-image sample, generating target question text information corresponding to the sub-image sample; the target question text information is used to indicate whether the sub-image sample contains content corresponding to the sample label;
[0031] Based on each of the sub-image samples and the corresponding target question text information, the model parameters of the initial multimodal large model are adjusted to obtain the multimodal large model.
[0032] As is well known, since the initial multimodal large model itself has the function of detecting and identifying image content, in the embodiment of the present disclosure, on this basis, based on the sample label of the sub-image sample, target question text information corresponding to the sub-image sample is generated, and the target question text information is used to indicate whether the sub-image sample contains content corresponding to the sample label, and then based on the sub-image sample and the corresponding target question text information, the model parameters of the initial multimodal large model are adjusted, so that the multimodal large model can learn the matching relationship between the sub-image sample and the target question text information, rather than just broadly identifying the image content. In this way, the detection accuracy of the multimodal large model can be improved.
[0033] The present disclosure also provides an image detection device, including:
[0034] An image acquisition module is used to acquire a candidate abnormal image; the candidate abnormal image is obtained by performing target detection on a target image by an abnormal target detection model;
[0035] A category prediction module, used to predict the abnormal category of the candidate abnormal image based on the abnormal classification model, and obtain the predicted abnormal category information of the candidate abnormal image;
[0036] A text generation module, configured to generate, based on the predicted abnormality category information, question text information for the candidate abnormal image and including the predicted abnormality category information; the question text information is used to indicate whether the candidate abnormal image contains content corresponding to the predicted abnormality category information;
[0037] The anomaly detection module is used to input the candidate abnormal image and the question text information into the multimodal large model to obtain an anomaly detection result of the candidate abnormal image; the anomaly detection result is used to characterize whether the candidate abnormal image is an abnormal image.
[0038] In an optional implementation, when the abnormality detection result indicates that the candidate abnormal image contains content corresponding to the predicted abnormality category information and the probability is greater than a preset probability, the candidate abnormal image is determined to be the abnormal image.
[0039] In an optional implementation, the image detection device further includes an image generation module, and the image generation module is used to:
[0040] Acquire the target image;
[0041] Detect the target image based on the abnormal target detection model to obtain candidate abnormal regions and detected abnormality category information corresponding to the candidate abnormal regions;
[0042] The candidate abnormal region in the target image is cropped, and the cropped image is used as the candidate abnormal image.
[0043] In an optional implementation manner, the text generation module is specifically used to:
[0044] When the predicted abnormality category information is consistent with the detected abnormality category information of the candidate abnormal region, question text information including the predicted abnormality category information is generated for the candidate abnormal image based on the predicted abnormality category information.
[0045] In an optional implementation, the abnormal target detection model, the abnormal classification model and the multimodal large model are trained based on the same training image sample set respectively.
[0046] In an optional implementation, the image detection device further includes a sample generation module, and the sample generation module is used to:
[0047] Acquire a first image and a second image; the first image refers to an image containing abnormal content, the first image carries a sample label annotated by a user, the sample label includes an abnormal area and an abnormal category of the abnormal area, and the second image refers to an image that does not contain abnormal content;
[0048] Cropping the abnormal area in the first image to obtain a sub-image, pasting the sub-image to a preset position of the second image, and using the preset position of the sub-image in the second image and the abnormal category corresponding to the sub-image as a sample label of the second image;
[0049] The training image sample set is generated based on the first image and the second image.
[0050] In an optional implementation, the image detection device further includes a model training module, and the model training module is used to:
[0051] For each of the training image samples in the training image sample set, based on the position information of the abnormal region in the training image sample, perform region cropping on the abnormal region in the training image sample to obtain a sub-image sample corresponding to the abnormal region, and determine the abnormal category of the abnormal region as a sample label of the sub-image sample;
[0052] For each of the sub-image samples, based on the sample label of the sub-image sample, generating target question text information corresponding to the sub-image sample; the target question text information is used to indicate whether the sub-image sample contains content corresponding to the sample label;
[0053] Based on each of the sub-image samples and the corresponding target question text information, the model parameters of the initial multimodal large model are adjusted to obtain the multimodal large model.
[0054] An embodiment of the present disclosure also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of any possible image detection method described above are performed.
[0055] The embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any possible image detection method described above are executed.
[0056] For a description of the effects of the above-mentioned image detection device, electronic device, and computer-readable storage medium, please refer to the description of the above-mentioned image detection method, which will not be repeated here.
[0057] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure.
[0058] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.
[0060] Figure 1 A flow chart of an image detection method provided by an embodiment of the present disclosure is shown;
[0061] Figure 2 A flow chart of a method for generating a training image sample set provided by an embodiment of the present disclosure is shown;
[0062] Figure 3 A schematic diagram of the structure of an image detection device provided by an embodiment of the present disclosure is shown;
[0063] Figure 4 A schematic diagram showing the structure of another image detection device provided by an embodiment of the present disclosure is shown;
[0064] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0065] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.
[0066] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0067] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.
[0068] Internet platforms provide users with convenient online interaction channels. In order to enrich the interaction methods and improve the efficiency of information sharing, most platforms support users to interact or share resources by uploading pictures.
[0069] Research has found that some images may contain abnormal content, such as uncivilized content. These abnormal contents have the risk of causing adverse guidance to other users. Therefore, the detection of image content has become particularly important.
[0070] Based on the above research, the present disclosure provides an image detection method, which first obtains a candidate abnormal image; the candidate abnormal image is obtained by performing target detection on a target image through an abnormal target detection model; secondly, based on an abnormal classification model, an abnormal category is predicted for the candidate abnormal image to obtain predicted abnormal category information of the candidate abnormal image; then, based on the predicted abnormal category information, question text information is generated for the candidate abnormal image and contains the predicted abnormal category information; the question text information is used to indicate whether the candidate abnormal image contains content corresponding to the predicted abnormal category information; finally, the candidate abnormal image and the question text information are input into a multimodal large model to obtain an abnormal detection result of the candidate abnormal image; the abnormal detection result is used to characterize whether the candidate abnormal image is an abnormal image.
[0071] In the disclosed embodiment, based on the predicted abnormality category information, question text information is generated for the candidate abnormal image and contains the predicted abnormality category information, and the question text information is used to indicate whether the candidate abnormal image contains content corresponding to the predicted abnormality category information. Compared with the solution in the related art in which the user inputs descriptive text information to the multimodal large model, the accuracy of the information description of the question text information can be improved. In addition, by inputting the question text information, the method can convert conventional questions and answers for the image into a matching method between the image and the text for solving it. In this way, the detection accuracy of the multimodal large model can be improved, thereby improving the detection accuracy of the candidate abnormal images.
[0072] Furthermore, for target images, the accuracy of image detection can be further improved through three stages of image detection (target detection, image classification, and anomaly detection of multimodal large models).
[0073] To facilitate understanding of this embodiment, an image detection method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the image detection method provided in the embodiment of the present disclosure is generally an electronic device, and the electronic device can be a server, which can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data and artificial intelligence platforms. In other embodiments, the electronic device can also be a terminal device, wherein the terminal device can be a mobile device, a user terminal, a terminal, a handheld device, a computing device, a vehicle-mounted device, a wearable device, and the like.
[0074] In other implementations, the method can also be applied to an implementation environment consisting of a terminal device and a server. In addition, the image detection method can also be implemented by a processor calling a computer-readable instruction stored in a memory.
[0075] An image detection method provided by an embodiment of the present disclosure is described below in conjunction with the accompanying drawings.
[0076] See also Figure 1 As shown, it is a flow chart of an image detection method provided by an embodiment of the present disclosure, such as Figure 1 As shown in , the method includes steps S101 to S104, wherein:
[0077] S101: Acquire candidate abnormal images; the candidate abnormal images are obtained by performing target detection on a target image using an abnormal target detection model.
[0078] Here, due to the influence of the detection accuracy of the abnormal target detection model, false detection may occur. Therefore, the candidate abnormal image may be an abnormal image or a non-abnormal image.
[0079] The number of candidate abnormal images may be one or more, which is not limited here.
[0080] The target image may be any type of image, for example, a landscape image, a portrait image, etc., which is not limited here.
[0081] Optionally, before obtaining the candidate abnormal image, the target image can be first obtained, and then the target image can be input into the abnormal target detection model to detect the target image based on the abnormal target detection model to obtain the candidate abnormal region and the detected abnormality category information corresponding to the candidate abnormal region. Finally, the candidate abnormal region in the target image is cropped, and the cropped image is used as the candidate abnormal image.
[0082] The number of the candidate abnormal regions may be 0 or at least one, which is not limited here.
[0083] Here, if the target image is an image that does not contain abnormal content (also called a normal image), then based on the abnormal target detection model, it may not detect the candidate abnormal region (that is, the number of candidate abnormal regions is 0). Of course, in some possible implementations, if the abnormal target detection model makes a false detection, it may also detect the candidate abnormal region for the target image that does not contain abnormal content.
[0084] In addition, if the target image includes at least one abnormal content, target detection is performed on the target image based on the abnormal target detection model to obtain candidate abnormal regions corresponding to each abnormal content, and at the same time, the detected abnormality category information corresponding to each candidate abnormal region is obtained. Moreover, each candidate abnormal region can be marked by a detection box in the target image, thereby improving the intuitiveness of the detection.
[0085] Exemplarily, for a target image A, whose image content includes smoking content, tattoo content, and a portrait, after performing target detection on the target image through an abnormal target detection model, candidate abnormal regions and detected abnormal category information corresponding to the candidate abnormal regions can be obtained. Here, the candidate abnormal regions may include candidate abnormal region 1, candidate abnormal region 2, and candidate abnormal region 3. Candidate abnormal region 1 corresponds to the smoking content in the target image A, and the detected abnormal category information of candidate abnormal region 1 is the smoking category; candidate abnormal region 2 corresponds to the tattoo content in the target image A, and the detected abnormal category information of candidate abnormal region 2 is the tattoo category; candidate abnormal region 3 corresponds to the portrait in the target image A, and the detected abnormal category information of candidate abnormal region 3 is the tattoo category. In this way, after obtaining candidate abnormal region 1, candidate abnormal region 2, and candidate abnormal region 3, candidate abnormal region 1, candidate abnormal region 2, and candidate abnormal region 3 can be cropped respectively to obtain candidate abnormal image 1 corresponding to candidate abnormal region 1, candidate abnormal image 2 corresponding to candidate abnormal region 2, and candidate abnormal image 3 corresponding to candidate abnormal region 3.
[0086] It should be noted that candidate abnormal region 3 is misdetected by the abnormal target detection model, and its actual abnormal category information is the normal category.
[0087] In this embodiment, the abnormal target detection model can be a yolov5 detection model. In other embodiments, it can also be a yolov8, yolov3, Region-CNN or other model, which is not limited here. The model training process of the abnormal target detection model will be described in detail later, and will not be repeated here.
[0088] S102: Based on the abnormal classification model, predict the abnormal category of the candidate abnormal image to obtain predicted abnormal category information of the candidate abnormal image.
[0089] Here, it can be understood as performing the second stage of anomaly detection on the target image. After the candidate abnormal image is detected, the candidate abnormal image can be input into the abnormal classification model to predict the abnormal category of the candidate abnormal image based on the abnormal classification model to obtain the predicted abnormal category information of the candidate abnormal image.
[0090] Exemplarily, the predicted abnormal category information may include smoking category, tattoo category, spitting category, uncivilized gesture category and normal category, etc., which are not limited here, wherein the normal category is used to indicate that the candidate abnormal image is a normal image, that is, the candidate abnormal image does not contain abnormal content.
[0091] Based on the above example, for candidate abnormal image 1, candidate abnormal image 2 and candidate abnormal image 3, abnormal category prediction is performed respectively, and it can be obtained that the predicted abnormal category information of candidate abnormal image 1 is the smoking category, the predicted abnormal category information of candidate abnormal image 2 is the tattoo category, and the predicted abnormal category information of candidate abnormal image 3 is the normal category.
[0092] Optionally, the predicted abnormal category information may also have a corresponding predicted probability, which is used to characterize the accuracy of the predicted abnormal category information. If the predicted probability is not less than a preset probability threshold, it means that the predicted abnormal category information is accurate. If the predicted probability is less than the preset probability threshold, it means that the predicted abnormal category information is wrong.
[0093] For example, if the preset probability threshold is 0.6, the predicted abnormal category information of the candidate abnormal image 3 is the normal category. If its predicted probability is 0.99, it means that the candidate abnormal image 3 is the normal category. If its predicted probability is 0.3, it means that the candidate abnormal image 3 is not the normal category.
[0094] In this implementation, the anomaly classification model may be a deep residual network (Resnet), or a Visual Transformer, MoblieNet or other model, which is not limited here.
[0095] S103: Based on the predicted abnormality category information, generate question text information for the candidate abnormal image and including the predicted abnormality category information; the question text information is used to indicate whether the candidate abnormal image contains content corresponding to the predicted abnormality category information.
[0096] Continuing with the above example, for candidate abnormal image 1, the predicted abnormal category information is the smoking category, and the generated question text information can be "Does the image contain a smoking area?" or "Does the image contain a smoking area?" and other similar expressions, which are not limited here.
[0097] Here, since the question text information is used to indicate whether the candidate abnormal image contains content corresponding to the predicted abnormal category information, that is, the question text information is targeted judgment text information rather than a broad problem description, in this way, the detection accuracy can be improved in the subsequent abnormality detection process.
[0098] In some embodiments, when generating question text information for the candidate abnormal image and including the predicted abnormal category information based on the predicted abnormal category information, it can be first determined whether the predicted abnormal category information is consistent with the detected abnormal category information. If they are consistent, question text information for the candidate abnormal image and including the predicted abnormal category information is generated based on the predicted abnormal category information. If they are inconsistent, the candidate abnormal image is eliminated.
[0099] It can be understood that if they are consistent, it means that the probability of the candidate abnormal image being an abnormal image is high and further detection is required. If they are inconsistent, it means that there is an error in the target detection stage and the candidate abnormal image is not an abnormal image. Therefore, the candidate abnormal image can be eliminated.
[0100] For example, for candidate abnormal image 1, its detected abnormal category information is the smoking category, and its predicted abnormal category information is also the smoking category, then candidate abnormal image 1 can be subjected to subsequent abnormality detection; for another example, for candidate abnormal image 3, its detected abnormal category information is the tattoo category, and its predicted abnormal category information is the normal category, then candidate abnormal image 3 is a normal image, therefore, candidate abnormal image 3 can be eliminated without the need to perform subsequent abnormality detection.
[0101] In this embodiment, by judging whether the predicted abnormal category information is consistent with the detected abnormal category information, the candidate abnormal images that are mistakenly detected by the abnormal target detection model can be eliminated. In this way, the number of candidate abnormal images for subsequent detection can be reduced, and invalid abnormality detection for normal images can be avoided, thereby improving the efficiency of abnormality detection.
[0102] As can be seen from the foregoing, the predicted abnormality category information also has a corresponding predicted probability. Therefore, in other embodiments, if the predicted abnormality category information of the candidate abnormal image is the same as the detected abnormality category information, and the predicted probability is less than the preset probability, the candidate abnormal image can also be eliminated.
[0103] For example, for candidate abnormal image 1, its detected abnormal category information is the smoking category, and its predicted abnormal category information is also the smoking category, but the predicted probability corresponding to the predicted abnormal category information is 0.2, then it can be said that candidate abnormal image 1 is not an abnormal image, and candidate abnormal image 1 can be eliminated without the need for subsequent abnormality detection.
[0104] S104: Inputting the candidate abnormal image and the question text information into a multimodal large model to obtain an abnormality detection result of the candidate abnormal image; the abnormality detection result is used to characterize whether the candidate abnormal image is an abnormal image.
[0105] A multimodal large model refers to a model that combines multimodal information such as text, images, videos, and audio for training.
[0106] It can be understood that since the question text information is used to indicate whether the candidate abnormal image contains content corresponding to the predicted abnormal category information, inputting the candidate abnormal image and the question text information into the multimodal large model can be understood as a problem of matching the image with the text, rather than the problem of using the multimodal large model for image recognition in the related art.
[0107] Exemplarily, the abnormality detection result of the candidate abnormal image may be “yes” or “includes”, etc., which is not limited here.
[0108] Here, the multimodal large model can be a blip2 model or an InstructBlip model, which is not limited here.
[0109] As we all know, the multimodal large model itself has the ability to detect (or recognize) the content of the image. For example, the conventional use of the multimodal large model is to detect the image and give a corresponding answer based on the text description entered by the user, and the text description entered by the user is usually a relatively broad description.
[0110] In the disclosed embodiment, based on the predicted abnormality category information, question text information is generated for the candidate abnormal image and contains the predicted abnormality category information, and the question text information is used to indicate whether the candidate abnormal image contains content corresponding to the predicted abnormality category information. Compared with the solution in the related art that the user inputs descriptive text information to the multimodal large model, the accuracy of the information description of the question text information can be improved. In addition, the accuracy of the answer given by the multimodal large model is low. The present method can convert the conventional question and answer for the image into a matching method between the image and the text by inputting the question text information. In this way, the detection accuracy of the multimodal large model can be improved, thereby improving the detection accuracy of the candidate abnormal images.
[0111] Furthermore, for target images, the accuracy of image detection can be further improved through three stages of image detection (target detection, image classification, and anomaly detection of multimodal large models).
[0112] In some implementations, when the abnormality detection result indicates that the candidate abnormal image contains content corresponding to the predicted abnormality category information and the probability is greater than a preset probability, the candidate abnormal image is determined to be the abnormal image.
[0113] Here, the multimodal large model can include probability prediction tasks, so that the anomaly detection result can also include probability. If the probability is not less than the preset probability, it means that the accuracy of the anomaly detection result is high. If the probability is less than the preset probability, it means that the accuracy of the anomaly detection result is low.
[0114] Exemplarily, if the preset probability is 0.5, the abnormality detection result of the candidate abnormal image includes "yes, 0.87", indicating that the candidate abnormal image is an abnormal image, and if the abnormality detection result of the candidate abnormal image includes "yes, 0.3", indicating that the candidate abnormal image is not an abnormal image.
[0115] In other examples, if the abnormality detection result of the candidate abnormal image can also include "yes, 0.87; no 0.13", it can also be said that the candidate abnormal image is an abnormal image; if the abnormality detection result of the candidate abnormal image includes "yes, 0.44; no 0.56", it can also be said that the candidate abnormal image is not an abnormal image.
[0116] The following is a detailed introduction to the training process of the abnormal target detection model, the abnormal classification model, and the multimodal large model.
[0117] In the disclosed embodiment, the abnormal target detection model, the abnormal classification model and the multimodal large model are respectively trained based on the same training image sample set, so that the reusability of the training image sample set can be improved.
[0118] Combine the following Figure 2 , the generation process of the training image sample set is introduced in detail, Figure 2 A flowchart of a method for generating a training image sample set provided by an embodiment of the present disclosure. Figure 2 As shown, the method includes S201-S203:
[0119] S201: Acquire a first image and a second image; the first image refers to an image containing abnormal content, the first image carries sample labels annotated by a user, the sample labels include abnormal areas and abnormal categories of the abnormal areas, and the second image refers to an image that does not contain abnormal content.
[0120] Here, the first image and the second image may be any type of image, for example, a landscape image, a scene image, a portrait image, an animal portrait image, etc., and are not limited here.
[0121] Among them, abnormal content may refer to content that does not meet preset requirements. For example, abnormal content may be content that contains uncivilized behavior (such as uncivilized gestures, smoking, etc.).
[0122] In this embodiment, for the first image, the first image can be annotated in advance by manual annotation, so that the first image can carry the sample labels annotated by the user, wherein the sample labels include abnormal areas and abnormal categories of the abnormal areas. It can be understood that since the first image contains abnormal content, the abnormal area is the area containing abnormal content, and the categories of the abnormal areas may include smoking categories, uncivilized gesture categories, etc., which are not limited here.
[0123] It should be noted that the number of abnormal regions in the first image may be one or more, which is not limited here.
[0124] S202: Crop the abnormal area in the first image to obtain a sub-image, paste the sub-image to a preset position of the second image, and use the preset position of the sub-image in the second image and the abnormal category corresponding to the sub-image as a sample label of the second image.
[0125] The preset position may be set according to actual needs, for example, it may be a random position, or the center position of the second image, etc., which is not limited here.
[0126] It can be understood that after the sub-image is pasted into the second image, the second image includes the abnormal area. At this time, the preset position of the sub-image in the second image and the abnormal category corresponding to the sub-image can be used as the sample label of the second image.
[0127] S203: Generate the training image sample set based on the first image and the second image.
[0128] It should be understood that after obtaining the second image containing the abnormal area, a training image sample set can be generated based on the first image and the second image, that is, the first image and the second image are used as training image samples.
[0129] In this embodiment, a sub-image is obtained by cropping the abnormal area in the first image, and the sub-image is pasted to a preset position of the second image. In this way, training image samples can be generated without user annotation, which is conducive to reducing the manpower cost required for image annotation and improving the efficiency of image annotation.
[0130] The following is a detailed description of the training processes of the abnormal target detection model, the abnormal classification model, and the multimodal large model.
[0131] Specifically, the abnormal target detection model can be trained in the following ways:
[0132] Based on the training image sample set, the initial abnormal target detection model is supervised trained to obtain the abnormal target detection model.
[0133] Here, it can be understood that the training image samples in the training image sample set can be input into the initial abnormal target detection model to obtain the target detection results, and based on the target detection results and the sample labels of the training image samples, the initial abnormal target detection model can be supervised trained until the training results meet the preset training requirements to obtain the abnormal target detection model.
[0134] As can be seen from the foregoing, the sample label includes the location information of the abnormal area and the abnormal category of the abnormal area. Therefore, the trained abnormal target detection model is able to detect the training image samples, output the abnormal area and output the detected abnormal category information.
[0135] In some implementations, the abnormal classification model can be trained in the following manner:
[0136] (A) For each training image sample in the training image sample set, crop the abnormal region according to the position information of the abnormal region in the training image sample to obtain a sub-image sample corresponding to the abnormal region, and determine the abnormal category of the abnormal region as the sample label of the sub-image sample.
[0137] As can be seen from the foregoing, the position information of the abnormal area in the training image sample is known (that is, it can be the position information of the target detection, or it can be the preset position of pasting). Therefore, the abnormal area can be cropped according to the position information to obtain a sub-image sample. At the same time, the abnormal category of the abnormal area can be determined as the sample label of the sub-image sample.
[0138] Exemplarily, for a training image sample M, the abnormal areas it contains include M1, M2 and M3, wherein the abnormal category of M1 is the smoking category, the abnormal category of M2 is the tattoo category, and the abnormal category of M3 is the tattoo category. Then, M1, M2 and M3 can be cropped respectively to obtain a sub-image sample N1 corresponding to M1, and the sample label of sub-image sample N1 is the smoking category, a sub-image sample N2 corresponding to M2, and the sample label of sub-image sample N2 is the tattoo category, and a sub-image sample N3 corresponding to M3, and the sample label of sub-image sample N3 is the tattoo category.
[0139] (B) Based on each of the sub-image samples and the corresponding sample labels, the initial anomaly classification model is trained to obtain the anomaly classification model.
[0140] In this way, after determining the sample label of each sub-image sample, the initial anomaly classification model can be supervised trained based on each sub-image sample and the corresponding sample label to obtain the anomaly classification model. In this way, the anomaly classification model can have the ability to classify images and determine the anomaly category of the image.
[0141] In the disclosed embodiment, a multimodal large model can be obtained by training in the following manner:
[0142] (1) For each of the sub-image samples, based on the sample label of the sub-image sample, generating target question text information corresponding to the sub-image sample; the target question text information is used to indicate whether the sub-image sample contains content corresponding to the sample label.
[0143] As can be seen from the foregoing, the sample label of a sub-image sample refers to the abnormal category of the sub-image sample. Therefore, the target question text information here can be used to indicate whether the sub-image sample contains content corresponding to the abnormal category.
[0144] Exemplarily, the sample label of sub-image sample 1 is the smoking category, and the corresponding target question text information may be “whether there is smoking content in sub-image sample 1”, or “whether sub-image sample 1 contains smoking content”.
[0145] (2) Based on each of the sub-image samples and the corresponding target question text information, the model parameters of the initial multimodal large model are adjusted to obtain the multimodal large model.
[0146] Here, it should be understood that since the initial multimodal large model usually has the ability to ask questions and answer visual images, for example, the user can input an image into the initial multimodal large model and ask it questions such as "Is this image an abnormal image" or "What abnormal content does this image contain?" The initial multimodal large model will give corresponding answers. However, since the question information input by the user is usually relatively broad, the initial multimodal large model cannot give answers that meet the user's requirements.
[0147] Based on the above research, in the embodiment of the present disclosure, the model parameters of the initial multimodal large model are adjusted based on each sub-image sample and the corresponding target question text information, so that the multimodal large model can align the sub-image samples with the target question text information (sample labels) to obtain the multimodal large model.
[0148] Specifically, the initial multimodal large model usually includes an image feature extraction model and a large language model. Since the large language model often requires a huge amount of data and computing power, when fine-tuning the initial multimodal large model, the three tasks in the image feature extraction model can be fine-tuned based on each sub-image sample and the corresponding target question text information: contrastive learning (ITC), image-text matching learning (ITM), and image-text generation learning (ITG), so that the multimodal large model can learn the relationship between the sub-image samples and the target question text information.
[0149] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.
[0150] Based on the same inventive concept, an image detection device corresponding to the above-mentioned image detection method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned image detection method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0151] See also Figure 3 , Figure 3 Schematic diagram of an image detection device provided by an embodiment of the present disclosure. Figure 3 As shown in , the image detection device 300 provided in the embodiment of the present disclosure includes:
[0152] An image acquisition module 310 is used to acquire a candidate abnormal image; the candidate abnormal image is obtained by performing target detection on a target image using an abnormal target detection model;
[0153] A category prediction module 320 is used to predict the abnormal category of the candidate abnormal image based on the abnormal classification model to obtain the predicted abnormal category information of the candidate abnormal image;
[0154] A text generation module 330 is used to generate question text information for the candidate abnormal image and including the predicted abnormal category information based on the predicted abnormal category information; the question text information is used to indicate whether the candidate abnormal image contains content corresponding to the predicted abnormal category information;
[0155] The anomaly detection module 340 is used to input the candidate abnormal image and the question text information into the multimodal large model to obtain an anomaly detection result of the candidate abnormal image; the anomaly detection result is used to characterize whether the candidate abnormal image is an abnormal image.
[0156] In an optional implementation, when the abnormality detection result indicates that the candidate abnormal image contains content corresponding to the predicted abnormality category information and the probability is greater than a preset probability, the candidate abnormal image is determined to be the abnormal image.
[0157] See also Figure 4 , Figure 4 FIG. 1 is a schematic diagram of another image detection device provided by an embodiment of the present disclosure. Figure 4 As shown in , the image detection device 300 provided in the embodiment of the present disclosure further includes an image generation module 350, and the image generation module 350 is used to:
[0158] Acquire the target image;
[0159] Detect the target image based on the abnormal target detection model to obtain candidate abnormal regions and detected abnormality category information corresponding to the candidate abnormal regions;
[0160] The candidate abnormal region in the target image is cropped, and the cropped image is used as the candidate abnormal image.
[0161] In an optional implementation manner, the text generation module 330 is specifically used for:
[0162] When the predicted abnormality type information is consistent with the detected abnormality type information of the candidate abnormality image, question text information for the candidate abnormality image and including the predicted abnormality type information is generated based on the predicted abnormality type information.
[0163] In an optional implementation, the abnormal target detection model, the abnormal classification model and the multimodal large model are trained based on the same training image sample set respectively.
[0164] In an optional implementation, the image detection device 300 further includes a sample generation module 360, and the sample generation module 360 is used to:
[0165] Acquire a first image and a second image; the first image refers to an image containing abnormal content, the first image carries a sample label annotated by a user, the sample label includes an abnormal area and an abnormal category of the abnormal area, and the second image refers to an image that does not contain abnormal content;
[0166] Cropping the abnormal area in the first image to obtain a sub-image, pasting the sub-image to a preset position of the second image, and using the preset position of the sub-image in the second image and the abnormal category corresponding to the sub-image as a sample label of the second image;
[0167] The training image sample set is generated based on the first image and the second image.
[0168] In an optional implementation, the image detection device 300 further includes a model training module 370, and the model training module 370 is used to:
[0169] For each of the training image samples in the training image sample set, based on the position information of the abnormal region in the training image sample, perform region cropping on the abnormal region in the training image sample to obtain a sub-image sample corresponding to the abnormal region, and determine the abnormal category of the abnormal region as a sample label of the sub-image sample;
[0170] For each of the sub-image samples, based on the sample label of the sub-image sample, generating target question text information corresponding to the sub-image sample; the target question text information is used to indicate whether the sub-image sample contains content corresponding to the sample label;
[0171] Based on each of the sub-image samples and the corresponding target question text information, the model parameters of the initial multimodal large model are adjusted to obtain the multimodal large model.
[0172] For descriptions of the processing flow of each module in the above-mentioned device and the interaction flow between each module, reference can be made to the relevant descriptions in the above-mentioned method embodiment, which will not be described in detail here.
[0173] Corresponding to the above-mentioned image detection method, the embodiment of the present disclosure further provides an electronic device 500, such as Figure 5 FIG. 1 is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present disclosure, including:
[0174] Processor 510, memory 520, and bus 530. The memory 520 is used to store execution instructions, including internal memory 521 and external memory 522; the internal memory 521 is also called internal memory, which is used to temporarily store the operation data in the processor 510 and the data exchanged with the external memory 522 such as a hard disk. The processor 510 exchanges data with the external memory 522 through the internal memory 521.
[0175] In the embodiment of the present application, the memory 520 is specifically used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 510. That is, when the electronic device 500 is running, the processor 510 communicates with the memory 520 through the bus 530, so that the processor 510 executes the application code stored in the memory 520, and then executes the image detection method described in the above method embodiment.
[0176] Among them, the memory 520 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.
[0177] The processor 510 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0178] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 500. In other embodiments of the present application, the electronic device 500 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0179] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, any one of the image detection methods described in the above method embodiments is executed. The storage medium may be a volatile or non-volatile computer-readable storage medium.
[0180] The method in the embodiment of the present disclosure can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in this application is executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, a core network device, an OAM or other programmable device.
[0181] The computer program or instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.
[0182] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. An image detection method, It is characterized in that include: Get the target image; Detect the target image based on an abnormal target detection model to obtain candidate abnormal regions and detected abnormality category information corresponding to the candidate abnormal regions; Cropping the candidate abnormal region in the target image, and using the cropped image as a candidate abnormal image; Based on the abnormal classification model, predict the abnormal category of the candidate abnormal image to obtain the predicted abnormal category information of the candidate abnormal image; Based on the predicted abnormality category information, generating question text information for the candidate abnormal image and including the predicted abnormality category information; the question text information is used to indicate whether the candidate abnormal image contains content corresponding to the predicted abnormality category information; The candidate abnormal image and the question text information are input into a multimodal large model to obtain an abnormality detection result of the candidate abnormal image; the abnormality detection result is used to characterize whether the candidate abnormal image is an abnormal image; the abnormal image refers to an image containing uncivilized content.
2. The method according to claim 1, It is characterized in that When the abnormality detection result indicates that the candidate abnormality image contains content corresponding to the predicted abnormality category information and the probability is greater than a preset probability, the candidate abnormality image is determined to be the abnormal image.
3. The method according to claim 2, It is characterized in that The step of generating, based on the predicted abnormality category information, question text information for the candidate abnormal image and including the predicted abnormality category information comprises: When the predicted abnormality type information is consistent with the detected abnormality type information of the candidate abnormality image, question text information for the candidate abnormality image and including the predicted abnormality type information is generated based on the predicted abnormality type information.
4. The method according to any one of claims 1 to 3, It is characterized in that The abnormal target detection model, the abnormal classification model and the multimodal large model are trained based on the same training image sample set respectively.
5. The method according to claim 4, It is characterized in that The training image sample set is generated by the following method: Acquire a first image and a second image; the first image refers to an image containing abnormal content, the first image carries a sample label annotated by a user, the sample label includes an abnormal area and an abnormal category of the abnormal area, and the second image refers to an image that does not contain abnormal content; Cropping the abnormal area in the first image to obtain a sub-image, pasting the sub-image to a preset position of the second image, and using the preset position of the sub-image in the second image and the abnormal category corresponding to the sub-image as a sample label of the second image; The training image sample set is generated based on the first image and the second image.
6. The method according to claim 5, It is characterized in that The multimodal large model is generated by the following method: For each of the training image samples in the training image sample set, based on the position information of the abnormal region in the training image sample, perform region cropping on the abnormal region in the training image sample to obtain a sub-image sample corresponding to the abnormal region, and determine the abnormal category of the abnormal region as a sample label of the sub-image sample; For each of the sub-image samples, based on the sample label of the sub-image sample, generating target question text information corresponding to the sub-image sample; the target question text information is used to indicate whether the sub-image sample contains content corresponding to the sample label; Based on each of the sub-image samples and the corresponding target question text information, the model parameters of the initial multimodal large model are adjusted to obtain the multimodal large model.
7. An image detection device, It is characterized in that include: An image generation module, used to obtain a target image; Detect the target image based on an abnormal target detection model to obtain candidate abnormal regions and detected abnormality category information corresponding to the candidate abnormal regions; Cropping the candidate abnormal region in the target image, and using the cropped image as a candidate abnormal image; A category prediction module, used to predict the abnormal category of the candidate abnormal image based on the abnormal classification model, and obtain the predicted abnormal category information of the candidate abnormal image; A text generation module, configured to generate, based on the predicted abnormality category information, question text information for the candidate abnormal image and including the predicted abnormality category information; the question text information is used to indicate whether the candidate abnormal image contains content corresponding to the predicted abnormality category information; The anomaly detection module is used to input the candidate abnormal image and the question text information into the multimodal large model to obtain the anomaly detection result of the candidate abnormal image; the anomaly detection result is used to characterize whether the candidate abnormal image is an abnormal image; the abnormal image refers to an image containing uncivilized content.
8. An electronic device, It is characterized in that It includes a processor, a memory and a bus, the memory stores machine-readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the image detection method according to any one of claims 1 to 6 is performed.
9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image detection method according to any one of claims 1 to 6 is executed.
