Image category judgment method and device, equipment and storage medium

Abstract

The invention discloses an image category judgment method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring an input image; obtaining a first image category of the input image according to the input image through a first neural network model, the first neural network model being used for classifying whether the input image is an image generated by artificial intelligence AI; under the condition that the confidence coefficient of the first neural network model to the first image category is smaller than a first set threshold value, a second image category of the input image is determined through a second neural network model according to the input image and a text question, and the text question is used for asking whether the input image is AI generation or not; the second neural network model is used for classifying the input image again in combination with the text problem; and determining a determination result of the input image according to the second image category, the determination result being a result of determining whether the input image is an AI generation result. According to the method, the image category judgment efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a method, device, equipment and storage medium for determining an image category. Background Art

[0002] With the development of information technology, the method of generating images through AI has become increasingly popular, such as automatically generating images through AIGC (AI Generated Content, artificial intelligence content automatic generation technology).

[0003] In the related art, when determining whether an image is generated by AI, it is generally determined manually by humans.

[0004] However, in the above-mentioned related technologies, it is necessary to manually determine whether the image is generated by AI, and the efficiency of determining the image category is low. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus, device and storage medium for determining the image category, which can improve the efficiency of determining the image category. The technical solutions proposed in the present application are as follows:

[0006] According to one aspect of an embodiment of the present application, a method for determining an image category is provided, the method comprising:

[0007] Get the input image;

[0008] Obtaining a first image category of the input image according to the input image through a first neural network model, wherein the first neural network model is used to classify whether the input image is an AI-generated image;

[0009] In a case where the confidence of the first neural network model for the first image category is less than a first set threshold, determining a second image category of the input image by a second neural network model according to the input image and a text question, wherein the text question is used to ask whether the input image is generated by AI, and the second neural network model is used to classify the input image again in combination with the text question;

[0010] According to the second image category, a determination result of the input image is determined, where the determination result is a result of determining whether the input image is generated by AI.

[0011] According to one aspect of an embodiment of the present application, a device for determining an image category is provided, the device comprising:

[0012] An acquisition module, used for acquiring an input image;

[0013] A classification module, configured to obtain a first image category of the input image according to the input image through a first neural network model, wherein the first neural network model is configured to classify whether the input image is an image generated by artificial intelligence (AI);

[0014] The classification module is further configured to determine, by using a second neural network model, a second image category of the input image according to the input image and a text question when the confidence of the first neural network model for the first image category is less than a first set threshold, wherein the text question is used to ask whether the input image is generated by AI, and the second neural network model is used to classify the input image again in combination with the text question;

[0015] A determination module is used to determine a determination result of the input image according to the second image category, wherein the determination result is a result of determining whether the input image is generated by AI.

[0016] According to one aspect of an embodiment of the present application, a computer device is provided, the computer device comprising a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-mentioned method for determining the image category.

[0017] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned method for determining the image category.

[0018] According to one aspect of an embodiment of the present application, a computer program product is provided. The computer program product includes a computer program. The computer program is loaded and executed by a processor to implement the above-mentioned method for determining the image category.

[0019] The technical solution provided in the embodiments of the present application can bring the following beneficial effects:

[0020] By using the first neural network model to determine whether the input image is AI generated, the efficiency of image category determination is improved compared to the manual determination method in the related art. Furthermore, when the confidence of the first image category determined by the first neural network model is less than the first set threshold, the second neural network model is continued to be used in combination with the text question to classify the input image again to obtain the final input image determination result. In this way, when the confidence of the first neural network model is low, the second neural network model is used to continue the determination, which is conducive to improving the accuracy of image category determination. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1is a schematic diagram of a computer system provided by an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of a method for determining an image category provided by an embodiment of the present application;

[0023] Figure 3 is a flow chart of a method for determining an image category provided by an embodiment of the present application;

[0024] Figure 4 is a flow chart of a method for determining an image category provided by another embodiment of the present application;

[0025] Figure 5 is a flow chart of a method for determining an image category provided by another embodiment of the present application;

[0026] Figure 6 is a block diagram of an image category determination device provided by an embodiment of the present application;

[0027] Figure 7 It is a structural block diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0029] Please refer to Figure 1 , which shows a schematic diagram of a computer system provided by an embodiment of the present application. The computer system can be implemented as an image category determination system. The computer system may include: a terminal device 10 and a server 20.

[0030] The number of terminal devices 10 can be one or more. The terminal device 10 can be an electronic device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a game console, an e-book reader, a multimedia player, a wearable device, a smart TV, a car terminal, etc. The client of the target application can be installed in the terminal device 10. Optionally, the target application can be an application that needs to be downloaded and installed, or it can be a point-and-click application, which is not limited in the embodiment of the present application. In the embodiment of the present application, the above-mentioned target application is an application for determining the image category. Of course, there is no limitation on the specific type of the target application, and the type of the target application can be an image application, a music application, a video application, an animation application, a social application, a question-and-answer application, a virtual reality application, an augmented reality application, etc. The embodiment of the present application does not limit the specific category of the target application. In other embodiments, the target application can be considered as a separate functional module, such as being implemented as one of the functional modules in the music application for realizing image category determination. Exemplarily, the client of the above-mentioned target application is running in the terminal device 10.

[0031] The server 20 is used to provide background services for the client of the target application installed and running in the terminal device 10. For example, the server 20 can be the background server of the above-mentioned target application. The server 20 can be a single server, or a server cluster composed of multiple servers, or a cloud computing service center. Optionally, the server 20 provides background services for the clients of the target application in multiple terminal devices 10 at the same time. The terminal device 10 and the server 20 can communicate with each other through the network.

[0032] When the above-mentioned target application is a music application, the music application obtains an input image; a first image category of the input image is obtained according to the input image through a first neural network model, and the first neural network model is used to classify whether the input image is an image generated by AI; when the confidence of the first neural network model for the first image category is less than a first set threshold, a second image category of the input image is determined according to the input image and a text question through a second neural network model, and the text question is used to ask whether the input image is generated by AI, and the second neural network model is used to classify the input image again in combination with the text question; according to the second image category, a judgment result of the input image is determined, and the judgment result is a result of determining whether the input image is generated by AI.

[0033] The method provided in the embodiment of the present application may be performed by a computer device. The computer device may be any electronic device that has the ability to store and process data. For example, the computer device may be Figure 1The terminal device 10 may also be a server 20.

[0034] In the related art, there are two ways to determine whether it is AI generated: 1. Through manual judgment; 2. Through simple picture (picture in this application is also called image) classification.

[0035] Then, the technical solutions in the related art have the following disadvantages: 1. Manual judgment: cannot be automated, has low precision and recall rate, and high cost; 2. Image 2 classification: low precision and low recall rate.

[0036] The technical solution provided by the embodiment of the present application solves the above-mentioned problem and proposes an AIGC-generated image determination system, which implements a serial determination method through multiple models, and improves precision and recall while being automated. Specifically, the first neural network model is used to determine whether the input image is AI-generated, which improves the efficiency of image category determination compared to the manual determination method in the related art. Furthermore, when the confidence of the first neural network model for the first image category determined by the first neural network model is less than the first set threshold, the second neural network model is continued to be used in combination with the text question to classify the input image again to obtain the final determination result of the input image. In this way, when the confidence of the first neural network model is low, the second neural network model is used to continue the determination, which is conducive to improving the accuracy of image category determination.

[0037] like Figure 2 As shown, the acquired input image 200 is input into the AIGC image classification model 210 (i.e., the first neural network model) to obtain the classification result (i.e., the first image category) and the confidence for the input image 200. In some embodiments, the AIGC image classification model 210 is trained jointly using normal images (i.e., non-AI generated images) and AI generated images.

[0038] In some embodiments, it is determined whether the confidence of the AIGC image classification model 210 for the classification result is greater than a threshold (i.e., a first set threshold). If the confidence of the AIGC image classification model 210 for the output classification result is greater than or equal to the threshold, the classification result output by the AIGC image classification model 210 is used as the determination result of the input image. If the confidence of the AIGC image classification model 210 for the output classification result is less than the threshold, AI dark watermark detection 220 is performed.

[0039] In some embodiments, the AI ​​dark watermark detection 220 is also detected by a neural network model, and the neural network model used to detect the dark watermark is also a model trained with AI dark watermark data. Exemplarily, if the confidence of the dark watermark detection result is less than or equal to the threshold, the multimodal model determination 230 is continued. Determine whether the confidence of the multimodal model (that is, the second neural network model) for the output classification result (that is, the second image category) is greater than the threshold. If the confidence of the multimodal model for the output classification result is greater than or equal to the threshold, the classification result output by the multimodal model is used as the determination result of the input image. If the confidence of the multimodal model for the output classification result is less than the threshold, the image inversion tags (that is, image labels) 240 is performed to obtain tags.

[0040] In some embodiments, a similarity determination 250 is performed between an image (i.e., an input image) and tags. It is determined whether the confidence of the similarity determination result is greater than a threshold. If the confidence of the similarity determination result is greater than or equal to the threshold, the determination result of the input image is determined based on the similarity determination result. If the confidence of the similarity determination result is less than the threshold, it is determined whether to perform manual determination. If manual determination is performed, manual determination 260 is performed, and the output result of the manual determination is used as the determination result of the input image. If no manual determination is performed, the determination result of the input image is directly determined to be a non-AIGC image.

[0041] See the following embodiments for details.

[0042] Please refer to Figure 3 , which shows a flow chart of a method for determining an image category provided by an embodiment of the present application. The execution subject of each step of the method may be the computer device introduced above. The method may include at least one of the following steps (310-340).

[0043] Step 310, obtaining an input image.

[0044] The embodiments of the present application do not limit the source of the input image, nor do they limit the content of the input image. The input image can be an image taken by yourself or downloaded from the Internet. For example, the input image can be an image sent by a friend account in a social application, an image in the comments under a song or video in a music application, an image posted by a user in a short video application, and so on. It should be noted that the collection of relevant data (including input images, etc.) in this application complies with legal requirements and is collected after the user's permission, and does not involve security and privacy issues.

[0045] Step 320, obtaining a first image category of the input image based on the input image through a first neural network model, and the first neural network model is used to classify whether the input image is an AI-generated image.

[0046] In some embodiments, the first neural network model is used to classify whether the input image is an AI-generated image. In an embodiment of the present application, the first neural network model is also referred to as an AIGC image classification model. In some embodiments, the first neural network model includes at least one of a feature extraction layer, a feature fusion layer, and a classifier. Exemplarily, the feature extraction layer is used to extract features of the input image, the feature fusion layer is used to fuse the extracted features to obtain fused features, and the classifier is used to classify the fused features to obtain a classification result. Exemplarily, the classification result is also the first image category.

[0047] In some embodiments, the first image category is used to indicate whether the input image is AI generated. In some embodiments, the first neural network model is a multi-classification classification model, and the first image category is one of the following: non-AI generated images (also referred to as normal images, real images, etc.), first type of AI generated images, second type of AI generated images, and the first type is different from the second type. That is, in an embodiment of the present application, if there are at least two image categories for AI-generated images. When there are two image categories for AI-generated images, they are the first type of AI-generated images and the second type of AI-generated images. When there are three image categories for AI-generated images, they are the first type of AI-generated images, the second type of AI-generated images, and the third type of AI-generated images. Among them, the first type, the second type, and the third type are different. And so on.

[0048] Compared with the simple two-classification in the related art (AI-generated images, not AI-generated images (or normal images)), this application increases the number of categories for AI-generated images. Exemplarily, the image categories of AI-generated images include: two-dimensional AI image category, realistic AI image category, landscape AI image category, and character AI image category. This application improves the classification accuracy of AI-generated images and refines the image categories for AI-generated images through a multi-classification approach.

[0049] In some embodiments, the first neural network model is a pre-trained model, or a pre-trained model obtained after fine-tuning. In some embodiments, the first neural network model is taken as a four-classification model. In some embodiments, normal images and AIGC images are collected by external image crawlers to train a 4-classification model. In order to improve the model capabilities, they can be divided into: two-dimensional AI image category, realistic AI image category, landscape AI image category and normal image category. Images identified by the model (with high confidence) can also be used as training data to form automated training.

[0050] In some embodiments, the following parts are required in the process of training the first neural network model.

[0051] Part 1: Data preparation.

[0052] 1. Sample collection: It is necessary to build a diverse dataset, including a large number of real images and AI-generated images. Real photos can be obtained from public datasets, and AI-generated images can be obtained from different AIGC generation models (such as DALL-E, StableDiffusion, Midjourney, etc.). 2. Annotated data: The type of images annotated manually or automatically to ensure that the training set contains high-quality label information.

[0053] The second part is feature extraction and model selection.

[0054] 1. Visual feature analysis: AIGC images usually have specific visual features, such as certain textures, pixel distribution, edge processing methods, etc., which are subtly different from real photos. Some traditional image features can be extracted, such as edge features, texture features, color histograms, etc. 2. Deep learning models: Convolutional neural networks or more complex models (such as VisionTransformer) can be used to capture the features of AIGC images. Fine-tuning pre-trained models: Models pre-trained on large datasets such as Imagenet can be used, and then fine-tuned on the AIGC dataset to identify real and AI-generated images.

[0055] The third part is model training and classification.

[0056] 1. Supervised learning: Use a four-category model (such as ResNet, EfficientNet) to train the model on a labeled dataset, and the model outputs "real" or "AI-generated" categories. 2. Feature fusion and classification: Through the fusion of multiple features, the classification performance is further optimized. Traditional image features and deep learning features can be combined to improve the accuracy of classification.

[0057] Part 4, model optimization and verification.

[0058] 1. Cross-validation: Cross-validate the model to ensure the classification accuracy on the training set and test set. 2. Confusion matrix and precision evaluation: Evaluate the accuracy, recall rate and F1 score of the model to ensure that the classifier has high judgment accuracy for both real images and AIGC images.

[0059] In some embodiments, when using the model to predict the four-classification model of the image, a confidence level and a category (one of the four) will be output. The greater the confidence level, the more accurate the prediction. The image and category are mapped and filled into the model training set for training. The training data structure is an image set and a text file (in the format of key and value, image name: category).

[0060] For details, please refer to the explanation of the following embodiments, which will not be described in detail here.

[0061] In some embodiments, after step 320, it also includes: when the confidence of the first neural network model for the first image category is greater than or equal to a first set threshold, determining the first image category as the judgment result of the input image.

[0062] In some embodiments, when the confidence of the first neural network model for the first image category is greater than or equal to the first set threshold, that is, when the first neural network has a greater prediction confidence in the judgment result of the image category of the input image, the first image category is directly determined as the judgment result of the input image. For example, the judgment result is at least one of the following: two-dimensional AI image category, realistic AI image category, landscape AI image category, and normal image category. This method can not only ensure the accuracy of image judgment, but also reduce the cost of image judgment.

[0063] Step 330, when the confidence of the first neural network model for the first image category is less than the first set threshold, the second image category of the input image is determined by the second neural network model based on the input image and the text question, the text question is used to ask whether the input image is AI generated, and the second neural network model is used to classify the input image again in combination with the text question.

[0064] In some embodiments, the confidence of the first neural network model for the first image category can also be output by the first neural network model when outputting the first image category. If the confidence of the first neural network model for the first image category is less than the first set threshold, it means that the first neural network model is not confident enough in predicting the first image category, and further determination is required to improve the accuracy of image classification.

[0065] In some embodiments, the second neural network model is used to reclassify the input image in combination with the text question. In an embodiment of the present application, the second neural network model is also referred to as a multimodal model. In some embodiments, the second neural network model includes at least one of a feature encoding layer, a feature extraction layer, a feature fusion layer, and a classifier. Exemplarily, the feature encoding layer is used to encode the text question to obtain text features. The feature extraction layer is used to extract features of the input image to obtain image features. The feature fusion layer is used to fuse the extracted text features and image features to obtain fused features, and the classifier is used to classify the fused features to obtain a classification result. Exemplarily, the classification result is also the second image category. In some embodiments, in some embodiments, the second neural network model is a pre-trained model, or a pre-trained model obtained after fine-tuning.

[0066] In some embodiments, the text question is used to ask whether the input image is generated by AI (i.e., the text query described below). Exemplarily, the text question is automatically generated by the neural network model, or it can be manually set in advance.

[0067] In some embodiments, step 330 is also called multimodal model judgment: while inputting the image, a text query (whether it is AI generated) is added, and the text (yes or no) is output, so as to further improve the recall capability through the multimodal large model.

[0068] In some embodiments, the input of the multimodal model is: 1. Image: Provide the image to be detected, and the model will extract the visual features therein. 2. Text query: Enter a specific text question (for example, "Is this image generated by AI?") to guide the model to pay more attention to the features generated by AI when making judgments. In some embodiments, the output of the multimodal model: 1. Yes / No judgment: The model will output a binary classification result, that is, "yes" or "no", which is used to directly determine whether the image is generated by AI. 2. Confidence score: The confidence of the judgment is also provided (usually a probability value between 0 and 1), indicating the confidence of the model in the judgment. This confidence can help filter out results with lower confidence levels to improve the overall reliability of judgments.

[0069] In some embodiments, the use of a multimodal model can further enhance the model's judgment ability, especially through the joint analysis of image and text information. The multimodal model can more accurately identify potential AIGC features and avoid misjudgments that may occur under a single modality. This architecture is usually suitable for pre-trained models that integrate vision and language (such as CLIP (Contrastive Language-Image Pre-Training), BLIP (Bidirectional Language Model Pre-training), etc.), which can process image and text inputs in the same framework and return binary classification results and their confidence that meet application requirements.

[0070] In some embodiments, when the confidence of the first neural network model for the first image category is less than a first set threshold, dark watermark detection can also be performed, and whether to perform multimodal model judgment can be determined based on the dark watermark detection result.

[0071] In some embodiments, the second neural network model can be a multi-class classification model or a binary classification model.

[0072] When the first neural network model is a multi-classification classification model, the second image category is one of the following: non-AI generated images, first type of AI generated images, second type of AI generated images, and the first type is different from the second type. That is, in an embodiment of the present application, if there are at least two image categories for AI-generated images. When there are two image categories for AI-generated images, they are the first type of AI-generated images and the second type of AI-generated images. When there are three image categories for AI-generated images, they are the first type of AI-generated images, the second type of AI-generated images, and the third type of AI-generated images. Among them, the first type, the second type, and the third type are different. And so on.

[0073] When the first neural network model is a binary classification model, the second image category is one of the following: non-AI generated image, AI generated image. In simple terms, the classification result can also be directly indicated by "yes (AI generated)" and "no (AI generated)".

[0074] Step 340: Determine a determination result of the input image according to the second image category, where the determination result is a result of determining whether the input image is generated by AI.

[0075] In some embodiments, if the confidence of the second neural network model for the second image category is greater than or equal to a set threshold, the second image category is directly used as the judgment result of the input image. In some embodiments, if the confidence of the second neural network model for the second image category is less than the set threshold, other judgments are continued. For details, please refer to the explanation of the following embodiment, which will not be repeated here.

[0076] The technical solution provided in the embodiment of the present application uses a first neural network model to determine whether the input image is generated by AI, which improves the efficiency of image category determination compared to the manual determination method in the related art. Furthermore, when the confidence of the first neural network model for the first image category determined is less than the first set threshold, the second neural network model is continued to be used in combination with the text question to classify the input image again to obtain the final input image determination result. In this way, when the confidence of the first neural network model is low, the second neural network model is used to continue the determination, which is conducive to improving the accuracy of image category determination.

[0077] Please refer to Figure 4 , which shows a flow chart of a method for determining an image category provided by another embodiment of the present application. The execution subject of each step of the method may be the computer device introduced above. The method may include at least one of the following steps (410-460).

[0078] Step 410: Obtain an input image.

[0079] In some embodiments, the first neural network model includes a first feature extraction sub-model, a second feature extraction sub-model, and a classifier, and the classifier is a multi-classification classifier.

[0080] Exemplarily, the first feature extraction submodel is a feature extraction layer (or feature extraction model) for extracting shallow image features. Exemplarily, the second feature extraction submodel is a feature extraction layer (or feature extraction model) for extracting deep learning features. Exemplarily, the classifier is a classification layer (or classification model) for implementing multi-classification.

[0081] Step 420, obtaining a first image feature of the input image according to the input image through the first feature extraction sub-model, where the first image feature is a shallow image feature of the input image extracted by the first feature extraction sub-model.

[0082] In some embodiments, the shallow image features are features related to the image extracted directly from the image. Exemplarily, the shallow image features include at least one of the following: edge features, texture features, and color histograms. In some embodiments, the input of the first feature extraction submodel is the input image, and the output is the first image feature of the input image. In some embodiments, the first feature extraction submodel is a pre-trained model, or a pre-trained model obtained by fine-tuning.

[0083] Step 430, obtaining a second image feature of the input image according to the input image through the second feature extraction sub-model, where the second image feature is a deep learning feature of the input image extracted by the second feature extraction sub-model.

[0084] In some embodiments, the deep learning features are deep features extracted from the image after deep learning. In some embodiments, the input of the second feature extraction submodel is the input image, and the output is the second image feature of the input image. In some embodiments, the second feature extraction submodel is a pre-trained model, or a pre-trained model is obtained by fine-tuning. In some embodiments, the second feature extraction submodel can use a convolutional neural network or a more complex model (such as Vision Transformer) to capture the deep learning features of the input image.

[0085] Step 440: Classify the input image according to the first image feature and the second image feature by a classifier to obtain a first image category.

[0086] In some embodiments, the input of the classifier is the first image feature and the second image feature, and the output is the first image category.

[0087] In some embodiments, a feature fusion layer is further included before the classifier. Exemplarily, the feature fusion layer is used to fuse the first image feature and the second image feature to obtain a fused feature. Exemplarily, in some embodiments, the input of the classifier is the fused feature, and the output is the first image category.

[0088] In some embodiments, the feature fusion layer sets different fusion weight parameters for the first image feature and the second image feature, respectively, and fuses the first image feature and the second image feature through different fusion weight parameters. Exemplarily, different input images correspond to different fusion weight parameters. Exemplarily, the fusion weight parameters are related to the content of the input image.

[0089] Step 450, when the confidence of the first neural network model for the first image category is less than a first set threshold, a second image category of the input image is determined by a second neural network model based on the input image and a text question, the text question is used to ask whether the input image is generated by AI, and the second neural network model is used to classify the input image again in combination with the text question.

[0090] In some embodiments, the training process of the first neural network model includes at least one of the following steps S1 to S6 (not shown in the figure).

[0091] Step S1, obtaining a plurality of non-AI generated images and a plurality of AI generated images to form an image set.

[0092] Exemplarily, multiple non-AI generated images and multiple AI generated images are collected to form an image set. Exemplarily, multiple non-AI generated images (i.e. Figure 2 ) and multiple AI-generated images.

[0093] Step S2, selecting a target pre-trained classification model from multiple pre-trained classification models as a base model.

[0094] In some embodiments, a pre-trained classification model with the best classification effect is selected from multiple pre-trained classification models as the target pre-trained classification model, thereby obtaining a base model.

[0095] In some embodiments, the hyperparameters of the base model are determined by a hyperparameter search technique, where the hyperparameters of the base model are parameters that remain unchanged during the training process of the base model.

[0096] Step S3, obtaining the image category of the ith image according to the ith image in the image set through the base model, where i is a positive integer.

[0097] In some embodiments, the base model is trained using the images in the above image set. Of course, the images in the image set can be directly labeled with image categories to obtain classification labels corresponding to each image to form a classification label set. Exemplarily, the base model is trained using the image set and the classification label set, and the trained base model is used as the first neural network model.

[0098] In some embodiments, manual annotation can be used, or classification labels can be automatically generated based on the source of the image. Exemplarily, different image generation models are used to generate images, and classification labels are automatically added (different image generation models correspond to different image categories of AI-generated images). Exemplarily, images taken by users are obtained, and classification labels are automatically added as non-AI generated.

[0099] Of course, it is also possible to not perform labeling, but directly decide whether to use the image as a sample to train the base model again based on the confidence of the image category predicted by the base model for the image in the image set, so as to achieve automatic training.

[0100] Step S4, when the confidence of the base model for the image category of the ith image is greater than or equal to the second set threshold, the ith image is taken as a sample image and placed in a sample image set; the image category of the ith image is determined as the classification label of the ith image; a mapping relationship is established between the ith image and the classification label of the ith image, and the classification label of the ith image is in the classification label set.

[0101] Exemplarily, the input of the base model is an image in the image set, and the output is the image category of the image. Of course, the confidence for the image category can also be output.

[0102] Exemplarily, the images in the image set are traversed, and the images whose confidence level output by the base model is greater than or equal to the second set threshold are screened out as sample images. At the same time, the image category output by the base model is used as a classification label without manual labeling. In this way, a sample image and the classification label corresponding to the sample image automatically constitute a training sample, and the training sample is used to train the base model again, thereby realizing automated training.

[0103] Step S5: when the confidence of the base model for the image category of the ith image is less than a second set threshold, the ith image is removed from the image set.

[0104] Similarly, if the confidence of the base model for the output image category is less than the second set threshold, the image is directly not used as a sample image, that is, the image is discarded and the base model is not used to train the image.

[0105] Step S6, using the sample image set and the classification label set to train the base model, there is a mapping relationship between a sample image in the sample image set and a classification label in the classification label set.

[0106] Exemplarily, all sample images finally screened from the image set are used to construct a sample image set. At the same time, a classification label set is constructed according to the image category output by the base model corresponding to each sample image in the sample image set, and a mapping relationship is constructed between the sample images and the classification labels that have a corresponding relationship, such as using QKV (query, index, content) technology to achieve a one-to-one mapping between sample images and classification labels.

[0107] Step 460: Determine a determination result of the input image according to the second image category, where the determination result is a result of determining whether the input image is generated by AI.

[0108] The technical solution provided in the embodiment of the present application realizes the detailed classification of the image categories of AI-generated images through multi-classification, thereby improving the classification accuracy. Furthermore, during classification, not only shallow image features but also deep image features are extracted to improve the classification accuracy of the classification results.

[0109] In addition, through the automated training of the base model, the training efficiency is improved, which is helpful to reduce the classification cost of image classification.

[0110] Please refer to Figure 5 , which shows a flow chart of a method for determining an image category provided by another embodiment of the present application. The execution subject of each step of the method may be the computer device introduced above. The method may include at least one of the following steps (510-570).

[0111] Step 510: Obtain an input image.

[0112] Step 520: Obtain a first image category of the input image based on the input image through a first neural network model, where the first neural network model is used to classify whether the input image is an AI-generated image.

[0113] Step 530, when the confidence of the first neural network model for the first image category is less than the first set threshold, the second image category of the input image is determined by the second neural network model based on the input image and the text question, the text question is used to ask whether the input image is AI generated, and the second neural network model is used to classify the input image again in combination with the text question.

[0114] In some embodiments, when the confidence of the first neural network model for the first image category is less than a first set threshold, dark watermark detection is performed on the input image to obtain a dark watermark detection result. Dark watermark detection refers to detecting whether there is a dark watermark in the input image. The dark watermark is used to indicate that the input image is generated by AI.

[0115] The dark watermark mentioned in the embodiments of the present application can be understood as a watermark image placed in an image or watermark information in a non-image form (such as watermark numbers, watermark text, etc.). Exemplarily, the dark watermark detection in the embodiments of the present application can detect whether there is a dark watermark in the input image.

[0116] The dark watermark detection result mentioned in the embodiments of the present application includes at least one of the presence of a dark watermark and the absence of a dark watermark.

[0117] In some embodiments, before performing dark watermark detection, AI dark watermark data is required to pre-train how to detect dark watermarks. AI dark watermark data is a watermark image, such as a png format, such as various logo images. To embed the watermark image into an image, dark watermark detection technology can be used to identify whether an image has a dark watermark.

[0118] The embedding principle of dark watermark is explained below. Dark watermark is a covert way to embed information into an image, usually by embedding tiny signal interference in the frequency domain to avoid affecting the visual effect of the image. The following are common embedding methods: DCT (discrete cosine transform) frequency domain embedding: In the DCT frequency domain, the low-frequency area affects the image clarity, and the high-frequency area is more easily compressed and lost. Therefore, the watermark information is usually embedded in the intermediate frequency area, and the watermark information is hidden in it by adjusting the intermediate frequency coefficient. DWT (discrete wavelet transform) frequency domain embedding: The image is decomposed into frequency components (low frequency and high frequency) of different scales through DWT. The watermark information can be embedded in the high-frequency sub-band (or intermediate frequency sub-band) to ensure the visual effect while reducing compression loss. Implicit embedding based on pixel arrangement: The watermark information is implied by changing the arrangement order of certain pixels or specific pixel positions. This method is suitable for low-interference covert watermarks.

[0119] The following is an explanation of the implementation steps of dark watermark embedding. Taking DCT as an example, the detailed steps of dark watermark embedding are described: Image segmentation: Divide the image into multiple small blocks (such as 8x8 or 16x16 blocks) to facilitate the application of frequency domain transform to each block. Frequency domain transformation: Apply DCT to each small block to transform the spatial domain to the frequency domain, so that the frequency components of each block can be obtained. Watermark information embedding: Select appropriate frequency domain components (such as intermediate frequency coefficients) to embed watermark information. The watermark information can be converted into binary values ​​(0 and 1), and the watermark information can be hidden in the frequency domain by fine-tuning the coefficients of the intermediate frequency. For example, if a bit of the watermark is 1, the DCT coefficient is increased by a certain value; if it is 0, the coefficient is reduced by a certain value. Inverse DCT transform: After the watermark is embedded, an inverse DCT transform is performed on each block to convert the frequency domain data back to the spatial domain, and finally a watermarked image is obtained.

[0120] In some embodiments, the judgment result of the input image is determined according to the dark watermark detection result.

[0121] Exemplarily, if the dark watermark detection result is that the dark watermark exists, the determination result of the input image is determined to be an AI generated image. Exemplarily, if the dark watermark detection result is that the dark watermark does not exist, the determination result of the input image is determined to be a non-AI generated image.

[0122] In some embodiments, the dark watermark detection method includes at least one of the following methods P1 to P3 (not shown in the figure).

[0123] Step P1, extracting the frequency characteristics of the input image; when the frequency characteristics indicate that dark watermark information exists in a preset frequency band, determining the dark watermark detection result is that the input image has a dark watermark; when the frequency characteristics indicate that no dark watermark information exists in the preset frequency band, determining the dark watermark detection result is that the input image does not have a dark watermark.

[0124] For example, the input image is transformed in the frequency domain (such as discrete cosine transform DCT or discrete wavelet transform DWT) to decompose the low-frequency, medium-frequency and high-frequency components (i.e., frequency characteristics). The watermark information is usually embedded in a specific frequency band in a specific pattern, and the dark watermark is identified by detecting the frequency characteristics of this frequency band.

[0125] Step P2, extracting the frequency domain features of the input image; when the frequency domain features match the preset frequency domain features of the dark watermark, determining the dark watermark detection result is that the input image has a dark watermark; when the frequency domain features do not match the preset frequency domain features of the dark watermark, determining the dark watermark detection result is that the input image does not have a dark watermark.

[0126] Exemplarily, after extracting the frequency domain features, they can be matched with the known AIGC watermark feature template, or a classifier can be used to determine whether there are dark watermark features in the frequency domain. Exemplarily, the AIGC watermark feature template is the preset dark watermark frequency domain feature, which is the preset frequency domain feature used to indicate the characteristics of the dark watermark. Exemplarily, a classifier can also be directly used to classify the frequency domain features, and the classification result is at least one of the following: the dark watermark exists, the dark watermark does not exist.

[0127] Exemplarily, the extracted frequency domain features of the input image are matched with the preset frequency domain features of the dark watermark to determine whether the frequency domain features match the frequency domain features of the dark watermark.

[0128] For example, a feature matching threshold is set on the frequency domain features. If the matching degree between the image frequency domain features and the watermark template exceeds the threshold, it is determined to be a dark watermark image. The optimal threshold is determined through experiments and tested on different AIGC images to ensure the accuracy and anti-interference of the model.

[0129] In some embodiments, a degree of match between the frequency domain features and the dark watermark frequency domain features is determined.

[0130] Exemplarily, the similarity between the frequency domain feature and the dark watermark frequency domain feature is calculated, and the similarity is used as the matching degree between the frequency domain feature and the dark watermark frequency domain feature. Exemplarily, the similarity here can be cosine similarity, Euclidean distance, Manhattan distance, etc.

[0131] In some embodiments, when the matching degree is greater than or equal to a third set threshold, it is determined that the frequency domain feature matches the dark watermark frequency domain feature.

[0132] Exemplarily, if the matching degree is greater than or equal to the third set threshold, the frequency domain feature matches the frequency domain feature of the dark watermark.

[0133] In some embodiments, when the degree of matching is less than a third set threshold, it is determined that the frequency domain feature does not match the frequency domain feature of the dark watermark.

[0134] Exemplarily, if the matching degree is less than the third set threshold, the frequency domain feature does not match the frequency domain feature of the dark watermark.

[0135] Step P3, obtaining a dark watermark detection result based on the input image through a third neural network model, where the third neural network model is used to detect whether a dark watermark exists in the input image.

[0136] In some embodiments, a convolutional neural network model is used to train a dark watermark detection model (i.e., a third neural network model). Some images with known watermarks can be generated in the training set, and the model can be trained to identify such specific watermark features. Exemplarily, the third neural network model is a pre-trained model, or a pre-trained model obtained after fine-tuning. Exemplarily, the input of the third neural network model is an input image, and the output is a dark watermark detection result.

[0137] In some embodiments, when the confidence of the third neural network model for the dark watermark detection result is greater than or equal to a fourth set threshold, a step of determining the judgment result of the input image based on the dark watermark detection result is performed.

[0138] Exemplarily, when the confidence of the third neural network model for the dark watermark detection result is greater than or equal to the fourth set threshold, the judgment result of the input image is determined according to the dark watermark detection result. That is, if the dark watermark detection result is that a dark watermark exists, the judgment result of the input image is determined to be an AI generated image. Exemplarily, if the dark watermark detection result is that a dark watermark does not exist, the judgment result of the input image is determined to be a non-AI generated image.

[0139] In some embodiments, when the confidence of the dark watermark detection result of the third neural network model is less than a fourth set threshold, a step of determining a second image category of the input image based on the input image and text questions through the second neural network model is performed.

[0140] Exemplarily, when the confidence level of the dark watermark detection result of the third neural network model is less than the fourth set threshold, the image category of the input image continues to be determined, such as using the second neural network model to perform multimodal model determination, that is, step 530.

[0141] Step 540, when the confidence of the second neural network model for the second image category is greater than or equal to the fifth set threshold, the second image category is determined as the determination result of the input image.

[0142] In some embodiments, if the confidence of the second neural network model for the second image category is greater than or equal to a fifth set threshold, the second image category is determined as the determination result of the input image.

[0143] Step 550, when the confidence of the second neural network model for the second image category is less than the fifth set threshold, obtain the image label of the input image, where the image label is text related to the input image obtained by recognizing the input image through the neural network model.

[0144] In some embodiments, if the confidence of the second neural network model for the second image category is less than the fifth set threshold, the image category of the input image continues to be determined, such as executing step 550.

[0145] In some embodiments, the image tag of the input image, that is, the image reverse deduction tags mentioned in the above embodiments, tags are keywords used to describe the content of the image, and may include objects in the image (such as "tree", "mountain", "beach"), scenes (such as "sunrise", "city", "forest"), styles (such as "cartoon", "realism") and other information. Source: These tags usually come from the metadata and descriptive text of the image, or are automatically extracted from the image content by an algorithm to identify the semantics of the image. In an embodiment of the present application, tags refer to a set of keywords or labels that describe the content of the image. For AI generated content (AIGC) classification, tags can help identify information such as the subject, object, scene, and style of the image, and then use this information to perform similarity matching with the actual content of the image to determine whether the image meets the description of these tags.

[0146] In some embodiments, the image tag of the input image is specifically obtained in the following manner.

[0147] The first one uses an image recognition model.

[0148] Use pre-trained image recognition models. Common models include ResNet, EfficientNet, or more specialized multi-label classification models (such as DETR, YOLO, etc.). They can detect and classify objects or scenes in images and return tags that match the image content. Specific operation: Input the image to the model, the model will recognize the objects or scenes in the image based on visual features and return possible tags. The advantages of this method are: high recognition accuracy and suitable for common objects or scenes.

[0149] The second one uses a multi-label image classification model.

[0150] Some image classification models can generate multiple labels to provide richer content descriptions. For example, multimodal models such as CLIP can annotate images with multiple labels. Specific operation: Input the image into the model, and the model will generate multiple relevant labels based on the image content to provide a more comprehensive description. The advantages of this method are: diversified labels, suitable for the description of complex image content.

[0151] The third method is to use reverse annotation generation of multimodal large models.

[0152] Multimodal models such as CLIP or BLIP can not only perform image content recognition, but also process the similarity between images and text. These models can reversely generate semantic descriptions from images and then convert the generated descriptions into tags. Specific operation: input an image, the model generates a semantic description (or a series of keywords), and then converts the description into tags. The advantage of this method is that it can generate more abstract and general tags, which is suitable for images containing complex semantic information.

[0153] Step 560, obtaining a similarity determination result according to the input image and the image label through a fourth neural network model, where the fourth neural network model is used to predict the similarity between the input image and the image label.

[0154] Exemplarily, the fourth neural network model is a pre-trained model, or a pre-trained model obtained after fine-tuning. Exemplarily, the input of the fourth neural network model is an input image and an image label, and the output is a similarity determination result.

[0155] Exemplarily, the similarity determination result refers to the similarity between the input image and the image label, such as a similarity value.

[0156] In some embodiments, once the tags of an image are obtained, these tags can be matched with the image content for similarity. If the similarity between the two is low, it may mean that the content of the image is unnatural and may be generated by AIGC. Similarity calculation: The tags text and image content are input into a multimodal model (such as CLIP) at the same time to calculate the similarity between the text and the image content. Judgment basis: If the image content does not match the reverse-generated tags description, it means that it may be an AIGC image. The reason is that AI-generated images often have some details or unnatural structures in the content that do not conform to the label description.

[0157] Step 570: Determine the determination result of the input image according to the similarity determination result.

[0158] In some embodiments, when the similarity determination result indicates that the similarity between the input image and the image label is greater than or equal to a sixth set threshold, the determination result of the input image is determined to be a non-AI generated image.

[0159] Exemplarily, if the similarity determination result indicates that the similarity between the input image and the image label is high, it means that the input image is more consistent with the image label, that is, the input image is more natural, and the input image is considered to be a non-AI generated image.

[0160] In some embodiments, when the similarity determination result indicates that the similarity between the input image and the image label is less than a sixth set threshold, the determination result of the input image is determined to be an AI-generated image.

[0161] Exemplarily, if the similarity determination result indicates that the similarity between the input image and the image label is low, it means that the input image does not conform to the image label, that is, the input image is unnatural, and the input image is considered to be an AI-generated image.

[0162] For example, the tags text and input image are simultaneously input into a multimodal model (such as CLIP) to calculate the similarity between the text and image content. If the input image does not match the reverse generated tags description, it means that it may be an AIGC image, because AI-generated images often have some details or unnatural structures in their content that do not match the tag description.

[0163] In some embodiments, if the similarity determination result output by the fourth neural network model indicates a still high similarity, subsequent manual determination is no longer performed, and the input image is considered to be a non-AI generated image.

[0164] In some embodiments, if the similarity indicated by the similarity determination result output by the fourth neural network model is still low, it may be considered to add manual determination to improve the determination accuracy of the image category. Exemplarily, a query is added as to whether manual determination is required. If manual determination is required, manual determination is performed on the input image. Exemplarily, if manual determination is not required, manual determination is not performed on the input image. The input image is directly considered to be a non-AI generated image.

[0165] For example, for business scenarios that require high accuracy, results with low confidence can be backed up by additional manual judgment. For example, the business scenario can be the platform's review of whether the image provided by the user is AI-generated in an exam scenario. For example, the business scenario can be the platform's review of whether the image provided by the user is AI-generated in a user qualification verification scenario.

[0166] The technical solution provided by the embodiment of the present application determines whether it is AI generated by the detection result obtained by dark watermark detection, which is conducive to further improving the classification accuracy. Furthermore, the present application provides a variety of dark watermark detection methods, reflecting the diversity and flexibility of dark watermark detection.

[0167] Finally, we can determine the image category of the input image again by inferring tags from the image and determining the similarity between the image and the tags, which helps to ensure the accuracy of the classification results. At the same time, we can add manual judgment as a backup, which is more friendly to business scenarios that require high accuracy.

[0168] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0169] Please refer to Figure 6 , which shows a block diagram of an apparatus for determining image categories provided by an embodiment of the present application. The apparatus has the function of implementing the above-mentioned method for determining image categories, and the function can be implemented by hardware or by hardware executing corresponding software. The apparatus can be the computer device described above, or it can be set in a computer device. Figure 6 As shown, the device 600 may include: an acquisition module 610 , a classification module 620 and a determination module 630 .

[0170] The acquisition module 610 is used to acquire an input image.

[0171] The classification module 620 is used to obtain a first image category of the input image according to the input image through a first neural network model, and the first neural network model is used to classify whether the input image is an image generated by artificial intelligence AI.

[0172] The classification module 620 is also used to determine the second image category of the input image based on the input image and a text question through a second neural network model when the confidence of the first neural network model for the first image category is less than a first set threshold, wherein the text question is used to ask whether the input image is generated by AI, and the second neural network model is used to classify the input image again in combination with the text question.

[0173] The determination module 630 is used to determine a determination result of the input image according to the second image category, wherein the determination result is a result of determining whether the input image is generated by AI.

[0174] In some embodiments, the first neural network model includes a first feature extraction sub-model, a second feature extraction sub-model, and a classifier, and the classifier is a multi-classification classifier.

[0175] In some embodiments, the classification module 620 is used to obtain a first image feature of the input image according to the input image through the first feature extraction submodel, where the first image feature is a shallow image feature of the input image extracted by the first feature extraction submodel; obtain a second image feature of the input image according to the input image through the second feature extraction submodel, where the second image feature is a deep learning feature of the input image extracted by the second feature extraction submodel; and classify the input image according to the first image feature and the second image feature through the classifier to obtain the first image category.

[0176] In some embodiments, the classification module 620 is used to perform several steps of the training process of the first neural network model: acquiring multiple non-AI generated images and multiple AI generated images to form an image set; selecting a target pre-trained classification model from multiple pre-trained classification models as a base model; obtaining the image category of the i-th image in the image set according to the i-th image through the base model, where i is a positive integer; when the confidence of the base model for the image category of the i-th image is greater than or equal to a second set threshold, taking the i-th image as a sample image and putting it into the sample image set; determining the image category of the i-th image as the classification label of the i-th image; establishing a mapping relationship between the i-th image and the classification label of the i-th image, and the classification label of the i-th image is in the classification label set; when the confidence of the base model for the image category of the i-th image is less than the second set threshold, removing the i-th image from the image set; using the sample image set and the classification label set to train the base model, there is a mapping relationship between a sample image in the sample image set and a classification label in the classification label set.

[0177] In some embodiments, the classification module 620 is used to perform dark watermark detection on the input image to obtain a dark watermark detection result when the confidence of the first neural network model for the first image category is less than the first set threshold. The dark watermark detection refers to detecting whether a dark watermark exists in the input image, and the dark watermark is used to indicate that the input image is generated by AI; based on the dark watermark detection result, the judgment result of the input image is determined.

[0178] In some embodiments, the classification module 620 is used to extract the frequency characteristics of the input image; when the frequency characteristics indicate that dark watermark information exists in a preset frequency band, determine that the dark watermark detection result is that the dark watermark exists in the input image; when the frequency characteristics indicate that dark watermark information does not exist in the preset frequency band, determine that the dark watermark detection result is that the dark watermark does not exist in the input image; or, extract the frequency domain characteristics of the input image; when the frequency domain characteristics match the preset dark watermark frequency domain characteristics, determine that the dark watermark detection result is that the dark watermark exists in the input image; when the frequency domain characteristics do not match the preset dark watermark frequency domain characteristics, determine that the dark watermark detection result is that the dark watermark does not exist in the input image; or, obtain the dark watermark detection result based on the input image through a third neural network model, and the third neural network model is used to detect whether the dark watermark exists in the input image.

[0179] In some embodiments, the classification module 620 is used to determine the degree of match between the frequency domain feature and the dark watermark frequency domain feature; when the degree of match is greater than or equal to a third set threshold, it is determined that the frequency domain feature matches the dark watermark frequency domain feature; when the degree of match is less than the third set threshold, it is determined that the frequency domain feature does not match the dark watermark frequency domain feature.

[0180] In some embodiments, the classification module 620 is used to execute the step of determining the judgment result of the input image according to the dark watermark detection result when the confidence of the third neural network model for the dark watermark detection result is greater than or equal to a fourth set threshold; and to execute the step of determining the second image category of the input image according to the input image and the text question through the second neural network model when the confidence of the third neural network model for the dark watermark detection result is less than the fourth set threshold.

[0181] In some embodiments, the determination module 630 is used to determine the second image category as the judgment result of the input image when the confidence of the second neural network model for the second image category is greater than or equal to a fifth set threshold; obtain the image label of the input image when the confidence of the second neural network model for the second image category is less than the fifth set threshold, the image label is text related to the input image obtained by recognizing the input image by the neural network model; obtain a similarity judgment result based on the input image and the image label through a fourth neural network model, the fourth neural network model is used to predict the similarity between the input image and the image label; and determine the judgment result of the input image based on the similarity judgment result.

[0182] In some embodiments, the determination module 630 is used to determine that the determination result of the input image is a non-AI generated image when the similarity determination result indicates that the similarity between the input image and the image label is greater than or equal to a sixth set threshold; and to determine that the determination result of the input image is an AI generated image when the similarity determination result indicates that the similarity between the input image and the image label is less than the sixth set threshold.

[0183] In some embodiments, the first neural network model is a multi-class classification model, and the first image category is one of the following: non-AI generated images, a first type of AI generated images, and a second type of AI generated images, and the first type is different from the second type.

[0184] In some embodiments, the determination module 630 is used to determine the first image category as the determination result of the input image when the confidence of the first neural network model for the first image category is greater than or equal to the first set threshold.

[0185] The technical solution provided in the embodiment of the present application uses a first neural network model to determine whether the input image is generated by AI, which improves the efficiency of image category determination compared to the manual determination method in the related art. Furthermore, when the confidence of the first neural network model for the first image category determined is less than the first set threshold, the second neural network model is continued to be used in combination with the text question to classify the input image again to obtain the final input image determination result. In this way, when the confidence of the first neural network model is low, the second neural network model is used to continue the determination, which is conducive to improving the accuracy of image category determination.

[0186] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0187] Please refer to Figure 7 , which shows a block diagram of a computer device 700 provided in one embodiment of the present application. The computer device 700 may be any electronic device with data calculation, processing and storage capabilities. The computer device 700 may be used to implement the method for determining the image category provided in the above embodiment.

[0188] Typically, the computer device 700 includes a processor 701 and a memory 702 .

[0189] The processor 701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 701 may also include an AI processor, which is used to process computing operations related to machine learning.

[0190] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include a high-speed random access memory and a non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 702 is used to store a computer program, which is configured to be executed by one or more processors to implement the above-mentioned method for determining the image category.

[0191] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the computer device 700, and the computer device 700 may include more or less components than shown in the figure, or combine some components, or adopt a different arrangement of components.

[0192] In an exemplary embodiment, a computer-readable storage medium is also provided, in which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned method for determining the image category is implemented. Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random Access Memory), SSD (Solid State Drives) or an optical disk, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0193] In an exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned method for determining the image category.

[0194] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0195] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining an image category, characterized in that: The method comprises: Get the input image; Obtaining a first image category of the input image according to the input image through a first neural network model, wherein the first neural network model is used to classify whether the input image is an image generated by artificial intelligence AI; In a case where the confidence of the first neural network model for the first image category is less than a first set threshold, determining a second image category of the input image by a second neural network model according to the input image and a text question, wherein the text question is used to ask whether the input image is generated by AI, and the second neural network model is used to classify the input image again in combination with the text question; According to the second image category, a determination result of the input image is determined, where the determination result is a result of determining whether the input image is generated by AI.

2. The method according to claim 1, characterized in that The first neural network model includes a first feature extraction sub-model, a second feature extraction sub-model and a classifier, and the classifier is a multi-classification classifier; The step of obtaining a first image category of the input image according to the input image by using a first neural network model includes: Obtaining, according to the input image, a first image feature of the input image through the first feature extraction submodel, wherein the first image feature is a shallow image feature of the input image extracted by the first feature extraction submodel; Obtaining, according to the input image, a second image feature of the input image through the second feature extraction sub-model, where the second image feature is a deep learning feature of the input image extracted by the second feature extraction sub-model; The input image is classified by the classifier according to the first image feature and the second image feature to obtain the first image category.

3. The method according to claim 2, characterized in that The training process of the first neural network model includes the following steps: Acquire multiple non-AI generated images and multiple AI generated images to form an image set; Select a target pre-trained classification model from multiple pre-trained classification models as a base model; Obtaining the image category of the i-th image according to the i-th image in the image set by using the base model, where i is a positive integer; When the confidence of the base model for the image category of the i-th image is greater than or equal to a second set threshold, the i-th image is taken as a sample image and put into a sample image set; the image category of the i-th image is determined as a classification label of the i-th image; Establishing a mapping relationship between the i-th image and the classification label of the i-th image, where the classification label of the i-th image is in a classification label set; When the confidence of the base model for the image category of the i-th image is less than the second set threshold, removing the i-th image from the image set; The base model is trained using the sample image set and the classification label set, and there is a mapping relationship between a sample image in the sample image set and a classification label in the classification label set.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: When the confidence of the first neural network model for the first image category is less than the first set threshold, performing dark watermark detection on the input image to obtain a dark watermark detection result, wherein the dark watermark detection refers to detecting whether a dark watermark exists in the input image, and the dark watermark is used to indicate that the input image is generated by AI; A determination result of the input image is determined according to the dark watermark detection result.

5. The method according to claim 4, characterized in that The performing dark watermark detection on the input image to obtain a dark watermark detection result includes: extracting the frequency characteristics of the input image; if the frequency characteristics indicate that dark watermark information exists in a preset frequency band, determining that the dark watermark detection result is that the input image exists the dark watermark; if the frequency characteristics indicate that dark watermark information does not exist in the preset frequency band, determining that the dark watermark detection result is that the input image does not exist the dark watermark; or, extracting frequency domain features of the input image; if the frequency domain features match the preset frequency domain features of the dark watermark, determining that the dark watermark detection result is that the input image has the dark watermark; if the frequency domain features do not match the preset frequency domain features of the dark watermark, determining that the dark watermark detection result is that the input image does not have the dark watermark; or, The dark watermark detection result is obtained according to the input image through a third neural network model, and the third neural network model is used to detect whether the dark watermark exists in the input image.

6. The method according to claim 5, characterized in that The method further comprises: Determining a degree of match between the frequency domain feature and the dark watermark frequency domain feature; When the matching degree is greater than or equal to a third set threshold, determining that the frequency domain feature matches the frequency domain feature of the dark watermark; When the matching degree is less than the third set threshold, it is determined that the frequency domain feature does not match the dark watermark frequency domain feature.

7. The method according to claim 6, characterized in that In the case where the confidence of the dark watermark detection result of the third neural network model is greater than or equal to a fourth set threshold, performing a step of determining a determination result of the input image according to the dark watermark detection result; In the case where the confidence of the dark watermark detection result of the third neural network model is less than the fourth set threshold, a step of determining a second image category of the input image according to the input image and text question through a second neural network model is performed.

8. The method according to any one of claims 1 to 3, characterized in that: The step of determining a determination result of the input image according to the second image category includes: When the confidence of the second neural network model for the second image category is greater than or equal to a fifth set threshold, determining the second image category as the determination result of the input image; When the confidence of the second neural network model for the second image category is less than the fifth set threshold, obtaining an image label of the input image, wherein the image label is text related to the input image obtained by recognizing the input image through the neural network model; Obtaining a similarity determination result according to the input image and the image label by a fourth neural network model, wherein the fourth neural network model is used to predict the similarity between the input image and the image label; According to the similarity determination result, a determination result of the input image is determined.

9. The method according to claim 8, characterized in that Determining the determination result of the input image according to the similarity determination result includes: When the similarity determination result indicates that the similarity between the input image and the image label is greater than or equal to a sixth set threshold, determining that the determination result of the input image is a non-AI generated image; When the similarity determination result indicates that the similarity between the input image and the image label is less than the sixth set threshold, it is determined that the determination result of the input image is an AI-generated image.

10. The method according to any one of claims 1 to 3, characterized in that: The first neural network model is a multi-class classification model, the first image category is one of the following: a non-AI generated image, a first type of AI generated image, and a second type of AI generated image, the first type being different from the second type; The method further comprises: When the confidence of the first neural network model for the first image category is greater than or equal to the first set threshold, the first image category is determined as the determination result of the input image.

11. A device for determining image category, characterized in that: The device comprises: An acquisition module, used for acquiring an input image; A classification module, configured to obtain a first image category of the input image according to the input image through a first neural network model, wherein the first neural network model is configured to classify whether the input image is an image generated by artificial intelligence (AI); The classification module is further configured to determine, by using a second neural network model, a second image category of the input image according to the input image and a text question when the confidence of the first neural network model for the first image category is less than a first set threshold, wherein the text question is used to ask whether the input image is generated by AI, and the second neural network model is used to classify the input image again in combination with the text question; A determination module is used to determine a determination result of the input image according to the second image category, wherein the determination result is a result of determining whether the input image is generated by AI.

12. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the method for determining the image category according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the method for determining the image category according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product includes a computer program, which is stored in a computer-readable storage medium. A processor reads and executes the computer program from the computer-readable storage medium to implement the method for determining an image category according to any one of claims 1 to 10.

Images