Image category detection method and device, equipment and storage medium
After judging the authenticity of the image through the style classification model, the corresponding detection model is selected for AI-generated image detection, which solves the problem of low detection accuracy in the prior art and improves the accuracy and robustness of image category detection.
Patent Information
- Application Number
- CN202411109944.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-13
AI Technical Summary
The prior art has low accuracy when detecting AI-generated image categories, especially due to the disappearance of feature defects caused by the evolution of AI models, the cracking of digital watermarks and the reduction in model detection accuracy caused by sample style differences.
The style classification model is used to initially judge the real style category or non-real style category of the image, and then select the corresponding image detection model based on the category for further detection to determine whether the image is a generated image.
The accuracy of image category detection is improved. By first judging the image style category and then performing specific detection, the accuracy and robustness of the detection are enhanced and the error rate is reduced.
Smart Images

Figure CN119027730A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for detecting image categories. Background Art
[0002] With the continuous development of artificial intelligence technology, by inputting instructions into the artificial intelligence content generation (AIDC) model, the AIGC model will automatically create text, images, audio or video content.
[0003] However, in some fields, due to authenticity and version issues, it is necessary to detect whether an image is generated by an AIGC model. The purpose of detecting the category of AIGC-generated images is to effectively detect whether the image or video is generated by AIGC. At present, according to the technical route, the detection methods of image categories can be divided into three categories. The first category is to distinguish AIGC-generated images based on physical features. This type of method mainly distinguishes AIGC-generated images by finding inconsistencies between artificial information and the physical world. For example, based on the physical rules of lighting and reflection in perspective, it can be determined whether the image is generated by AIGC. The second category is to add a digital watermark to the image generated by AIGC. A digital watermark is added at the beginning of AI image generation. Subsequently, only this digital watermark needs to be detected to determine whether the image is generated by the AIGC model. The third category is a method for detecting AIGC images based on a deep learning model. The detection effect of this method usually depends on the effectiveness of network feature extraction and appropriate and diverse training data.
[0004] As the AIGC model continues to evolve, the defects in the image will gradually disappear, and the first type of detection method will become ineffective. Since there is no completely reliable digital watermarking technology at present, the second type can be easily cracked, resulting in reduced detection accuracy. The third type of detection method relies on the extracted sample features. When the sample styles vary greatly, the model detection accuracy decreases. Summary of the invention
[0005] In order to solve the above technical problems, the present application provides a method, device, equipment and storage medium for detecting image categories to improve the detection accuracy of AIGC image categories.
[0006] In a first aspect, the present application provides a method for detecting an image category, the method comprising: obtaining an image to be detected; classifying the image to be detected using a style classification model to obtain a first target category of the image to be detected, the first target category including a real style category or a non-real style category; using an image detection model corresponding to the first target category as a target detection model; detecting the image to be detected using the target image detection model to obtain a second target category of the image to be detected, the second target category including a generated category or a non-generated category.
[0007] In a second aspect, the present application provides an image category detection device, which includes: an image acquisition module, used to acquire an image to be detected; a style detection module, used to classify the image to be detected using a style classification model, and obtain a first target category corresponding to the image to be detected, the first target category including a real style category or a non-real style category; a model selection module, used to use the image detection model corresponding to the first target category as a target detection model; a generation detection module, used to detect the image to be detected using a target image detection model, and obtain a second target category corresponding to the image to be detected, the second target category including a generated category or a non-generated category.
[0008] In a third aspect, the present application provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by one or more processors, the one or more processors implement the image category detection method as described in the first aspect above.
[0009] In a fourth aspect, the present application provides a storage medium, which may be a computer-readable storage medium, storing a computer program thereon, which, when executed by a processor, implements the method for detecting image categories as described in the first aspect above.
[0010] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the method for detecting image categories as described in the first aspect above is implemented.
[0011] Compared with the prior art, the technical solution provided by the embodiments of the present application has the following advantages:
[0012] The embodiments of the present application provide a method, apparatus, device and storage medium for detecting image categories, the method comprising: obtaining an image to be detected; classifying the image to be detected using a style classification model to obtain a first target category of the image to be detected, the first target category including a real style category or a non-real style category; using an image detection model corresponding to the first target category as a target detection model; detecting the image to be detected using the target image detection model to obtain a second target category of the image to be detected, the second target category including a generated category or a non-generated category. In the embodiments of the present application, the style category of the image to be detected is first determined using a style classification model, and then the style category is sent to the corresponding image detection model to determine whether the image to be detected is a generated image, thereby solving the problem of low image detection accuracy and improving the accuracy of image category detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0015] Figure 1 A schematic diagram of a flow chart of a method for detecting image categories provided in an embodiment of the present application;
[0016] Figure 2 A schematic diagram of the structure of each model relationship provided in the embodiment of the present application;
[0017] Figure 3 A flowchart of a model training method provided in an embodiment of the present application;
[0018] Figure 4 A schematic diagram of a flow chart of another method for detecting image categories provided in an embodiment of the present application;
[0019] Figure 5 A schematic diagram of parameter configuration of an image feature extraction network provided in an embodiment of the present application;
[0020] Figure 6 A schematic diagram of the structure of a texture feature extraction network provided in an embodiment of the present application;
[0021] Figure 7 A schematic diagram of the structure of the image detection model provided in the embodiment of the present application;
[0022] Figure 8 A comparison diagram of a set of experimental activation diagrams provided in the embodiments of the present application;
[0023] Fig. 9 A schematic diagram of the structure of an image category detection device provided in an embodiment of the present application;
[0024] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0025] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0026] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.
[0027] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0028] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0029] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0030] The following is a detailed description of the image category detection method provided by the present application in conjunction with the accompanying drawings and specific implementation methods.
[0031] Figure 1 This is a flow chart of an image category detection method in an embodiment of the present application. This embodiment can be applied to detecting whether an image is an AI-generated image. The method can be executed by an image category detection device. The image category detection device can be implemented in software and / or hardware. The image category detection device can be configured in an electronic device.
[0032] like Figure 1 As shown, the image category detection method provided in the embodiment of the present application mainly includes steps S101-S104.
[0033] S101, obtaining an image to be detected.
[0034] The image to be detected may be a single image. The image to be detected may be any one of the four images: a real non-generated image, a real generated image, a non-real generated image, and a non-real non-generated image. The image category detection method provided in the embodiment of the present application can identify the category of the image to be detected.
[0035] The image to be detected may also be a set of multiple images to be detected, and the set may be composed of any one of the four types of images: real non-generated images, real generated images, non-real generated images, and non-real non-generated images; or it may be a set composed of a mixture of two or more of the above four types of images. The image category detection method provided in the embodiment of the present application can identify the category of each image to be detected in the set.
[0036] Among them, real images refer to images of real objects or scenes in the real world, and generated images refer to images created by synthesis using computer algorithms.
[0037] A real non-generated image is an image directly captured by a camera or other image capture device. For example, a photo of a running puppy taken by a mobile phone is captured directly from the real world without being processed by any image generation algorithm. Therefore, this is a real non-generated image.
[0038] A truly generated image is an image generated by a computer algorithm based on real-world data or information. In other words, although the image is generated by a computer algorithm, the image content is real, or at least may exist in the real world. For example, if a large number of real cat images are used as training data to train an AIGC model, then after the AIGC model is trained, it generates an image that looks very much like a real cat. This image is a truly generated image.
[0039] Non-realistically generated images are those generated by computer algorithms, and the image content may be completely fictional and not based on any real-world data. In other words, the image content does not reflect the real world and is completely imagined or created. For example, a generative model may create a fantasy creature, such as a dragon or a unicorn, or an abstract artistic image. For example, a generative model is used to generate a virtual character in a game, or a production model is used to produce a scene from a science fiction movie. Then, the image of the virtual character or the image of the science fiction scene is a non-realistically generated image.
[0040] Non-realistic, non-generated images are images that are neither generated by an algorithm nor represent any real scene or object in the real world. For example: a fantasy landscape hand-painted by an artist, a logo created by a designer using drawing software, etc.
[0041] S102: Classify the image to be detected by using a style classification model to obtain a first target category of the image to be detected, where the first target category includes a true style category or a non-true style category.
[0042] The style classification model is mainly used to determine whether the image to be detected is a real style image. The first target category includes a real style category or a non-real style category. The real style category is used to indicate that the image to be detected is a real image, and the non-real style category is used to indicate that the image to be detected is a non-real image.
[0043] Furthermore, the style classification model is a binary classification model. After the image to be detected is input into the style classification model, the style classification model classifies the image to be detected to determine whether the image to be detected is a real image or a non-real image.
[0044] S103: Use the image detection model corresponding to the first target category as the target detection model.
[0045] First, the detection system of the entire image category is explained, such as Figure 2 As shown, the image category detection system includes a style classification model 21 and two image detection models, the two image detection models are a real image detection model 22 and a non-real image detection model 23; the real image detection model 22 is used to detect whether a real image is a generated image, and the non-real image detection model 23 is used to detect whether a non-real image is a generated image.
[0046] Among them, one image detection model corresponds to a first target category. In other words, the image detection model corresponding to the real style category is the real image detection model, and the image detection model corresponding to the non-real style category is the non-real image detection model. Whether the real image detection model is selected as the subsequent detection model or the non-real image detection model is selected as the subsequent detection model is determined according to whether the image to be detected is a real image or a non-real image.
[0047] In one possible implementation, the image detection model corresponding to the first target category is used as the target detection model, including: when the first target category is a real style category, selecting a detection model of the real image category as the target detection model; when the second target category is a non-real style category, selecting a detection model of the non-real image category as the target detection model.
[0048] Specifically, Figure 2 As shown, both the real image detection model and the non-real image detection model are connected to the style classification model. When the style classification model determines that the image to be detected is a real style category, that is, the image to be detected is a real image, the image to be detected can be directly input into the real image detection model. When the style classification model determines that the image to be detected is a non-real style category, that is, the image to be detected is a non-real image, the image to be detected can be directly input into the non-real image detection model.
[0049] S104 , using the target image detection model to detect the image to be detected, and obtaining a second target category of the image to be detected, where the second target category includes a generated category or a non-generated category.
[0050] The image detection model includes a real image detection model and a non-real image detection model, and the target image detection model is a real image detection model or a non-real image detection model.
[0051] Furthermore, the image detection model is also a binary classification model, and the image detection model is used to determine whether the image to be detected is a generated image. Specifically, the real image detection model is used to detect whether the real style image is a generated image, and the non-real image detection model is used to detect whether the non-real style image is a generated image.
[0052] Specifically, when the image to be detected is a real image, the real image detection model is used to detect and identify the image to be detected to determine whether the real image is a generated image. When the image to be detected is a non-real image, the non-real image detection model is used to detect and identify the image to be detected to determine whether the non-real image is a generated image.
[0053] The image category detection system in the embodiment of the present application includes a style classification detection model, a real image detection model, and a non-real image detection model. In the image category detection process, the style classification model is first used to detect whether the image to be detected is a real image or a non-real image, and then the image to be detected of the real style category is input into the real image detection model, and the real image detection model is used to determine whether the image to be detected is a generated image or a non-generated image, and the image to be detected of the non-real style category is input into the non-real image detection model, and the non-real image detection model is used to determine whether the image to be detected is a generated image or a non-generated image. After determining the authenticity of the image to be detected, whether the image to be detected is a generated class is detected, thereby improving the accuracy of image detection.
[0054] Based on the above embodiments, the present application provides a model training method, such as Figure 3As shown, the model training method provided in the embodiment of the present application mainly includes S201-S205.
[0055] S201. Acquire sample images, where the sample images include: real non-generated sample images, real generated sample images, non-real generated sample images, and non-real non-generated sample images.
[0056] Sample images refer to training images used for model training, and the images include: real non-generated sample images, real generated sample images, non-real generated sample images, and non-real non-generated sample images. The differences between the sample images can be referred to the discussion of the various types of images in the image to be detected in the above embodiment, and will not be discussed in the embodiments of this application.
[0057] Specifically, real non-generated images and non-real non-generated images can be obtained through open source data sets or web crawlers. Various public AIGC tools or various image generation platforms can be used to generate a large number of real generated images and non-real generated images. The method of obtaining sample images is no longer specifically limited in the embodiments of the present application.
[0058] S202. Divide the sample images into a first sample image set, a second sample image set and a third sample image set, wherein the first sample image set includes real generated sample images and non-real generated sample images, the second sample image set includes real non-generated sample images and real generated sample images, and the third sample image set includes non-real non-generated sample images and non-real generated sample images.
[0059] Since the image category detection system includes three models, all sample images are divided into different sample image sets in the embodiment of the present application, and each sample image set is used to train its corresponding network model.
[0060] Among them, the first sample image set includes real generated sample images and non-real generated sample images, which are used to train the style classification model, the second sample image set includes real non-generated sample images and real generated sample images, which are used to train the real image detection model, and the third sample image set includes non-real non-generated sample images and non-real generated sample images, which are used to train the non-real image detection model.
[0061] After a large number of sample images are obtained in S201, each sample image is annotated to determine the category of the sample image. The categories of the sample images include: real non-generated sample images, real generated sample images, non-real generated sample images, and non-real non-generated sample images. Each sample image corresponds to only one category. The annotation can be performed manually or by using an existing annotation program.
[0062] Furthermore, the sample images can be annotated according to their sources. For example, if the sample images are images generated by a generative model, such images are annotated as generated images. For example, if the sample images are images acquired by a camera platform, such images are annotated as real images. Another example is that if the sample images are images acquired by a comic platform, such images are annotated as non-real images. After the sample images are annotated according to their sources, the annotation information of the sample images can be refined and modified.
[0063] Specifically, the sample images may be divided into a first sample image set, a second sample image set, and a third sample image set according to the annotation information of each sample image.
[0064] S203: Using the first sample image set to train the first network model to obtain a style classification model.
[0065] The first network model is an initial model of the style classification model, that is, a style classification model that has not been trained using the first sample image set. The first network model is a binary classification network model.
[0066] Exemplarily, the first network model may be a DaViT network model, which utilizes two mechanisms, spatial attention and channel attention, to more effectively capture global and local information in an image and has a better effect on style classification.
[0067] Among them, the DaViT network model uses window attention to limit the scope of calculation, thereby reducing the computational complexity of global self-attention. This local attention mechanism helps the model capture local features of the image. In addition to attention in the spatial dimension, the DaViT network model also considers attention in the channel dimension, which helps the model learn the correlation between different feature channels, thereby strengthening the modeling of global information.
[0068] In order to solve the problem that the sample data in the first sample image set may have class imbalance, the original loss function of the DaViT network model is modified in this embodiment.
[0069] Specifically, the total loss function of the style classification model is determined by the cross entropy loss function and the center loss function, as shown in formula (1).
[0070] L total =L CE +αL center (1)
[0071] Among them, L total represents the total loss function of the style classification model, L CE represents the cross entropy loss function, L centerRepresents the center loss function, α represents the weight of the center loss function, which is used to enhance the intra-class consistency of images of different styles.
[0072] In the classification task, the cross entropy loss function is used to measure the difference between the predicted probability distribution of the style classification model and the probability distribution of the true label. The true label is usually represented as a one-hot encoding, that is, except for the position of the true category, all other positions are 0.
[0073] Furthermore, if the model predicts a low probability for the correct category, the loss value of the cross entropy loss function will be very high. At this time, the model weights are adjusted during the back propagation process to reduce the loss. When the model predicts the probability of the correct category close to 1, the loss value of the cross entropy loss function tends to 0, indicating that the model's prediction is consistent with the true label.
[0074] The center loss function first calculates the square of the difference between the predicted value and the actual value of all samples, then adds the squared difference and finally divides it by the number of sample images N. This gives an indicator that reflects the prediction accuracy of the model, namely the mean square error. The smaller the mean square error, the better the prediction effect of the model.
[0075] Specifically, the cross entropy loss function is expressed by formula (2);
[0076]
[0077] Wherein, N represents the total number of sample images in the first sample image set, C represents the number of categories in the first sample image set; i represents the sample image index in the first sample image set, i represents the category index in the first sample image set, and λ i Represents the weight value of the i-th sample image, p ij is the probability that the i-th sample image is of category j predicted by the style classification model, and ε represents the label smoothing coefficient.
[0078] Specifically, C represents the number of categories in the first sample image set, and optional C is 2. ε represents a label smoothing coefficient, which is used to prevent the style classification model from being overconfident. Specifically, in order to make the style classification model more robust, ε is used to smooth the probability value to reduce overfitting and improve the generalization ability of the model.
[0079] λ i Represents the weight value of the i-th sample image, which is used to balance the influence of different categories. ij Indicates that the i-th sample image is a one-hot vector of the true label of category i.
[0080] Specifically, the center loss function is expressed by formula (3);
[0081]
[0082] Among them, y i The one-hot vector representing the true label of the i-th sample image. i represents the feature vector of the i-th sample image before the feature extractor extracts it, f(x i ) represents the feature vector extracted by the feature extractor from the i-th sample image. The explanation of other parameters can refer to the description in the above embodiment.
[0083] The above cross entropy loss function and center loss function are added to the DaViT network model, and the appropriate parameters are adjusted for model training. After training, a style classification model is obtained, which can classify the image to be detected into one of the two types: real style category or non-real style. In other words, the style classification model can be used to determine whether the image to be detected is a real style category or a non-real style.
[0084] S204: Use the second sample image set to train the second network model to obtain a real image detection model.
[0085] S205: Train the third network model using the third sample image set to obtain a non-realistic image detection model.
[0086] Among them, the model structures of the second network model and the third network model are the same. In other words, the model structures of the real image detection model and the non-real image detection model are the same, and the difference between the two is that different training sample images are used, resulting in different model parameters. The model structures of the second network model and the third network model can refer to the description in the following embodiments.
[0087] The second network model is trained using real generated sample images and real generated sample images to obtain a real image detection model; the third network model is trained using non-real non-generated sample images and non-real generated sample images to obtain a real image detection model.
[0088] In the embodiment of the present application, a binary classification network for image detection of real style images is generated using real generated sample images and real non-generated sample images, and a binary classification network for generated image detection of non-real style images is trained using non-real style generated data and non-real style non-generated data. Compared with directly using 4 types of classification data to train a multi-classification network, this training scheme has a simpler decision boundary and a lower demand for training data. At the same time, special training can be performed for real style generated images to improve the detection effect of such data.
[0089] In the embodiment of the present application, a detection system for image categories with three binary classification models is obtained. When performing detection, the authenticity style of the image is first determined, and then the corresponding image detection model is sent to detect according to the authenticity style to improve the accuracy of image detection. The other two image detection models have strong robustness, and the real image detection model also has a good detection accuracy for the generated images of non-realistic styles, and vice versa, ensuring that even if the style classification is wrong, the detection accuracy can still maintain a high level.
[0090] On the basis of the above embodiments, the detection method of image categories is further optimized in the embodiments of the present application, such as Figure 4 As shown, the optimized image category detection method of the embodiment of the present application mainly includes:
[0091] S301, obtaining an image to be detected.
[0092] S302: Classify the image to be detected by using a style classification model to obtain a first target category of the image to be detected, where the first target category includes a real style category or a non-real style category.
[0093] S303: Use the image detection model corresponding to the first target category as a target detection model.
[0094] S301-S303 provided in the embodiment of the present application is the same as the execution process of S101-S103 provided in the above embodiment. For details, please refer to the description in the above embodiment, and no specific limitations are given in the embodiment of the present application.
[0095] S304: Using an image feature extraction network to extract image features of different scales in the image to be detected.
[0096] In the embodiment of the present application, the image detection model adopts an improved DaViT network model. Specifically, the DaViT-Tiny framework network is loaded using the timm library and then transformed and optimized. The DaViT-Tiny network input image size is 224×224×3. Among them, the DaViT-Tiny network combines a hybrid model of a convolutional neural network (CNN) and a Transformer.
[0097] In one possible implementation, the image feature extraction network has at least two stages, including a block embedding layer and multiple dual attention blocks. The block embedding layer is used to divide the input image into a series of non-overlapping small blocks; the dual attention block includes a spatial attention layer and a channel attention layer.
[0098] See also Figure 5As shown, the improved DaViT network model includes 4 stages, which are connected in sequence, wherein each stage includes a patch embedding layer and multiple dual attention blocks.
[0099] The block embedding layer divides the input image or feature map into a series of non-overlapping small blocks, each of which is then flattened and converted into a fixed-length vector. In other words, each small block is converted into a higher-dimensional feature vector to facilitate processing by the subsequent dual attention block.
[0100] The dual attention block consists of a spatial attention layer and a channel attention layer. The spatial attention layer focuses on which locations in the image are more important, while the channel attention layer focuses on which channels (or feature maps) of the image features are more critical. The spatial attention layer and the channel attention layer can be viewed as weighted operations on feature maps in two different dimensions. The spatial attention layer generates a weight matrix of the same size as the input feature map to highlight important areas in the image, while the channel attention layer generates a weight vector that matches the number of channels to emphasize or suppress specific feature channels. The spatial attention layer and the channel attention layer are used alternately to capture both global and local features of the image.
[0101] Specifically, Figure 5 As shown in Figure 1, the output image size of the first stage (stage 1) is 56×56, and the parameters of the block embedding layer are convolution kernel 7, stride 4, pixel width added to the edge of the image (pad) 3, and output channel number C. 1 is 96. vin.ze.7x7 represents a 7x7 local attention window used by the dual attention block in the first stage, P w =49 The local attention window includes 49 pixels. h 1 =N g 1 =3 means that the dual attention block in the first stage includes 3 spatial attention layers and 3 channel attention layers; C h 1 =C g 1 =32 means that the output channels of each spatial attention layer and each spatial attention layer are 32.
[0102] The output size of the second stage (stage 2) is 28×28, and the parameters of the block embedding layer are convolution kernel 2, stride 2, pixel width added to the edge of the image (pad) is 0, and the number of output channels is C. 2 is 192. vin.ze.7x7 represents a 7x7 local attention window used by the dual attention block in the second stage, P w =49 The local attention window includes 49 pixels. h 2 =N g 2 =6 means that the dual attention block in the second stage includes 6 spatial attention layers and 6 channel attention layers; C h 2 =C g 2 =32 means that the output channels of each spatial attention layer and each spatial attention layer are 32.
[0103] The output size of the image in the third stage (stage3) is 14×14, and the parameters of the block embedding layer are convolution kernel 2, stride 2, pixel width added to the edge of the image (pad) 0, and output channel number C. 3 is 384. vin.ze.7x7 represents a 7x7 local attention window used by the dual attention block in the third stage, P w =49 The local attention window includes 49 pixels. h 3 =N g 3 =12 means that the dual attention block in the third stage includes 12 spatial attention layers and 12 channel attention layers; C h 3 =C g 3 =32 means that the output channels of each spatial attention layer and each spatial attention layer are 32.
[0104] The output size of the image in stage 4 is 7×7, and the parameters of the block embedding layer are convolution kernel 2, stride 2, pixel width added to the edge of the image (pad) 0, and output channel number C. 4 is 168. vin.ze.7x7 represents a 7x7 local attention window used by the dual attention block in the fourth stage, P w =49 The local attention window includes 49 pixels. 4=Ng 4 =24 means that the dual attention block in the fourth stage includes 24 spatial attention layers and 24 channel attention layers; C h 3 =C g 3 =32 means that the output channels of each spatial attention layer and each spatial attention layer are 32.
[0105] The network parameters adopted by the spatial attention layers and channel attention layers at different stages are different, so that the stacked dual attention modules can capture the layout features and global features of the image.
[0106] S305 . For each scale of image features, extract texture features from the image features using a texture feature extraction network.
[0107] For AIGC-generated images with realistic styles, they usually have obvious edges and details in order to simulate realistic styles; for AIGC-generated images with non-realistic styles, they usually have texture and brushstroke features of artistic styles. Therefore, texture features play an important role in the detection of AIGC-generated images. Its dual attention mechanism can ensure that the image detection model pays attention to both the global and local features of the image. On this basis, a texture feature extraction network is added to extract the texture features of the image separately, and finally merged with the features extracted by the DaViT network to improve the detection effect.
[0108] In one possible implementation, Figure 6 As shown, the texture feature extraction network includes a first convolution layer 61, a Gram matrix 62, a second convolution layer 63, a third convolution layer 64 and a first average pooling layer 65 which are connected in sequence; wherein, the convolution kernels of the first convolution layer, the second convolution layer and the third convolution layer are the same, the number of channels of the first convolution layer and the third convolution layer are the same, and the step size of the number of channels of the first convolution layer and the second convolution layer are the same.
[0109] Specifically, the first convolution layer 61 is a conv convolution, the convolution kernel of which is 3, the number of channels is 48, and the step size is 1. The second convolution layer 63 is a conv convolution, the convolution kernel of which is 3, the number of channels is 24, and the step size is 2. The third convolution layer 64 is a conv convolution, the convolution kernel of which is 3, the number of channels is 24, and the step size is 4.
[0110] S306 , using a fusion layer to fuse image features of different scales and texture features corresponding to image features of each scale to obtain fused features.
[0111] In the embodiment of the present application, an image feature extraction network including four stages is used as an example for explanation, such as Figure 7As shown, the image to be detected is input as an input image into the input layer 71 in the image detection model, and the size of the input image is 224×224×224. Then, after feature extraction in the first stage 72, a first feature map of 56×56×94 is obtained. Then, the first feature map is input into the second stage 73 for feature extraction to obtain a second feature map of 28×28×192. The first feature map is input into the first texture feature extraction network 76 to obtain a first texture feature of 1×1×48. The second feature map is input into the second texture feature extraction network 77 to obtain a second texture feature of 1×1×48. The first texture feature and the second texture feature are merged to obtain a third texture feature.
[0112] The second feature map is input into the third stage 74 for feature extraction to obtain a third feature map of 14×14×384; the third feature map is input into the third texture feature extraction network 78 to obtain a fourth texture feature of 1×1×48; the third texture feature is combined with the fourth texture feature to obtain a fifth texture feature.
[0113] The third feature map is input to the fourth stage 75 for feature extraction to obtain a 7×7×768 fourth feature map. After the fourth feature map passes through the second average pooling layer 79 and the first fully connected (FC) layer 710, the fifth feature is obtained.
[0114] After the fifth feature and the fifth texture feature are combined and passed through the second FC layer 711 and the output layer 712, the second target category of the image to be detected is obtained.
[0115] S307 , using a fully connected layer to perform binary classification processing on the fused features to obtain a second target category of the image to be detected.
[0116] The fused features are sent to the fully connected layer for binary classification, and the classification function has a label-smoothed cross entropy binary classification loss function.
[0117] An output layer is also included after the fully connected layer. The output layer adopts the Softmax function, which is used to convert the original score into the probability that the image to be detected belongs to the generated class image and the probability that it belongs to the non-generated class image.
[0118] In order to verify the effectiveness of the network, a generated image was visualized for category activation and comparison, as follows Figure 8As shown, the left picture is an image generated using an existing generative model, the middle picture is the category activation map of DaViT, and the right picture is the category activation map of the image category detection system provided in the embodiment of the present application. It can be seen that compared with DaViT, the image category detection system pays more attention to the edges and texture areas of the image due to the addition of the texture feature extraction network, which helps to better distinguish the generated image from the real image.
[0119] Fig. 9 is a schematic diagram of the structure of a detection device for an image category in an embodiment of the present application, such as Fig. 9 As shown, the image category detection device 90 provided in the embodiment of the present application mainly includes: an image acquisition module 91, a style detection module 92, a model selection module 93 and a generation detection module 94.
[0120] Among them, the image acquisition module 91 is used to acquire the image to be detected; the style detection module 92 is used to classify the image to be detected by using the style classification model to obtain the first target category corresponding to the image to be detected, and the first target category includes a real style category or a non-real style category; the model selection module 93 is used to use the image detection model corresponding to the first target category as the target detection model; the generation detection module 94 is used to detect the image to be detected by using the target image detection model to obtain the second target category corresponding to the image to be detected, and the second target category includes a generated category or a non-generated category.
[0121] The embodiment of the present application provides a device for detecting image categories, which is used to execute the following process: obtaining an image to be detected; classifying the image to be detected using a style classification model to obtain a first target category of the image to be detected, the first target category including a real style category or a non-real style category; using an image detection model corresponding to the first target category as a target detection model; detecting the image to be detected using the target image detection model to obtain a second target category of the image to be detected, the second target category including a generated category or a non-generated category. Since the style category of the image to be detected is first determined using the style classification model, and then sent to the corresponding image detection model according to the style category to determine whether the image to be detected is a generated image, the accuracy of image category detection is improved.
[0122] In a possible implementation, it also includes: a model training module, used to obtain sample images, the sample images include: real non-generated sample images, real generated sample images, non-real generated sample images, and non-real non-generated sample images; dividing the sample images into a first sample image set, a second sample image set, and a third sample image set, wherein the first sample image set includes real generated sample images and non-real generated sample images, the second sample image set includes real generated sample images and real generated sample images, and the third sample image set includes non-real non-generated sample images and non-real generated sample images; using the first sample image set to train the first network model to obtain a style classification model; using the second sample image set to train the second network model to obtain a real image detection model; using the third sample image set to train the third network model to obtain a non-real image detection model.
[0123] In one possible implementation, the model selection module 93 is specifically used to select the real image detection model as the target detection model when the first target category is a real style category; and to select the non-real image detection model as the target detection model when the second target category is a non-real style category.
[0124] In a possible implementation, the total loss function of the style classification model is determined by a cross entropy loss function and a center loss function, and the cross entropy loss function is expressed by the following formula:
[0125]
[0126] Wherein, N represents the total number of sample images in the first sample image set, C represents the number of categories in the first sample image set; i represents the sample image index in the first sample image set, i represents the category index in the first sample image set, and λ i Represents the weight value of the i-th sample image, p ij is the probability that the i-th sample image is of category i predicted by the style classification model, ε represents the label smoothing coefficient, and y ij Indicates that the i-th sample image is a one-hot vector of the true label of category i;
[0127] The center loss function is expressed by the following formula:
[0128]
[0129] y i The one-hot vector representing the authenticity label of the i-th sample image, x i represents the feature vector of the i-th sample image before the feature extractor extracts it, f(x i) represents the feature vector extracted by the feature extractor from the i-th sample image.
[0130] In one possible implementation, the target image detection model includes an image feature extraction network, a texture feature extraction network, a fusion layer and a fully connected layer; a detection module 94 is generated, which is specifically used to use the image feature extraction network to extract image features of different scales in the image to be detected; for image features of each scale, a texture feature extraction network is used to extract texture features from the image features; a fusion layer is used to fuse image features of different scales and texture features corresponding to image features of each scale to obtain fused features; and a fully connected layer is used to perform binary classification on the fused features to obtain a second target category of the image to be detected.
[0131] In one possible implementation, the texture feature extraction network includes a first convolutional layer, a Gram matrix, a second convolutional layer, a third convolutional layer and a first average pooling layer connected in sequence; wherein the convolution kernels of the first convolutional layer, the second convolutional layer and the third convolutional layer are the same, the number of channels of the first convolutional layer and the third convolutional layer are the same, and the step size of the number of channels of the first convolutional layer and the second convolutional layer is the same.
[0132] In one possible implementation, the image feature extraction network has at least two stages, each of which includes a block embedding layer and a plurality of dual attention blocks, wherein the block embedding layer is used to divide the input image into a series of non-overlapping small blocks; and the dual attention block includes a spatial attention layer and a channel attention layer.
[0133] The image category detection device provided in the embodiment of the present application can execute the image category detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0134] Fig.10 is a schematic diagram of the structure of an electronic device provided in this embodiment. The electronic device may include an image category detection device, such as Fig.10 As shown, the electronic device 1000 includes a processor 1010, a memory 1020, an input device 1030, and an output device 1040; the number of processors 1010 in the electronic device can be one or more. Fig.10 A processor 1010 is taken as an example; the processor 1010, the memory 1020, the input device 1030 and the output device 1040 in the electronic device can be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.
[0135] The memory 1020, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the image category detection method in the embodiment of the present invention. The processor 1010 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 1020, that is, implements the image category detection method provided in the embodiment of the present invention.
[0136] The memory 1020 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 1020 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 1020 may further include a memory remotely arranged relative to the processor 1010, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0137] The input device 1030 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device, and may include a keyboard, a mouse, etc. The output device 1040 may include a display device such as a display screen.
[0138] This embodiment further provides a storage medium containing computer executable instructions, and when the computer executable instructions are executed by a computer processor, they are used to implement the image category detection method provided by the embodiment of the present invention.
[0139] Of course, the computer executable instructions of a storage medium including computer executable instructions provided by an embodiment of the present invention are not limited to the operations of the method described above, but can also execute related operations in the image category detection method provided by any embodiment of the present invention.
[0140] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.
[0141] It is worth noting that in the embodiment of the above-mentioned image category detection device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.
[0142] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0143] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for detecting image categories, characterized in that: The method comprises: Acquire the image to be detected; Classifying the image to be detected by using a style classification model to obtain a first target category of the image to be detected, where the first target category includes a real style category or a non-real style category; Using the image detection model corresponding to the first target category as the target image detection model; The target image detection model is used to detect the image to be detected to obtain a second target category of the image to be detected, where the second target category includes a generated category or a non-generated category.
2. The method according to claim 1, characterized in that: Also includes: Acquire sample images, wherein the sample images include: real non-generated sample images, real generated sample images, unreal generated sample images, and unreal non-generated sample images; Dividing the sample images into a first sample image set, a second sample image set and a third sample image set, wherein the first sample image set includes real generated sample images and non-real generated sample images, the second sample image set includes real generated sample images and real generated sample images, and the third sample image set includes non-real non-generated sample images and non-real generated sample images; Using the first sample image set to train a first network model to obtain a style classification model; Using the second sample image set to train a second network model to obtain a real image detection model; The third network model is trained using the third sample image set to obtain a non-realistic image detection model.
3. The method according to claim 2, characterized in that Using the image detection model corresponding to the first target category as a target detection model includes: When the first target category is a real style category, selecting the real image detection model as the target detection model; When the second target category is a non-realistic style category, the non-realistic image detection model is selected as the target detection model.
4. The method according to any one of claims 1 to 3, characterized in that: The total loss function of the style classification model is determined by a cross entropy loss function and a center loss function, and the cross entropy loss function is expressed by the following formula: Wherein, N represents the total number of sample images in the first sample image set, C represents the number of categories in the first sample image set; i represents the sample image index in the first sample image set, i represents the category index in the first sample image set, and λ i Represents the weight value of the i-th sample image, p ij is the probability that the i-th sample image is of category i predicted by the style classification model, ε represents the label smoothing coefficient, and y ij Indicates that the i-th sample image is a one-hot vector of the true label of category i; The center loss function is expressed by the following formula: y i The one-hot vector representing the authenticity label of the i-th sample image, x i represents the feature vector of the i-th sample image before the feature extractor extracts it, f(x i ) represents the feature vector extracted by the feature extractor from the i-th sample image.
5. The method according to any one of claim 1, characterized in that The target image detection model includes an image feature extraction network, a texture feature extraction network, a fusion layer and a fully connected layer; Detecting the image to be detected by using the target image detection model to obtain a second target category of the image to be detected, including: Extracting image features of different scales in the image to be detected using the image feature extraction network; For each scale of image features, extracting texture features from the image features using a texture feature extraction network; The fusion layer is used to fuse the image features of different scales and the texture features corresponding to the image features of each scale to obtain the fusion features; The fused features are subjected to binary classification processing using a fully connected layer to obtain a second target category of the image to be detected.
6. The method according to claim 5, characterized in that The texture feature extraction network includes a first convolutional layer, a Gram matrix, a second convolutional layer, a third convolutional layer and a first average pooling layer connected in sequence; Among them, the convolution kernels of the first convolution layer, the second convolution layer and the third convolution layer are the same, the number of channels of the first convolution layer and the third convolution layer are the same, and the step size of the number of channels of the first convolution layer and the second convolution layer are the same.
7. The method according to claim 5, characterized in that The image feature extraction network has at least two stages, each of which includes a block embedding layer and a plurality of dual attention blocks. The block embedding layer is used to divide the input image into a series of non-overlapping small blocks; the dual attention block includes a spatial attention layer and a channel attention layer.
8. A device for detecting image categories, characterized in that: The method comprises: An image acquisition module, used for acquiring an image to be detected; A style detection module, used to classify the image to be detected by using a style classification model to obtain a first target category corresponding to the image to be detected, where the first target category includes a real style category or a non-real style category; A model selection module, used to use the image detection model corresponding to the first target category as the target detection model; A generation detection module is used to detect the image to be detected using the target image detection model to obtain a second target category corresponding to the image to be detected, where the second target category includes a generated category or a non-generated category.
9. An electronic device, characterized in that: The device comprises: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image category detection method according to any one of claims 1 to 7.
10. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for detecting the image category as described in any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Image sentiment analysis method and related device
CN112488214A
Structure and texture feature guided double-encoder image restoration method
CN116523985A
Image semantic segmentation method and device, electronic equipment and storage medium
CN117011520A
Character recognition method, electronic equipment and storage medium
CN117935245A
AIGC image detection model training method and AIGC image detection method
CN118230057A
