Method and device for screening AI generated pictures based on CLIP

Abstract

The invention provides a method and a device for screening AI generated pictures based on CLIP. The method comprises the following steps: setting conditions and generating a plurality of images through an AI model; encoding the set condition through the CLIP to obtain a first encoding value, and encoding each image through the CLIP to obtain a plurality of second encoding values; the cosine similarity of each second coding value and the first coding value is calculated, the image corresponding to the second coding value with the highest cosine similarity is taken out, the image is output, the whole AI drawing process does not need manual participation, the drawing result can be automatically detected, and the drawing efficiency is greatly improved.

Description

Technical Field

[0001] The present invention relates to a method and device for screening AI-generated pictures based on CLIP. Background Art

[0002] When using AI large models for drawing, whether it is text-to-image or image-to-image, the results randomly generated by AI often require human participation in screening or multiple runs to finally obtain the ideal drawing effect. Therefore, the AI ​​drawing process cannot be fully automated, and the generated image effect is not stable enough, resulting in a significant reduction in production efficiency. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a method and device for screening AI-generated pictures based on CLIP. The entire AI drawing process does not require human participation and the drawing results can be automatically detected, which greatly improves the drawing efficiency.

[0004] In a first aspect, the present invention provides a method for screening AI-generated images based on CLIP, comprising:

[0005] Step 1: Set conditions to generate multiple images through the AI ​​model;

[0006] Step 2: Encode the set condition through CLIP to obtain a first encoding value, and encode each image through CLIP to obtain multiple second encoding values;

[0007] Step 3: Calculate the cosine similarity between each second code value and the first code value, take out the image corresponding to the second code value with the highest cosine similarity, and output the image.

[0008] In a second aspect, the present invention provides a device for screening AI-generated images based on CLIP, comprising:

[0009] The generation module sets conditions to generate multiple images through the AI ​​model;

[0010] An encoding module, encoding the set condition through CLIP to obtain a first encoding value, and encoding each image through CLIP to obtain a plurality of second encoding values;

[0011] The output module calculates the cosine similarity between each second coding value and the first coding value, takes out the image corresponding to the second coding value with the highest cosine similarity, and outputs the image.

[0012] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0013] 1. The entire AI drawing process does not require human intervention and can automatically detect the drawing results, greatly improving the drawing efficiency.

[0014] 2. This method is added when AI is drawing, which can effectively remove pictures that do not meet expectations, thereby improving the quality of AI drawing.

[0015] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0017] Figure 1 is a flow chart of the method in Embodiment 1 of the present invention;

[0018] Figure 2 It is a schematic diagram of the structure of the device in the second embodiment of the present invention. DETAILED DESCRIPTION

[0019] The technical solution in the embodiment of the present application has the following general idea:

[0020] Overview of implementation principle: CLIP encode the AI-generated image, then CLIP encode the prompt word or reference image, calculate the relevance score, and then sort the scores from high to low. A high relevance score can be judged as meeting the expectations of the prompt word or reference image.

[0021] Specific implementation of application scenarios:

[0022] 1. Text-to-Image

[0023] step:

[0024] Input the prompt word (such as "a scenic mountain landscape").

[0025] Generate corresponding images through the existing AI large model.

[0026] Use CLIP to encode the generated image and the original prompt word separately.

[0027] Calculate the cosine similarity between the two. The value range of cosine similarity is -1 to 1. The closer the cosine similarity is to -1, the higher the oppositeness is. A cosine similarity of 0 means that they have no directional similarity or correlation. The closer the cosine similarity is to 1, the higher the similarity is.

[0028] If the similarity score is high, it means that the generated image meets the expectations of the prompt word.

[0029] Application scenarios: Evaluate whether the generated image accurately reflects the description in the prompt word, such as images generated by models such as DALL-E and Stable Diffusion.

[0030] 2. Image-to-Image

[0031] step:

[0032] Provide a reference image (such as a black and white photo).

[0033] Generate target images (such as colorized photos) through existing AI large models.

[0034] The generated image and the original reference image are encoded separately using CLIP.

[0035] Calculate the cosine similarity between the two, that is, the color style similarity.

[0036] In order to adapt to the application scenario of AI line drawing, the image two-dimensional similarity algorithm is used. Because image similarity has structural similarity (marked as S) and color style similarity (marked as C), it is necessary to calculate the similarity from these two dimensions separately, and then weight them (the weighting ratio can be dynamically adjusted according to the application scenario) to obtain the final similarity. When calculating structural similarity, this solution uses the SSIM algorithm, which is more in line with the human visual system's perception of image quality, especially the sensitivity to structural information.

[0037] The SSIM calculation steps are as follows:

[0038] Grayscale processing: If the images are in color, both images need to be converted into grayscale images first, because SSIM is usually calculated for single-channel images (grayscale images).

[0039] Sliding window calculation: SSIM is a metric based on a local window. SSIM is calculated in units of local windows and then averaged to obtain the overall SSIM indicator.

[0040] Specific calculation process:

[0041] Brightness comparison: Calculate the similarity of the average brightness of two images.

[0042] Contrast comparison: Calculate the similarity of the contrast (i.e. standard deviation) of two images.

[0043] Structural comparison: Compare the structures based on the brightness mean and contrast of the images.

[0044] The above three parts are combined into the SSIM index.

[0045] Final similarity score: S*n+C*m

[0046] in:

[0047] S represents structural similarity;

[0048] C represents color similarity;

[0049] n, m are weighting coefficients, which can be adjusted according to specific application scenarios. For example, n accounts for 70-80% of the weight of AI line drawing; for other drawings, m accounts for 60-70% of the weight; line drawings are images that use lines to outline the contours, shapes and structures during the drawing process. Common ones include: (sketch) images, black and white line outlines, character sketches; AI models can refer to line drawings for coloring or painting, which is collectively referred to as AI line drawings.

[0050] If the similarity score is high, it means that the generated image retains the characteristics of the original image to some extent and conforms to the expected transformation.

[0051] Application scenarios: Evaluate the effects of tasks such as image style transfer, image restoration, and image enhancement, such as images generated using models such as StyleGAN, Pix2Pix, and CycleGAN.

[0052] Embodiment 1

[0053] like Figure 1 As shown, this embodiment provides a method for screening AI-generated images based on CLIP, including:

[0054] Step 1: Set conditions to generate multiple images through the AI ​​model;

[0055] Step 2: Encode the set condition through CLIP to obtain a first encoding value, and encode each image through CLIP to obtain multiple second encoding values;

[0056] Step 3: Calculate the cosine similarity between each second code value and the first code value, take out the image corresponding to the second code value with the highest cosine similarity, and output the image.

[0057] In this implementation, preferably, the step 1 is specifically: generating multiple images through an AI model based on the text; or generating multiple images through an AI model based on the original image.

[0058] In this implementation, preferably, the step 2 is specifically: encoding the text or the original image through CLIP to obtain a first encoding value, and encoding each image through CLIP to obtain multiple second encoding values.

[0059] In this implementation, preferably, step 3 is specifically as follows: if multiple images are generated according to the text through the AI ​​model, the cosine similarity between each second coding value and the first coding value is calculated, and the image corresponding to the second coding value with the highest cosine similarity is output;

[0060] If multiple images are generated through the AI ​​model based on the original image, the cosine similarity between each second coding value and the first coding value is calculated, and the structural similarity of the original image and the multiple images is calculated through the SSIM algorithm; then the final similarity between the original image and the multiple images is calculated through S*n+C*m, where S represents structural similarity; C represents cosine similarity; n, m are weighting coefficients; and the corresponding image with the highest final similarity is output.

[0061] In this implementation, preferably, when drawing AI line drafts, the weight of n accounts for 70-80%; when drawing other drawings, the weight of m accounts for 60-70%; m+n=1.

[0062] Based on the same inventive concept, the present application also provides a device corresponding to the method in Example 1, see Example 2 for details.

[0063] Embodiment 2

[0064] like Figure 2 As shown, in this embodiment, a device for screening AI-generated pictures based on CLIP is provided, including:

[0065] The generation module sets conditions to generate multiple images through the AI ​​model;

[0066] An encoding module, encoding the set condition through CLIP to obtain a first encoding value, and encoding each image through CLIP to obtain a plurality of second encoding values;

[0067] The output module calculates the cosine similarity between each second coding value and the first coding value, takes out the image corresponding to the second coding value with the highest cosine similarity, and outputs the image.

[0068] In this implementation, preferably, the generation module specifically generates multiple images through an AI model based on text; or generates multiple images through an AI model based on the original image.

[0069] In this implementation, preferably, the encoding module specifically: encodes the text or the original image through CLIP to obtain a first encoding value, and encodes each image through CLIP to obtain multiple second encoding values.

[0070] In this implementation, preferably, the output module is specifically: if multiple images are generated according to the text through the AI ​​model, the cosine similarity between each second coding value and the first coding value is calculated, and the image corresponding to the second coding value with the highest cosine similarity is output;

[0071] If multiple images are generated through the AI ​​model based on the original image, the cosine similarity between each second coding value and the first coding value is calculated, and the structural similarity of the original image and the multiple images is calculated through the SSIM algorithm; then the final similarity between the original image and the multiple images is calculated through S*n+C*m, where S represents structural similarity; C represents cosine similarity; n, m are weighting coefficients; and the corresponding image with the highest final similarity is output.

[0072] In this implementation, preferably, when drawing AI line drafts, the weight of n accounts for 70-80%; when drawing other drawings, the weight of m accounts for 60-70%; m+n=1.

[0073] Since the device introduced in the second embodiment of the present invention is a device used to implement the method of the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the device based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0074] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for screening AI-generated images based on CLIP, characterized in that: include: Step 1: Set conditions to generate multiple images through the AI ​​model; Step 2: Encode the set condition through CLIP to obtain a first encoding value, and encode each image through CLIP to obtain multiple second encoding values; Step 3: Calculate the cosine similarity between each second code value and the first code value, take out the image corresponding to the second code value with the highest cosine similarity, and output the image.

2. A method for generating pictures based on CLIP screening AI according to claim 1, characterized in that: The step 1 specifically includes: generating multiple images through an AI model based on the text; or generating multiple images through an AI model based on the original image.

3. A method for generating pictures based on CLIP screening AI according to claim 2, characterized in that: The step 2 specifically includes: encoding the text or the original image through CLIP to obtain a first encoding value, and encoding each image through CLIP to obtain multiple second encoding values.

4. A method for generating pictures based on CLIP screening AI according to claim 2, characterized in that: The step 3 is specifically as follows: if multiple images are generated according to the text through the AI ​​model, the cosine similarity between each second coding value and the first coding value is calculated, and the image corresponding to the second coding value with the highest cosine similarity is output; If multiple images are generated through the AI ​​model based on the original image, the cosine similarity between each second coding value and the first coding value is calculated, and the structural similarity of the original image and the multiple images is calculated through the SSIM algorithm; then the final similarity between the original image and the multiple images is calculated through S*n+C*m, where S represents structural similarity; C represents cosine similarity; n, m are weighting coefficients; and the corresponding image with the highest final similarity is output.

5. A method for generating pictures based on CLIP screening AI according to claim 4, characterized in that: When drawing AI line drafts, the weight of n accounts for 70-80%; in other drawings, the weight of m accounts for 60-70%; m+n=1.

6. A device for screening AI-generated images based on CLIP, characterized in that: include: The generation module sets conditions to generate multiple images through the AI ​​model; An encoding module, encoding the set condition through CLIP to obtain a first encoding value, and encoding each image through CLIP to obtain a plurality of second encoding values; The output module calculates the cosine similarity between each second coding value and the first coding value, takes out the image corresponding to the second coding value with the highest cosine similarity, and outputs the image.

7. The device for generating pictures based on CLIP screening AI according to claim 6, characterized in that: The generation module specifically generates multiple images through an AI model based on text; or generates multiple images through an AI model based on the original image.

8. The device for generating pictures based on CLIP screening AI according to claim 7, characterized in that: The encoding module specifically encodes the text or the original image through CLIP to obtain a first encoding value, and encodes each image through CLIP to obtain multiple second encoding values.

9. The device for generating pictures based on CLIP screening AI according to claim 7, characterized in that: The output module is specifically: if multiple images are generated according to the text through the AI ​​model, the cosine similarity between each second coding value and the first coding value is calculated, and the image corresponding to the second coding value with the highest cosine similarity is output; If multiple images are generated through the AI ​​model based on the original image, the cosine similarity between each second coding value and the first coding value is calculated, and the structural similarity of the original image and the multiple images is calculated through the SSIM algorithm; then the final similarity between the original image and the multiple images is calculated through S*n+C*m, where S represents structural similarity; C represents cosine similarity; n, m are weighting coefficients; and the corresponding image with the highest final similarity is output.

10. The device for generating pictures based on CLIP screening AI according to claim 9, characterized in that: When drawing AI line drafts, the weight of n accounts for 70-80%; in other drawings, the weight of m accounts for 60-70%; m+n=1.

Images