Image optimization method based on CLIP model

Abstract

The invention discloses an image optimization method based on a CLIP model, and the method comprises the steps: S1, carrying out the coding initial processing of characters and images, inputting a character description and an initial image code into a CLIP model, and obtaining a corresponding semantic vector and a visual vector; s2, cross-modal contrast learning: based on positive and negative sample contrast and an adaptive attention mechanism, aligning the semantic vector and the visual vector obtained in the S1; s3, self-supervised learning is carried out, and precision improvement of semantic-to-visual mapping is completed through self-supervised learning; s4, feature refinement generation: based on the optimized semantic and visual feature vectors, generating a preliminary concept conforming to description; and S5, image output: repeating the steps S1 to S4 until a preset matching value is met, and outputting a conception image matched with the text description. According to the image optimization method based on the CLIP model, semantic mapping from characters to images is improved, so that the semantic mapping is more in line with the expectation of natural language description, and a visual conception graph with rich details is generated.

Description

Technical Field

[0001] The present application relates to the field of AI visual image generation, and in particular to an image optimization method based on a CLIP model. Background Art

[0002] The CLIP model (Contrastive Language–Image Pretraining) is a multimodal model developed by OpenAI that can embed text descriptions and images into a shared feature space. However, CLIP is not precise enough in details during the generation process, especially in the expression of complex scenes, fine textures, and light and shadow relationships.

[0003] There is an urgent need for a new method to make the generated images more consistent with the expectations of natural language descriptions and generate detailed visual concept maps. Summary of the invention

[0004] The purpose of this application is to overcome the problems of the prior art and disclose an image optimization method based on the CLIP model to generate a visual concept map with rich details.

[0005] The purpose of this application is achieved through the following technical solutions:

[0006] A CLIP model-based image optimization method, the CLIP model-based image optimization method comprising:

[0007] S1: Initial processing of text and image coding: input text description and preliminary image coding into the CLIP model to obtain the corresponding semantic vector and visual vector;

[0008] S2: Cross-modal contrastive learning, based on positive and negative sample comparison and adaptive attention mechanism, aligns the semantic vector and visual vector obtained in S1;

[0009] S3: Self-supervised learning, which improves the accuracy of semantic-to-visual mapping through self-supervised learning;

[0010] S4: Feature refinement generation, based on the optimized semantic and visual feature vectors, generates preliminary ideas that match the description;

[0011] S5: Image output, repeating steps S1 to S4 until a preset matching value is met, and outputting a conceptual image matching the text description.

[0012] According to a preferred embodiment, step S1 includes: encoding the input text based on the text encoder of the CLIP model to generate semantic vectors at the vocabulary level and sentence level;

[0013] Furthermore, the visual encoder based on the CLIP model processes the input image, extracts its visual features, and generates a visual vector consistent with the semantic vector space.

[0014] According to a preferred embodiment, step S2 includes: using P image pairs corresponding to the matched text as positive samples and N image pairs corresponding to the unmatched text as negative samples, and shortening the embedding distance of the positive sample pairs and pushing the distance of the negative sample pairs farther by contrast loss;

[0015] And the sensitivity adjustment of contrast loss is completed by introducing temperature parameters.

[0016] According to a preferred embodiment, step S2 further includes: adding an adaptive attention mechanism between the semantic vector and the visual vector so as to focus on preset features of the text and the image during the comparison process.

[0017] According to a preferred embodiment, step S2 further includes: in the comparison loss process, using cosine similarity to measure the similarity between the semantic vector and the visual vector, and calculating the loss based on the similarity.

[0018] According to a preferred embodiment, in step S3, the self-supervised learning process includes:

[0019] The masked language model randomly masks some words in the input semantic vector, so that the masked language model can infer image features under incomplete text information;

[0020] The image mask model randomly masks some pixel blocks in the input image vector, allowing the model to restore visual details under the guidance of text.

[0021] According to a preferred embodiment, step S4 comprises:

[0022] Based on the optimized semantic and visual feature vectors, a preliminary visual feature map is generated based on the generative adversarial network GAN or diffusion model;

[0023] And use multi-layer convolution to gradually add color, light, shadow, and texture details to the generated image;

[0024] And through multiple iterations of generation and feedback, the image details are optimized to generate a preliminary image concept that meets the description.

[0025] The aforementioned main scheme of the present application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection in the present application. After understanding the scheme of the present application, those skilled in the art can understand that there are multiple combinations based on the prior art and common knowledge, all of which are technical schemes to be protected by the present application, and they are not exhaustively listed here.

[0026] Beneficial effects of this application:

[0027] The image optimization method based on the CLIP model in this application generates a detailed visual concept map by improving the semantic mapping from text to image to make it more consistent with the expectations of natural language description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is the CLIP model architecture diagram of this application. DETAILED DESCRIPTION

[0029] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0030] It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. In addition, the present application should point out that in the present application, if the specific structure, connection relationship, position relationship, power source relationship, etc. are not specifically written, the structure, connection relationship, position relationship, power source relationship, etc. involved in the present application are all known by those skilled in the art based on the existing technology without creative work.

[0031] Embodiment 1:

[0032] refer to Figure 1 As shown, the present application discloses an image optimization method based on the CLIP model, and the image optimization method based on the CLIP model includes the following steps.

[0033] Step S1: Initial processing of text and image coding: input text description and preliminary image coding into the CLIP model to obtain corresponding semantic vectors and visual vectors.

[0034] Step S1 is implemented based on the text encoding module and the image encoding module. The text encoding module encodes the input text content into a semantic vector to extract key information. The image encoding module encodes the input image or reference visual features into a visual vector.

[0035] Specifically, CLIP's text encoder (based on Transformer) is used to encode the text input and generate semantic vectors at the word level and sentence level. The output of this stage is the semantic vector related to the text description (such as scene, object, emotion, etc.). In addition, CLIP's visual encoder (based on ResNet or Vision Transformer) is used to process the input image, extract its visual features, and generate a visual vector consistent with the semantic vector space.

[0036] Step S2: Cross-modal contrastive learning, based on positive and negative sample comparison and adaptive attention mechanism, aligns the semantic vector and visual vector obtained in S1.

[0037] Preferably, step S2 adopts chain positive-negative sample comparison, including: in the training data, matching texts correspond to P image pairs as positive samples, and mismatching texts correspond to N image pairs as negative samples, and the embedding distance of the positive sample pairs is shortened and the distance of the negative sample pairs is pushed away through contrast loss.

[0038] Furthermore, the sensitivity adjustment of contrast loss is completed by introducing a temperature parameter. The temperature parameter (usually in the range of 0.05-0.1) is introduced to adjust the sensitivity of contrast loss, so that the model can more smoothly push or pull sample pairs between different modes.

[0039] Step S2 also includes: adding an adaptive attention mechanism between the semantic vector and the visual vector, so that the preset features of the text and image are focused during the comparison process to optimize the multimodal alignment effect.

[0040] Preferably, in step S2, the cosine similarity is used to measure the similarity between the semantic vector and the visual vector, and the loss is calculated based on the similarity. The higher the similarity, the more matched the positive sample pair is, and the model will further narrow the distance between them; the lower the similarity, the farther away they are.

[0041] In step S2, the positive sample pairs are: the correct text-image pairs in the training data are used as positive sample pairs, such as "cat on the grass" and images containing cats and grass. The negative sample pairs are: for each text description, some unrelated images are randomly paired as negative sample pairs, and vice versa. These mismatched sample pairs are used to guide the model to increase the distance in the feature space.

[0042] The sample pair generation strategy is:

[0043] (1) Random negative samples: Randomly select image and text combinations from different categories to generate mismatched sample pairs.

[0044] (2) Category-perturbed negative samples: negative sample pairs are selected from categories that are semantically close but do not completely match the description, such as “dog on the beach” and “cat on the beach”.

[0045] (3) Partially similar negative samples: Generate negative samples for images with similar scenes or features to increase the model’s discrimination ability.

[0046] Contrastive loss formula: Contrastive loss or InfoNCE loss with temperature parameter is usually used.

[0047] The formula is as follows:

[0048]

[0049] Where: x i Represents the vector of the i-th text description, y i Represents the image vector that matches it. sim(x i ,y i ) represents the similarity function, usually cosine similarity. τ is the temperature parameter, which is used to balance the contrast between different sample pairs.

[0050] y i→N =(y i ,.....,y N ) indicates that the text corresponds to N matching images, forming a linked list according to the attention scores.

[0051] Preferably, step S2 can be implemented based on a cross-modal contrastive learning module. By using a contrastive learning method, the similarity between text and visual feature vectors is enhanced, so that matching text-image pairs are closer in the embedding space.

[0052] The cross-modal contrastive learning module optimizes the model weights through back propagation and gradient descent algorithms based on the calculated contrastive loss. It is usually trained with large batches of data, and by comparing more positive and negative sample pairs, the module learns cross-modal text and image features on a wide range of data. In each round of iteration, the weights are updated according to the contrastive loss of the positive and negative sample pairs, and the embedding space of text and images is continuously adjusted to gradually optimize them in the feature space.

[0053] After multiple rounds of training, the cross-modal contrastive learning module can optimize the alignment effect of the text-image embedding space. The final output model can generate highly matching image features that match the input text description in the visual feature space. The optimized embedding space has the following characteristics:

[0054] Semantic consistency: Matching text-image pairs have high similarity in the embedding space, ensuring consistency between descriptions and generated images.

[0055] Enhanced robustness: For diverse and complex text descriptions (such as emotions, lighting, location, and other details), the model has stronger recognition capabilities and can capture detailed information more accurately.

[0056] Efficient generation: When generating images, the model can directly generate ideas based on the optimized embedding space, thereby accelerating the image generation process.

[0057] Step S3: self-supervised learning, the accuracy of semantic-to-visual mapping is improved through self-supervised learning. In step S3, the self-supervised learning process includes:

[0058] The masked language model (MLM) is used to randomly mask out some words in the input semantic vector, so that the masked language model can infer image features under incomplete text information.

[0059] The image mask model MIM randomly masks some pixel blocks in the input image vector, allowing the model to restore visual details under the guidance of text.

[0060] Preferably, step S3 can be implemented by a self-supervised learning optimization module, taking key words (such as color, position, light and shadow information, etc.) of the text description as the generation target, and allowing the self-supervised learning optimization module to learn to accurately restore the corresponding details when generating images.

[0061] Step S4: Feature refinement generation, based on the optimized semantic and visual feature vectors, generates a preliminary concept that conforms to the description.

[0062] Step S4 includes:

[0063] Based on the semantic and visual feature vectors optimized in step S3, a preliminary visual feature map is generated based on the generative adversarial network GAN or the diffusion model;

[0064] Multi-layer convolution is used to gradually add color, light and shadow, and texture details to the generated image; for example, for the description of "white clouds on the blue sky", the generated image will refine the shape and light and shadow of the white clouds on the blue background.

[0065] And through multiple iterations of generation and feedback, the image details are optimized to generate a preliminary image concept that meets the description.

[0066] Step S5: Image output, repeating steps S1 to S4 until a preset matching value is met, and outputting a conceptual image matching the text description.

[0067] The image optimization method based on the CLIP model in this application generates a detailed visual concept map by improving the semantic mapping from text to image to make it more consistent with the expectations of natural language description.

[0068] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An image optimization method based on CLIP model, characterized in that: The image optimization method based on the CLIP model comprises: S1: Initial processing of text and image coding: input text description and preliminary image coding into the CLIP model to obtain the corresponding semantic vector and visual vector; S2: Cross-modal contrastive learning, based on positive and negative sample comparison and adaptive attention mechanism, aligns the semantic vector and visual vector obtained in S1; S3: Self-supervised learning, which improves the accuracy of semantic-to-visual mapping through self-supervised learning; S4: Feature refinement generation, based on the optimized semantic and visual feature vectors, generates preliminary ideas that match the description; S5: Image output, repeating steps S1 to S4 until a preset matching value is met, and outputting a conceptual image matching the text description.

2. The image optimization method according to claim 1, characterized in that: Step S1 includes: encoding the input text based on the text encoder of the CLIP model to generate semantic vectors at the word level and sentence level; Furthermore, the visual encoder based on the CLIP model processes the input image, extracts its visual features, and generates a visual vector consistent with the semantic vector space.

3. The image optimization method according to claim 2, characterized in that: Step S2 includes: using P image pairs corresponding to the matched text as positive samples and N image pairs corresponding to the unmatched text as negative samples, and using contrast loss to shorten the embedding distance of the positive sample pairs and to increase the distance of the negative sample pairs; And the sensitivity adjustment of contrast loss is completed by introducing temperature parameters.

4. The image optimization method according to claim 3, characterized in that: Step S2 also includes: adding an adaptive attention mechanism between the semantic vector and the visual vector so as to focus on the preset features of the text and the image during the comparison process.

5. The image optimization method according to claim 3, characterized in that: Step S2 also includes: in the comparison loss process, using cosine similarity to measure the similarity between the semantic vector and the visual vector, and calculating the loss based on the similarity.

6. The image optimization method according to claim 3, characterized in that: In step S3, the self-supervised learning process includes: The masked language model randomly masks some words in the input semantic vector, so that the masked language model can infer image features under incomplete text information; The image mask model randomly masks some pixel blocks in the input image vector, allowing the model to restore visual details under the guidance of text.

7. The image optimization method according to claim 6, characterized in that: Step S4 includes: Based on the optimized semantic and visual feature vectors, a preliminary visual feature map is generated based on the generative adversarial network GAN or diffusion model; And use multi-layer convolution to gradually add color, light, shadow, and texture details to the generated image; And through multiple iterations of generation and feedback, the image details are optimized to generate a preliminary image concept that meets the description.

Images