Image generation and image-text alignment methods, apparatus, terminal devices and storage media

Abstract

This invention discloses an image generation and image-text alignment method, apparatus, terminal device, and storage medium. In the training process of a visual language model, training images and training text are input into the visual language model. The visual language model simultaneously performs image generation and image-text alignment tasks to obtain a unified visual language model, which can realize image-text alignment and multimodal generation functions, solving the technical problem that the prior art cannot achieve a unified discrimination and generation model.

Description

Technical Field

[0001] This application relates to the field of neural networks, and more particularly to an image generation and image-text alignment method, apparatus, terminal device, and storage medium. Background Technology

[0002] The CLIP (Contrastive Language-Image Pre-Training) model aligns the semantics between images and text through contrastive learning and cue modeling. It exhibits significant discriminative power and, after fine-tuning, achieves state-of-the-art performance on various cross-modal tasks, including zero-shot classification and image-text retrieval. A well-known and typical unified discriminative and generative model is the Generative Adversarial Network (GAN), which uses a discriminator to guide the adversarial process of the generator to achieve better natural image synthesis. However, this training mode often leads to model crashes and training instability. Existing techniques attempt to combine CLIP models to achieve zero-shot image processing guided by text cues; however, current CLIP ensemble models only focus on preserving the semantics of individual image-text pairs, ignoring auxiliary information from other image-text pairs, thus failing to achieve discriminative tasks such as cross-modal retrieval and zero-shot classification.

[0003] In conclusion, how to achieve a unified discrimination and generation model has become a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This invention provides an image generation and image-text alignment method, apparatus, terminal device, and storage medium to solve the technical problem that existing technologies cannot achieve a unified discrimination and generation model.

[0005] In a first aspect, embodiments of the present invention provide an image generation and image-text alignment method, including:

[0006] When an image generation task is received, the first input text and the noise image are obtained;

[0007] The first input text and the noise image are input into the visual language model, so that the visual language model extracts the first text features and the first image features from the first input text and the noise image, respectively, and generates a target image corresponding to the first input text based on the first text features and the first image features.

[0008] When an image-text alignment task is received, the second input text and at least one first input image are obtained.

[0009] The second input text and the first input image are input into the visual language model, so that the visual language model extracts the second text features and the second image features from the second input text and the first input image, respectively, and determines the target image corresponding to the second input text in the first input image based on the second image features and the second text features;

[0010] During the training process of the visual language model, the training image and the training text are input into the visual language model. The visual language model simultaneously performs the image generation task and the image-text alignment task, and adjusts the parameters of the visual language model until the output error of the visual language model is within a preset range.

[0011] Secondly, embodiments of the present invention provide an image generation and image-text alignment apparatus, comprising:

[0012] The first task receiving module is used to obtain the first input text and the noise image when it receives the image generation task;

[0013] An image output module is used to input the first input text and the noise image into a visual language model, so that the visual language model extracts first text features and first image features from the first input text and the noise image respectively, and generates a target image corresponding to the first input text based on the first text features and the first image features.

[0014] The second task receiving module is used to obtain the second input text and at least one first input image when it receives an image-text alignment task;

[0015] An alignment module is used to input the second input text and the first input image into the visual language model, so that the visual language model extracts second text features and second image features from the second input text and the first input image, respectively, and determines the target image corresponding to the second input text in the first input image based on the second image features and the second text features;

[0016] The model training module is used to input the training image and the training text into the visual language model during the training process of the visual language model, and to simultaneously perform the image generation task and the image-text alignment task through the visual language model, and to adjust the parameters of the visual language model until the output error of the visual language model is within a preset range.

[0017] Thirdly, embodiments of the present invention provide a terminal device, the terminal device including a processor and a memory;

[0018] The memory is used to store computer programs and to transfer the computer programs to the processor;

[0019] The processor is configured to execute an image generation and image-text alignment method as described in the first aspect, according to instructions in the computer program.

[0020] Fourthly, embodiments of the present invention provide a storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform an image generation and image-text alignment method as described in the first aspect.

[0021] As described above, in the training process of the visual language model, the present invention inputs training images and training text into the visual language model, and performs image generation and image-text alignment tasks simultaneously through the visual language model, thereby obtaining a unified visual language model that can realize image-text alignment and multimodal generation functions, solving the technical problem that the prior art cannot achieve a unified discrimination and generation model. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating an image generation and text alignment method provided in an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram illustrating a visual language model performing an image generation task, as provided in an embodiment of the present invention.

[0024] Figure 3 This is a schematic diagram illustrating a visual language model performing an image-text alignment task, as provided in an embodiment of the present invention.

[0025] Figure 4 This is a schematic diagram illustrating a process for training a visual language model, as provided in an embodiment of the present invention.

[0026] Figure 5 This is a schematic diagram of the structure of a visual language model provided in an embodiment of the present invention.

[0027] Figure 6 This is a schematic diagram of the structure of a fusion module provided in an embodiment of the present invention.

[0028] Figure 7 This is a schematic diagram of an image generation and text alignment device provided in an embodiment of the present invention.

[0029] Figure 8 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0030] The following description and accompanying drawings fully illustrate specific embodiments of this application to enable those skilled in the art to practice them. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of embodiments of this application includes the entire scope of the claims and all available equivalents of the claims. In this document, each embodiment may be referred to individually or collectively by the term "invention," which is merely for convenience and is not intended to automatically limit the scope of the application to any single invention or inventive concept if more than one invention is disclosed. Relational terms such as "first" and "second" are used herein only to distinguish one entity or operation from another, without requiring or implying any actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed. The various embodiments in this document are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the structures, products, etc., disclosed in the embodiments, since they correspond to the disclosed parts, the descriptions are relatively simple; relevant details can be found in the method section.

[0031] Current advancements in computer vision are driven by scaling models on large datasets of captioned images collected from the internet. CLIP demonstrates remarkable discriminative power by aligning the semantics between images and text through contrastive learning and cue modeling. With fine-tuning, CLIP can achieve robust performance across various cross-modal tasks. Furthermore, fine-tuning techniques show significant performance on domain-specific data, facilitating large-scale pre-trained discriminative and generative models, thus eliminating the need for retraining.

[0032] A review of research on unified discriminative and generative models reveals that a prominent and typical example is the Generative Adversarial Network (GAN), which uses a discriminator to guide the adversarial process of the generator to achieve better natural image synthesis. However, this training modality often leads to model collapse and training instability. In contrast, diffusion-based generative models have demonstrated their advantages in unconditional and multiconditional image generation tasks such as text-to-image, painting, and style transfer. Some works, such as DiffusionCLIP and DiffuseIT, attempt to combine CLIP models to achieve zero-shot image processing guided by textual cues. However, current CLIP ensemble models focus only on preserving the semantics of individual image-text pairs, ignoring auxiliary information from other image-text pairs, and fail to achieve discriminative tasks such as cross-modal retrieval and zero-shot classification. To fill this gap, a simple and effective solution is to use a pre-trained CLIP model to guide the training of the diffusion model in an end-to-end manner. However, this training method may lead to problems in semantic alignment learning due to the small training dataset (semantic collapse) of the CLIP model.

[0033] In summary, to address the technical problem of the inability to achieve a unified discrimination and generation model in existing technologies, this invention provides an image generation and image-text alignment method, such as... Figure 1 As shown, Figure 1 This is a flowchart illustrating an image generation and image-text alignment method provided in an embodiment of the present invention. The image generation and image-text alignment method provided in this embodiment can be executed by a terminal device, which can be implemented through software and / or hardware. The terminal device can consist of two or more physical entities, or it can consist of a single physical entity. For example, the terminal device can be a computer, a host computer, or a server. The method includes the following steps:

[0034] Step 101: When the image generation task is received, obtain the first input text and the noise image.

[0035] In this embodiment, when a user needs to generate an image based on text, the user can send an image generation task to the terminal device. Upon receiving the image generation task, the terminal device obtains the first input text entered by the user and a noise image. The first input text includes information about the content of the image the user expects to generate. For example, if the first input text contains the word "apple," then the user expects to generate an image containing an apple. The noise image refers to an image containing noise. The noise image can be generated randomly, and this embodiment does not specifically limit the method of generating the noise image.

[0036] Step 102: Input the first input text and the noisy image into the visual language model, so that the visual language model extracts the first text features and the first image features from the first input text and the noisy image respectively, and generates a target image corresponding to the first input text based on the first text features and the first image features.

[0037] After receiving the first input text and the noisy image, the terminal device needs to further invoke a pre-trained visual language model and input the first input text and the noisy image into the visual language model. Upon receiving the first input text and the noisy image, the language model can extract first text features and first image features from the first input text and the noisy image, respectively. Then, the visual language model further generates a target image corresponding to the first input text based on the first text features and the first image features. For example... Figure 2 As shown, after the user inputs an image and first input text including the word "apple," the visual language model can generate a target image including an apple. In one embodiment, the visual language model includes a text encoder, a visual autoencoder, and an image generation module. The first input text and the noisy image can be input into the text encoder and the visual autoencoder, respectively, so that the text encoder extracts first text features from the first input text, and the visual autoencoder extracts first image features from the noisy image. Then, the first text features and the first image features are input together into the image generation module, which generates a target image corresponding to the first input text based on the first image features and the first text features.

[0038] Step 103: When the image-text alignment task is received, obtain the second input text and at least one first input image.

[0039] Additionally, when a user requires image-text alignment, they can send an image-text alignment task to the terminal device. Image-text alignment refers to matching an image with text describing the image's content; for example, matching an image depicting an apple with text containing the word "apple." Upon receiving the image-text alignment task, the terminal device acquires the second input text entered by the user and at least one first input image. The number of first input images can be set according to actual needs.

[0040] Step 104: Input the second input text and the first input image into the visual language model so that the visual language model extracts the second text features and the second image features from the second input text and the first input image, respectively, and determines the target image corresponding to the second input text in the first input image based on the second image features and the second text features.

[0041] After acquiring the second input text and the first input image, the terminal device further extracts second text features and second image features from the second input text and the first input image, respectively. Based on the second image features and the second text features, it determines the target image corresponding to the second input text among all the first input images. For example... Figure 3 As shown, the user input includes a first input image containing "apple" and a second input image containing "pear," while the second input text is "apple." The visual language model can align the second input text "apple" with the first input image containing "apple." In one embodiment, the visual language model also includes a CLIP visual encoder. The visual language model can input the second input text and the first input image into a text encoder and a CLIP visual encoder, respectively, so that the text encoder extracts second text features from the second input text, and the CLIP visual encoder extracts second image features from the first input image. Then, the visual language model can determine the target image corresponding to the second input text among all the first input images by calculating the similarity between the second text features and the second image features.

[0042] In the training process of the visual language model, training images and training text are input into the visual language model. The visual language model simultaneously performs image generation and image-text alignment tasks, and the parameters of the visual language model are adjusted until the output error of the visual language model is within a preset range.

[0043] Furthermore, it should be noted that during the training process of the visual language model, the user needs to acquire training images and training text, input them into the visual language model, and have the model simultaneously perform image generation and image-text alignment tasks. After obtaining the output results of the image generation and alignment tasks, the user can determine whether the errors of both tasks are within preset ranges. If not, the parameters of the visual language model are adjusted, and new training images and text are input again for training until the errors of both tasks are within the preset range, thus obtaining a well-trained visual language model.

[0044] As described above, in the training process of the visual language model, the present invention inputs training images and training text into the visual language model, and performs image generation and image-text alignment tasks simultaneously through the visual language model, thereby obtaining a unified visual language model that can realize image-text alignment and multimodal generation functions, solving the technical problem that the prior art cannot achieve a unified discrimination and generation model.

[0045] like Figure 4 As shown, Figure 4 This is a schematic diagram illustrating a process for training a visual language model according to an embodiment of the present invention. The steps for training the visual language model in this embodiment include:

[0046] Step 201: Input the training images and training text into the visual language model so that the visual language model can perform both image generation and image-text alignment tasks simultaneously.

[0047] In this embodiment, when training the visual language model, it is first necessary to acquire training images and training text. In one embodiment, after acquiring the training text, for the training text used in the image-text alignment task, a CLIP-tokenizer can be used to tag the training text, thereby obtaining a tag sequence, represented as follows: Where ω i Let represent the i-th label, and m represent the number of labels. Then, the training images and training text can be input into the visual language model, allowing the model to simultaneously perform image generation and image-text alignment tasks.

[0048] Step 202: When performing the image generation task, extract the first training image features from the training images and the first training text features from the training text using a visual language model.

[0049] During training, when performing the image generation task, the visual language model can extract first training image features from the training images and first training text features from the training text. In one embodiment, such as... Figure 5 As shown, the visual language model includes a visual autoencoder and a CLIP text encoder. The visual autoencoder is a neural network model capable of unsupervised learning. It mainly consists of two parts: an encoder for core feature extraction and a decoder for data reconstruction. The encoder is primarily responsible for compressing the original input data and extracting its core features, while the decoder expands and reconstructs the input data based on the extracted core features. It's important to note that the parameters of the visual autoencoder remain unchanged during training. The CLIP text encoder is a Transformer model, specifically a Decoder structure, and is a GPT model with 12 Transformer blocks and 512 hidden nodes.

[0050] In step 202, during the image generation task, the first training image features are extracted from the training images and the first training text features are extracted from the training text using a visual language model, including:

[0051] Step 2021: Input the training text into the CLIP text encoder so that the CLIP text encoder can extract the first training text features from the training text.

[0052] To obtain the entire sentence and each token ω in the training text i In this embodiment, the text encoder from CLIP (Stable Diffusion Model) is used to extract the first training text feature T from the training text. f The first training text feature T f This is a token-level feature. It should be noted that the parameters of the CLIP text encoder in this embodiment are adjustable during training.

[0053] Step 2022: Input the training image into the visual autoencoder so that the visual autoencoder extracts the first training image features from the training image.

[0054] For a given image, in this embodiment of the invention, a stable diffusion (SD) visual autoencoder is used to perform a diffusion process in the latent image space. The visual language model is able to extract the first training image feature V from each training image. a ∈R c×h×w The three-dimensional representation of V, where c, h, and w represent the features of the first training image, respectively. a The channel, height, and width.

[0055] Step 203: When performing the image-text alignment task, extract the second training image features and the third training image features from the training images through the visual language model, and extract the second training text features from the training text.

[0056] During training, when performing the image-text alignment task, the visual language model extracts second and third training image features from the training images, and extracts second training text features from the training text in the visual language model. In one embodiment, such as... Figure 5 As shown, the visual language model includes a CLIP visual encoder, which is used to learn image-text alignment. The parameters of the CLIP visual encoder can be adjusted during training. In this embodiment, the CLIP visual encoder is mainly used to obtain global features and token-level features of the image.

[0057] Specifically, in step 203, when performing the image-text alignment task, the visual language model extracts second and third training image features from the training images, and extracts second training text features from the training text, including:

[0058] Step 2031: Input the training text into the CLIP text encoder so that the CLIP text encoder can extract the second training text features from the training text.

[0059] In this embodiment, after the training text is input into the CLIP text encoder, the CLIP text encoder is also used to extract the second training text feature T from the training text. g The second training text feature is the [EOS] token feature. It should be noted that after extracting the first and second training text features, a post-processing step is required, including layemorm (a normalization method) and L2 normalization.

[0060] Step 2032: Input the training image into the CLIP visual encoder so that the CLIP visual encoder extracts the second training image features and the third training image features from the training image.

[0061] After inputting the training images into the CLIP visual encoder, the CLIP visual encoder first obtains the second training image feature V of the training images. g ∈R d and the third training image feature V f ∈R l×d The second training image feature V g For global features, the third training image feature V f For token-level features, l represents the number of image patches, and d represents the dimension of each token feature. Global feature V g Corresponding to the [CLS] token feature, embodiments of the present invention learn image-text alignment by generating normalized [CLS] features.

[0062] Step 204: Adjust the parameters of the visual language model based on the first training image features, the second training image features, the third training image features, the first training text features, and the second training text features until the output error of the visual language model is within a preset range.

[0063] After obtaining the first training image features, the second training image features, the third training image features, the first training text features, and the second training text features, the parameters of the visual language model can be adjusted based on these features. In one embodiment, the image-text contrast loss, text-conditional image synthesis loss, and reciprocal semantic consistency loss of the visual language model can be calculated based on the first training image features, the second training image features, the third training image features, the first training text features, and the second training text features, and the parameters of the visual language model can be adjusted based on these three losses.

[0064] Based on the above embodiments, such as Figure 5 As shown, the visual language model includes a fusion module and an image generation module.

[0065] Step 204 involves adjusting the parameters of the visual language model based on the features of the first training image, the second training image, the third training image, the first training text, and the second training text until the output error of the visual language model is within a preset range. This includes:

[0066] Step 2041: Input the first training image features and the third training image features into the fusion module, so that the fusion module generates fused semantic features based on the first training image features and the third training image, and generates the first target image features based on the third training image features.

[0067] In this embodiment, the fusion module uses two different feature inputs: the third training image feature V f and the first training image features V a (Obtained from the CLIP visual encoder and visual autoencoder, respectively). Because the original visual language models have different training tasks (image generation and image-text alignment), these features exhibit unique characteristics, leading to variations in feature representation and semantic understanding. Specifically, the third training image feature V... f It primarily focuses on capturing the overall meaning of sentences, while also providing a relatively coarse token-level understanding, and the first training image feature V a This represents three-dimensional features and excels at capturing detailed information about each marker. Therefore, to leverage the strengths of two features from different visual encoders, this embodiment uses a fusion module to fuse the third training image feature V. f and the first training image features V a This generates fused semantic features, and at the same time, it generates first target image features based on the features of the third training image.

[0068] Based on the above embodiments, the fusion module includes a pooling layer, a first linear layer, a stacking layer, and a second linear layer.

[0069] Step 2041: Input the first training image features and the third training image features into the fusion module, so that the fusion module generates fused semantic features based on the first training image features and the third training image features, and generates the first target image features based on the third training image features, including:

[0070] Step 20411: Input the features of the first training image into the pooling layer to obtain the enhanced features output by the pooling layer.

[0071] In this embodiment, as Figure 6 As shown, the fusion module includes a pooling layer, a first linear layer, a stacked layer, and a second linear layer. First, the features of the first training image are input into the pooling layer, which enhances the edge features of the training image. In one embodiment, a max pooling operation with a stride of 2 and a kernel size of 2 can be used to enhance the edge features.

[0072] Step 20412: Input the enhanced features into the first linear layer to obtain the first target image features output by the first linear layer.

[0073] After obtaining the enhanced features, these enhanced features can be regarded as a set of mappings of channel c (∈R). h / 2×w / 2 To ensure consistency with the third training image features V f To ensure compatibility, this embodiment uses a first linear layer to reduce the dimension of the dimension-enhancing feature from (h / 2)×(w / 2) to d, thereby obtaining the first target image feature V′. a .

[0074] Step 20413: Input the first target image features and the third training image features into the stacked layer for fusion to obtain the fused features output by the stacked layer.

[0075] After obtaining the first target image feature V′ a Then, the first target image features V′ can be further processed. a and the third training image features V f The inputs are fused into a stacked layer to obtain the fused features output by the stacked layer. Specifically, the first target image features and the third training image features V f Connect along the first dimension, in R (l+c)×d The space is connected to form a fusion feature.

[0076] Step 20414: Input the fused features into the second linear layer to obtain the fused semantic features output by the second linear layer.

[0077] Finally, the fused features are input into the second linear layer, which transforms the fused features into a fused semantic feature. The linear layer runs along the first dimension, where n u It is a fusion of semantic features V u The channel size.

[0078] Step 2042: Input the first training image features and the first training text features into the image generation module, so that the image generation module generates a potential image and extracts the second target image features based on the first training image features and the first training text features.

[0079] In this embodiment, the first training image V is used during the training process. a and the first training text features T f After being input into the image generation module, the image generation module will use the features V of the first training image. a and the first training text features T f Generate a latent image, and from the features V of the first training image. a and the first training text features T f Extract the second target image feature V r In one embodiment, considering the first training image V a and the first training text features T f The image generation module uses a stable diffusion model's UNet module. During training, only the parameters of the cross-attention module within the UNet module are adjustable, while the parameters of other modules are fixed. The purpose of the UNet module is to generate images from the first training text features T. f The underlying image V is denoised using t-step noise reduction. a，t The results are then fed back into the UNet module to predict noise. And generate a latent image V with t steps of noise. a，t , where ε θ (V a，t ,t,T f ) indicates that in parameter V a，t ,t,T f The noise generated below.

[0080] Step 2043: Calculate the first loss based on the fused semantic features, the first target image features, and the second target image features.

[0081] After obtaining the fused semantic features, the first target image features, and the second target image features, a first loss, namely the reciprocal semantic consistency loss, is calculated. Since the main objective of this embodiment is to improve the accuracy and effectiveness of image-text alignment in the discrimination task by combining label-level and text-level information, aligned image-text representations are used to guide the image generation task of the stable diffusion model, aiming to generate semantically matched and realistic images. To address the semantic collapse problem caused by limited data in image-text contrastive learning and to enhance the overall capability of the visual language model, reciprocal semantic consistency (RSC) modeling is introduced in this embodiment. Reciprocal semantic consistency (RSC) modeling bridges the gap between the image features extracted by the CLIP visual encoder and the visual autoencoder of the SD model, effectively solving the semantic collapse problem. Specifically, based on the fused semantic features, the first target image features, and the second target image features, the first loss is calculated, including:

[0082] Step 20431: Determine the first dimension of the fused semantic features, the first dimension of the first target image features, and the first dimension of the second target image features.

[0083] In this embodiment, the first step is to determine the fused semantic feature V. u The first dimension, the first target image feature V′ a The first dimension and the second target image feature V r The first dimension.

[0084] Step 20432: Calculate the first loss based on the first dimension of the fused semantic features, the first dimension of the first target image features, and the first dimension of the second target image features, as well as the fused semantic features, the corresponding first target image features, and the corresponding second target image features corresponding to each training sample. A training sample includes the associated training text and training image.

[0085] Then, the first loss can be calculated based on the first dimension of the fused semantic features, the first dimension of the first target image features, and the first dimension of the second target image features, as well as the fused semantic features, the corresponding first target image features, and the corresponding second target image features corresponding to each training sample. A training sample includes an associated training text and a training image.

[0086] In this embodiment, the fused semantic feature V output from the fusion module is introduced. u And by using the first training image features V of the visual autoencoder a The first target image feature V′ is generated by feeding it back into the first linear layer of the fusion module. a These features are used to force the reconstruction of the second target image features V.r Through calculation Prediction is performed in a single step, where This represents a difference over t steps. In one embodiment, the second target image feature V r The first linear layer of the fusion module is also projected onto the feature space, therefore, in this embodiment, the reciprocal semantic consistency loss is... The calculation formula is as follows:

[0087]

[0088] Where n is the batch size, i.e., the number of training samples. A training sample includes a training text and a training image associated with the training text. t V represents u The first dimension, V r The first dimension and V a The first dimension. Let V represent the fused semantic features, the second target image features, and the first target image features of the i-th training sample, respectively. It should be noted that, due to the fused semantic features V... u The first dimension and the second target image feature V r This is not a perfect correspondence, therefore in this embodiment, V is first calculated in the first dimension. u The average value is then repeated n times along the same dimension. t This ensures consistency. By striking a balance between fused semantic information and token-level semantic information in this way, CLIP-enhanced semantic guidance is achieved.

[0089] Step 2044: Calculate the second loss based on the latent image and the features of the first training text.

[0090] For the first training image feature V a and its corresponding first training text feature T f Add t steps of random noise to the first training image feature V a In the process, the latent image V is obtained. a，t And combined with textual conditional effects on the latent image V a，t Progressive denoising, predicting the text conditional image synthesis loss generated from the image with t steps of noise, i.e., the second loss, is expressed as: The details are as follows:

[0091]

[0092] in, is the loss function, which in this embodiment is the MSE loss function, and ∈ represents noise.

[0093] Step 2045: Calculate the third loss based on the second training text features and the second training image features.

[0094] When calculating the third loss, namely the image-text contrast loss, this embodiment first calculates the second training image feature of the i-th training sample among n training samples. Second training text features The similarity between them is calculated. After calculating the similarity, an n×n similarity score map is generated. Based on the similarity score map, the learnable temperature τ can be further calculated. c Image-to-text InfoNCE loss And text-to-image InfoNCE loss

[0095]

[0096]

[0097] Finally, according to and The third loss can then be calculated.

[0098]

[0099] Third loss The goal is to achieve image-text alignment within a specific domain.

[0100] Step 2046: Adjust the parameters of the visual language model according to the first loss, the second loss and the third loss until the output error of the visual language model is within the preset range.

[0101] The parameters of the visual language model are adjusted based on the first, second, and third losses, and it is determined whether the output error of the visual language model is within a preset range. If not, training images and training text are input to continue adjusting the visual language model. When the output error of the visual language model is within the preset range, the trained visual language model is obtained.

[0102] In one embodiment, the training text and training images used to train the visual language model are sourced from the following:

[0103] To focus on personalized design and creation, particularly in the fashion and product fields, two publicly available datasets were selected: the CM-Fashion dataset and the M5Product dataset. Both contain 500,000 images covering various clothing categories and include detailed descriptions. From the CM-Fashion dataset, two subsets, Fashion-man and Fashion-woman, were selected, and a training set with 100 training samples per class and a test set with 20 training samples per class were created. Furthermore, this embodiment also utilizes the M5Product dataset, which consists of over 5 million image-text pairs across more than 6,000 categories. For the M5Product dataset, samples from 10 categories were randomly selected, and the same number of samples as in the fashion field were used to construct the training and test sets.

[0104] The details of training the visual language model are as follows:

[0105] This embodiment utilizes Stable Diffusion v1-4 and the pre-trained CLIP-ViT-L / 14 as the base models to construct the visual language model. This visual language model is fine-tuned on three specific datasets: Fashion-man, Fashion-woman, and E-commerce Product. The input resolutions of the visual autoencoder and CLIP visual encoder are 512×512 and 224×224, respectively. This embodiment uses a batch size of 6 and fine-tunes the visual language model for 200 epochs. An AdamW optimizer with a learning rate of 1e-4 is used. Eight steps of gradient accumulation are used during training. For V... u This invention sets the channel size to 4, matching the number of channels in the Stable Diffusion autoencoder. Furthermore, we incorporate an exponential moving average (EMA) model with a decay factor of 0.9999 in each iteration; the EMA model is used to evaluate the performance of downstream tasks.

[0106] The evaluation of this visual language model is as follows:

[0107] This embodiment further evaluates the performance of the visual language model and compares it with existing methods on cross-modal image-text alignment and image generation tasks. For the image-text alignment task, this invention uses the Recall@N metric to evaluate the performance of the visual language model; for the image generation task, this embodiment uses the FID score metric to quantify the quality and diversity of the reference image library. Experimental results highlight the superiority of the visual language model and demonstrate its potential to advance visual and language tasks.

[0108] As described above, in the training process of the visual language model, the present invention inputs training images and training text into the visual language model, and performs image generation and image-text alignment tasks simultaneously through the visual language model, thereby obtaining a unified visual language model that can realize multimodal generation functions and solves the technical problem that the prior art cannot realize a unified discrimination and generation model.

[0109] Furthermore, this invention proposes Reciprocal Semantic Consistency (RSC) modeling during the establishment of the visual language model. RSC modeling enhances the semantic alignment of image features extracted from the CLIP visual encoder and the visual autoencoder, effectively bridging the semantic gap between these two visual encoders. This is achieved by fusing semantic features V... u With the reconstructed second target image features V r Semantic-level feature alignment ensures strong and stable semantic understanding capabilities. Furthermore, this embodiment of the invention utilizes a fusion module that plays a crucial role in reconciling and integrating image features from the CLIP visual encoder and visual autoencoder, bridging feature gaps between image features. By integrating RSC, it addresses the challenge of semantic collapse during fine-tuning on small datasets and improves the quality of generated images and image-text semantic alignment.

[0110] like Figure 7 As shown, Figure 7 This is a schematic diagram of the structure of an image generation and image-text alignment device provided in an embodiment of the present invention, as shown below. Figure 7 As shown, the image generation and image-text alignment device includes:

[0111] The first task receiving module 301 is used to acquire the first input text and the noise image when it receives the image generation task;

[0112] The image output module 302 is used to input the first input text and the noisy image into the visual language model, so that the visual language model extracts the first text features and the first image features from the first input text and the noisy image respectively, and generates a target image corresponding to the first input text based on the first text features and the first image features.

[0113] The second task receiving module 303 is used to obtain a second input text and at least one first input image when it receives an image-text alignment task;

[0114] Alignment module 304 is used to input the second input text and the first input image into the visual language model, so that the visual language model extracts the second text features and the second image features from the second input text and the first input image, respectively, and determines the target image corresponding to the second input text in the first input image based on the second image features and the second text features;

[0115] The model training module 305 is used to input training images and training text into the visual language model during the training process, and to simultaneously perform image generation and image-text alignment tasks through the visual language model, and adjust the parameters of the visual language model until the output error of the visual language model is within a preset range.

[0116] Based on the above embodiments, the model training module 305 includes:

[0117] The first input submodule is used to input training images and training text into the visual language model so that the visual language model can simultaneously perform image generation tasks and image-text alignment tasks.

[0118] The first feature extraction submodule is used to extract first training image features from training images and first training text features from training text using a visual language model when performing image generation tasks.

[0119] The second feature extraction submodule is used to extract second training image features and third training image features from training images and second training text features from training text when performing image-text alignment tasks.

[0120] The parameter adjustment submodule is used to adjust the parameters of the visual language model based on the features of the first training image, the second training image, the third training image, the first training text, and the second training text until the output error of the visual language model is within a preset range.

[0121] Based on the above embodiments, the visual language model includes a visual autoencoder and a CLIP text encoder;

[0122] The first feature extraction submodule includes:

[0123] The first text feature extraction unit is used to input the training text into the CLIP text encoder so that the CLIP text encoder can extract the first training text features from the training text.

[0124] The first image feature extraction unit is used to input the training image into the visual autoencoder so that the visual autoencoder extracts the first training image features from the training image.

[0125] Based on the above embodiments, the visual language model includes a CLIP visual encoder;

[0126] The second feature extraction submodule includes:

[0127] The second text feature extraction unit is used to input the training text into the CLIP text encoder so that the CLIP text encoder can extract the second training text features from the training text.

[0128] The second image feature extraction unit is used to input the training image into the CLIP visual encoder so that the CLIP visual encoder can extract the second training image features and the third training image features from the training image.

[0129] Based on the above embodiments, the visual language model includes a fusion module and an image generation module;

[0130] The parameter adjustment submodule includes:

[0131] The feature fusion unit is used to input the first training image features and the third training image features into the fusion module, so that the fusion module generates fused semantic features based on the first training image features and the third training image, and generates the first target image features based on the third training image features;

[0132] An image generation unit is used to input first training image features and first training text features into an image generation module, so that the image generation module generates a potential image and extracts second target image features based on the first training image features and first training text features;

[0133] The first loss calculation unit is used to calculate the first loss based on the fused semantic features, the first target image features, and the second target image features;

[0134] The second loss calculation unit is used to calculate the second loss based on the latent image and the features of the first training text.

[0135] The third loss calculation unit is used to calculate the third loss based on the features of the second training text and the features of the second training image.

[0136] The parameter adjustment unit is used to adjust the parameters of the visual language model according to the first loss, the second loss, and the third loss until the output error of the visual language model is within a preset range.

[0137] Based on the above embodiments, the fusion module includes a pooling layer, a first linear layer, a stacking layer, and a second linear layer;

[0138] The feature fusion unit includes:

[0139] The feature enhancement subunit is used to input the features of the first training image into the pooling layer to obtain the enhanced features output by the pooling layer;

[0140] The first linear connection subunit is used to input the enhanced features into the first linear layer to obtain the first target image features output by the first linear layer;

[0141] The fusion subunit is used to input the features of the first target image and the features of the third training image into the stacked layer for fusion, so as to obtain the fused features output by the stacked layer;

[0142] The second linear connection subunit is used to input the fused features into the second linear layer to obtain the fused semantic features output by the second linear layer.

[0143] Based on the above embodiments, the first loss calculation unit includes:

[0144] The dimension determination subunit is used to determine the first dimension of the fused semantic features, the first dimension of the first target image features, and the first dimension of the second target image features;

[0145] The first loss calculation subunit is used to calculate the first loss based on the first dimension of the fused semantic features, the first dimension of the first target image features, and the first dimension of the second target image features, as well as the fused semantic features, the corresponding first target image features, and the corresponding second target image features corresponding to each training sample. A training sample includes the associated training text and training image.

[0146] The image generation and image-text alignment apparatus provided in this embodiment of the invention is included in a terminal device and can be used to execute the image generation and image-text alignment methods provided in the above embodiments, possessing corresponding functions and beneficial effects.

[0147] It is worth noting that in the embodiments of the above-mentioned image generation and image-text alignment device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy distinction between each other and are not used to limit the scope of protection of the present invention.

[0148] This embodiment also provides a terminal device, such as Figure 8 As shown, the terminal device 40 includes a processor 400 and a memory 401;

[0149] Memory 401 is used to store computer program 402 and transfer computer program 402 to processor 400;

[0150] The processor 400 is used to execute the steps in the above-described embodiment of an image generation and image-text alignment method according to the instructions in the computer program 402.

[0151] For example, computer program 402 may be divided into one or more modules / units, one or more of which are stored in memory 401 and executed by processor 400 to complete this application. One or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 402 in terminal device 40.

[0152] Terminal device 40 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. Terminal device 40 may include, but is not limited to, processor 400 and memory 401. Those skilled in the art will understand that... Figure 8 This is merely an example of terminal device 40 and does not constitute a limitation on terminal device 40. It may include more or fewer components than shown, or combine certain components, or different components. For example, terminal device 40 may also include input / output devices, network access devices, buses, etc.

[0153] The processor 400 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0154] The memory 401 can be an internal storage unit of the terminal device 40, such as a hard disk or RAM of the terminal device 40. The memory 401 can also be an external storage device of the terminal device 40, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal device 40. Furthermore, the memory 401 can include both internal and external storage units of the terminal device 40. The memory 401 is used to store computer programs and other programs and data required by the terminal device 40. The memory 401 can also be used to temporarily store data that has been output or will be output.

[0155] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0157] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0158] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0159] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing computer programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] This invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform an image generation and image-text alignment method, the method comprising the following steps:

[0161] When an image generation task is received, the first input text and the noise image are obtained;

[0162] The first input text and the noisy image are input into the visual language model, so that the visual language model extracts the first text features and the first image features from the first input text and the noisy image, respectively, and generates a target image corresponding to the first input text based on the first text features and the first image features.

[0163] When an image-text alignment task is received, the second input text and at least one first input image are obtained.

[0164] The second input text and the first input image are input into the visual language model so that the visual language model extracts the second text features and the second image features from the second input text and the first input image, respectively, and determines the target image corresponding to the second input text in the first input image based on the second image features and the second text features.

[0165] In the training process of the visual language model, training images and training text are input into the visual language model. The visual language model simultaneously performs image generation and image-text alignment tasks, and the parameters of the visual language model are adjusted until the output error of the visual language model is within a preset range.

[0166] Note that the above are merely preferred embodiments and the technical principles applied in this invention. Those skilled in the art will understand that the embodiments of this invention are not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the protection scope of this invention. Therefore, although the embodiments of this invention have been described in detail above, the embodiments of this invention are not limited to the above embodiments. More other equivalent embodiments may be included without departing from the concept of the embodiments of this invention, and the scope of the embodiments of this invention is determined by the scope of the appended claims.

Claims

1. A method of image generation and text alignment, characterized by, The method comprises: when receiving an image generation task, obtaining first input text and a noise image; inputting the first input text and the noise image into a visual language model, so that the visual language model extracts first text features and first image features from the first input text and the noise image respectively, and generates a target image corresponding to the first input text according to the first text features and the first image features; when receiving an image-text alignment task, obtaining second input text and at least one first input image; inputting the second input text and the first input image into the visual language model, so that the visual language model extracts second text features and second image features from the second input text and the first input image respectively, and determines a target image corresponding to the second input text in the first input image according to the second image features and the second text features; wherein, in the training process of the visual language model, training images and training texts are input into the visual language model, so that the visual language model simultaneously performs the image generation task and the image-text alignment task; when performing the image generation task, the visual language model extracts first training image features from the training images and first training text features from the training texts; when performing the image-text alignment task, the visual language model extracts second training image features and third training image features from the training images and second training text features from the training texts; adjust the parameters of the visual language model according to the first training image features, the second training image features, the third training image features, the first training text features and the second training text features until the output error of the visual language model is within a preset range.

2. The method of image generation and text alignment according to claim 1, wherein, The visual language model comprises a visual autoencoder and a CLIP text encoder; when performing the image generation task, the visual language model extracts first training image features from the training images and first training text features from the training texts, comprising: inputting the training text into the CLIP text encoder, so that the CLIP text encoder extracts first training text features from the training text; inputting the training image into the visual autoencoder, so that the visual autoencoder extracts first training image features from the training image.

3. The method of image generation and text alignment according to claim 2, wherein, The visual language model comprises a CLIP visual encoder; when performing the image-text alignment task, the visual language model extracts second training image features and third training image features from the training images and second training text features from the training texts, comprising: inputting the training text into the CLIP text encoder, so that the CLIP text encoder extracts second training text features from the training text; inputting the training image into the CLIP visual encoder to enable the CLIP visual encoder to extract a second training image feature and a third training image feature from the training image.

4. The method of claim 1, wherein, The visual language model comprises a fusion module and an image generation module; The adjusting of the parameters of the visual language model according to the first training image feature, the second training image feature, the third training image feature, the first training text feature and the second training text feature until the output error of the visual language model is within a preset range comprises: inputting the first training image feature and the third training image feature into the fusion module to enable the fusion module to generate a fusion semantic feature according to the first training image feature and the third training image, and to generate a first target image feature according to the third training image feature; inputting the first training image feature and the first training text feature into the image generation module to enable the image generation module to generate a latent image and extract a second target image feature according to the first training image feature and the first training text feature; calculating a first loss according to the fusion semantic feature, the first target image feature and the second target image feature; calculating a second loss according to the latent image and the first training text feature; calculating a third loss according to the second training text feature and the second training image feature; adjusting the parameters of the visual language model according to the first loss, the second loss and the third loss until the output error of the visual language model is within a preset range.

5. The method of image generation and image-text alignment according to claim 4, wherein, The fusion module comprises a pooling layer, a first linear layer, a stacking layer and a second linear layer; The inputting of the first training image feature and the third training image feature into the fusion module to enable the fusion module to generate a fusion semantic feature according to the first training image feature and the third training image, and to generate a first target image feature according to the third training image feature comprises: inputting the first training image feature into the pooling layer to obtain an enhanced feature output by the pooling layer; inputting the enhanced feature into the first linear layer to obtain a first target image feature output by the first linear layer; inputting the first target image feature and the third training image feature into the stacking layer for fusion to obtain a fusion feature output by the stacking layer; inputting the fusion feature into the second linear layer to obtain a fusion semantic feature output by the second linear layer.

6. The method of image generation and image-text alignment according to claim 4, wherein, The calculating of the first loss according to the fusion semantic feature, the first target image feature and the second target image feature comprises: determining a first dimension of the fusion semantic feature, a first dimension of the first target image feature and a first dimension of the second target image feature; According to the first dimension of the fusion semantic feature, the first dimension of the first target image feature, and the first dimension of the second target image feature, and the fusion semantic feature corresponding to each training sample, the corresponding first target image feature, and the corresponding second target image feature, a first loss is calculated, and one of the training samples includes an associated training text and a training image.

7. An image generation and justification apparatus, characterized by Comprise: The first task receiving module is configured to, when receiving an image generation task, acquire a first input text and a noise image; The image output module is configured to input the first input text and the noise image into a visual language model, so that the visual language model extracts a first text feature and a first image feature from the first input text and the noise image, respectively, and generates a target image corresponding to the first input text according to the first text feature and the first image feature; The second task receiving module is configured to, when receiving an image-text alignment task, acquire a second input text and at least one first input image; The alignment module is configured to input the second input text and the first input image into the visual language model, so that the visual language model extracts a second text feature and a second image feature from the second input text and the first input image, respectively, and determines a target image corresponding to the second input text in the first input image according to the second image feature and the second text feature; The model training module is configured to, in a training process of the visual language model, input a training image and a training text into the visual language model, so that the visual language model simultaneously performs the image generation task and the image-text alignment task; In the execution of the image generation task, the first training image feature is extracted from the training image by the visual language model, and the first training text feature is extracted from the training text; in the execution of the image-text alignment task, the second training image feature and the third training image feature are extracted from the training image by the visual language model, and the second training text feature is extracted from the training text; the parameters of the visual language model are adjusted according to the first training image feature, the second training image feature, the third training image feature, the first training text feature, and the second training text feature, until the output error of the visual language model is within a preset range.

8. A terminal device, comprising: The terminal device comprises a processor and a memory; The memory is configured to store a computer program and transmit the computer program to the processor; The processor is configured to execute the image generation and image-text alignment method according to the instructions in the computer program.

9. A storage medium storing computer-executable instructions, wherein: The computer executable instructions are configured to execute the image generation and image-text alignment method according to the instructions in the computer program.

Images