Image generation model training method and device, terminal and storage medium

Abstract

Embodiments of the present application relate to the field of artificial intelligence technology, and specifically provide a training method and device of an image generation model, a terminal and a storage medium. The method comprises: obtaining a first image set output by an initial image generation model to be trained; obtaining image similarity of each first image in the first image set, the image similarity comprising at least color similarity, grayscale similarity and contour similarity; calculating a loss value of a loss function of the initial image generation model according to the image similarity; updating a model parameter of the initial image generation model according to the loss value until a target image generation model is obtained. The training method of the image generation model provided by the present application can be applied to the generation of disease-related images in the medical field, solves the problem that the image set generated by the existing image generation model is relatively single, improves the diversity of the image set generated by the image generation model, and helps doctors to more vividly help patients understand relevant information of the disease, thereby better assisting the treatment of doctors.

Description

Training methods, devices, terminals, and storage media for image generation models Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, terminal and storage medium for an image generation model. Background Technology

[0002] With the development of internet technology, people can search for images corresponding to various medical cases online to understand the specific progression of the disease. With advancements in image processing technology, generating desired images from text using models is no longer a challenge. For example, Generative Adversarial Networks (GANs) can be used to generate images corresponding to text content, or diffusion models can be used.

[0003] Therefore, during diagnosis and treatment, doctors can describe the patient's symptoms in text form, and then use a text-to-image model to generate images related to the patient's symptoms. This helps doctors better assist patients in understanding their condition more accurately and vividly. However, both generative adversarial networks (GANs) and diffusion models are prone to overfitting or underfitting due to the limited training samples used. This results in relatively homogeneous generated images, severely reducing the diversity of the output images needed and diminishing the effectiveness of the model in helping doctors understand patients' conditions. Summary of the Invention

[0004] The main objective of this application is to provide a training method, apparatus, terminal, and storage medium for an image generation model. This aims to solve the problem of the image generation model generating a single type of image in the image set, thereby increasing the diversity of images in the generated image set. This allows doctors to better utilize the patient's symptom information to obtain medical images corresponding to the patient during diagnosis and treatment, assisting doctors in helping patients better understand their symptoms and improving the efficiency of doctors in explaining the severity of symptoms to patients.

[0005] In a first aspect, embodiments of this application provide a method for training an image generation model, comprising:

[0006] Obtain the first set of images output by the initial image generation model to be trained;

[0007] Obtain the image similarity of each first image in the first image set. The image similarity includes at least color similarity, grayscale similarity, and contour similarity.

[0008] The loss value of the loss function of the initial image generation model is calculated based on the image similarity.

[0009] The model parameters of the initial image generation model are updated based on the loss value until the target image generation model is obtained.

[0010] Secondly, embodiments of this application also provide an image generation model training apparatus, comprising:

[0011] The data acquisition module is used to acquire the first set of images output by the initial image generation model to be trained;

[0012] The data calculation module is used to obtain the image similarity of each first image in the first image set. The image similarity includes at least color similarity, grayscale similarity and contour similarity.

[0013] The data processing module is used to calculate the loss value of the loss function of the initial image generation model based on the image similarity.

[0014] The data update module is used to update the model parameters of the initial image generation model according to the loss value until the target image generation model is obtained.

[0015] Thirdly, embodiments of this application also provide a terminal device, the terminal including a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for connecting and communicating between the processor and the memory, wherein when the computer program is executed by the processor, it implements the steps of a training method for any of the image generation models provided in this specification.

[0016] Fourthly, embodiments of this application also provide a storage medium for computer-readable storage, characterized in that the storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the training method for the image generation model as provided in any of the embodiments of this application.

[0017] This application provides a training method, apparatus, terminal, and storage medium for an image generation model. The training method includes training an initial image generation model based on a large amount of training data; performing incremental training on the initial image generation model; acquiring a first image set output by the initial image generation model during incremental training; calculating the image similarity of each first image in the first image set, where image similarity includes at least color similarity, grayscale similarity, and contour similarity; calculating the loss value of the loss function of the initial image generation model based on the image similarity; and updating the model parameters of the initial image generation model based on the loss value to minimize the similarity between the first images in the first image set, until a target image generation model is obtained. This allows for further optimization of the image generation model while reducing the computational resources required for training, solving the problem of single-image generation and increasing the diversity of generated images. This enables doctors to better utilize patient information to obtain corresponding medical images during diagnosis, assisting doctors in helping patients better understand their symptoms and improving the efficiency of explaining the severity of symptoms to patients. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 is a flowchart illustrating a training method for an image generation model provided in an embodiment of this application;

[0020] Figure 2 is a schematic diagram of the structure of an image generation model provided in an embodiment of this application;

[0021] Figure 3 is a schematic diagram of the structure of an image generation model provided in an embodiment of this application;

[0022] Figure 4 is a schematic diagram of the module structure of a training device for an image generation model provided in an embodiment of this application;

[0023] Figure 5 is a schematic block diagram of a terminal device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0026] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0027] This application provides a training method, apparatus, terminal, and storage medium for an image generation model. The training method for the image generation model can be applied to a terminal device, which can be a tablet computer, laptop computer, personal digital assistant, wearable device, or server. The server can be a standalone server or a server cluster.

[0028] This application provides a training method, apparatus, terminal, and storage medium for an image generation model. The training method includes training an initial image generation model based on a large amount of training data; performing incremental training on the initial model; acquiring a first image set output by the initial model during incremental training; calculating the image similarity of each first image in the first image set, where image similarity includes at least color similarity, grayscale similarity, and contour similarity; calculating the loss value of the loss function of the initial image generation model based on the image similarity; and updating the model parameters of the initial image generation model based on the loss value to minimize the similarity between the first images in the first image set, until a target image generation model is obtained. This reduces the computational resources required for training the image generation model while further optimizing it, solving the problem of single-image generation and increasing the diversity of generated images. This allows doctors to better utilize patient information to obtain corresponding medical images during diagnosis, assisting doctors in helping patients better understand their symptoms and improving the efficiency of explaining the severity of symptoms.

[0029] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0030] Please refer to Figure 1, which is a flowchart illustrating a training method for an image generation model provided in an embodiment of this application.

[0031] As shown in Figure 1, the training method of this image generation model includes steps S1 to S4.

[0032] Step S1: Obtain the first set of images output by the initial image generation model to be trained.

[0033] For example, image generation has become a key research area in deep learning in recent years. Since the introduction of generative adversarial networks (GANs), generating desired images through models has become relatively straightforward. However, image generation relies heavily on the input image and lacks control over its elements, limiting its commercial applications. While image generation models have seen significant development in recent years, they tend to generate homogeneous images. This problem is generally caused by overfitting due to excessive training epochs, insufficient training samples, or high complexity. Furthermore, image generation models do not discriminate against the diversity of generated images during training, making it difficult to address the issue of generating uniform images. This directly results in existing image generation models still struggling to produce satisfactory target images, leading to low-quality generated images and insufficient efficiency and functionality for commercial projects.

[0034] To address the issue of the uniformity of each first image in the first image set when the image generation model generates the first image set, this application proposes an incremental training method based on the initial image generation model, and introduces a determination of the diversity of the first image set during the training process. The optimization based on the initial image generation model is intended to reduce computational resources and achieve the goal of cost reduction and efficiency improvement.

[0035] For example, image generation models include two categories: unconditional generation and conditional generation. Unconditional generation refers to generating samples from the dataset without conditions, while conditional image generation refers to generating images from the dataset conditionally based on labels, such as generating images based on text content. This application mainly focuses on conditional image generation. When an image generation model generates multiple images based on conditional constraints, it can ensure that the generated images satisfy the conditional constraints while maintaining low similarity between the images.

[0036] For example, when generating images from text, if the text content is "generate the state of the lungs when the fever is 40 degrees Celsius", then each of the first images in the generated first image set can contain the lungs. However, the state of the lungs is diverse, such as the lungs of a normal person, the lungs of a smoker, the lungs of an infant, etc. A fever of 40 degrees Celsius can also include a variety of different physical states. The goal is to minimize the similarity between the first images while satisfying the condition "fever of 40 degrees Celsius - lungs".

[0037] For example, as image generation models continue to develop, more and more excellent models have emerged, such as generative adversarial networks, diffusion models, etc. The implementation methods of the models are different. Therefore, the initial image generation model can be an existing open-source image generation model, or you can set up your own model structure for training. No specific restrictions are imposed here.

[0038] In some implementations, as shown in Figure 2, the initial image generation model includes a text encoding layer, an image generation layer, and an image discrimination layer. Obtaining the first image set output by the initial image generation model to be trained includes: converting the text data of the image to be generated into a first text vector according to the text encoding layer of the initial image generation model; using the first text vector to obtain a first generated image set that conforms to the text data using the image generation layer of the initial image generation model; and using the image discrimination layer of the initial image generation model to determine whether the first generated image set conforms to the text data, thereby obtaining the first image set.

[0039] For example, to generate images from text, the text data is first converted into a machine-readable language and encoded in the text encoding layer. In the image generation layer, multiple images satisfying the text encoding are generated based on the text encoding, forming a first generated image set. This first generated image set is then input to the image discrimination layer to determine whether the images in the first generated image set satisfy the condition constraints, i.e., the text data.

[0040] The text encoding layer is primarily used to represent text in a computer. Existing techniques include discrete representation and word vector generation. The most commonly used technique in discrete representation is one-hot encoding, which assigns an ID to each word. However, when the number of words is large, it is prone to dimensionality explosion and cannot describe the relationships between words. Word vector generation, on the other hand, predicts words appearing in the context during the training of a neural network language model, thereby obtaining the vector representation of each word and the dependencies between words at different positions. This allows for better utilization of the text's contextual information, resulting in a more comprehensive text vector representation. For example, the text encoding layer can be composed of Bi-LSTM or RNN.

[0041] In the image generation layer, the direction of image generation is restricted by the first text vector to obtain an image that satisfies the text data. The image generation layer includes an attention module, a fully connected module, and a convolution module. The first text vector is input to the attention module of the image generation layer. The attention module extracts the dependencies between different positions of the first text vector and inputs the output to the fully connected module of the image generation layer. The fully connected module fuses the features at different positions of the first text vector and then inputs them to the convolution module of the image generation layer. The convolution module obtains an image set that conforms to the text data.

[0042] The image discrimination layer is mainly used to determine whether the images generated by the image generation layer conform to the semantic requirements of the text data. It primarily judges between the real and generated images corresponding to the text data to improve the generation capability of the image generation layer. The real and generated images can be judged based on image similarity; the higher the similarity between the generated and real images, the better. However, to ensure the model's creativity, the generated and real images do not need to be exactly the same.

[0043] When generating images from text, a model can rely on a piece of text and learn through continuous learning to determine the keywords needed in the text for image generation. Alternatively, the model can be directly provided with keywords for image generation, and the model can be generated directly based on the provided keywords.

[0044] In some implementations, as shown in Figure 3, the initial image generation model includes a text encoding layer, an autoencoding layer, an image generation layer, and an image classification layer. Obtaining the first image set output by the initial image generation model to be trained further includes: converting the text data of the images to be generated into second text vectors using the text encoding layer of the initial image generation model; obtaining images unrelated to the text data and using the autoencoding layer of the initial image generation model to obtain image encoding vectors for these images; concatenating the two text vectors and the image encoding vectors to obtain a concatenated vector and using the image generation layer of the initial image generation model to obtain a second generated image set that conforms to the text data; using the image classification layer of the initial image generation model to obtain the probability that each image in the second generated image set conforms to the text data, and determining the first image set based on the probability that each image in the second generated image set conforms to the text data.

[0045] For example, the text used in text-to-image generation is a keyword. A text encoding layer converts the keyword into machine-readable language to obtain a text vector. An image is randomly generated, and this randomly generated image is processed by an autoencoder layer to extract image features, obtaining an image encoding vector. The text vector and the image encoding vector are concatenated to obtain a concatenated vector, which is then input into the image generation layer to obtain a second set of generated images matching the text keywords. An image classification layer calculates the class probability of each image in the second set corresponding to the keyword, sorts the class probabilities of each image in the second set from high to low, and outputs the required number of images according to the output image quantity requirement, thus obtaining multiple images matching the text keywords. By optimizing the similarity between the required number of output images and the real images, the initial image generation model is continuously optimized to generate an image generation model that meets the requirements.

[0046] Furthermore, the entire training process of the initial image generation model involves a continuous interplay and optimization between the generated images and text data. The distribution of images generated by the image generation model continuously meets the requirements of the text data, thereby improving the model's generation capability. To enhance the generalization ability of the initial image generation model, some data that does not match the real images can be added as negative samples during model training to improve the model's resistance to interference. The specific amount of data can be set according to requirements.

[0047] The initial image generation model can be trained according to the model structure set above, or it can be an open-source image generation model, such as a Latent Diffusion Model (LDM).

[0048] After obtaining the initial image generation model, it is incrementally trained to ensure the diversity of images generated. The first image set is obtained by inputting the training text into the initial image generation model.

[0049] Step S2: Obtain the image similarity of each first image in the first image set. The image similarity includes at least color similarity, grayscale similarity, and contour similarity.

[0050] For example, to ensure that each first image in the first image set meets the requirements of the training text while also possessing diversity, the image similarity of each first image in the first image set is calculated. Image similarity is not calculated using cosine similarity between any two first images; instead, it analyzes the color similarity, grayscale similarity, and contour similarity between any two first images. Specifically, color similarity characterizes the color similarity between any two images, grayscale similarity characterizes the similarity of image details after excluding color interference, and contour similarity characterizes the similarity of image layout between any two images. Thus, considering the color, detail, and layout of the images, the image similarity between any two images is obtained.

[0051] For example, given the text "lungs with a 40-degree fever," the initial image generation model generates a first image set containing multiple images that meet the requirement of "lungs with a 40-degree fever." Images with a 40-degree fever can be categorized into different RGB values ​​representing the different colors present under high fever. Therefore, the color similarity between the first images can be calculated based on the RGB similarity. The images in the first image set are then converted to grayscale images to eliminate color interference. Details are extracted from the resulting grayscale images. For example, the lungs could be those of a smoker or a healthy person; there are significant differences in detail between the lungs of smokers and healthy individuals, allowing for the assessment of detail similarity between the first images. Contour extraction is then performed on the converted grayscale images from the first image set to obtain the similarity between image layouts. For example, the lungs in the generated image could be in the lower left corner or in the center. Finally, by combining the color similarity, grayscale similarity, and contour similarity among the first images, the overall image similarity is obtained.

[0052] In some implementations, the first image set includes multiple sub-images. Obtaining the image similarity of each first image in the first image set, where the image similarity includes at least color similarity, grayscale similarity, and contour similarity, includes: performing a color comparison operation on an image control group in the first image set to compare the colors of the sub-images within the image control group to obtain the color similarity of the image control group, wherein the image control group is composed of any two sub-images in the first image set; performing a detail extraction operation on the image control group to compare the details of the sub-images within the image control group to obtain the grayscale similarity of the image control group; performing a contour extraction operation on the image control group to compare the contours of the sub-images within the image control group to obtain the contour similarity of the image control group; and repeatedly obtaining the color similarity, the grayscale similarity, and the contour similarity, and determining the image similarity based on the repeatedly obtained color similarity, grayscale similarity, and contour similarity.

[0053] For example, any two sub-images are obtained from the first image set to form an image control group, and then the image similarity between the two sub-images in the image control group is calculated. Image similarity includes at least color similarity, grayscale similarity, and contour similarity. Image pixel values ​​are obtained from the image control group in the first image set, and the color similarity of the image control group is obtained by comparing the image pixel values. Here, the image contains three channels: RGB. The color similarity of a single channel can be compared, or the average value can be calculated based on the color similarity of the three RGB channels, or different weights can be assigned to the three RGB channels to calculate the color similarity, thereby obtaining the color similarity between the two sub-images in the image control group.

[0054] A detail extraction operation is performed on the image control group in the first image set to obtain content differences between two sub-images in the image control group that can clearly represent the differences between the two sub-images in the image control group, thereby obtaining the detail differences between the two sub-images in the image control group, and finally determining the similarity of image details between the two sub-images in the image control group.

[0055] Contour extraction is performed on the image control group in the first image set to obtain the contour information of the sub-images in the image control group. Based on the contour information in the sub-images, the layout differences of different images are known, and then the similarity of the image layout between two images in the image control group is obtained.

[0056] After obtaining the color similarity, grayscale similarity, and contour similarity of the image control group, the average value of these three similarities is calculated to obtain the image similarity of the image control group. Color extraction, detail extraction, and contour extraction operations are then performed on any two images in the first image set forming the image control group to obtain the corresponding image similarity. This leads to the acquisition of the image similarity for each first image in the first image set.

[0057] For example, if the first image set includes image A, image B, and image C, then the image control groups that need to be formed are: image A and image B, image B and image C, and image A and image C.

[0058] In some implementations, a color comparison operation is performed on the image control group in the first image set to compare the colors of the sub-images within the image control group and obtain the color similarity of the image control group. This includes: obtaining the first pixel vector and the second pixel vector corresponding to two sub-images in the image control group, respectively; and calculating the cosine similarity based on the first pixel vector and the second pixel vector to obtain the color similarity of the two sub-images in the image control group.

[0059] For example, the pixel values ​​of two sub-images in the image control group are obtained, the corresponding pixel vectors are obtained based on the pixel values ​​of the sub-images, and the cosine similarity is calculated based on the pixel vectors to obtain the color similarity between the two sub-images.

[0060] For example, an image control group contains three channels: RGB. Taking one channel as an example, obtain the pixel values ​​of the sub-image in that channel, and then arrange the sub-images in order from left to right and top to bottom to form the pixel vector of the sub-image in that channel. Taking a four-grid image as an example, if the top-left pixel value is 0, the top-right pixel value is 1, the bottom-left pixel value is 2, and the bottom-right pixel value is 3, then the pixel vector corresponding to this four-grid image is...

[0061] <0,1,2,3>. Calculate the cosine similarity of the pixel vectors corresponding to the sub-images. The smaller the difference between the pixel values ​​in the images, the greater the color similarity of the images.

[0062] In some implementations, performing detail extraction on the image control group to compare the details of the sub-images within the image control group and obtain the grayscale similarity of the image control group includes: performing grayscale transformation on two sub-images in the image control group to obtain a first grayscale image and a second grayscale image; obtaining a third pixel vector of the first grayscale image and a fourth pixel vector of the second grayscale image; and calculating a cosine similarity based on the third pixel vector and the fourth pixel vector to obtain the grayscale similarity of the two sub-images in the image control group.

[0063] For example, to eliminate the interference of image color on image detail extraction, the sub-images in the image control group are first subjected to grayscale transformation to obtain the first grayscale image and the second grayscale image corresponding to the two sub-images. The grayscale transformation can use binarization technology to convert the color image into a grayscale image, and the image value contains only two data types. The threshold for binarization transformation can be set by the user.

[0064] When a grayscale image contains only 0 and 1, the location information corresponding to 0 represents the background, and the location information corresponding to 1 represents the target object, such as the lungs. Therefore, after obtaining the third pixel vector of the first grayscale image and the fourth pixel vector of the second grayscale image, the larger the calculation result of the cosine similarity, the greater the grayscale similarity.

[0065] In some implementations, performing contour extraction on the image control group to compare the contours of sub-images within the image control group and obtain the contour similarity of the image control group includes: adding noise data to the first grayscale image for blurring to obtain a first noisy image, and adding noise data to the second grayscale image for blurring to obtain a second noisy image; extracting the contour of the first noisy image using binarization to obtain a first contour vector, and extracting the contour of the second noisy image using binarization to obtain a second contour vector; and calculating the cosine similarity based on the first contour vector and the second contour vector to obtain the contour similarity between the two sub-images in the image control group.

[0066] For example, to reduce the interference of color on image contour similarity, the images in the image control group are processed into grayscale to obtain a first grayscale image and a second grayscale image. Then, noise data is added to both the first and second grayscale images for blurring, where the noise data can be white noise or random noise, resulting in a first noise image corresponding to the first grayscale image and a second noise image corresponding to the second grayscale image. Based on this, the first and second noise images are binarized to obtain the image contour information. Cosine similarity is calculated based on the first and second contour vectors to obtain the contour similarity between the two sub-images in the image control group.

[0067] Step S3: Calculate the loss value of the loss function of the initial image generation model based on the image similarity.

[0068] For example, image similarity represents the similarity between two sub-images in the image control group. After obtaining the image similarity of each first image in the first image set, the total similarity of the first image set is obtained by summing and averaging the image similarities of all image control groups. The loss function is the total similarity of the first image set. That is, the total similarity of the first image set is used as the loss function of the initial image generation model.

[0069] The loss function's loss value is positively correlated with the total similarity; the higher the total similarity, the higher the loss value; conversely, the lower the total similarity, the lower the loss value. In other words, the lower the similarity between images, the better the diversity, the more the requirements are met, and the smaller the loss value.

[0070] Step S4: Update the model parameters of the initial image generation model according to the loss value until the target image generation model is obtained.

[0071] For example, when the loss value does not meet the requirements, the model parameters of the initial image generation model are updated, and the steps in Step 1 to Step 3 are executed again until the loss value meets the requirements. The corresponding model parameters of the current initial image generation model are then saved as the target image generation model.

[0072] Among these applications, the target image generation model can be used in the field of digital healthcare to generate relevant images based on text. This allows doctors to better utilize patient information to obtain corresponding medical images during diagnosis and treatment, assisting doctors in helping patients better understand their symptoms and improving the efficiency of explaining the severity of their conditions. Furthermore, it can provide patients with a more vivid understanding of their condition, encouraging them to actively cooperate with doctors in treatment or to change unhealthy habits, such as smoking.

[0073] Please refer to Figure 4, which illustrates an image generation model training device 200 provided in an embodiment of this application. The image generation model training device 200 includes a data acquisition module 201, a data calculation module 202, a data processing module 203, and a data update module 204. Specifically, the data acquisition module 201 acquires a first image set output by an initial image generation model to be trained; the data calculation module 202 acquires the image similarity of each first image in the first image set, where image similarity includes at least color similarity, grayscale similarity, and contour similarity; the data processing module 203 calculates the loss value of the loss function of the initial image generation model based on the image similarity; and the data update module 204 updates the model parameters of the initial image generation model based on the loss value until a target image generation model is obtained.

[0074] In some implementations, the initial image generation model includes a text encoding layer, an image generation layer, and an image discrimination layer. During the process of acquiring the first image set output by the initial image generation model to be trained, the data acquisition module 201 performs the following:

[0075] The text data of the image to be generated is converted into a first text vector according to the text encoding layer of the initial image generation model;

[0076] The first text vector is used to obtain a first generated image set that conforms to the text data through the image generation layer of the initial image generation model;

[0077] The first generated image set is obtained by using the image discrimination layer of the initial image generation model to determine whether it conforms to the text data.

[0078] In some implementations, the initial image generation model includes a text encoding layer, an autoencoding layer, an image generation layer, and an image classification layer. During the process of acquiring the first image set output by the initial image generation model to be trained, the data acquisition module 201 also performs the following:

[0079] The text data of the image to be generated is converted into a second text vector according to the text encoding layer of the initial image generation model;

[0080] Obtain an image unrelated to the text data, and use the autoencoder layer of the initial image generation model to obtain an image encoding vector from the image;

[0081] The two text vectors and the image encoding vector are concatenated to obtain a concatenated vector. The concatenated vector is then used in the image generation layer of the initial image generation model to obtain a second generated image set that conforms to the text data.

[0082] The probability that each image in the second generated image set matches the text data is obtained by using the image classification layer of the initial image generation model, and the first image set is determined based on the probability that each image in the second generated image set matches the text data.

[0083] In some implementations, the first image set includes multiple sub-images. During the process of acquiring the image similarity of each first image in the first image set, where image similarity includes at least color similarity, grayscale similarity, and contour similarity, the data calculation module 202 performs the following:

[0084] A color comparison operation is performed on the image control group in the first image set to compare the colors of the sub-images within the image control group and obtain the color similarity of the image control group, wherein the image control group is composed of any two sub-images in the first image set;

[0085] A detail extraction operation is performed on the image control group to compare the details of the sub-images within the image control group and obtain the grayscale similarity of the image control group.

[0086] A contour extraction operation is performed on the image control group to compare the contours of the sub-images within the image control group and obtain the contour similarity of the image control group.

[0087] The color similarity, grayscale similarity, and contour similarity are obtained multiple times, and the image similarity is determined based on the obtained color similarity, grayscale similarity, and contour similarity.

[0088] In some implementations, the data calculation module 202 performs a color comparison operation on the image control group in the first image set to compare the colors of the sub-images within the image control group and obtain the color similarity of the image control group. During this process, the module executes:

[0089] Obtain the first pixel vector and the second pixel vector corresponding to the two sub-images in the image control group, respectively;

[0090] The color similarity between two sub-images in the image control group is obtained by calculating the cosine similarity based on the first pixel vector and the second pixel vector.

[0091] In some implementations, the data calculation module 202 performs the following during the process of performing detail extraction on the image control group to compare the details of sub-images within the image control group and obtain the grayscale similarity of the image control group:

[0092] The two sub-images in the image control group are transformed by grayscale to obtain the first grayscale image and the second grayscale image;

[0093] Obtain the third pixel vector of the first grayscale image and the fourth pixel vector of the second grayscale image;

[0094] The grayscale similarity between the two sub-images in the image control group is obtained by calculating the cosine similarity based on the third pixel vector and the fourth pixel vector.

[0095] In some implementations, the data calculation module 202 performs the following during the process of performing contour extraction on the image control group to compare the contours of sub-images within the image control group and obtain the contour similarity of the image control group:

[0096] The first grayscale image is blurred by adding noise data, and the second grayscale image is blurred by adding noise data, to obtain the second noise image.

[0097] Contour extraction is performed on the first noisy image using binarization technology to obtain a first contour vector, and contour extraction is performed on the second noisy image using binarization to obtain a second contour vector;

[0098] Based on the first contour vector and the second contour vector, cosine similarity is calculated to obtain the contour similarity between two sub-images in the image control group.

[0099] In some implementations, the image generation model training device 200 is applied to a terminal device.

[0100] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned training method embodiment of the image generation model, and will not be repeated here.

[0101] Please refer to Figure 5, which is a schematic block diagram of the structure of the terminal device provided in the embodiment of this application.

[0102] As shown in Figure 5, the terminal device 300 includes a processor 301 and a memory 302. The processor 301 and the memory 302 are connected through a bus 303, such as an I2C (Inter-integrated Circuit) bus.

[0103] Specifically, processor 301 provides computing and control capabilities to support the operation of the entire server. Processor 301 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0104] Specifically, the memory 302 can be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a portable hard drive, etc.

[0105] Those skilled in the art will understand that the structure shown in Figure 5 is merely a block diagram of a portion of the structure related to the embodiments of this application, and does not constitute a limitation on the terminal device to which the embodiments of this application are applied. A specific terminal device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0106] The processor 301 is used to run a computer program stored in the memory, and implements the training method of the image generation model provided in any embodiment of this application when executing the computer program.

[0107] In some implementations, the processor 301 is used to run a computer program stored in memory, applied to a terminal device, and performs the following steps when executing the computer program:

[0108] Obtain the first set of images output by the initial image generation model to be trained;

[0109] Obtain the image similarity of each first image in the first image set. The image similarity includes at least color similarity, grayscale similarity, and contour similarity.

[0110] The loss value of the loss function of the initial image generation model is calculated based on the image similarity.

[0111] The model parameters of the initial image generation model are updated based on the loss value until the target image generation model is obtained.

[0112] In some implementations, the initial image generation model includes a text encoding layer, an image generation layer, and an image discrimination layer. During the process of acquiring the first image set output by the initial image generation model to be trained, the processor 301 executes:

[0113] The text data of the image to be generated is converted into a first text vector according to the text encoding layer of the initial image generation model;

[0114] The first text vector is used to obtain a first generated image set that conforms to the text data through the image generation layer of the initial image generation model;

[0115] The first generated image set is obtained by using the image discrimination layer of the initial image generation model to determine whether it conforms to the text data.

[0116] In some implementations, the initial image generation model includes a text encoding layer, an autoencoding layer, an image generation layer, and an image classification layer. During the process of acquiring the first image set output by the initial image generation model to be trained, the processor 301 also performs the following:

[0117] The text data of the image to be generated is converted into a second text vector according to the text encoding layer of the initial image generation model;

[0118] Obtain an image unrelated to the text data, and use the autoencoder layer of the initial image generation model to obtain an image encoding vector from the image;

[0119] The two text vectors and the image encoding vector are concatenated to obtain a concatenated vector. The concatenated vector is then used in the image generation layer of the initial image generation model to obtain a second generated image set that conforms to the text data.

[0120] The probability that each image in the second generated image set matches the text data is obtained by using the image classification layer of the initial image generation model, and the first image set is determined based on the probability that each image in the second generated image set matches the text data.

[0121] In some implementations, the first image set includes multiple sub-images. During the process of acquiring the image similarity of each first image in the first image set, where the image similarity includes at least color similarity, grayscale similarity, and contour similarity, the processor 301 performs the following:

[0122] A color comparison operation is performed on the image control group in the first image set to compare the colors of the sub-images within the image control group and obtain the color similarity of the image control group, wherein the image control group is composed of any two sub-images in the first image set;

[0123] A detail extraction operation is performed on the image control group to compare the details of the sub-images within the image control group and obtain the grayscale similarity of the image control group.

[0124] A contour extraction operation is performed on the image control group to compare the contours of the sub-images within the image control group and obtain the contour similarity of the image control group.

[0125] The color similarity, grayscale similarity, and contour similarity are obtained multiple times, and the image similarity is determined based on the obtained color similarity, grayscale similarity, and contour similarity.

[0126] In some embodiments, during the process of performing a color comparison operation on the image control group in the first image set to compare the colors of sub-images within the image control group and obtain the color similarity of the image control group, the processor 301 performs the following:

[0127] Obtain the first pixel vector and the second pixel vector corresponding to the two sub-images in the image control group, respectively;

[0128] The color similarity between two sub-images in the image control group is obtained by calculating the cosine similarity based on the first pixel vector and the second pixel vector.

[0129] In some embodiments, during the process of performing a detail extraction operation on the image control group to compare the details of sub-images within the image control group and obtain the grayscale similarity of the image control group, the processor 301 performs the following:

[0130] The two sub-images in the image control group are transformed by grayscale to obtain the first grayscale image and the second grayscale image;

[0131] Obtain the third pixel vector of the first grayscale image and the fourth pixel vector of the second grayscale image;

[0132] The grayscale similarity between the two sub-images in the image control group is obtained by calculating the cosine similarity based on the third pixel vector and the fourth pixel vector.

[0133] In some embodiments, during the process of performing a contour extraction operation on the image control group to compare the contours of sub-images within the image control group and obtain the contour similarity of the image control group, the processor 301 performs the following:

[0134] The first grayscale image is blurred by adding noise data, and the second grayscale image is blurred by adding noise data, to obtain the second noise image.

[0135] Contour extraction is performed on the first noisy image using binarization technology to obtain a first contour vector, and contour extraction is performed on the second noisy image using binarization to obtain a second contour vector;

[0136] Based on the first contour vector and the second contour vector, cosine similarity is calculated to obtain the contour similarity between two sub-images in the image control group.

[0137] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the terminal device described above can be referred to the corresponding process in the aforementioned training method embodiment of the image generation model, and will not be repeated here.

[0138] This application also provides a storage medium for computer-readable storage, which stores one or more programs that can be executed by one or more processors to implement the steps of a training method for any image generation model provided in the embodiments of this application.

[0139] The storage medium can be an internal storage unit of the terminal device described in the aforementioned embodiments, such as the terminal device's memory. Alternatively, the storage medium can be an external storage device of the terminal device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card.

[0140] It will be understood by those skilled in the art that all or some of the steps in the methods disclosed above, and the functional modules / units in the apparatus, can be implemented as software, firmware, hardware, and suitable combinations thereof. In hardware embodiments, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0141] It should be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, herein, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0142] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training method for an image generation model, characterized in that, The training method includes: obtaining a first image set output by an initial image generation model to be trained; obtaining the image similarity of each first image in the first image set, wherein the image similarity includes at least color similarity, grayscale similarity, and contour similarity; calculating the loss value of the loss function of the initial image generation model based on the image similarity; updating the model parameters of the initial image generation model based on the loss value until a target image generation model is obtained; wherein the initial image generation model includes a text encoding layer, an image generation layer, and an image discrimination layer, and obtaining the first image set output by the initial image generation model to be trained includes: converting the text data of the image to be generated into a first text vector according to the text encoding layer of the initial image generation model; using the image generation layer of the initial image generation model to obtain a first generated image set that conforms to the text data; using the image discrimination layer of the initial image generation model to determine whether the first generated image set conforms to the text data, thereby obtaining the first image set.

2. The method according to claim 1, characterized in that, The initial image generation model includes a text encoding layer, an autoencoding layer, an image generation layer, and an image classification layer. Obtaining the first image set output by the initial image generation model to be trained further includes: converting the text data of the images to be generated into second text vectors using the text encoding layer of the initial image generation model; obtaining images unrelated to the text data and using the autoencoding layer of the initial image generation model to obtain image encoding vectors for these images; concatenating the second text vector and the image encoding vector to obtain a concatenated vector and using the image generation layer of the initial image generation model to obtain a second generated image set that conforms to the text data; using the image classification layer of the initial image generation model to obtain the probability that each image in the second generated image set conforms to the text data, and determining the first image set based on the probability that each image in the second generated image set conforms to the text data.

3. The method according to any one of claims 1-2, characterized in that, The first image set includes multiple sub-images. The step of obtaining the image similarity of each first image in the first image set, where image similarity includes at least color similarity, grayscale similarity, and contour similarity, includes: performing a color comparison operation on an image control group in the first image set to compare the colors of the sub-images within the image control group to obtain the color similarity of the image control group, wherein the image control group is composed of any two sub-images in the first image set; performing a detail extraction operation on the image control group to compare the details of the sub-images within the image control group to obtain the grayscale similarity of the image control group; performing a contour extraction operation on the image control group to compare the contours of the sub-images within the image control group to obtain the contour similarity of the image control group; and repeatedly obtaining the color similarity, the grayscale similarity, and the contour similarity, and determining the image similarity based on the repeatedly obtained color similarity, grayscale similarity, and contour similarity.

4. The method according to claim 3, characterized in that, The step of performing a color comparison operation on the image control group in the first image set to compare the colors of the sub-images within the image control group and obtain the color similarity of the image control group includes: obtaining the first pixel vector and the second pixel vector corresponding to the two sub-images in the image control group respectively; and calculating the cosine similarity based on the first pixel vector and the second pixel vector to obtain the color similarity of the two sub-images in the image control group.

5. The method according to claim 3, characterized in that, The step of performing detail extraction on the image control group to compare the details of the sub-images within the image control group and obtain the grayscale similarity of the image control group includes: performing grayscale transformation on two sub-images in the image control group to obtain a first grayscale image and a second grayscale image; obtaining the third pixel vector of the first grayscale image and the fourth pixel vector of the second grayscale image; and calculating the cosine similarity based on the third pixel vector and the fourth pixel vector to obtain the grayscale similarity of the two sub-images in the image control group.

6. The method according to claim 5, characterized in that, The step of performing contour extraction on the image control group to compare the contours of sub-images within the image control group and obtain the contour similarity of the image control group includes: adding noise data to the first grayscale image for blurring to obtain a first noisy image, and adding noise data to the second grayscale image for blurring to obtain a second noisy image; extracting the contour of the first noisy image using binarization technology to obtain a first contour vector, and extracting the contour of the second noisy image using binarization to obtain a second contour vector; and calculating the cosine similarity based on the first contour vector and the second contour vector to obtain the contour similarity between the two sub-images in the image control group.

7. A training device for an image generation model, characterized in that, include: The data acquisition module is used to acquire the first set of images output by the initial image generation model to be trained; The data calculation module is used to obtain the image similarity of each first image in the first image set. The image similarity includes at least color similarity, grayscale similarity and contour similarity. The data processing module is used to calculate the loss value of the loss function of the initial image generation model based on the image similarity. The data update module is used to update the model parameters of the initial image generation model according to the loss value until the target image generation model is obtained; wherein, the initial image generation model includes a text encoding layer, an image generation layer, and an image discrimination layer, and obtaining the first image set output by the initial image generation model to be trained includes: converting the text data of the image to be generated into a first text vector according to the text encoding layer of the initial image generation model; using the image generation layer of the initial image generation model to obtain a first generated image set that conforms to the text data; using the image discrimination layer of the initial image generation model to determine whether the first generated image set conforms to the text data, thereby obtaining the first image set.

8. A terminal device, characterized in that, The terminal device includes a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the training method of the image generation model as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, When the computer-readable storage medium is executed by one or more processors, the one or more processors perform the steps of the training method for the image generation model as described in any one of claims 1 to 6.

Images