Image generation model training method and device, electronic equipment and storage medium

Abstract

The invention provides an image generation model training method and device, electronic equipment and a storage medium, and belongs to the technical field of computers. In the method, image deduplication and text addition are carried out on a first training data set, a first intermediate data set and a second intermediate data set are obtained, and then a second training data set is obtained. The images in the second training data set are reserved sample images, and each sample image corresponds to a plurality of sample texts, so that when the second training data set is used for training, the interference of low-quality data is reduced, and the diversity of training samples is improved. Compared with the image generation model trained by using the first training data set, the image generation model trained by using the above mode not only avoids the overfitting problem of the image generation model and improves the generalization performance of the model, but also avoids the problems of overexposure, supersaturation, incomplete details and the like, and improves the image quality and diversity of the generated image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a training method, device, electronic device and storage medium for an image generation model. Background Art

[0002] With the development of computer technology, image generation models based on diffusion models are also improving. Compared with traditional image generation models based on generative adversarial networks, image generation models based on diffusion models can obtain higher quality and more diverse generated images by gradually denoising. However, image generation models based on diffusion models may have overfitting problems when dealing with tasks in certain specific fields, resulting in poor results in generated images. For example, when an image of an animal is generated by an image generation model, the generated image of the image generation model may have quality problems such as overexposure, oversaturation, and incomplete details. Therefore, how to train image generation models based on diffusion models to improve the quality of generated images is an urgent problem to be solved. Summary of the invention

[0003] The present disclosure provides a training method, device, electronic device and storage medium for an image generation model, which avoids the overfitting problem of the image generation model by performing image deduplication and text addition on a training data set, and also avoids problems such as overexposure, oversaturation and incomplete details, thereby improving the image quality and diversity of the generated images. The technical solutions of the present disclosure are as follows:

[0004] According to one aspect of an embodiment of the present disclosure, a method for training an image generation model is provided, the method comprising:

[0005] Deduplication of images is performed on a first training data set to obtain a first intermediate data set, wherein the first training data set includes a plurality of sample images and an original description text of each sample image, the images in the first intermediate data set are sample images retained from the plurality of sample images, and each sample image in the first intermediate data set corresponds to an original description text;

[0006] Adding text to the first training data set to obtain a second intermediate data set, wherein the images in the second intermediate data set are the multiple sample images, each sample image in the second intermediate data set corresponds to multiple sample texts, and the multiple sample texts of each sample image include an original description text of the sample image and an additional description text of the sample image, and the additional description text and the original description text are used to describe the sample image from different angles;

[0007] Based on the first intermediate data set and the second intermediate data set, determining a second training data set, wherein images in the second training data set are sample images retained from the plurality of sample images, and each sample image in the second training data set corresponds to a plurality of sample texts;

[0008] Based on the second training data set, an image generation model is trained, where the image generation model is used to generate an image that satisfies the description of the text based on the text input into the image generation model.

[0009] According to another aspect of an embodiment of the present disclosure, a training device for an image generation model is provided, the device comprising:

[0010] an image processing unit configured to perform image deduplication on a first training data set to obtain a first intermediate data set, wherein the first training data set includes a plurality of sample images and an original description text of each sample image, the images in the first intermediate data set are sample images retained from the plurality of sample images, and each sample image in the first intermediate data set corresponds to an original description text;

[0011] a text processing unit configured to add text to the first training data set to obtain a second intermediate data set, wherein the images in the second intermediate data set are the multiple sample images, each sample image in the second intermediate data set corresponds to multiple sample texts, and the multiple sample texts of each sample image include an original description text of the sample image and an additional description text of the sample image, wherein the additional description text and the original description text are used to describe the sample image from different angles;

[0012] a determining unit configured to determine a second training data set based on the first intermediate data set and the second intermediate data set, wherein images in the second training data set are sample images retained from the plurality of sample images, and each sample image in the second training data set corresponds to a plurality of sample texts;

[0013] The training unit is configured to train an image generation model based on the second training data set, wherein the image generation model is used to generate an image that satisfies the description of the text based on the text input into the image generation model.

[0014] In some embodiments, the image processing unit includes:

[0015] a feature extraction subunit, configured to input each sample image in the first training data set into an image extraction model for feature extraction, and obtain image features of each sample image in the first training data set, the image extraction model being used to output image features based on the image input to the image extraction model, the image features including at least one of color features, texture features, and shape features;

[0016] a clustering subunit configured to perform multi-level clustering on the plurality of sample images in the first training data set based on the image features of each sample image in the first training data set to obtain a plurality of cluster groups, wherein the similarity between the plurality of sample images in each cluster group is not less than a similarity threshold;

[0017] The image processing subunit is configured to retain, for the multiple cluster groups, sample images that meet preset conditions in each cluster group and discard other sample images in each cluster group to obtain the first intermediate data set.

[0018] In some embodiments, the clustering subunit is configured to perform first-level clustering on multiple sample images in the first training data set based on image features of each sample image in the first training data set to obtain a preset number of intermediate cluster groups, and the sample images in each intermediate cluster group belong to the same field; for the preset number of intermediate cluster groups, second-level clustering is performed on the sample images in each intermediate cluster group to obtain the multiple cluster groups.

[0019] In some embodiments, the clustering subunit is configured to, for the multiple cluster groups, input multiple sample images in each cluster group into an image scoring model for scoring, and obtain scores for the multiple sample images in each cluster group, respectively, where the score of each sample image is determined by at least one of an aesthetic score, a resolution score, and an image quality score; and in descending order of the scores of the multiple sample images in each cluster group, retain the sample images whose ranking is not lower than a ranking threshold in each cluster group, and discard the sample images whose ranking is lower than the ranking threshold in each cluster group.

[0020] In some embodiments, the image processing subunit is configured to, for the plurality of cluster groups, input a plurality of sample images in each cluster group into an image scoring model for scoring, and obtain scores for the plurality of sample images in each cluster group, respectively, wherein the score of each sample image is determined by at least one of an aesthetic score, a resolution score, and an image quality score;

[0021] Based on the scores of the multiple sample images in each cluster group, the sample images with scores not less than the score threshold are retained, and the sample images with scores less than the score threshold are discarded.

[0022] In some embodiments, the text processing unit is configured to input the instruction text and each sample image in the first training data set into the image description model for text description, and obtain additional description text for each sample image in the first training data set, respectively, wherein the instruction text is used to control the image description model to generate text that meets the requirements, and the image description model is used to generate text based on the image input into the image description model; the additional description text for each sample image is added to the first training data set to obtain the second intermediate data set.

[0023] In some embodiments, the training unit is configured to sample multiple sample images in the second training data set, and use the sampled sample images as sample input images; for any sample input image, based on preset sampling probabilities corresponding to different sample texts, sample multiple sample texts corresponding to the sample input image, and use the sampled sample texts as sample input texts of the sample input image; and train the image generation model based on the sample input images and the sample input texts of each sample input image.

[0024] In some embodiments, the apparatus further comprises:

[0025] an acquisition unit, configured to acquire a time step threshold and a text guide coefficient, wherein the time step threshold is a threshold for distinguishing a first time step interval from a second time step interval, the first time step interval is before the second time step interval, the size of the text guide coefficient is positively correlated with the degree of influence of the text on the image generation model, the text guide coefficient includes a first guide coefficient and a second guide coefficient, and the first guide coefficient is smaller than the second guide coefficient;

[0026] A denoising unit is configured to input the prompt text into the image generation model for denoising processing in multiple time step intervals to obtain a result image, wherein the multiple time step intervals include the first time step interval and the second time step interval, the text guidance coefficient of the image generation model in the first time step interval is the first guidance coefficient, the text guidance coefficient of the image generation model in the second time step interval is the second guidance coefficient, and the result image meets the description of the prompt text.

[0027] In some embodiments, the denoising unit is configured to, when any time step is not greater than the time step threshold, perform denoising on the intermediate image of the previous time step through the image generation model based on the first guided coefficient to obtain the intermediate image of the time step; when any time step is greater than the time step threshold, perform denoising on the intermediate image of the previous time step through the image generation model based on the second guided coefficient to obtain the intermediate image of the time step.

[0028] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0029] one or more processors;

[0030] A memory for storing program codes executable by the processor;

[0031] Among them, the processor is configured to execute the program code to implement the training method of the above-mentioned image generation model.

[0032] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When a program code in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device is enabled to perform the above-mentioned training method of the image generation model.

[0033] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the above-mentioned training method of the image generation model when executed by a processor.

[0034] The disclosed embodiment provides a method for training an image generation model, in which a first training data set is subjected to image deduplication and text addition to obtain a first intermediate data set and a second intermediate data set, and then a second training data set is obtained. Since the images in the second training data set are sample images retained after image deduplication, and each sample image in the second training data set corresponds to multiple sample texts, the use of the second training data set for training not only reduces the interference of low-quality data, but also improves the richness and diversity of training samples. Compared with the image generation model trained using the first training data set, the image generation model trained using the above method not only reduces the interference of low-quality data, avoids the overfitting problem of the image generation model, improves the generalization performance of the model, but also avoids problems such as overexposure, oversaturation, and incomplete details, and improves the image quality and diversity of the generated images.

[0035] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0037] Figure 1 It is a schematic diagram of an implementation environment of a training method for an image generation model according to an exemplary embodiment.

[0038] Figure 2The figure is a flowchart of a method for training an image generation model according to an exemplary embodiment.

[0039] Figure 3 The figure is a flowchart of another method for training an image generation model according to an exemplary embodiment.

[0040] Figure 4 It is a block diagram of a training device for an image generation model according to an exemplary embodiment.

[0041] Figure 5 It is a block diagram of another training device for an image generation model according to an exemplary embodiment.

[0042] Figure 6 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0043] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0044] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0045] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the multiple training data sets, instruction texts, time step thresholds and text guidance coefficients involved in this disclosure are all obtained with full authorization.

[0046] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a training method for an image generation model according to an exemplary embodiment. Figure 1 The implementation environment specifically includes: a terminal 101 and a server 102. The terminal 101 can be connected to the server 102 via a wireless network or a wired network.

[0047] The terminal 101 may be at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), and a laptop computer. An application may be installed and run on the terminal 101, and the application may input text of an image generation model to generate an image. The application is associated with the server 102, and the server 102 provides background services to the terminal 101.

[0048] The terminal 101 may generally refer to one of a plurality of terminals. This embodiment is described by taking the terminal 101 as an example. Those skilled in the art may know that the number of the above terminals may be more or less. For example, the above terminals may be several, or the above terminals may be dozens or hundreds, or a greater number. The embodiment of the present disclosure does not limit the number and device type of the terminals.

[0049] The server 102 is at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Optionally, the number of the above servers may be more or less, which is not limited in the embodiments of the present disclosure. Of course, the server 102 may also include other functional servers to provide more comprehensive and diversified services. In some embodiments, the server 102 undertakes the main computing work, and the terminal 101 undertakes the secondary computing work; or, the server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or, the server 102 and the terminal 101 use a distributed computing architecture for collaborative computing. The server 102 can be connected to the terminal 101 and other terminals via a wireless network or a wired network. Optionally, the number of the above servers may be more or less, which is not limited in the embodiments of the present disclosure.

[0050] Figure 2 is a flowchart of a method for training an image generation model according to an exemplary embodiment. Figure 2 As shown, the method is executed by the terminal and includes the following steps:

[0051] In step S201, the terminal deduplicates images of the first training data set to obtain a first intermediate data set. The first training data set includes multiple sample images and original description text for each sample image. The images in the first intermediate data set are sample images retained from the multiple sample images. Each sample image in the first intermediate data set corresponds to an original description text.

[0052] In an embodiment of the present disclosure, the sample images in the first intermediate data set are fewer than the sample images in the first training data set. Since the terminal screens and discards the sample images in each cluster group, the images in the first intermediate data set are only the retained sample images; since the sample images retained by the terminal are images that meet the preset conditions, the sample images in the first intermediate data set are of higher quality and more representative. The training data in the first intermediate data set is in the form of image-text pairs. In other words, the first intermediate data set includes multiple groups of training data, each group of training data includes a retained sample image and an original description text of the retained sample image. Among them, the original description text is used to describe the image content of the sample image. It should be noted that the original description text may include not only words associated with the image content of the sample image, but also words that are unrelated to the image content of the sample image. For example, a sample image is an image of animal P, and the original description text is "Selected wallpapers of animal P".

[0053] In step S202, the terminal adds text to the first training data set to obtain a second intermediate data set. The images in the second intermediate data set are multiple sample images. Each sample image in the second intermediate data set corresponds to multiple sample texts. The multiple sample texts of each sample image include the original description text of the sample image and the additional description text of the sample image. The additional description text and the original description text are used to describe the sample image from different angles.

[0054] In the disclosed embodiment, the text data in the second intermediate data set is more than the text data in the first training data set. Since the terminal adds the additional description text of each sample image generated by the model to the training data set, the sample text associated with each sample image in the second intermediate data set is richer and more diverse. The training data in the second intermediate data set is in the form of image-text pairs. In other words, the second intermediate data set includes multiple sets of training data, each set of training data includes a sample image and multiple sample texts corresponding to the sample image.

[0055] The additional description text and the original description text usually describe the sample image from different angles. For example, the original description text of a sample image is "Animal P's selected mobile wallpaper", which is used to describe the main elements and usage scenarios of the sample image; an additional description text of the sample image is "A brown animal P is running towards the distance on the grassland", which is used to describe the detailed content of the sample image.

[0056] In step S203, the terminal determines a second training data set based on the first intermediate data set and the second intermediate data set, where the images in the second training data set are sample images retained from multiple sample images, and each sample image in the second training data set corresponds to multiple sample texts.

[0057] In the disclosed embodiment, the images in the second training data set are the same as the images in the first intermediate data set, and are both sample images retained in the first training data set; the multiple sample texts of each sample image in the second training data set are the same as the multiple sample texts of the sample image in the second intermediate data set, and are both multiple sample texts obtained after adding additional descriptive text.

[0058] In step S204, the terminal trains an image generation model based on the second training data set, where the image generation model is used to generate an image that satisfies the description of the text based on the text input into the image generation model.

[0059] In the disclosed embodiment, the second training data set is a training data set obtained by removing duplicate images and adding text to the first training data set. Sampling is performed in the second training data set, and the sampled sample images and sample texts are input into the image generation model for training.

[0060] The disclosed embodiment provides a method for training an image generation model, in which a first training data set is subjected to image deduplication and text addition to obtain a first intermediate data set and a second intermediate data set, and then a second training data set is obtained. Since the images in the second training data set are sample images retained after image deduplication, and each sample image in the second training data set corresponds to multiple sample texts, the use of the second training data set for training not only reduces the interference of low-quality data, but also improves the richness and diversity of training samples. Compared with the image generation model trained using the first training data set, the image generation model trained using the above method not only reduces the interference of low-quality data, avoids the overfitting problem of the image generation model, improves the generalization performance of the model, but also avoids problems such as overexposure, oversaturation, and incomplete details, and improves the image quality and diversity of the generated images.

[0061] In some embodiments, performing image deduplication on the first training data set to obtain a first intermediate data set includes:

[0062] Input each sample image in the first training data set into the image extraction model for feature extraction, and obtain image features of each sample image in the first training data set, respectively, the image extraction model is used to output image features based on the image input to the image extraction model, and the image features include at least one of color features, texture features and shape features;

[0063] Based on the image features of each sample image in the first training data set, multi-level clustering is performed on the multiple sample images in the first training data set to obtain multiple cluster groups, and the similarity between the multiple sample images in each cluster group is not less than a similarity threshold;

[0064] For multiple cluster groups, sample images that meet preset conditions in each cluster group are retained, and other sample images in each cluster group are discarded to obtain a first intermediate data set.

[0065] In the embodiment of the present application, the sample images in the first training data set are effectively screened and optimized in the above manner, and the sample images in the first training data set are deduplicated. By performing multi-level clustering on multiple sample images, sample images with high similarity are clustered in the same clustering group, which not only improves the data processing efficiency, but also ensures the accuracy and reliability of clustering. At the same time, by retaining only sample images that meet preset conditions in each clustering group, redundant data and low-quality data can be removed, thereby improving the image diversity and image quality in the training data.

[0066] In some embodiments, based on the image features of each sample image in the first training data set, a plurality of sample images in the first training data set are subjected to multi-level clustering to obtain a plurality of cluster groups, including:

[0067] Based on the image features of each sample image in the first training data set, performing first-level clustering on the plurality of sample images in the first training data set to obtain a preset number of intermediate clustering groups, wherein the sample images in each intermediate clustering group belong to the same field;

[0068] For a preset number of intermediate clustering groups, second-level clustering is performed on the sample images in each intermediate clustering group to obtain multiple clustering groups.

[0069] In the embodiment of the present application, through the first-level clustering, the sample images in the first training data set are divided according to the fields to which they belong, which not only reduces the amount of calculation in the subsequent clustering process, but also helps to improve the accuracy and reliability of clustering. By performing the second-level clustering on the sample images in each intermediate clustering group on the basis of the first-level clustering, a plurality of more subdivided clustering groups are obtained, so that the images in each clustering group are highly similar sample images, which not only further improves the accuracy and reliability of clustering, but also facilitates the subsequent image screening and retention in each clustering group.

[0070] In some embodiments, for multiple cluster groups, retaining sample images that meet preset conditions in each cluster group, discarding other sample images in each cluster group, and obtaining a first intermediate data set includes:

[0071] For multiple cluster groups, multiple sample images in each cluster group are input into the image scoring model for scoring, and scores of the multiple sample images in each cluster group are obtained respectively, and the score of each sample image is determined by at least one of an aesthetic score, a resolution score, and an image quality score;

[0072] According to the order of scores of multiple sample images in each cluster group from high to low, the sample images whose ranking is not lower than the ranking threshold in each cluster group are retained, and the sample images whose ranking is lower than the ranking threshold in each cluster group are discarded.

[0073] In an embodiment of the present application, sample images are scored through an image scoring model, which achieves quantitative evaluation of multiple aspects such as aesthetics, resolution, and image quality of the sample images, which is conducive to retaining high-quality sample images and discarding low-quality sample images, thereby reducing redundant data and low-quality data in the training data set, and improving the richness and diversity of sample images in the training data set, which is conducive to subsequently alleviating the overfitting phenomenon in the image generation model and improving the quality and diversity of the generated images of the image generation model.

[0074] In some embodiments, for multiple cluster groups, retaining sample images that meet preset conditions in each cluster group, discarding other sample images in each cluster group, and obtaining a first intermediate data set includes:

[0075] For multiple cluster groups, multiple sample images in each cluster group are input into the image scoring model for scoring, and scores of the multiple sample images in each cluster group are obtained respectively, and the score of each sample image is determined by at least one of an aesthetic score, a resolution score, and an image quality score;

[0076] Based on the scores of multiple sample images in each cluster group, sample images with scores not less than the score threshold are retained, and sample images with scores less than the score threshold are discarded.

[0077] In an embodiment of the present application, sample images are scored through an image scoring model, which achieves quantitative evaluation of multiple aspects such as aesthetics, resolution, and image quality of the sample images, which is conducive to retaining high-quality sample images and discarding low-quality sample images, thereby reducing redundant data and low-quality data in the training data set, and improving the richness and diversity of sample images in the training data set, which is conducive to subsequently alleviating the overfitting phenomenon in the image generation model and improving the quality and diversity of the generated images of the image generation model.

[0078] In some embodiments, adding text to the first training data set to obtain a second intermediate data set includes:

[0079] Inputting the instruction text and each sample image in the first training data set into the image description model for text description, respectively obtaining additional description text for each sample image in the first training data set, wherein the instruction text is used to control the image description model to generate text that meets the requirements, and the image description model is used to generate text based on the image input into the image description model;

[0080] The additional description text of each sample image is added to the first training data set to obtain a second intermediate data set.

[0081] In the embodiment of the present application, multiple additional description texts can be obtained by combining different instruction texts and different image description models. At the same time, by adding multiple additional description texts to each sample image, the second intermediate data set not only contains the original description text, but also adds other text descriptions related to the sample image content, enriching the information dimension of the training data set and improving the richness and diversity of the training data set.

[0082] In some embodiments, training the image generation model based on the second training data set includes:

[0083] Sampling a plurality of sample images in the second training data set, and using the sampled sample images as sample input images;

[0084] For any sample input image, based on preset sampling probabilities corresponding to different sample texts, multiple sample texts corresponding to the sample input image are sampled, and the sampled sample texts are used as sample input texts of the sample input image;

[0085] Train an image generation model based on sample input images and sample input text for each sample input image.

[0086] In the embodiment of the present application, by sampling different pairs of images and texts, the image generation model can be exposed to diverse input data during the training process, which is conducive to the image generation model learning more abundant data, not only reducing the overfitting phenomenon, but also improving the diversity and flexibility of the image generation model when generating images, and improving the generalization performance of the image generation model. At the same time, by adjusting the preset sampling probability, the sampling ratio of the text generated by different image description models can be adjusted, which is conducive to controlling the distribution of the input data of the image generation model during the training process, thereby further improving the performance and generalization performance of the image generation model.

[0087] In some embodiments, the method further comprises:

[0088] Obtaining a time step threshold and a text guidance coefficient, wherein the time step threshold is a threshold for distinguishing a first time step interval from a second time step interval, wherein the first time step interval is before the second time step interval, and the size of the text guidance coefficient is positively correlated with the degree of influence of the text on the image generation model, and the text guidance coefficient includes a first guidance coefficient and a second guidance coefficient, and the first guidance coefficient is smaller than the second guidance coefficient;

[0089] The prompt text is input into the image generation model for denoising processing in multiple time step intervals to obtain a result image. The multiple time step intervals include a first time step interval and a second time step interval. The text guidance coefficient of the image generation model in the first time step interval is a first guidance coefficient, and the text guidance coefficient of the image generation model in the second time step interval is a second guidance coefficient. The result image meets the description of the prompt text.

[0090] In an embodiment of the present application, by setting different text guidance coefficients in different time step intervals, the degree of influence of text on the image generation model can be controlled. By using a smaller first guidance coefficient in the early stage of image generation (i.e., the first time step interval), the image generation model does not rely too much on the prompt text, which is conducive to improving the diversity of model-generated images; and by using a larger second guidance coefficient in the later stage of image generation (i.e., the second time step interval), the generated image is more consistent with the description of the prompt text, thereby improving the text relevance of the generated image. Therefore, by adjusting the time step threshold and the text guidance coefficient, the diversity of image generation and the relevance to the text can be more flexibly balanced, thereby ensuring the quality and diversity of the generated image.

[0091] In some embodiments, the prompt text is input into the image generation model for denoising at multiple time step intervals to obtain a result image, including:

[0092] When any time step is not greater than the time step threshold, based on the first guidance coefficient, the intermediate image of the previous time step is denoised by the image generation model to obtain the intermediate image of the time step;

[0093] When any time step is greater than the time step threshold, the intermediate image of the previous time step is denoised through the image generation model based on the second guidance coefficient to obtain the intermediate image of the time step.

[0094] In the embodiment of the present application, the image generation process is clearly divided into two stages by setting a time step threshold. By performing denoising on the image according to the preset guidance coefficient in each stage, not only can the image generation model maintain good performance and stability in each stage, but it is also conducive to balancing the diversity of image generation and text relevance, thereby ensuring the quality and diversity of the generated images.

[0095] Above Figure 2 Shown is a flow chart of a training method for an image generation model of the present disclosure. The training scheme for the image generation model provided by the present disclosure is further elaborated below. Figure 3 is a flowchart of another method for training an image generation model according to an exemplary embodiment. Figure 3 The method is executed by the terminal. The method comprises the following steps:

[0096] In step S301, the terminal inputs each sample image in the first training data set into the image extraction model for feature extraction, and obtains the image features of each sample image in the first training data set. The first training data set includes multiple sample images and the original description text of each sample image.

[0097] In an embodiment of the present disclosure, an image extraction model is used to output image features based on an image input to the image extraction model. Typically, an image is input into the image extraction model, and output data of a certain layer of the image extraction model is obtained as a representation vector of the image features. The image features of each sample image include at least one of color features, texture features, and shape features. For ease of description, the representation vector of image features may also be referred to as a feature vector. The sample image is converted into a feature vector through the image extraction model. A feature vector is typically a high-dimensional vector, and each dimension is used to represent a type of feature of the image, such as color features, texture features, and shape features.

[0098] Among them, the first training data set is the original training data set that has not been processed. The training data in the first training data set is in the form of image-text pairs. In other words, the first training data set includes multiple groups of training data, and each group of training data includes a sample image and an original description text of the sample image. The original description text is used to describe the image content of the sample image. It should be noted that the original description text includes not only words associated with the image content of the sample image, but also words that are not related to the image content of the sample image. For example, the original description text includes words such as "wallpaper", "photographic work" and "painting creation" that describe the image type or application scenario of the sample image. Usually, the original description text is not directly obtained by the image description model to perform text description on the sample image, and therefore may include redundant information.

[0099] In step S302, the terminal performs multi-level clustering on multiple sample images in the first training data set based on the image features of each sample image in the first training data set to obtain multiple cluster groups, and the similarity between multiple sample images in each cluster group is not less than a similarity threshold.

[0100] In the disclosed embodiment, different clustering groups contain feature vectors of different sample images. The similarity between feature vectors in the same clustering group is greater than the similarity between feature vectors in different clustering groups. The similarity between feature vectors in the same clustering group is not less than a similarity threshold. Since feature vectors are used to represent sample images, the terminal performs multi-level clustering on multiple sample images in the first training data set, that is, the terminal classifies the feature vectors of each sample image in the first training data set into multiple clustering groups through multi-level clustering. For ease of description, clustering feature vectors is also referred to as clustering sample images. Generally, multiple sample images in each clustering group are used to describe the same main content, and the differences between the multiple sample images are small. For example, there are differences in resolution, aspect ratio, color temperature, etc. between multiple sample images in a clustering group, but the sample images all describe the same animal and the same background.

[0101] In some embodiments, the terminal first performs first-level clustering on multiple sample images, and then performs second-level clustering on each of them to obtain multiple cluster groups. Accordingly, based on the image features of each sample image in the first training data set, the terminal performs first-level clustering on multiple sample images in the first training data set to obtain a preset number of intermediate cluster groups, and the sample images in each intermediate cluster group belong to the same field; for the preset number of intermediate cluster groups, the terminal performs second-level clustering on the sample images in each intermediate cluster group to obtain multiple cluster groups.

[0102] Among them, different intermediate clustering groups contain feature vectors of sample images of different fields. The sample images represented by the feature vectors in the same intermediate clustering group belong to the same field. For the convenience of description, the field of the sample image can also be referred to as the vertical category to which the sample image belongs. For example, in the case of dividing the sample images according to the image theme, the fields to which the multiple sample images belong may include the landscape field, the animal field, the mechanical field, and the architectural field, etc. Correspondingly, the sample images in the intermediate clustering group A all belong to the landscape field, and the sample images in the intermediate clustering B all belong to the animal field. For another example, in the case of further subdividing the sample images according to the image subject on the basis of dividing the sample images according to the image theme, the fields to which the multiple sample images belong may include the insect field, the fish field, the mountain field, the bridge field, the automobile field, etc. Correspondingly, the sample images in the intermediate clustering group a all belong to the insect field, and the sample images in the intermediate clustering b all belong to the automobile field. It should be noted that the division of the above intermediate clustering groups is exemplary and not limited to this.

[0103] In some embodiments, the first level clustering is coarse-grained clustering, and the second level clustering is fine-grained clustering. Accordingly, the first level clustering adopts a clustering algorithm that requires a preset number of intermediate cluster groups, and the number of intermediate cluster groups corresponds to the number of fields. The second level clustering adopts a clustering algorithm that does not require a preset number of cluster groups, so that feature vectors with high similarity are classified into the same cluster group.

[0104] For example, the terminal uses a first clustering algorithm to perform first-level clustering on the feature vectors of multiple sample images, thereby classifying the multiple sample images into N intermediate clustering groups. For example, N is pre-set to 100 or 200, and the terminal uses the first clustering algorithm to perform first-level clustering, and obtains 100 or 200 intermediate clustering groups. The terminal uses a second clustering algorithm to perform second-level clustering on each intermediate clustering group. At this time, there is no need to preset the number of clustering groups. In each intermediate clustering group, the terminal uses the second clustering algorithm to classify feature vectors whose distance in the feature space is less than the distance threshold into the same clustering group. For example, the sample images in cluster group 1 are all animal A on the beach, and the sample images in cluster group 2 are all animal B on the grass.

[0105] The first clustering algorithm is used to divide the data points into K groups, so that the data points in the same group are as similar as possible, and the data points in different groups are as different as possible. The second clustering algorithm is used to divide the data points into density-based groups, which makes it easy to identify groups of arbitrary shapes.

[0106] It should be noted that, through the first-level clustering, the sample images in the first training data set are divided according to their fields, which not only reduces the amount of calculation in the subsequent clustering process, but also helps to improve the accuracy and reliability of clustering. Based on the first-level clustering, the sample images in each intermediate cluster group are clustered at the second level, and multiple cluster groups with more subdivision are obtained, so that the images in each cluster group are highly similar sample images, which not only further improves the accuracy and reliability of clustering, but also facilitates the subsequent image screening and retention in each cluster group.

[0107] In step S303, for multiple clustering groups, the terminal retains sample images that meet preset conditions in each clustering group, discards other sample images in each clustering group, and obtains a first intermediate data set. The images in the first intermediate data set are sample images retained from multiple sample images, and each sample image in the first intermediate data set corresponds to an original description text.

[0108] In the disclosed embodiment, the sample images in the first intermediate data set are fewer than the sample images in the first training data set. Since the terminal screens and discards the sample images in each cluster group, the images in the first intermediate data set are only the retained sample images; since the sample images retained by the terminal are images that meet the preset conditions, the sample images in the first intermediate data set are of higher quality and more representative. The training data in the first intermediate data set is in the form of image-text pairs. In other words, the first intermediate data set includes multiple groups of training data, each group of training data includes a retained sample image and an original description text of the retained sample image.

[0109] It should be noted that, through the above method, the sample images in the first training data set are effectively screened and optimized, and the sample images in the first training data set are deduplicated. By performing multi-level clustering on multiple sample images, sample images with high similarity are clustered in the same clustering group, which not only improves the data processing efficiency, but also ensures the accuracy and reliability of clustering. At the same time, by retaining only sample images that meet the preset conditions in each clustering group, redundant data and low-quality data can be removed, thereby improving the image diversity and image quality in the training data.

[0110] In some embodiments, the terminal determines whether to retain or discard the sample images by scoring each sample image in the same cluster group. Accordingly, for multiple cluster groups, the terminal inputs multiple sample images in each cluster group into the image scoring model for scoring, and obtains the scores of the multiple sample images in each cluster group respectively, and the score of each sample image is determined by at least one of the aesthetic score, the resolution score, and the image quality score. The terminal retains the sample images whose ranking is not less than the ranking threshold in each cluster group and discards the sample images whose ranking is less than the ranking threshold in each cluster group according to the order of the scores of the multiple sample images in each cluster group from high to low; or, based on the scores of the multiple sample images in each cluster group, retains the sample images whose scores are not less than the score threshold and discards the sample images whose scores are less than the score threshold.

[0111] Optionally, the score of each sample image can be obtained by weighted sum of multiple values ​​among the aesthetic score, resolution score and image quality score, or can be determined by one of the aesthetic score, resolution score and image quality score. Among them, the aesthetic score (Aesthetic Score) is used to indicate the quality of parameters related to image aesthetics in the sample image, such as color distribution, picture composition, etc.; the resolution score is used to indicate the quality of the resolution (Resolution) of the sample image, such as scoring in pixels; the image quality score is used to indicate the quality of parameters related to image quality (Image Quality) in the sample image, such as clarity, noise level, etc.

[0112] It should be noted that the above-mentioned method of confirming the score of the sample image is only an exemplary description. The terminal can also determine the score of the sample image by considering whether there is overexposure, underexposure, lens distortion, detail defects, etc. in the sample image, which will not be repeated here.

[0113] Optionally, after determining the scores of multiple sample images in each cluster group, the terminal may retain a preset number of sample images with higher scores, or retain sample images with scores at or above a score threshold. For example, the terminal retains the sample image with the highest score, or the terminal retains the sample images ranked in the top three in terms of scores, or the terminal retains the sample images ranked in the top 5% in terms of scores. For another example, the terminal retains sample images with scores of 90 or above.

[0114] It should be noted that the above method of determining whether to retain or discard the sample images by the scores of the sample images is only an exemplary description. The terminal can also directly determine the sample images with better quality through the neural network model and retain them. Alternatively, the terminal supports retaining and discarding sample images by manual annotation, which will not be repeated here.

[0115] It should be noted that by scoring the sample images through the image scoring model, a quantitative evaluation of the sample images in terms of aesthetics, resolution, image quality, etc. is achieved, which is conducive to retaining high-quality sample images and discarding low-quality sample images, thereby reducing redundant data and low-quality data in the training data set, reducing the data repeatability of the sample images, and ensuring the richness and diversity of the sample images in the training data set, which is conducive to subsequently alleviating the overfitting phenomenon of the image generation model and improving the quality and diversity of the images generated by the image generation model.

[0116] In step S304, the terminal inputs the instruction text and each sample image in the first training data set into the image description model for text description, and obtains additional description text for each sample image in the first training data set. The instruction text is used to control the image description model to generate text that meets the requirements, and the additional description text and the original description text are used to describe the sample image from different angles.

[0117] In the disclosed embodiment, the image description model is used to generate text based on the image input into the image description model. By using different image description models and different instruction text combinations, the terminal can obtain multiple additional description texts for the same sample image, making the text description of the sample image richer and the text format more diverse.

[0118] For example, the image description model is a large language model. The terminal uses multiple large language models to describe each sample image in text, and uses multiple instruction texts to guide the model to generate text that meets the requirements during each model processing. For example, the instruction text is "Please describe the scene in the image, including the main elements and the relationship between the elements." For another example, the instruction text is "Please generate the key points in the image".

[0119] The additional description text and the original description text usually describe the sample image from different angles. For example, the original description text of a sample image is "Animal P's selected mobile wallpaper", which is used to describe the main elements and usage scenarios of the sample image; an additional description text of the sample image is "A brown animal P is running towards the distance on the grassland", which is used to describe the detailed content of the sample image.

[0120] In step S305, the terminal adds the additional descriptive text of each sample image to the first training data set to obtain a second intermediate data set. The images in the second intermediate data set are multiple sample images. Each sample image in the second intermediate data set corresponds to multiple sample texts. The multiple sample texts of each sample image include the original description text of the sample image and the additional description text of the sample image.

[0121] In the disclosed embodiment, the text data in the second intermediate data set is more than the text data in the first training data set. Since the terminal adds the additional description text of each sample image generated by the model to the training data set, the sample text associated with each sample image in the second intermediate data set is richer and more diverse. The training data in the second intermediate data set is in the form of image-text pairs. In other words, the second intermediate data set includes multiple sets of training data, each set of training data includes a sample image and multiple sample texts corresponding to the sample image.

[0122] Optionally, each sample text may be associated with a label, and the label is used to indicate the source type of the sample text. For example, the label of a sample text is used to indicate that the sample text is an original description text, or the label of a sample text is used to indicate that the sample text is generated by image description model X, or the label of a sample text is used to indicate that the sample text is generated by image description model Y. In the above manner, it is convenient to uniformly set the preset sampling probabilities corresponding to different labels to regulate the data sampling process during the training process.

[0123] In some embodiments, the terminal uses a trained model to detect sample images. For sample images involving specific text, the terminal associates and adds specific text to the sample text of the sample image. For example, the sample image involving specific text is a sample image of a representative building in the fields of literature, film, television, games, etc. In this case, the specific text is the name of the representative building. The terminal uses a trained model to detect whether a sample image contains a representative building Q. When it is detected that a sample image contains a representative building Q, the terminal adds the name of the representative building Q to the multiple sample texts of the sample image. In other words, the use of text combination ensures that the sample text of the sample image includes specific text, thereby improving the quality of the training data and facilitating the subsequent improvement of the performance of the image generation model.

[0124] It should be noted that multiple additional description texts can be obtained through the combination of different instruction texts and different image description models. At the same time, by adding multiple additional description texts to each sample image, the second intermediate data set not only contains the original description text, but also adds other text descriptions related to the sample image content, enriching the information dimension of the training data set and improving the richness and diversity of the training data set.

[0125] In step S306, the terminal determines a second training data set based on the first intermediate data set and the second intermediate data set, where the images in the second training data set are sample images retained from multiple sample images, and each sample image in the second training data set corresponds to multiple sample texts.

[0126] In the disclosed embodiment, the images in the second training data set are the same as the images in the first intermediate data set, and are both sample images retained in the first training data set; the multiple sample texts of each sample image in the second training data set are the same as the multiple sample texts of the sample image in the second intermediate data set, and are both multiple sample texts obtained after adding additional descriptive text.

[0127] It should be noted that the above steps 301, 302 and 303 are the process of performing image deduplication on the first training data set, and the above steps 304 and 305 are the process of adding text to the first training data. The embodiment of the present application does not limit the order of the image deduplication process and the text addition process, and the simultaneous image deduplication process and text addition process are used as an example for illustration. In the above manner, since the image deduplication process and the text addition process are processed in parallel, processing time can be saved and the processing efficiency of the training data is improved.

[0128] In some embodiments, the terminal first performs an image deduplication process and then performs a text addition process. Accordingly, the terminal performs image deduplication on the first training data set to obtain an intermediate training data set, wherein the images in the intermediate training data set are retained sample images, and the sample texts of the sample images in the intermediate training data set are original description texts; the terminal adds text to the intermediate training data set to obtain a second training data set. In the above manner, since there is no need to generate and add additional description text to the discarded sample images, resource consumption is reduced, the amount of calculation is saved, and the efficiency of forming training data is improved.

[0129] In step S307, the terminal trains an image generation model based on the second training data set, where the image generation model is used to generate an image that satisfies the description of the text based on the text input into the image generation model.

[0130] In the disclosed embodiment, the second training data set is a training data set obtained by removing duplicate images and adding text to the first training data set. Sampling is performed in the second training data set, and the sampled sample images and sample texts are input into the image generation model for training.

[0131] In some embodiments, after sampling a sample image, the terminal samples the sample text corresponding to the sample image. Accordingly, the terminal samples multiple sample images in the second training data set, and uses the sampled sample images as sample input images; for any sample input image, the terminal samples multiple sample texts corresponding to the sample input image based on preset sampling probabilities corresponding to different sample texts, and uses the sampled sample texts as sample input texts of the sample input image; the terminal trains the image generation model based on the sample input images and the sample input texts of each sample input image.

[0132] When the same sample image is sampled in different iterative training processes, the sample text sampled by the terminal in each iterative training process may be the same or different. That is, in the process of training the image generation model, for the same sample image, the sample text corresponding to the sample image is not completely the same.

[0133] It should be noted that by sampling different pairs of images and texts, the image generation model can be exposed to diverse input data during the training process, which is conducive to the image generation model learning richer data, not only reducing the overfitting phenomenon, but also improving the diversity and flexibility of the image generation model when generating images, and improving the generalization performance of the image generation model. At the same time, by adjusting the preset sampling probability, it is possible to adjust the sampling ratio of the text generated by different image description models, which is conducive to controlling the distribution of the input data of the image generation model during the training process, thereby further improving the performance and generalization performance of the image generation model.

[0134] In some embodiments, in the process of using the trained image generation model, the terminal controls the image generation process through different text guidance coefficients. Accordingly, the terminal obtains a time step threshold and a text guidance coefficient, the time step threshold is a threshold for distinguishing the first time step interval from the second time step interval, the first time step interval is before the second time step interval, the size of the text guidance coefficient is positively correlated with the degree of influence of the text on the image generation model, the text guidance coefficient includes the first guidance coefficient and the second guidance coefficient, and the first guidance coefficient is less than the second guidance coefficient; the terminal inputs the prompt text into the image generation model for denoising of multiple time step intervals to obtain a result image, the multiple time step intervals include the first time step interval and the second time step interval, the text guidance coefficient of the image generation model of the first time step interval is the first guidance coefficient, the text guidance coefficient of the image generation model of the second time step interval is the second guidance coefficient, and the result image meets the description of the prompt text.

[0135] Among them, the first time step interval is the early stage of image generation, and for the image generation model based on the diffusion model, the first time step interval is a high noise interval; the second time step interval is the middle and late stages of image generation, and for the image generation model based on the diffusion model, the second time step interval is a medium and low noise interval. For example, the first time step interval indicates the reasoning process of the first 20% of the time steps, and the second time step interval indicates the reasoning process of the remaining 80% of the time steps; or, the first time step interval indicates the reasoning process of the first 10% of the time steps, and the second time step interval indicates the reasoning process of the remaining 90% of the time steps.

[0136] Among them, since the size of the text guidance coefficient is positively correlated with the degree of influence of the text on the image generation model, the larger the value of the text guidance coefficient, the better the text relevance of the image generation. Since the first guidance coefficient is smaller than the second guidance coefficient, when the image generation model uses the first guidance coefficient, the text relevance of the image generation is poor, and when the image generation model uses the second guidance coefficient, the text relevance of the image generation is good.

[0137] In some embodiments, the terminal determines the time step threshold and the number of text guides by reasoning in the validation set. Accordingly, different preset time step thresholds are used, and based on the validation set, denoising is performed by the image generation model to obtain the result image corresponding to each preset time step threshold. For the result image corresponding to each time step threshold, the model is used to score or the manual annotation method is used to score, so as to select a suitable time step threshold from multiple preset time step thresholds.

[0138] It should be noted that dividing the different intervals in the training process based on the time step threshold is only an exemplary description. Optionally, the terminal obtains the iteration number threshold and the text guidance coefficient; the terminal inputs the prompt text into the image generation model for multiple rounds of iterative generation process to obtain the result image.

[0139] In some embodiments, the influence of the prompt text on the image generation model is adjusted by the text guidance coefficient. Accordingly, based on the first output value and the second output value, a first intermediate value is determined, the first output value is the output of the image generation model when the prompt text is used, and the second output value is the output of the image generation model when the prompt text is not used, that is, the basic output of the image generation model, the first intermediate value is the difference between the first output value and the second output value, and the first intermediate value is used to reflect the influence of the prompt text on the image generation model, that is, to indicate the additional gain output corresponding to the prompt text; based on the first intermediate value and the text guidance coefficient, a second intermediate value is determined, the text guidance coefficient is used to adjust the influence of the prompt text on the image generation model, and the second intermediate value is the product value of the text guidance coefficient and the first intermediate value; based on the second intermediate value and the second output value, a third output value is determined, the third output value is the sum of the second intermediate value and the second output value, and the third output value is the output of the image generation model after the text guidance coefficient is adjusted.

[0140] For ease of description, the above-mentioned method of using the text guidance coefficient to adjust the influence of the prompt text on the image generation model is called an adjustment strategy. In some embodiments, this strategy is not used in the first time step interval; this strategy is used in the second time step interval. That is, this strategy is not used in the high noise interval, and this strategy is used in the medium and low noise interval, so as to avoid the problems of low diversity and overexposure caused by this strategy on the basis of ensuring the quality of image generation, which will not be repeated here.

[0141] It should be noted that by setting different text guidance coefficients at different time step intervals, the influence of text on the image generation model can be controlled. By using a smaller first guidance coefficient in the early stage of image generation, the image generation model does not rely too much on the prompt text, which is conducive to improving the diversity of the model-generated images; and by using a larger second guidance coefficient in the middle and late stages of image generation, the generated image is more consistent with the description of the prompt text, which improves the text relevance of the generated image. Therefore, by adjusting the time step threshold and the text guidance coefficient, the diversity of image generation and the relevance to the text can be more flexibly balanced, thereby ensuring the quality and diversity of the generated images.

[0142] In some embodiments, different text guidance coefficients are used at different stages of image generation. Accordingly, when any time step is not greater than the time step threshold, the terminal performs denoising on the intermediate image of the previous time step through the image generation model based on the first guidance coefficient to obtain the intermediate image of the time step; when any time step is greater than the time step threshold, the terminal performs denoising on the intermediate image of the previous time step through the image generation model based on the second guidance coefficient to obtain the intermediate image of the time step.

[0143] It should be noted that by setting the time step threshold, the image generation process is clearly divided into two stages. By denoising the image according to the preset guidance coefficient in each stage, not only can the image generation model maintain good performance and stability in each stage, but it is also conducive to balancing the diversity of image generation and text relevance, thereby ensuring the quality and diversity of generated images.

[0144] The disclosed embodiment provides a method for training an image generation model, in which a first training data set is subjected to image deduplication and text addition to obtain a first intermediate data set and a second intermediate data set, and then a second training data set is obtained. Since the images in the second training data set are sample images retained after image deduplication, and each sample image in the second training data set corresponds to multiple sample texts, the use of the second training data set for training not only reduces the interference of low-quality data, but also improves the richness and diversity of training samples. Compared with the image generation model trained using the first training data set, the image generation model trained using the above method not only reduces the interference of low-quality data, avoids the overfitting problem of the image generation model, improves the generalization performance of the model, but also avoids problems such as overexposure, oversaturation, and incomplete details, and improves the image quality and diversity of the generated images.

[0145] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0146] Figure 4 FIG. 1 is a block diagram of a training device for an image generation model according to an exemplary embodiment. Figure 4 As shown, the device includes: an image processing unit 401, a text processing unit 402, a determination unit 403 and a training unit 404.

[0147] The image processing unit 401 is configured to perform image deduplication on the first training data set to obtain a first intermediate data set, wherein the first training data set includes a plurality of sample images and an original description text of each sample image, wherein the images in the first intermediate data set are sample images retained from the plurality of sample images, and each sample image in the first intermediate data set corresponds to an original description text;

[0148] The text processing unit 402 is configured to add text to the first training data set to obtain a second intermediate data set, wherein the images in the second intermediate data set are multiple sample images, each sample image in the second intermediate data set corresponds to multiple sample texts, and the multiple sample texts of each sample image include an original description text of the sample image and an additional description text of the sample image, and the additional description text and the original description text are used to describe the sample image from different angles;

[0149] The determining unit 403 is configured to determine a second training data set based on the first intermediate data set and the second intermediate data set, where the images in the second training data set are sample images retained from the multiple sample images, and each sample image in the second training data set corresponds to multiple sample texts;

[0150] The training unit 404 is configured to train an image generation model based on a second training data set, where the image generation model is used to generate an image that satisfies the description of the text based on the text input into the image generation model.

[0151] In some embodiments, Figure 5 is a block diagram of another image generation model training device according to an exemplary embodiment. Figure 5 As shown, the image processing unit 401 includes:

[0152] The feature extraction subunit 4011 is configured to input each sample image in the first training data set into the image extraction model for feature extraction, and obtain image features of each sample image in the first training data set, respectively, the image extraction model is used to output image features based on the image input to the image extraction model, and the image features include at least one of color features, texture features and shape features;

[0153] The clustering subunit 4012 is configured to perform multi-level clustering on the multiple sample images in the first training data set based on the image features of each sample image in the first training data set to obtain multiple cluster groups, and the similarity between the multiple sample images in each cluster group is not less than a similarity threshold;

[0154] The image processing subunit 4013 is configured to retain, for multiple cluster groups, sample images that meet preset conditions in each cluster group and discard other sample images in each cluster group to obtain a first intermediate data set.

[0155] In some embodiments, the clustering subunit 4012 is configured to perform first-level clustering on multiple sample images in the first training data set based on image features of each sample image in the first training data set to obtain a preset number of intermediate cluster groups, and the sample images in each intermediate cluster group belong to the same field; for the preset number of intermediate cluster groups, second-level clustering is performed on the sample images in each intermediate cluster group to obtain multiple cluster groups.

[0156] In some embodiments, the clustering subunit 4012 is configured to, for multiple cluster groups, input multiple sample images in each cluster group into an image scoring model for scoring, and obtain scores for the multiple sample images in each cluster group, respectively, where the score of each sample image is determined by at least one of an aesthetic score, a resolution score, and an image quality score; and in descending order of the scores of the multiple sample images in each cluster group, retain the sample images whose ranking is not lower than the ranking threshold in each cluster group, and discard the sample images whose ranking is lower than the ranking threshold in each cluster group.

[0157] In some embodiments, the image processing subunit 4013 is configured to, for multiple cluster groups, input multiple sample images in each cluster group into the image scoring model for scoring, and obtain scores for the multiple sample images in each cluster group, respectively, where the score of each sample image is determined by at least one of an aesthetic score, a resolution score, and an image quality score;

[0158] Based on the scores of multiple sample images in each cluster group, sample images with scores not less than the score threshold are retained, and sample images with scores less than the score threshold are discarded.

[0159] In some embodiments, the text processing unit 402 is configured to input the instruction text and each sample image in the first training data set into the image description model for text description, and obtain additional description text for each sample image in the first training data set, respectively. The instruction text is used to control the image description model to generate text that meets the requirements, and the image description model is used to generate text based on the image input to the image description model; the additional description text of each sample image is added to the first training data set to obtain a second intermediate data set.

[0160] In some embodiments, the training unit 403 is configured to sample multiple sample images in the second training data set, and use the sampled sample images as sample input images; for any sample input image, based on preset sampling probabilities corresponding to different sample texts, sample multiple sample texts corresponding to the sample input image, and use the sampled sample texts as sample input texts of the sample input image; and train the image generation model based on the sample input images and the sample input texts of each sample input image.

[0161] In some embodiments, the apparatus further comprises:

[0162] An acquisition unit 501 is configured to acquire a time step threshold and a text guidance coefficient, wherein the time step threshold is a threshold for distinguishing a first time step interval from a second time step interval, the first time step interval is before the second time step interval, the size of the text guidance coefficient is positively correlated with the degree of influence of the text on the image generation model, the text guidance coefficient includes a first guidance coefficient and a second guidance coefficient, and the first guidance coefficient is smaller than the second guidance coefficient;

[0163] The denoising unit 502 is configured to input the prompt text into the image generation model for denoising processing in multiple time step intervals to obtain a result image. The multiple time step intervals include a first time step interval and a second time step interval. The text guidance coefficient of the image generation model in the first time step interval is a first guidance coefficient, and the text guidance coefficient of the image generation model in the second time step interval is a second guidance coefficient. The result image meets the description of the prompt text.

[0164] In some embodiments, the denoising unit 502 is configured to, when any time step is not greater than the time step threshold, perform denoising on the intermediate image of the previous time step through the image generation model based on the first guided coefficient to obtain the intermediate image of the time step; when any time step is greater than the time step threshold, perform denoising on the intermediate image of the previous time step through the image generation model based on the second guided coefficient to obtain the intermediate image of the time step.

[0165] The disclosed embodiment provides a training device for an image generation model, wherein a terminal performs image deduplication and text addition on a first training data set to obtain a first intermediate data set and a second intermediate data set, and then obtains a second training data set. Since the images in the second training data set are sample images retained after image deduplication, and each sample image in the second training data set corresponds to multiple sample texts, the use of the second training data set for training not only reduces the interference of low-quality data, but also improves the richness and diversity of training samples. Compared with the image generation model trained using the first training data set, the image generation model trained using the above method not only reduces the interference of low-quality data, avoids the overfitting problem of the image generation model, and improves the generalization performance of the model, but also avoids problems such as overexposure, oversaturation, and incomplete details, and improves the image quality and diversity of the generated images.

[0166] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

[0167] It should be noted that the training device for the image generation model provided in the above embodiment is only illustrated by the division of the above functional units. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the training device for the image generation model provided in the above embodiment and the training method embodiment of the image generation model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0168] Regarding the training device for the image generation model in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0169] Figure 6 6 is a block diagram of an electronic device according to an exemplary embodiment. Generally, the electronic device 600 includes: a processor 601 and a memory 602 .

[0170] The processor 601 may include one or more processing cores, such as a 4-core processor, a 6-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0171] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one program code, which is used to be executed by the processor 601 to implement the training method of the image generation model provided by the method embodiment of the present disclosure.

[0172] In some embodiments, the electronic device 600 may further optionally include: a peripheral device interface 603 and at least one peripheral device. The processor 601, the memory 602 and the peripheral device interface 603 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 603 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607 and a power supply 608.

[0173] The peripheral device interface 603 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 601 and the memory 602. In some embodiments, the processor 601, the memory 602, and the peripheral device interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 601, the memory 602, and the peripheral device interface 603 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0174] The radio frequency circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 604 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 604 can communicate with other electronic devices through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 604 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0175] The display screen 605 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 605 is a touch display screen, the display screen 605 also has the ability to collect touch signals on the surface or above the surface of the display screen 605. The touch signal can be input to the processor 601 as a control signal for processing. At this time, the display screen 605 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 605 can be one, and the front panel of the electronic device 600 is set; in other embodiments, the display screen 605 can be at least two, which are respectively set on different surfaces of the electronic device 600 or are folded; in some other embodiments, the display screen 605 can be a flexible display screen, which is set on the curved surface or folded surface of the electronic device 600. Even, the display screen 605 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 605 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0176] The camera assembly 606 is used to capture images or videos. Optionally, the camera assembly 606 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the electronic device, and the rear camera is arranged on the back of the electronic device. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0177] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 601 for processing, or input them into the radio frequency circuit 604 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the electronic device 600. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 607 may also include a headphone jack.

[0178] The power supply 608 is used to power various components in the electronic device 600. The power supply 608 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 608 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0179] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the electronic device 600, and may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0180] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 602 including instructions, and the above instructions can be executed by a processor 601 of an electronic device 600 to complete the above-mentioned image generation model training method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0181] A computer program product includes a computer program, which implements the above-mentioned training method of the image generation model when executed by a processor.

[0182] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0183] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for an image generation model, characterized in that: The method comprises: Deduplication of images is performed on a first training data set to obtain a first intermediate data set, wherein the first training data set includes a plurality of sample images and an original description text of each sample image, the images in the first intermediate data set are sample images retained from the plurality of sample images, and each sample image in the first intermediate data set corresponds to an original description text; Adding text to the first training data set to obtain a second intermediate data set, wherein the images in the second intermediate data set are the multiple sample images, each sample image in the second intermediate data set corresponds to multiple sample texts, and the multiple sample texts of each sample image include an original description text of the sample image and an additional description text of the sample image, and the additional description text and the original description text are used to describe the sample image from different angles; Based on the first intermediate data set and the second intermediate data set, determining a second training data set, wherein images in the second training data set are sample images retained from the plurality of sample images, and each sample image in the second training data set corresponds to a plurality of sample texts; Based on the second training data set, an image generation model is trained, where the image generation model is used to generate an image that satisfies the description of the text based on the text input into the image generation model.

2. The training method of the image generation model according to claim 1, characterized in that: The step of performing image deduplication on the first training data set to obtain a first intermediate data set includes: Input each sample image in the first training data set into an image extraction model for feature extraction, and obtain image features of each sample image in the first training data set, the image extraction model is used to output image features based on the image input to the image extraction model, and the image features include at least one of color features, texture features, and shape features; Based on the image features of each sample image in the first training data set, multi-level clustering is performed on the multiple sample images in the first training data set to obtain multiple cluster groups, and the similarity between the multiple sample images in each cluster group is not less than a similarity threshold; For the multiple cluster groups, sample images that meet preset conditions in each cluster group are retained, and other sample images in each cluster group are discarded to obtain the first intermediate data set.

3. The training method of the image generation model according to claim 2, characterized in that: The method of performing multi-level clustering on the plurality of sample images in the first training data set based on the image features of each sample image in the first training data set to obtain a plurality of cluster groups includes: Based on the image features of each sample image in the first training data set, performing first-level clustering on the plurality of sample images in the first training data set to obtain a preset number of intermediate clustering groups, wherein the sample images in each intermediate clustering group belong to the same field; For the preset number of intermediate cluster groups, second-level clustering is performed on the sample images in each intermediate cluster group to obtain the multiple cluster groups.

4. The training method of the image generation model according to claim 2, characterized in that: The step of retaining, for the plurality of cluster groups, sample images that meet a preset condition in each cluster group and discarding other sample images in each cluster group to obtain the first intermediate data set includes: For the plurality of cluster groups, inputting a plurality of sample images in each cluster group into an image scoring model for scoring, and obtaining scores of the plurality of sample images in each cluster group, respectively, wherein the score of each sample image is determined by at least one of an aesthetic score, a resolution score, and an image quality score; According to the order of scores of multiple sample images in each cluster group from high to low, the sample images whose ranking in each cluster group is not lower than the ranking threshold are retained, and the sample images whose ranking in each cluster group is lower than the ranking threshold are discarded.

5. The method for training an image generation model according to claim 2, characterized in that: The step of retaining, for the plurality of cluster groups, sample images that meet a preset condition in each cluster group and discarding other sample images in each cluster group to obtain the first intermediate data set includes: For the plurality of cluster groups, inputting a plurality of sample images in each cluster group into an image scoring model for scoring, and obtaining scores of the plurality of sample images in each cluster group, respectively, wherein the score of each sample image is determined by at least one of an aesthetic score, a resolution score, and an image quality score; Based on the scores of the multiple sample images in each cluster group, the sample images with scores not less than the score threshold are retained, and the sample images with scores less than the score threshold are discarded.

6. The training method of the image generation model according to claim 1, characterized in that: The step of adding text to the first training data set to obtain a second intermediate data set includes: Inputting the instruction text and each sample image in the first training data set into the image description model for text description, and obtaining additional description text for each sample image in the first training data set, wherein the instruction text is used to control the image description model to generate text that meets the requirements, and the image description model is used to generate text based on the image input into the image description model; The additional description text of each sample image is added to the first training data set to obtain the second intermediate data set.

7. The method for training an image generation model according to claim 1, characterized in that: The step of training the image generation model based on the second training data set includes: Sampling a plurality of sample images in the second training data set, and using the sampled sample images as sample input images; For any sample input image, based on preset sampling probabilities corresponding to different sample texts, multiple sample texts corresponding to the sample input image are sampled, and the sampled sample texts are used as sample input texts of the sample input image; An image generation model is trained based on the sample input images and the sample input text of each sample input image.

8. The method for training an image generation model according to claim 1, characterized in that: The method further comprises: Obtaining a time step threshold and a text guidance coefficient, wherein the time step threshold is a threshold for distinguishing a first time step interval from a second time step interval, the first time step interval is before the second time step interval, the size of the text guidance coefficient is positively correlated with the degree of influence of the text on the image generation model, the text guidance coefficient includes a first guidance coefficient and a second guidance coefficient, and the first guidance coefficient is smaller than the second guidance coefficient; The prompt text is input into the image generation model for denoising processing in multiple time step intervals to obtain a result image, the multiple time step intervals include the first time step interval and the second time step interval, the text guidance coefficient of the image generation model in the first time step interval is the first guidance coefficient, the text guidance coefficient of the image generation model in the second time step interval is the second guidance coefficient, and the result image meets the description of the prompt text.

9. The method for training an image generation model according to claim 8, characterized in that: The prompt text is input into the image generation model to perform denoising processing in multiple time step intervals to obtain a result image, including: When any time step is not greater than the time step threshold, based on the first guidance coefficient, the image generation model is used to perform denoising on the intermediate image of the previous time step to obtain the intermediate image of the time step; When any time step is greater than the time step threshold, based on the second guidance coefficient, the intermediate image of the previous time step is denoised by the image generation model to obtain the intermediate image of the time step.

10. A training device for an image generation model, characterized in that: The device comprises: an image processing unit configured to perform image deduplication on a first training data set to obtain a first intermediate data set, wherein the first training data set includes a plurality of sample images and an original description text of each sample image, the images in the first intermediate data set are sample images retained from the plurality of sample images, and each sample image in the first intermediate data set corresponds to an original description text; a text processing unit configured to add text to the first training data set to obtain a second intermediate data set, wherein the images in the second intermediate data set are the multiple sample images, each sample image in the second intermediate data set corresponds to multiple sample texts, and the multiple sample texts of each sample image include an original description text of the sample image and an additional description text of the sample image, wherein the additional description text and the original description text are used to describe the sample image from different angles; a determining unit configured to determine a second training data set based on the first intermediate data set and the second intermediate data set, wherein images in the second training data set are sample images retained from the plurality of sample images, and each sample image in the second training data set corresponds to a plurality of sample texts; The training unit is configured to train an image generation model based on the second training data set, wherein the image generation model is used to generate an image that satisfies the description of the text based on the text input into the image generation model.

11. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; Wherein, the processor is configured to execute the program code to implement the training method of the image generation model as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method for the image generation model as described in any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the training method for the image generation model as described in any one of claims 1 to 9.

Images