Picture generation model training method and generation method, electronic equipment and storage medium

Abstract

The invention provides a picture generation model training method, a picture generation model generation method, electronic equipment and a storage medium. According to the picture generation model training method, the picture text description information is added to the sample picture, then the sample picture is subjected to resolution clustering and size readjustment, the adjusted picture is obtained, the adjusted picture serves as input, the resolution of the adjusted picture serves as a first input condition of model training, and the model training efficiency is improved. Adjusting the picture text description information of the picture as a second input condition of model training, and enabling the trained picture generation model to establish a corresponding relation among the picture, the resolution and the picture text description information through model training, and the trained picture generation model can generate pictures with different resolutions according to the text description information.

Description

Technical Field

[0001] The present disclosure relates to the technical field of picture generation, and in particular to a picture generation model training method, a picture generation method, an electronic device, a storage medium, and a program product. Background Art

[0002] Text-to-image is a popular area of ​​generative artificial intelligence applications. In the existing technology, diffusion models are used to input text and output corresponding images. The model is generally divided into two steps: training and reasoning. The training stage generally collects enough image-text data to train the model to learn the connection between images and text; in the reasoning stage, any text is input into the model at random, and after several diffusion and denoising steps, a satisfactory image is finally obtained.

[0003] However, the existing image generation model can only accurately generate images of one resolution. Images of other resolutions will be distorted, resulting in picture distortion, unreasonable layout and other problems. Summary of the invention

[0004] The present disclosure provides a picture generation model training and picture generation method, electronic equipment, storage medium and program product.

[0005] According to one aspect of the present disclosure, a method for training a picture generation model is provided, comprising: adding corresponding picture text description information to pictures in a first picture sample set; clustering the pictures in the first picture sample set according to N resolutions to obtain N first picture sample subsets, wherein N≥2, and the N first picture sample subsets correspond to N resolutions respectively; adjusting the resolutions of the pictures in the N first picture sample subsets to obtain adjusted pictures, wherein the resolutions of the adjusted pictures are the resolutions corresponding to the first picture sample subsets to which the adjusted pictures are located; performing model training according to the adjusted pictures of the N first picture sample subsets, wherein the resolutions of the adjusted pictures serve as the first input condition for the model training, and the picture text description information serves as the second input condition for the model training; and obtaining a trained picture generation model, wherein the picture generation model serves as a target picture generation model.

[0006] According to at least one embodiment of the present disclosure, the image text description information is used as the second input condition for model training, including: dividing each image text description information into M segments of fragment text description information according to a unit text length, wherein M≥2, the text length of the image text description information is greater than the unit text length, and the text length of each fragment text description information is less than or equal to the unit text length; and using the M segments of fragment text description information corresponding to the adjusted image as the second input condition for model training of the image generation model.

[0007] According to at least one embodiment of the present disclosure, M segments of text description information corresponding to the adjusted picture are used as the second input condition for model training of the picture generation model, including: converting the M segments of text description information into M segments of text embedding vectors respectively; splicing the M segments of text embedding vectors into a picture text embedding vector; and using the picture text embedding vector corresponding to the adjusted picture as the second input condition for model training of the picture generation model.

[0008] According to at least one embodiment of the present disclosure, after obtaining the trained image generation model, the method further includes: performing model training on the image generation model using a second image sample set that meets a first quality condition, so that the image generation model generates images that meet the first quality condition.

[0009] According to at least one embodiment of the present disclosure, after the picture generation model is trained using the second picture sample set, the method further includes: fusing the first model parameters of i first models with the second model parameters of the picture generation model to obtain third model parameters, so that the picture generation model generates a picture that meets the second quality condition based on the third model parameters, wherein i≥1, the first model and the picture generation model have the same model architecture, the first model parameters of the first model are obtained after model training using the first picture sample set or the second picture sample set, and the third model parameters are used as parameters actually used by the picture generation model.

[0010] According to at least one embodiment of the present disclosure, the first model parameters of i first models are fused with the second model parameters of the image generation model to obtain third model parameters, including: using i first weight coefficients and i first model parameters to obtain i first weighted results, the i first weight coefficients corresponding to the i first model parameters one by one; using the second weight coefficient and the second model parameter corresponding to the second weight coefficient to obtain the second weighted result; using the i first weighted results and the second weighted result to obtain the third model parameters.

[0011] According to at least one embodiment of the present disclosure, after obtaining the third model parameters, it also includes: step S110: using the teacher model and the noise picture to perform denoising training on the first student model to obtain a first result, wherein the teacher model is the picture generation model, the first student model is generated based on the picture generation model, and the noise picture is obtained by denoising the sample picture; step S111: using the noise picture to perform denoising training on the second student model to obtain a third result, wherein the second student model is generated based on the picture generation model, and the parameter iteration algorithm of the first student model is different from that of the second student model; step S112: iterating the first student model and the second student model according to the loss functions of the first result and the third result respectively, repeating steps S110 to S112 until the first result and the third result satisfy the consistency function, taking the first student model as the target picture generation model, and the picture generation time of the first student model is less than the picture generation time of the picture generation model.

[0012] According to at least one embodiment of the present disclosure, a first student model is denoised using a teacher model and a noise picture to obtain a first result, including: using the teacher model to restore the noise picture to the picture before k noise additions as a training target to obtain a second result, wherein the noise picture is obtained by performing n+k noise additions on a sample picture, n≥1, k≥1; using the first student model to restore the second result to the picture before n noise additions as a training target to obtain the first result.

[0013] According to at least one embodiment of the present disclosure, the second student model is trained to denoise using the noisy image to obtain a third result, including: using the second student model to restore the noisy image to the image before n+k times of noise addition as a training target to obtain the third result.

[0014] According to at least one embodiment of the present disclosure, the teacher model is used to restore the noisy image to the image before k noise additions as a training target to obtain a second result, including: using the teacher model, with the picture text description information corresponding to the sample picture as a control condition, to restore the noisy image to the image before k noise additions as a training target to obtain a fourth result; using the teacher model, without using the picture text description information corresponding to the sample picture as a control condition, to restore the noisy image to the image before k noise additions to obtain a fifth result; obtaining the second result based on the fourth result and the fifth result.

[0015] According to another aspect of the present disclosure, a method for training a picture generation model is provided, including: obtaining an original picture sample set, screening out a first picture sample set from the original picture sample set, and screening out a second picture sample set from the original picture sample set according to a first quality condition; performing model training using the resolution of pictures in the first picture sample set and picture text description information corresponding to the pictures to obtain a trained picture generation model; performing model training on the picture generation model using the second picture sample set so that the picture generation model generates pictures that meet the first quality condition; fusing first model parameters of i first models with second model parameters of the picture generation model to obtain third model parameters so that the picture generation model generates pictures that meet the second quality condition based on the third model parameters, wherein i≥1, the first model has the same model architecture as the picture generation model, and the first model parameters of the first model are obtained by training the picture generation model using the first picture sample set or the second picture sample set. The third model parameters are obtained after model training, and are used as parameters actually used by the image generation model; the first student model is denoised by using the teacher model and the noise picture to obtain a first result, wherein the teacher model is the image generation model, the first student model is generated based on the image generation model, and the noise picture is obtained by denoising the sample picture; the second student model is denoised by using the noise picture to obtain a third result, wherein the second student model is generated based on the image generation model, and the parameter iteration algorithms of the first student model and the second student model are different; the first student model and the second student model are iterated according to the loss functions of the first result and the third result respectively, until the first result and the third result satisfy the consistency function, and the first student model is used as the target image generation model, wherein the image generation time of the first student model is less than the image generation time of the image generation model.

[0016] According to another aspect of the present disclosure, there is provided a method for generating an image, comprising: receiving text information; inputting the text information into a target image generation model obtained by the image generation model training method as described in any one of claims 1 to 11, to obtain an image corresponding to the text information.

[0017] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the image generation model training method as described in any one of the above items, or executes the image generation method as described above.

[0018] According to another aspect of the present disclosure, a readable storage medium is provided, in which execution instructions are stored. When the execution instructions are executed by a processor, they are used to implement the image generation model training method as described in any one of the above items, or to implement the image generation method as described above.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements the image generation model training method as described in any one of the above items, or implements the image generation method as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0021] Figure 1 It is a schematic diagram of a data pipeline in one embodiment of the present disclosure.

[0022] Figure 2 It is a flowchart of a method for training a picture generation model according to an embodiment of the present invention.

[0023] Figure 3 It is a schematic diagram of the resolution training process of the image generation model according to one embodiment of the present disclosure.

[0024] Figure 4 It is a flowchart of a text segmentation method according to an embodiment of the present invention.

[0025] Figure 5 It is a flowchart of a text segmentation method according to another embodiment of the present invention.

[0026] Figure 6 It is a flow chart of a model parameter fusion method in one embodiment of the present disclosure.

[0027] Figure 7 It is a schematic diagram of the model optimization process of one embodiment of the present disclosure.

[0028] Figure 8 It is a schematic diagram of a model optimization process according to another embodiment of the present invention.

[0029] Fig. 9 It is a flowchart of a method for training a picture generation model according to another embodiment of the present invention.

[0030] Fig.10 It is a schematic diagram of an application scenario of the picture generation model training method disclosed in the present invention.

[0031] Fig.11It is a comparative bar chart of the target image generation model disclosed in the present invention and the SDXL refiner model.

[0032] Fig.12 It is a comparative bar chart showing the target image generation model of the present disclosure and the Epic Realism model.

[0033] Fig.13 It is a comparative bar chart showing the target image generation model disclosed in the present invention and the Realistic Vision model.

[0034] Fig.14 It is a comparative bar chart showing the target image generation model disclosed in the present invention and the Stable Diffusion model.

[0035] Fig.15 It is a flowchart of a method for generating an image according to an embodiment of the present disclosure.

[0036] Fig.16 It is a structural schematic diagram of a picture generation model training device according to an embodiment of the present invention.

[0037] Fig.17 It is a schematic diagram of the structure of a picture generating device according to one embodiment of the present disclosure.

[0038] Fig.18 It is a structural schematic diagram of a picture generation model training device according to another embodiment of the present invention.

[0039] Fig.19 It is a schematic block diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] The present disclosure is further described in detail below in conjunction with the accompanying drawings and examples. It is understood that the specific examples described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0041] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0042] With the growth of home improvement demand and the growth and competition in the home improvement industry, in order to enable users to better choose home improvement agencies, the method of only providing renderings by designers in the past has been difficult to meet the diverse needs of users. With the development of artificial intelligence technology, especially generative artificial intelligence technology, it has gradually been adopted that users provide requirements and generative artificial intelligence technology automatically generates renderings to meet users' personalized needs, provide effect previews, and enhance user experience.

[0043] Existing image generation models can only accurately generate images with fixed resolutions. For example, existing image generation models such as DCGAN (Deep Convolutional Generative Adversarial Network), StyleGAN-XL (Extended Large-Scale Style-Based Generator Architecture), Diffusion Models, etc. can only generate images with fixed resolutions. For example, only images with a resolution of 512×512 can be accurately generated. Images with fixed resolutions can meet some simple needs of users. For example, users' requirements for living room layout only include clear requirements for a small number of large items such as sofas and TV cabinets, and no requirements for small decorations and ornaments. In this case, since there are fewer items, images with limited resolution can be clearly generated and meet user requirements. However, home decoration renderings cannot meet the requirements of every user. For example, in the same living room, users may require a large number of items such as coffee tables, sofas, built-in TV cabinets and display cabinets, chandeliers, and plant landscapes. At this time, due to the resolution limit, the renderings generated at a fixed resolution cannot clearly show every detail. At the same time, it cannot be applied to large indoor spaces due to the limited resolution. In the case of low resolution, large indoor space objects cannot be displayed clearly, and the display effect is poor.

[0044] Therefore, the existing fixed-resolution image generation model can no longer meet actual needs when generating renderings in the field of interior decoration.

[0045] The present invention proposes a method for training an image generation model, so that the image generation model can accurately output images of multiple resolutions to meet practical application needs.

[0046] The present disclosure discloses an application scenario of a method for training a picture generation model according to an embodiment. In the application scenario, the training process of the picture generation model can be run on a server, which can be a single server or a clustered server, etc. The training process of the picture generation model can also be run on a terminal device such as a personal computer.

[0047] In order to improve the model training effect, it is necessary to screen and process the image sample set used in the training, so as to obtain sample images with sufficient size and good quality, so as to improve the training quality of the image generation model and ensure that the image generation model can achieve the training objectives after training.

[0048] In order to improve the applicability of the model, original image sample sets can be obtained from multiple sources to enrich the types and quantity of training data, thereby ensuring that the trained image generation model can cover a large enough scene to meet user requirements.

[0049] The original image sample set can be obtained from the Internet, for example, by searching for interior decoration renderings on major search engines through crawler software, and obtaining images marked with tags related to interior decoration renderings. For example, images marked with interior decoration renderings, living room renderings, bedroom decoration pictures, and / or bathroom display pictures related to interior decoration renderings are obtained to form network image samples.

[0050] The source of the original image sample set can also include private domain image samples. In order to improve the exquisiteness and accuracy of the images generated by the image generation model, you can purchase private domain image samples. Compared with online image samples, such data can be of higher quality and can enrich the training samples of the model, thereby improving the final image generation effect.

[0051] The source of the original image sample set can also include open source image samples. Open source image samples have also been reviewed and have higher image quality than network image samples. At the same time, open source image samples can also be images trained and / or generated by other open source models. Such images are applied to other cultural image models. Therefore, the feature distribution, that is, the types and styles of scenery, scenes, renderings, and distribution methods in the picture, are more in line with the understanding of artificial intelligence, and are easier to be understood by the image generation model to be trained during training, which can speed up the training speed, enrich the content of the training samples, and ultimately increase the quality of the output results of the image generation model.

[0052] In order to increase the diversity of training samples and improve the output quality of the final trained image generation model, the original image sample set includes a large number of network image samples, private domain image samples and / or open source image samples, which also makes the original image sample set cluttered. Therefore, the original image sample set is preprocessed to obtain an image sample set that is easy to train.

[0053] The preprocessing process may include operations such as denoising and deduplication on the original image sample set, thereby improving the quality of the final image sample set and improving the training quality of the image generation model.

[0054] The images in the original image sample set can be classified and screened according to the screening conditions to eliminate low-quality images. For example, in order to support subsequent resolution training, images with too low resolution, such as images with a resolution lower than 320×240, and some images that are not suitable for the subsequent training process, such as images with too many elements and images that are easily deformed after adjusting the resolution, can be eliminated. The screening conditions can refer to the screening condition table shown in Table 1. It can be divided into five major categories, called first-class classification, namely low-quality images, basic attributes, aesthetics, spatial composition and content. Under these five major categories, the images can be further subdivided to obtain multiple second-class classifications. For example, the second-class classification under the basic attributes can include four categories: resolution, clarity, brightness and saturation. Based on aesthetics, it can be divided into three categories according to color, style and sense of reality. Through these classification criteria, the images that do not meet the conditions can be eliminated from the original image sample set, and the first image sample set used in the subsequent embodiments can be obtained, thereby ensuring the training effect of the subsequent model.

[0055] Table 1 Screening conditions

[0056]

[0057] While filtering according to the filtering conditions, you can also further divide according to the image quality. Set the first quality condition with high image quality requirements. Figure 1 As shown, based on the original picture sample set, it can be divided into 2 groups. The first group is the first picture sample set that can be screened by the picture recognition model according to the preset screening conditions, and the second group can be obtained by screening the picture recognition model and / or manually screening using a more stringent first quality condition. The high-quality picture data set, that is, the second picture sample set. It can be understood that as the quality of the picture data improves, the amount of each group of picture data is smaller. For example, the original picture sample set may include millions of low-quality pictures, the first picture sample set may include 600,000 or 700,000 pictures, and the second picture sample set only includes more than 100,000 screened high-quality pictures. The second picture sample set can be selected from the first picture sample according to the first quality condition, such as the decorative color, furniture selection, hard decoration design and other angles, and the selection process can be performed manually or by the picture recognition model, or it can be selected by both manual and picture recognition models, and finally the intersection of the selected pictures is taken as the second picture sample set. In summary, the original picture sample set is obtained, the original picture sample set is screened according to the screening conditions to obtain the first picture sample set, and the original picture sample set is screened according to the first quality condition to obtain the second picture sample set.

[0058] See also Figure 1 ,In this embodiment, a complete set of data pipelines for processing images is implemented for sample data, including image data acquisition, image quality screening, image text acquisition, and image data stratification.

[0059] Figure 2 FIG. 1 is a schematic diagram showing the overall process of a method M100 for training a picture generation model according to an embodiment of the present disclosure. Figure 2 The method shown includes steps S101 to S105. The method can be executed by an electronic device such as a server or a personal computer.

[0060] In step S101, corresponding picture text description information is added to pictures in the first picture sample set.

[0061] In order to realize the text-generated image function and enable the image generation model to generate images according to the text description information, it is necessary to add corresponding image text description information to each image in the first image sample set. The image text description information may include image semantic tags and / or natural language description information.

[0062] Image semantic tags can be found in Figure 1 , you can add image semantic tags to the image by including room, style, color scheme, soft decoration elements and / or hard decoration elements. For example, in the interior decoration rendering scenario, the image is an interior decoration rendering of a living room with a modern decoration style. The objects include floor-to-ceiling windows, a TV, a TV cabinet, a sofa, a storage cabinet, and a coffee table. You can add objects, style, and location (living room) as image semantic tags to the image. The image semantic tags of the image may include "modern decoration style", "living room", "floor-to-ceiling windows", "TV", "TV cabinet", "sofa", "storage cabinet", and "coffee table". You can add natural language description information corresponding to the image. See Figure 1 , the content of natural language description information can include various information such as layout, atmosphere, details and material or texture. For example, the natural language description information of the picture described above can be "a living room decorated in a modern style, with a sofa, TV cabinet, coffee table and storage cabinet, giving an overall open feeling". Of course, a picture can correspond to both picture semantic labels and natural language description information.

[0063] The image semantic tag can use a pre-built image recognition model to identify feature information in the image and use the feature information as the image semantic tag. The natural language description information can use a natural language model to identify the image and generate natural language description information corresponding to the image.

[0064] In step S102, clustering is performed on the pictures in the first picture sample set according to N resolutions to obtain N first picture sample subsets, where N≥2, N is a positive integer, and the N first picture sample subsets correspond to N resolutions respectively.

[0065] The first image sample set includes a large number of images with different resolutions. If these images are directly input into the image generation model for training, it is easy to cause the image generation model results to be discrete and the training effect to be poor. Therefore, according to the preset N resolutions, the images in the first image sample set are clustered to obtain N first image sample subsets. For example, Figure 3 As shown, five resolutions can be preset, namely 448×896, 576×768, 768×768, 768×576 and 896×448. The five resolutions correspond to five first image sample subsets. Each image in the first image sample set will be clustered into the first image sample subset with the smallest difference from the original resolution of the image. For example, a 768×512 image will be clustered into a first image sample subset of 768×576, and a 768×812 image will be clustered into a first image sample subset of 768×768. The number of images in each first image sample subset may not be equal, which is determined by the resolution distribution of the images in the image sample set. Clustering facilitates more targeted processing and training of images of different resolutions.

[0066] In addition, in order to avoid distortion of the clustered images during the subsequent resizing process, the N resolutions can be set in multiples of 64. For example, the difference between the resolution sizes 768 and 576 in the above example is 3 64s, and the difference between 896 and 768 is 1 64.

[0067] In step S103, the resolutions of the pictures in the N first picture sample subsets are adjusted to obtain adjusted pictures, wherein the resolution of the adjusted pictures is the resolution corresponding to the first picture sample subset in which the adjusted pictures are located.

[0068] After each image is assigned to a corresponding first image sample subset, each image in the first image sample subset is set to the target resolution with the resolution corresponding to the first image sample subset. The resolution of each image in the first image sample subset is readjusted to the target resolution, and the image after the resolution adjustment is called an adjusted image. After the images in the image sample set are screened, each image in the first image sample subset can be guaranteed not to be severely distorted or twisted during the resolution readjustment process. Therefore, cropping is not required to avoid information loss caused by cropping, ensure the integrity of the image content, and improve the training effect.

[0069] In step S104, model training is performed based on the adjusted images of the N first image sample subsets, wherein the resolution of the adjusted images is used as the first input condition for the model training, and the image text description information added to the adjusted images before adjustment is used as the second input condition for the model training.

[0070] It is understandable that the corresponding picture text description information has been added to the first picture sample set before the resolution size is adjusted. Therefore, even if the picture clustering is adjusted to the corresponding first picture sample subset and the resolution is adjusted, the corresponding picture text description information remains unchanged.

[0071] During the model training process, in order to increase the model's ability to understand each resolution, several adjusted images can be extracted from a first image sample subset during each training, and the adjusted images can be input into the image generation model for training, and the resolution of the adjusted images can be used as the first input condition and the image text description information corresponding to the adjusted images can be used as the second input condition to constrain the training process. After the training is completed, several adjusted images are extracted from another first image sample subset again, and the resolution of the adjusted images can be used as the first input condition and the image text description information corresponding to the adjusted images can be used as the second input condition to constrain the training process. The above process is continued until the preset number of cycles is reached, the training quantity is reached, or all samples are used to train the model, and the training process is completed.

[0072] For example, in the case of a first image sample subset with a resolution of 576×768 and a first image sample subset with a resolution of 768×768, during the training process, several adjusted images can be randomly extracted from the first image sample subset with a resolution of 576×768 and input into the image generation model, and the resolution of the adjusted image of 576×768 is used as the first input condition to train the image generation model, thereby increasing the understanding of the image generation model for images with a resolution of 576×768. Then, the next round of training is carried out, and several adjusted images are randomly extracted from the first image sample subset with a resolution of 768×768 and input into the image generation model, and the resolution of the adjusted image of 768×768 is used as the first input condition of the new round to train the image generation model, thereby increasing the understanding of the image generation model for images with a resolution of 768×768. At this time, several adjusted images are extracted from the first image sample subset of 576×768 for training, and the cycle is repeated until the training end condition is met. The training is terminated to obtain a trained image generation model.

[0073] The number of adjusted images extracted from the first image sample subset in each round of training, the training end condition, and the number of the first image sample subset can be set according to actual application requirements and are not limited here.

[0074] In step S105, a trained picture generation model is obtained, wherein the picture generation model is used as a target picture generation model.

[0075] After the training is completed, the trained image generation model, that is, the target image generation model, can be obtained. After the target image generation model is trained using the adjusted images and resolutions in the above-mentioned multiple first image sample subsets, it can generate images of any resolution between N resolutions. For example, the N resolutions corresponding to the first image sample subset are 448×896, 576×768, 768×768, 768×576 and 896×448, respectively. Then the target image generation model can output images of any resolution from 448×896 to 896×448, instead of only outputting images of resolutions of 448×896, 576×768, 768×768, 768×576 and 896×448. In this embodiment, the target image generation model preferably generates images of any resolution between 512×768 and 768×512.

[0076] The image generation model training method provided in this embodiment adjusts the image as a sample input, adjusts the resolution of the image as the first input condition for model training, and adjusts the image text description information of the image as the second input condition for model training, so that the trained image generation model can establish a correspondence between the image, the resolution, and the image text description information. The trained image generation model can generate images of different resolutions according to the text description information.

[0077] In some embodiments of the present disclosure, in step S104, the process of adjusting the picture text description information added before the adjustment as the second input condition for model training may include: Figure 4 Step S1041 and step S1042 are shown.

[0078] S1041, dividing the picture text description information corresponding to the adjusted picture into M segments of text description information according to the unit text length, where M≥2, the text length of the picture text description information is greater than the unit text length, and the text length of each segment text description information is less than or equal to the unit text length.

[0079] In order to increase the accuracy of the images generated by the image generation model, it is necessary to increase the amount of text that the image generation model can support input. For this reason, the aforementioned process of adding image text description information to the image greatly enriches the image text description information corresponding to the image, so that the target image generation model can support the recognition of longer text description information.

[0080] The text processing capabilities of the models are all limited. For example, models such as CLIP (Contrastive Language-Image Pre-training), BLIP (Base Language-Image Pre-training) and Parti (Pathways Auto-regressive Transformer) can only support text description information of a certain length (unit text length) at a time without affecting the accuracy of text recognition. For example, only 75 words of text input are supported. In order to support longer text description information, the image text description information corresponding to the adjusted image is divided into M segments of fragment text description information. The length of each segment of text description information is less than or equal to the unit text length. By segmenting the long text to obtain multiple segments of text that are less than or equal to the unit text length, the model can process longer text description information.

[0081] For example, the length of the picture text description information corresponding to the adjusted picture is 225 words, and the unit text length is 75 words. The picture text description information corresponding to the adjusted picture needs to be split into 3 segments of fragment text description information based on the unit text length. Each fragment text description information corresponds to 75 words. When splitting the picture text description information, the picture text description information can be split in sequence to obtain fragment text description information including numbering, so that when multiple segments of text are subsequently embedded in the vector splicing, the text order can be known by the numbering. The specific length of the unit text length can be set according to actual application requirements and is not limited here.

[0082] S1042: Use the M segments of text description information corresponding to the adjusted image as the second input condition for training the image generation model.

[0083] After the long text image text description information is divided into M segments of text description information, the model can process multiple segments of text description information, so that the M segments of text description information are used as the second input condition for model training of the image generation model for subsequent training, so that the image generation model can support long text input.

[0084] Further, regarding step S1042, in some embodiments of the present disclosure, it may include: Figure 5 Steps S10421 to S10423 are shown.

[0085] S10421. Convert the M segments of text description information into M segments of text embedding vectors respectively. Convert the M segments of text description information into M segments of text embedding vectors that the model can understand, thereby avoiding the direct conversion of embedding vectors for long texts and ensuring the accuracy of vector conversion. In one embodiment, the text encoder in the CLIP model can be used to convert the M segments of text description information into M segments of text embedding vectors respectively.

[0086] S10422, concatenate M text embedding vectors into a picture text embedding vector. In order for the model to understand the input long text, it is still necessary to reconcatenate the multiple text embedding vectors into a whole to obtain the picture text embedding vector. The picture text embedding vector reconstructed using the concat text embedding method can correspond to the picture text information of the picture adjusted before splitting, and still retain the original meaning, so that the subsequent picture generation model can understand the meaning of the long text.

[0087] When re-stitching, a number can be set in the text description information of each fragment, so as to ensure that the re-stitched image text embedding vector is spliced ​​in order and the meaning of the original image text description information is retained.

[0088] S10423, using the picture text embedding vector corresponding to the adjusted picture as the second input condition for model training of the picture generation model. Using the picture text embedding vector corresponding to the adjusted picture as the second input condition for model training of the picture generation model, and the resolution of the adjusted picture as the first input condition for model training, are input into the picture generation model for model training. Thus, a correspondence between the picture, resolution, and picture text description information is established, so that the picture generation model can generate a picture of the corresponding resolution according to the picture text description information.

[0089] In one embodiment, the first input condition and the second input condition can be trained together as control conditions of a U-Net model (fully convolutional neural network).

[0090] In another embodiment of the present disclosure, based on the aforementioned steps S101 to S107, step S108 may be further included.

[0091] S108. Perform model training on the picture generation model using the second picture sample set that meets the first quality condition, so that the picture generation model generates pictures that meet the first quality condition. After the picture generation model is trained, in order to further improve the picture generation quality of the picture generation model, the picture generation model can be trained using pictures with higher quality to improve the picture generation effect of the picture generation model. To this end, the picture generation model is further trained using the second picture sample set that meets the first quality condition. The picture generation model can be tuned using the EMU (Enhanced ModelUpdate) algorithm during the training process.

[0092] The first quality condition may be at least one condition selected from conditions including aesthetics, clarity, saturation, occlusion, and perspective. The second picture sample set that meets the first quality condition may be selected from the original picture sample set using a picture recognition model, or may be selected from the original picture sample set manually, or may be selected from the first picture sample set manually or using a picture recognition model. When the first quality condition is aesthetics, the aesthetics of the pictures generated by the picture generation model may be improved after the picture generation model is trained using the second picture sample set.

[0093] In another embodiment of the present disclosure, based on the aforementioned step S108, step S109 may also be included.

[0094] S109. Fusing the first model parameters of the i first models with the second model parameters of the picture generation model to obtain third model parameters, where i≥1. In this way, the picture generation model can generate pictures that meet the second quality condition (e.g., practicality) based on the third model parameters. The first model and the picture generation model have the same model architecture, the first model parameters of the first model are obtained after model training using the picture sample set, and the third model parameters are used as parameters actually used by the picture generation model.

[0095] The pictures output by the picture generation model after the above training can meet the first quality condition. For example, the picture aesthetics condition is met. However, the scene of the generated picture may not meet the second quality condition (such as practicality). As an example, in the interior decoration rendering scene, the objects and decoration styles in the pictures generated by the model are beautiful, but some furniture is too long and too large, which does not conform to the display scene, or some home decoration is difficult to implement in the real scene. Therefore, in order to further improve the picture quality and make the generated pictures meet the actual requirements, the picture generation model is integrated with the first model with higher authenticity to further improve the raw picture quality of the picture generation model.

[0096] In order to fuse the model parameters, the model architecture of the first model needs to be consistent with the model architecture of the image generation model. In this way, the model parameters of the first model and the image generation model can be fused one by one. For example, if the model architecture of the image generation model is Stable Diffusion, the architecture of the first model should also be StableDiffusion.

[0097] Since the first model can use a different sample set from the image generation model during the initial training, before fusion, each first model needs to be retrained using the first image sample set and / or the second image sample set. During the training process, the small learning rate of each first model can be fine-tuned to ensure that the model parameters of the first model are compatible with the first image sample set and / or the second image sample set. Ensure that the third model parameters obtained after the subsequent fusion of the model parameters of the first model with the parameters of the image generation model will not cause problems such as model non-convergence.

[0098] The image generation capability of the first model can make up for the deficiencies of the image generation model. The image generated by the fused image generation model using the third model parameters meets the second quality condition.

[0099] Finally, the target image generation model using the third model parameters can combine the advantages of each model and generate images with higher overall quality.

[0100] In some embodiments of the present disclosure, Figure 6 As shown, step S109 may also include steps S1091 to S1093.

[0101] S1091. Using i first weight coefficients and i first model parameters, i first weighted results are obtained, and the i first weight coefficients correspond to the i first model parameters one by one. The first model parameter of each first model corresponds to a first weight coefficient. Of course, the first model parameters may include multiple parameters. Each parameter naturally also corresponds to a corresponding weight coefficient. The first model parameters are weighted using the first weight coefficients according to the corresponding relationship to obtain a first weighted result corresponding to the first model parameter of each first model.

[0102] S1092. Obtain a second weighted result using the second weight coefficient and the second model parameter corresponding to the second weight coefficient. The second model parameter in the picture generation model may also include multiple parameters, and each parameter naturally corresponds to a corresponding weight coefficient. Similarly, the second model parameter is weighted using the second weight coefficient according to the corresponding relationship to obtain a second weighted result corresponding to the second model parameter of the picture generation model.

[0103] S1093. Obtain third model parameters using the i first weighted results and the second weighted results. According to the correspondence between the i first weighted results and the second weighted results, a one-to-one correspondence between each parameter in the first weighted result and each parameter in the second weighted result is used to perform averaging processing to obtain third model parameters with more balanced values, thereby achieving a fusion of the first model parameters and the second model parameters, using the third model parameters as parameters actually used by the image generation model, and using the image generation model using the third model parameters as the target image generation model.

[0104] In some embodiments of the present disclosure, after step S109, steps S110 to S112 may also be included. Figure 7 shown.

[0105] S110, using the teacher model and the noise picture to perform denoising training on the first student model to obtain a first result, wherein the teacher model is a picture generation model, the first student model is generated based on the picture generation model, and the noise picture is obtained by adding noise to the sample picture.

[0106] The sample images may be selected from the first image sample set or the second image sample set, and different sample images may be selected for each training.

[0107] Add noise to the sample image, and use the noise image and the image text description information corresponding to the noise image as input to the teacher model. The teacher model outputs the restored intermediate noise image. The intermediate noise image is used as the input of the first student model. The first student model is trained to restore the intermediate noise image to the sample image as the training goal, so that the first student model obtains the first result.

[0108] The image generation model is used as the teacher model. The first student model can also be an image generation model at the beginning. In subsequent iterations, due to different inputs, the parameters of the first student model after iterative update are no longer the same as those of the initial image generation model, and it becomes another model with different parameters.

[0109] S111. Perform denoising training on the second student model using the noisy image to obtain a third result.

[0110] Among them, the second student model is generated based on the image generation model, and the parameter iteration algorithm of the first student model is different from that of the second student model.

[0111] The second student model can be the same as the first student model at the beginning, and is also a picture generation model. In subsequent iterations, due to different inputs, the parameters of the second student model after iterative update are no longer the same as those of the initial picture generation model and the first student model, and become another model with different parameters. The second student model is not guided by the teacher model, and directly takes the noisy picture as input, with the training goal of restoring the noisy picture to the sample picture, performs denoising training, and outputs the third result.

[0112] S112, iterating the first student model and the second student model according to the loss functions of the first result and the third result respectively, repeating steps S110 to S112 until the first result and the third result satisfy the consistency function, and using the first student model as the target image generation model. The image generation time of the first student model is less than the image generation time of the image generation model, that is, the image generation time of the first student model is faster.

[0113] According to the first loss function corresponding to the first result and the second loss function corresponding to the third result, the respective first student model and the second student model are iteratively updated. The first student model and the second student model adopt different iterative update algorithms. For example, the first student model can use the EMA algorithm (EMA, Exponential Moving Average) to iteratively update model parameters. The second student model can be iteratively updated using the gradient back propagation algorithm. The first student model can also be iteratively updated using algorithms such as SMA (Simple Moving Average). Then, the first student model and the second student model after the model parameters are updated continue to execute steps S110 to S112 to perform denoising training on the new noisy image.

[0114] The third result and the first result are both used to reflect the sample image, but due to different inputs, the two results are not the same during the initial training process, but after training, the third result should be consistent with the first result. Therefore, through continuous training, until the first result and the third result meet the preset consistency function, the first student model that outputs the first result that meets the consistency function is used as the target image generation model. After the above-mentioned reverse noise training, denoising is completed through only two steps, step S111 and step S112, which shortens the training process, so that the image generation speed of the trained first student model is lower than that of the original image generation model, thereby improving the image generation efficiency.

[0115] Further, step S110 may include step S1101 and step S1102.

[0116] S1101, using the teacher model to restore the noisy image to the image before k times of noise addition, for denoising, wherein the noisy image is obtained by performing n+k times of noise addition on the sample image, where n≥1, k≥1.

[0117] The noise image is obtained by adding noise to the sample image multiple times. Two nodes are selected from the process of adding noise to the sample image, namely, adding noise to the sample image n times and adding noise to the sample image n+k times. After adding noise to the sample image n+k times, the noise image is obtained.

[0118] Using the teacher model, the training goal is to restore the noisy image to the image before k noise additions, that is, to restore the noisy image to the image after adding n noises to the sample image, and take the restored image with n noises added as the second result.

[0119] S1102, using the first student model, with the second result restored to the image before the n-times of noise addition as the training target, performing denoising to obtain the first result.

[0120] Since there are still n times of noise left in the second result that have not been restored, the first student model is used to continue restoring the second result, and denoising is performed with the goal of restoring the second result to the image before n times of noise addition, that is, denoising is performed with the goal of restoring the second result to the sample image to obtain the first result.

[0121] In some embodiments of the present disclosure, step S111 may include step S1111.

[0122] S1111. Using the second student model, denoising is performed with the goal of restoring the noisy image to the image before adding noise n+k times, to obtain a third result.

[0123] The second student model directly restores the noisy image to the image before n+k times of noise addition, and performs denoising. That is, the training goal is to directly restore the noisy image to the sample image, and denoise to obtain the third result. The input of the second student model is different from the input of the first student model, so the first result and the third result cannot meet the consistency requirements at the beginning of training, so continuous training is required.

[0124] Before the end of training, the direct outputs of the first student model and the second student model are difficult to meet the consistency requirements. In order to speed up the training and avoid model non-convergence, the initial first result output by the first student model can be mapped to the first result, and the initial third result output by the second student model can be mapped to the third result through a preset consistency function (which can adopt a function of the prior art).

[0125] In some embodiments of the present disclosure, regarding step S1101, Figure 8As shown, it may include steps S11011 to S11013.

[0126] S11011. Utilize the teacher model to use the picture text description information corresponding to the sample picture as a control condition to restore the noisy picture to the picture before k times of noise addition, perform denoising, and obtain a fourth result.

[0127] The teacher model uses the image text description information corresponding to the sample image as a control condition, and can more accurately denoise and restore the noisy image to obtain the fourth result. The fourth result meets specific conditions to ensure that the output of the subsequent first student model has a certain consistency.

[0128] S11012. Using the teacher model without using the picture text description information corresponding to the sample picture as a control condition to restore the noisy picture to the picture before k times of noise addition as a training target, perform denoising, and obtain the fifth result.

[0129] The teacher model is used again without using the image text description information as the control condition. The control condition is input into the teacher model in the form of empty characters. The training goal is to restore the noisy image to the image before k times of noise addition, and denoising is performed to obtain the fifth result. This makes the fifth result diverse and avoids the monotony of the generated result caused by overfitting of the output of the subsequent first student model.

[0130] S11013. Obtain a second result according to the fourth result and the fifth result.

[0131] The second result obtained by combining the fourth result and the fifth result has higher robustness, so that the subsequent first student model can generate reasonable pictures even when the conditional information is missing or unreliable, thereby ensuring the quality of the model.

[0132] Fig. 9 FIG. 2 is a schematic diagram showing the overall process of a method M200 for training a picture generation model according to another embodiment of the present disclosure. Fig. 9 The method shown includes steps S210 to S280. The method can be executed by an electronic device such as a server or a personal computer.

[0133] S210: Acquire an original picture sample set, filter out a first picture sample set from the original picture sample set according to a filtering condition, and filter out a second picture sample set from the original picture sample set according to a first quality condition.

[0134] S220: Perform model training using the resolution of the images in the first image sample set and the image text description information corresponding to the images to obtain a trained image generation model.

[0135] The specific training process can refer to the process from step S101 to step S105 mentioned above, which will not be described in detail here.

[0136] S230: Use the second picture sample set to perform model training on the picture generation model, so that the picture generation model generates pictures that meet the first quality condition.

[0137] S240, fusing the first model parameters of the i first models with the second model parameters of the picture generation model to obtain third model parameters, so that the picture generation model generates a picture that meets the second quality condition based on the third model parameters. Wherein, i≥1, the model architecture of the first model and the picture generation model is the same, the first model parameters of the first model are obtained after model training using the picture sample set, and the third model parameters are used as parameters actually used by the picture generation model.

[0138] S250: Use the image sample set to train the original first image generation model to obtain a trained first image generation model, and use the first image generation model as a target image generation model.

[0139] S260: Perform denoising training on the first student model using the teacher model and the noisy image to obtain a first result.

[0140] Among them, the teacher model is a picture generation model, the first student model is generated based on the picture generation model, and the noise picture is obtained by adding noise to the sample picture.

[0141] S270, using the noisy image to perform denoising training on the second student model to obtain a third result.

[0142] Among them, the second student model is generated based on the image generation model, and the parameter iteration algorithm of the first student model is different from that of the second student model.

[0143] S280. Iterate the first student model and the second student model according to the loss functions of the first result and the third result respectively, repeat steps S260 to S280 until the first result and the third result satisfy the consistency function, and use the first student model as the target image generation model.

[0144] Among them, the image generation time of the first student model is less than the image generation time of the image generation model.

[0145] The specific implementation process of step S210 to step S280 in this embodiment has corresponding content and detailed explanation in the above embodiments, which will not be repeated here.

[0146] like Fig.10 As shown, the present disclosure provides a method for training a picture generation model in the field of interior decoration renderings.

[0147] like Fig.10 As shown, first, for the sample, a data engineering plan is formulated to build a complete data pipeline for processing sample images. The data sources are divided into network data crawled from the Internet, that is, network image samples, purchased private domain data, that is, private domain image samples, and open source data sets, that is, open source image samples. The obtained images include network image samples, private domain image samples, and open source image samples. Then, the quality of the images in the original image sample set is evaluated. According to the preset screening conditions, as shown in the screening condition table of Table 1, according to 5 first-class classifications and 19 second-class labels, the images that meet the conditions are screened out by artificial intelligence to obtain the first screened image sample set. Then, the image semantic labels and natural language description information are used to describe the content of each image in the image sample set, and the corresponding image text description information is added to each image. The image semantic labels can be added based on each recognition small model. The natural language description information can be based on the existing suitable open source large model. Finally, the quality is stratified, and the second image sample set with higher image quality is screened out from the first image sample set through the first quality condition. The second image sample set can be manually screened.

[0148] The images in the first image sample set are clustered into 5 resolutions (such as Figure 3 ), and obtain 5 resolution buckets, i.e., 5 first image sample subsets. The resolution of each image in the 5 first image sample subsets is adjusted to the resolution corresponding to the first image sample subset in which it is located. The adjusted image is not cropped during the adjustment, and the complete information of the image is retained. During the subsequent training, several adjusted images in a first image sample subset are randomly selected to be trained in the image generation model. At the same time, the resolution of the adjusted image and the image text description information are used as input conditions. The image generation model after training is Fig.10 It can also be called the basic model. After training, the image generation model has the ability to generate interior decoration renderings of different resolutions based on text description information.

[0149] At the same time, during the training process using the bucketing strategy, in order to support longer text description information, the image text description information of the adjusted image that exceeds the unit length is split into multiple segments for training, so that the trained image generation model can support longer text description information.

[0150] Then, according to the judgment conditions such as the decoration color, furniture selection and hard decoration design in the first quality condition, the second picture sample set is selected from the original picture sample set. The picture generation model is trained using the second picture sample set. The trained picture generation model corresponds to Fig.10 The intermediate model in the training, the pictures generated by the image generation model are more beautiful.

[0151] In addition, in order to further enhance the realism of the images generated by the image generation model, the model parameters of the first model with the same model structure as the image generation model are fused with the model parameters of the first model that generates multiple images with stronger realism. Before fusion, the first model is trained using the first image sample set and / or the second image sample set, and the small learning rate of the first model is fine-tuned to ensure that the model parameters of the first model match the image generation model. The images generated by the image generation model using the fused third model parameters are more realistic. The image generation model using the third model parameters corresponds to Fig.10 The fused model in .

[0152] Finally, the model can be optimized based on the LCD (Learning using CategoricalDistillation) model distillation technology and the reverse denoising training process to train a more efficient target image generation model for birth charts.

[0153] See also Fig.10 In the three stages of quality tuning, model fusion and model optimization, automatic evaluation systems and manual evaluation systems are used to determine the quality and efficiency of images generated by the trained model, so that the image generation model can be adjusted in real time to ensure the training quality of the final model.

[0154] See Table 2 for a technical indicator comparison. Compared with the existing Stablediffusion model, Epic Realism model, Realistic Vision model, SDXL (StableDiffusion Extra Large) model, and SDXL+refiner model, the target image generation model of this application is superior to other models in terms of aesthetic index, soft decoration accuracy, style accuracy, hard decoration accuracy, similarity with the original image, and image-text matching. Figures 11 to 14 ,Compared with SDXL refiner model, Epic Realism model, Realistic Vision model and ,Stablediffusion model, the target image generation model outperforms other models in ,three evaluation indicators: aesthetics, text matching and layout rationality.

[0155] Table 2 Technical index comparison table

[0156]

[0157] Fig.15 FIG. 2 shows a schematic diagram of the overall process of a method M300 for generating an image in accordance with an embodiment of the present disclosure. Fig.15The method shown includes step S310 and step S320. The method can be executed by an electronic device such as a server or a personal computer. S310, receiving text information input by a user; S320, inputting the text information into a target image generation model disclosed in any of the above embodiments to obtain an image corresponding to the text description information.

[0158] Specifically, when it is necessary to use the trained target image generation model disclosed in any of the aforementioned embodiments in actual applications, first accept the text information input by the user, the text information can support text input of 227 words, and then input the text information into the target image generation model. The target image generation model will first determine the resolution of the image to be generated based on the text description input information, and then generate the corresponding image after determining the resolution. It can be seen that in this embodiment, due to the use of the target image generation model disclosed in any of the aforementioned embodiments, it is possible to generate high-quality images with the best resolution corresponding to the text information, instead of only being able to generate images with a fixed resolution, which improves the use scenarios and raw image quality of the model and improves the user experience.

[0159] Based on any of the above implementations, the present disclosure also provides a picture generation model training device. Fig.16 FIG. 1 is a schematic block diagram of the structure of a picture generation model training device according to an embodiment of the present disclosure. Fig.16 As shown, the image generation model training device includes: a description information adding module 1610, a clustering module 1620, a resolution adjustment module 1630 and a model training module 1640.

[0160] A description information adding module is used to add corresponding picture text description information to the pictures in the first picture sample set. A clustering module is used to cluster the pictures in the first picture sample set according to N resolutions to obtain N first picture sample subsets, wherein N≥2, and the N first picture sample subsets correspond to N resolutions respectively. A resolution adjustment module is used to adjust the resolutions of the pictures in the N first picture sample subsets respectively to obtain adjusted pictures, wherein the resolutions of the adjusted pictures are the resolutions corresponding to the first picture sample subsets where the adjusted pictures are located. A model training module is used to perform model training based on the adjusted pictures of the N first picture sample subsets, wherein the resolutions of the adjusted pictures are used as the first input condition for model training, and the picture text description information added to the adjusted pictures before adjustment is used as the second input condition for model training; and to obtain a trained picture generation model, wherein the picture generation model is used as the target picture generation model. In some embodiments of the present disclosure, the model training module is further used to divide the image text description information corresponding to the adjusted image into M segments of segment text description information based on the unit text length, wherein M≥2, the text length of the image text description information is greater than the unit text length, and the text length of each segment text description information is less than or equal to the unit text length; the M segments of segment text description information corresponding to the adjusted image are used as the second input condition for model training of the image generation model. In some embodiments of the present disclosure, the model training module is further used to convert the M segments of segment text description information into M segments of text embedding vectors respectively; concatenate the M segments of text embedding vectors into image text embedding vectors; and use the image text embedding vector corresponding to the adjusted image as the second input condition for model training of the image generation model. In some embodiments of the present disclosure, the model training module is used to perform model training on the image generation model using a second image sample set that meets the first quality condition, so that the image generation model generates images that meet the first quality condition.

[0161] In some embodiments of the present disclosure, the model training module is used to fuse the first model parameters of i first models with the second model parameters of the picture generation model to obtain third model parameters, so that the picture generation model generates a picture that meets the second quality condition based on the third model parameters, wherein i≥1, the first model and the picture generation model have the same model architecture, the first model parameters of the first model are obtained after model training using a picture sample set, and the third model parameters are used as parameters actually used by the picture generation model. In some embodiments of the present disclosure, the model training module is also used to obtain i first weighted results using i first weight coefficients and i first model parameters, and the i first weight coefficients correspond one-to-one to the i first model parameters; obtain a second weighted result using a second weight coefficient and a second model parameter corresponding to the second weight coefficient; and obtain a third model parameter using the i first weighted results and the second weighted results.

[0162] In some embodiments of the present disclosure, the model training module is used to: perform denoising training on a first student model using a teacher model and a noise picture to obtain a first result, wherein the teacher model is a picture generation model, the first student model is generated based on the picture generation model, and the noise picture is obtained by denoising a sample picture (S110); perform denoising training on a second student model using the noise picture to obtain a third result, wherein the second student model is generated based on the picture generation model, and the parameter iteration algorithms of the first student model and the second student model are different (S111); iterate the first student model and the second student model according to the loss functions of the first result and the third result, respectively (S112), repeat S110 to S112 until the first result and the third result satisfy the consistency function, and use the first student model as the target picture generation model, and the picture generation time of the first student model is less than the picture generation time of the picture generation model.

[0163] In some embodiments of the present disclosure, the model training module is also used to use the teacher model to restore the noise image to the state before k times of noise addition for denoising, and obtain a second result, wherein the noise image is obtained by performing n+k noise addition processing on the sample image, n≥1, k≥1; and use the first student model to restore the second result to the state before n times of noise addition for denoising, and obtain a first result. In some embodiments of the present disclosure, the model training module is also used to use the second student model to restore the noise image to the state before n+k times of noise addition for denoising, and obtain a third result. In some embodiments of the present disclosure, the model training module is also used to use the teacher model to use the picture text description information corresponding to the sample image as a control condition to restore the noise image to the state before k times of noise addition for denoising, and obtain a fourth result; use the teacher model without using the picture text description information corresponding to the sample image as a control condition to restore the noise image to the state before k times of noise addition for denoising, and obtain a fifth result; and obtain the second result based on the fourth result and the fifth result.

[0164] The present disclosure also provides a picture generating device. Fig.17 FIG. 1 is a schematic block diagram of the structure of an image generating device according to an embodiment of the present disclosure. Fig.17 As shown, the image generation device includes: a text information receiving module 1710 and an image generation module 1720. The text information receiving module 210 is used to receive text information input by a user; the image generation module 220 is used to input the text information into the target image generation model as described above to obtain an image corresponding to the text description information.

[0165] The present disclosure also provides a picture generation model training device. Fig.18 FIG. 1 is a schematic block diagram of the structure of a picture generation model training device according to an embodiment of the present disclosure. Fig.18 As shown, the image generating device includes: a sample set screening module 1810, a resolution training module 1820, a second training module 1830, a third training module 1840, a fourth training module 1850, a fifth training module 1860 and an iteration module 1870.

[0166] A sample set screening module is used to obtain an original image sample set, screen out a first image sample set from the original image sample set according to a screening condition, and screen out a second image sample set from the original image sample set according to a first quality condition. A first training module is used to perform model training using the resolution of the images in the first image sample set and the image text description information corresponding to the images to obtain a trained image generation model. A second training module is used to perform model training on the image generation model using the second image sample set so that the image generation model generates images that meet the first quality condition. A third training module is used to merge the first model parameters of i first models with the second model parameters of the image generation model to obtain third model parameters so that the image generation model generates images that meet the second quality condition based on the third model parameters, wherein i≥1, the first model and the image generation model have the same model architecture, the first model parameters of the first model are obtained after model training using the image sample set, and the third model parameters are used as parameters actually used by the image generation model. The fourth training module is used to perform denoising training on the first student model using the teacher model and the noise picture to obtain a first result, wherein the teacher model is the picture generation model, the first student model is generated based on the picture generation model, and the noise picture is obtained by adding noise to the sample picture. The fifth training module is used to perform denoising training on the second student model using the noise picture to obtain a third result, wherein the second student model is generated based on the picture generation model, and the parameter iteration algorithm of the first student model and the second student model is different. The iteration module is used to iterate the first student model and the second student model according to the loss functions of the first result and the third result respectively, until the first result and the third result satisfy the consistency function, and use the first student model as the target picture generation model, wherein the picture generation time of the first student model is less than the picture generation time of the picture generation model.

[0167] The present disclosure also provides an electronic device, which can execute the image generation model training method and / or image generation method of any one of the embodiments described above in the present disclosure, and the image generation model training device and / or image generation device of any one of the embodiments described above can be configured on the electronic device.

[0168] Fig.19It is a schematic block diagram of the structure of an electronic device equipped with a picture generation model training device, a picture generation device, or a picture generation model training device according to an embodiment of the present disclosure. Fig.19 It is shown in Fig.16 The modules of the image generation model training device, Fig.17 The modules of the image generating device, and Fig.18 The modules of the image generation model training device are only for convenience of representation. The electronic device only includes Fig.16 Image generation model training device, Fig.17 The image generating device Fig.18 A device in a picture generation model training device.

[0169] The hardware structure of the electronic device can be implemented using a bus architecture. The bus architecture can include any number of interconnected buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0170] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure only uses one connecting line, but does not mean that there is only one bus or one type of bus.

[0171] The present disclosure also provides a readable storage medium, in which a computer program is stored, and the computer program is used to implement the above method when executed by a processor. "Readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use in an instruction execution system, device or equipment or in combination with these instruction execution systems, devices or equipment. More specific examples of readable storage media include the following: an electrical connection portion (electronic device) with one or more wirings, a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM), etc.

[0172] The present disclosure also provides a computer program product. The method of the present disclosure can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed, the process or function of the present disclosure is executed in whole or in part.

[0173] A computer program or instruction may be stored in a readable storage medium or transmitted from one readable storage medium to another readable storage medium, for example, a computer program or instruction may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The readable storage medium may be any available medium that can be accessed or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid state drive. The computer readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0174] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0176] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0178] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Claims

1. A method for training an image generation model, characterized in that: include: Adding corresponding picture text description information to the pictures in the first picture sample set; Clustering the images in the first image sample set according to N resolutions to obtain N first image sample subsets, where N ≥ 2, and the N first image sample subsets correspond to the N resolutions respectively; Adjusting the resolutions of the pictures in the N first picture sample subsets respectively to obtain adjusted pictures, wherein the resolutions of the adjusted pictures are the resolutions corresponding to the first picture sample subsets in which the adjusted pictures are located; Performing model training according to the adjusted pictures of the N first picture sample subsets, wherein the resolution of the adjusted pictures is used as a first input condition for model training, and the picture text description information is used as a second input condition for model training; and A trained image generation model is obtained, wherein the image generation model is used as a target image generation model.

2. The image generation model training method according to claim 1, characterized in that: The image text description information is used as the second input condition for model training, including: Divide each image text description information into M segments of text description information based on the unit text length, where M≥2, the text length of the image text description information is greater than the unit text length, and the text length of each segment text description information is less than or equal to the unit text length; The M segments of text description information corresponding to the adjusted image are used as a second input condition for training the image generation model.

3. The image generation model training method according to claim 2, characterized in that: Using the M segments of text description information corresponding to the adjusted image as a second input condition for training the image generation model includes: Convert the M segments of text description information into M segments of text embedding vectors respectively; Concatenate the M text embedding vectors into the image text embedding vector; The picture text embedding vector corresponding to the adjusted picture is used as a second input condition for model training of the picture generation model.

4. The image generation model training method according to claim 1, characterized in that: After obtaining the trained image generation model, it also includes: The picture generation model is trained using a second picture sample set that meets a first quality condition, so that the picture generation model generates pictures that meet the first quality condition.

5. The image generation model training method according to claim 4, characterized in that: After the image generation model is trained using the second image sample set, the method further includes: The first model parameters of i first models are merged with the second model parameters of the picture generation model to obtain third model parameters, so that the picture generation model generates a picture that meets the second quality condition based on the third model parameters, wherein i≥1, the first model and the picture generation model have the same model architecture, the first model parameters of the first model are obtained after model training using the first picture sample set or the second picture sample set, and the third model parameters are used as parameters actually used by the picture generation model.

6. A method for training an image generation model, characterized in that: include: Acquire an original picture sample set, filter out a first picture sample set from the original picture sample set, and filter out a second picture sample set from the original picture sample set according to a first quality condition; Performing model training using the resolution of the images in the first image sample set and the image text description information corresponding to the images to obtain a trained image generation model; Using the second picture sample set to perform model training on the picture generation model, so that the picture generation model generates pictures that meet the first quality condition; fusing the first model parameters of i first models with the second model parameters of the picture generation model to obtain third model parameters, so that the picture generation model generates a picture that meets the second quality condition based on the third model parameters, wherein i≥1, the first model and the picture generation model have the same model architecture, the first model parameters of the first model are obtained after model training using the first picture sample set or the second picture sample set, and the third model parameters are used as parameters actually used by the picture generation model; Performing denoising training on a first student model using a teacher model and a noise picture to obtain a first result, wherein the teacher model is the picture generation model, the first student model is generated based on the picture generation model, and the noise picture is obtained by adding noise to a sample picture; Performing denoising training on a second student model using the noisy picture to obtain a third result, wherein the second student model is generated based on the picture generation model, and parameter iteration algorithms of the first student model and the second student model are different; The first student model and the second student model are iterated according to the loss functions of the first result and the third result respectively until the first result and the third result satisfy the consistency function, and the first student model is used as the target image generation model, wherein the image generation time of the first student model is less than the image generation time of the image generation model.

7. A method for generating an image, characterized in that: include: Receive text messages; The text information is input into a target image generation model obtained by the image generation model training method as described in any one of claims 1 to 6 to obtain an image corresponding to the text information.

8. An electronic device, characterized in that: include: A memory storing execution instructions; as well as A processor, wherein the processor executes the execution instructions stored in the memory, so that the processor executes the image generation model training method as described in any one of claims 1 to 6, or executes the image generation method as described in claim 7.

9. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, which, when executed by the processor, are used to implement the image generation model training method as described in any one of claims 1 to 6, or to implement the image generation method as described in claim 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the image generation model training method according to any one of claims 1 to 6, or implements the image generation method according to claim 7.

Images