Image generation model training method and device, image generation method and device and electronic equipment

Abstract

The invention relates to an image generation model training method and device, an image generation method and device, electronic equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a first sample image of a first resolution, and first text description information and a first sample noise adding image corresponding to the first sample image; performing iterative training on a to-be-trained image generation model according to the first sample noise-added image and the first text description information to obtain an initial image generation model; obtaining a second sample image of a second resolution, and second text description information and a second sample noise adding image corresponding to the second sample image; the second resolution is higher than the first resolution; the total noise adding step number of the second image noise adding model is greater than the total noise adding step number of the first image noise adding model; and performing iterative training on the initial image generation model according to the second sample noise-added image and the second text description information to obtain a target image generation model. By adopting the method, the image generation accuracy can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image generation technology, and in particular to an image generation model training method, an image generation method, an apparatus, an electronic device, a storage medium, and a computer program product. Background Art

[0002] With the development of image generation technology, many image generation models have emerged. Some image generation models can generate corresponding images based on text description information.

[0003] In the related art, in order to train a model for high-resolution image generation, noise is generally added to the high-resolution image to generate training data, so that the trained model can generate corresponding images based on text description information; however, during the training process, there may be incomplete noise addition, that is, the image after noise addition at the last step still retains some low-frequency features of the original image, such as contours, colors, etc., rather than a pure noise image. However, the inference of the model starts from a pure noise image, resulting in inconsistency in the initial state of the model during training and inference at high resolution, which leads to a decrease in the image-text correlation and image quality of the image generated by the trained model, and further leads to a low image generation accuracy. Summary of the invention

[0004] The present disclosure provides an image generation model training method, an image generation method, an apparatus, an electronic device, a storage medium, and a computer program product to at least solve the problem of low image generation accuracy in the related art. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image generation model training method, comprising:

[0006] Acquire a first sample image of a first resolution, as well as first text description information and a first sample noisy image corresponding to the first sample image; the first sample noisy image is obtained by performing random noise processing on the first sample image using a first image noisy model;

[0007] Iteratively training the image generation model to be trained according to the first sample noisy image and the first text description information to obtain an initial image generation model;

[0008] Acquire a second sample image of a second resolution, and second text description information and a second sample noisy image corresponding to the second sample image; the second resolution is higher than the first resolution; the second sample noisy image is obtained by performing random noisy processing on the second sample image by a second image noisy model; the total number of noisy steps of the second image noisy model is greater than the total number of noisy steps of the first image noisy model;

[0009] The initial image generation model is iteratively trained according to the second sample noisy image and the second text description information to obtain a target image generation model.

[0010] In an exemplary embodiment, the iterative training of the image generation model to be trained according to the first sample noisy image and the first text description information to obtain the initial image generation model includes:

[0011] Inputting the first sample noisy image and the first text description information into the image generation model to be trained for denoising, thereby obtaining a first predicted denoised image corresponding to the first sample noisy image;

[0012] Acquire a first target denoised image corresponding to the first sample denoised image, and determine a first loss value according to a difference between the first target denoised image and the first predicted denoised image;

[0013] According to the first loss value, the image generation model to be trained is iteratively trained to obtain a trained image generation model, and the trained image generation model is used as the initial image generation model.

[0014] In an exemplary embodiment, the step of obtaining a first target denoised image corresponding to the first sample denoised image includes:

[0015] Obtaining a first number of noise adding steps corresponding to the first sample noise adding image;

[0016] Performing a decrement process on the first noise adding step number to obtain a processed first noise adding step number;

[0017] The first processed denoising step number and the first sample image are input into the first image denoising model for denoising processing to obtain a first processed image, and the first processed image is used as the first target denoised image; the first processed image is used to represent the image obtained after denoising the first sample image to the first processed denoising step number.

[0018] In an exemplary embodiment, the iterative training of the initial image generation model according to the second sample noisy image and the second text description information to obtain a target image generation model includes:

[0019] Inputting the second sample noisy image and the second text description information into the initial image generation model for denoising, to obtain a second predicted denoised image corresponding to the second sample noisy image;

[0020] Acquire a second target denoised image corresponding to the second sample denoised image, and determine a second loss value according to a difference between the second target denoised image and the second predicted denoised image;

[0021] According to the second loss value, the initial image generation model is iteratively trained to obtain a trained initial image generation model, and the trained initial image generation model is used as the target image generation model.

[0022] In an exemplary embodiment, the step of acquiring a second target denoised image corresponding to the second sample denoised image includes:

[0023] Obtaining a second number of noise adding steps corresponding to the second sample noise adding image;

[0024] Performing a decrement process on the second noise adding step number to obtain a processed second noise adding step number;

[0025] The processed second denoising step number and the second sample image are input into the second image denoising model for denoising processing to obtain a second processed image, and the second processed image is used as the second target denoised image; the second processed image is used to represent the image obtained after denoising the second sample image to the processed second denoising step number.

[0026] In an exemplary embodiment, acquiring a second sample image with a second resolution includes:

[0027] Acquire an original image at a second resolution, and image scores of the original image under a plurality of preset indicators;

[0028] Performing fusion processing on the image scores of the original image under multiple preset indicators to obtain a fusion score of the original image;

[0029] Filtering out candidate images whose fusion scores satisfy a first preset score condition from the original images;

[0030] The annotation scores of the candidate images are obtained, a target image whose annotation score meets a second preset score condition is screened out from the candidate images, and the target image is used as the second sample image.

[0031] According to a second aspect of an embodiment of the present disclosure, there is provided an image generating method, comprising:

[0032] Acquire the text description information to be processed and the target noisy image; the target noisy image is used to represent the image obtained after the second sample image is noisy to the total number of noisy steps of the second image noisy model;

[0033] The text description information to be processed and the target noisy image are input into a target image generation model for multiple denoising processes to obtain a second resolution image corresponding to the text description information to be processed; the target image generation model is trained by the image generation model training method described in any one of the embodiments of the first aspect.

[0034] In an exemplary embodiment, the step of inputting the to-be-processed text description information and the target noisy image into a target image generation model for multiple denoising processes to obtain a second resolution image corresponding to the to-be-processed text description information includes:

[0035] Input the to-be-processed text description information and the target noisy image into a target image generation model for denoising, thereby obtaining a processed image corresponding to the to-be-processed text description information;

[0036] The processed image corresponding to the text description information to be processed is used as the target noisy image, and the process jumps to the step of inputting the text description information to be processed and the target noisy image into the target image generation model for denoising to obtain the processed image corresponding to the text description information to be processed, until the cumulative number of denoising processes of the target image generation model reaches the total number of denoising steps of the second image denoising model;

[0037] The processed image obtained after the target image generation model performs the last denoising process is used as the second resolution image.

[0038] According to a third aspect of an embodiment of the present disclosure, there is provided an image generation model training device, comprising:

[0039] A first acquisition unit is configured to acquire a first sample image of a first resolution, first text description information corresponding to the first sample image, and a first sample noisy image; the first sample noisy image is obtained by performing random noise processing on the first sample image by a first image noisy model;

[0040] A first training unit is configured to perform iterative training on the image generation model to be trained according to the first sample noisy image and the first text description information to obtain an initial image generation model;

[0041] A second acquisition unit is configured to acquire a second sample image of a second resolution, second text description information corresponding to the second sample image, and a second sample noisy image; the second resolution is higher than the first resolution; the second sample noisy image is obtained by randomly noisy processing the second sample image by a second image noisy model; the total number of noisy steps of the second image noisy model is greater than the total number of noisy steps of the first image noisy model;

[0042] The second training unit is configured to perform iterative training on the initial image generation model according to the second sample noisy image and the second text description information to obtain a target image generation model.

[0043] In an exemplary embodiment, the first training unit is further configured to perform denoising processing on the first sample noisy image and the first text description information, input the first sample noisy image and the first text description information into the image generation model to be trained, and obtain a first predicted denoised image corresponding to the first sample noisy image; obtain a first target denoised image corresponding to the first sample noisy image, and determine a first loss value according to the difference between the first target denoised image and the first predicted denoised image; according to the first loss value, iteratively train the image generation model to be trained to obtain a trained image generation model, and use the trained image generation model as the initial image generation model.

[0044] In an exemplary embodiment, the first training unit is further configured to execute the steps of obtaining a first denoising step corresponding to the first sample denoised image; performing a decrement process on the first denoising step to obtain a first denoising step after processing; inputting the first denoising step after processing and the first sample image into the first image denoising model for denoising to obtain a first processed image, and using the first processed image as the first target denoised image; the first processed image is used to represent an image obtained by denoising the first sample image to the first denoising step after processing.

[0045] In an exemplary embodiment, the second training unit is further configured to perform denoising processing on the second sample denoised image and the second text description information, input the initial image generation model to obtain a second predicted denoised image corresponding to the second sample denoised image; obtain a second target denoised image corresponding to the second sample denoised image, and determine a second loss value based on the difference between the second target denoised image and the second predicted denoised image; iteratively train the initial image generation model based on the second loss value to obtain a trained initial image generation model, and use the trained initial image generation model as the target image generation model.

[0046] In an exemplary embodiment, the second training unit is further configured to execute the steps of obtaining a second denoising step corresponding to the second sample denoised image; performing a decrement process on the second denoising step to obtain a processed second denoising step; inputting the processed second denoising step and the second sample image into the second image denoising model for denoising to obtain a second processed image, and using the second processed image as the second target denoised image; the second processed image is used to represent an image obtained by denoising the second sample image to the processed second denoising step.

[0047] In an exemplary embodiment, the second acquisition unit is further configured to execute the acquisition of an original image of a second resolution, and an image score of the original image under a plurality of preset indicators; perform a fusion process on the image scores of the original image under a plurality of preset indicators to obtain a fusion score of the original image; screen out candidate images whose fusion scores satisfy a first preset score condition from the original images; acquire the annotation scores of the candidate images, screen out a target image whose annotation scores satisfy a second preset score condition from the candidate images, and use the target image as the second sample image.

[0048] According to a fourth aspect of an embodiment of the present disclosure, there is provided an image generating device, including:

[0049] A data acquisition unit is configured to acquire text description information to be processed and a target noisy image; the target noisy image is used to represent an image obtained after the second sample image is noisy to the total number of noisy steps of the second image noisy model;

[0050] An image generation unit is configured to execute multiple denoising processes on the text description information to be processed and the target noisy image, and to obtain a second resolution image corresponding to the text description information to be processed; the target image generation model is trained by the internal image generation model training method described in any one of the embodiments of the first aspect.

[0051] In an exemplary embodiment, the image generation unit is further configured to execute the step of inputting the text description information to be processed and the target noisy image into a target image generation model for denoising processing to obtain a processed image corresponding to the text description information to be processed; using the processed image corresponding to the text description information to be processed as the target noisy image, and jumping to the step of inputting the text description information to be processed and the target noisy image into a target image generation model for denoising processing to obtain a processed image corresponding to the text description information to be processed, until the cumulative number of denoising processing times of the target image generation model reaches the total number of denoising steps of the second image denoising model; and using the processed image obtained after the target image generation model performs the last denoising processing as the second resolution image.

[0052] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0053] processor;

[0054] a memory for storing instructions executable by the processor;

[0055] The processor is configured to execute the instructions to implement the image generation model training method as described in any one of the embodiments of the first aspect, or the image generation method as described in any one of the embodiments of the second aspect.

[0056] According to the sixth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image generation model training method described in any one of the embodiments of the first aspect, or the image generation method described in any one of the embodiments of the second aspect.

[0057] According to the seventh aspect of the embodiments of the present disclosure, a computer program product is provided, which includes instructions. When the instructions are executed by a processor of an electronic device, the electronic device can execute the image generation model training method described in any one of the embodiments of the first aspect, or the image generation method described in any one of the embodiments of the second aspect.

[0058] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0059] First, a first sample image of a first resolution, as well as first text description information and a first sample noisy image corresponding to the first sample image are obtained; the first sample noisy image is obtained by randomly noising the first sample image through a first image noisy model; then, according to the first sample noisy image and the first text description information, the image generation model to be trained is iteratively trained to obtain an initial image generation model; then, a second sample image of a second resolution, as well as second text description information and a second sample noisy image corresponding to the second sample image are obtained; the second resolution is higher than the first resolution; the second sample noisy image is obtained by randomly noising the second sample image through a second image noisy model; the total number of noisy steps of the second image noisy model is greater than the total number of noisy steps of the first image noisy model; finally, according to the second sample noisy image and the second text description information, the initial image generation model is iteratively trained to obtain a target image generation model. Thus, when training the model, a two-stage training scheme is designed. In the first stage, the image generation model to be trained is iteratively trained according to the first text description information and the first sample noisy image corresponding to the first sample image of the first resolution, so as to learn the knowledge of the text-image in the low-resolution training stage, thereby improving the image-text consistency of the image generated by the model; in the second stage, the initial image generation model trained in the first stage is iteratively trained according to the second text description information and the second sample noisy image corresponding to the second sample image of the second resolution, so as to learn the image quality improvement in the high-resolution training stage, thereby improving the image quality corresponding to the image generated by the model; and, in the second training stage, the introduction of the second image denoising model with a total denoising step greater than the first image denoising model used in the first training stage is conducive to extending the denoising time of the model, so that the initial state of the model training and reasoning is consistent, thereby improving the image-text relevance and image quality of the image generated by the training model, thereby improving the image generation accuracy, and at the same time avoiding the incomplete denoising of the model at high resolution, resulting in the inconsistency of the initial state during model training and reasoning, thereby causing the image-text relevance and image quality of the image generated by the training model to decrease, thereby causing the defect of low image generation accuracy.

[0060] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0062] Figure 1 The figure is a flowchart of a method for training an image generation model according to an exemplary embodiment.

[0063] Figure 2 is a schematic diagram showing the noise adding effect of a scaled linear noise scheduler at different resolutions according to an exemplary embodiment.

[0064] Figure 3 is a flowchart showing the steps of obtaining an initial image generation model according to an exemplary embodiment.

[0065] Figure 4 The figure is a flowchart showing the steps of obtaining a target image generation model according to an exemplary embodiment.

[0066] Figure 5 is a flowchart showing steps of acquiring a second sample image with a second resolution according to an exemplary embodiment.

[0067] Figure 6 The figure is a flowchart of an image generating method according to an exemplary embodiment.

[0068] Figure 7 is a flowchart of another image generating method according to an exemplary embodiment.

[0069] Figure 8 It is a block diagram of an image generation model training device according to an exemplary embodiment.

[0070] Fig. 9 The figure is a block diagram of an image generating device according to an exemplary embodiment.

[0071] Fig.10 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0072] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0073] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0074] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0075] With the rapid development of AIGC (Artificial Intelligence Generated Content) technology, the ability to generate and edit images has gradually become an everyday tool for ordinary users and professional designers. High-quality images not only enhance the visual experience, but also play a key role in information communication, branding and marketing. Image quality directly affects the user's perception and interaction effect. Especially in the fields of e-commerce, advertising and social media, high-definition and detailed images can significantly improve user engagement and willingness to use. In addition, the relevance of images and text is also a crucial factor. The close combination of images and text content can enhance the transmission effect of information and make it easier for users to understand and remember. For professional designers, the organic combination of images and text not only enhances the beauty of the work, but also more effectively conveys the design concept and brand story. In news reports, educational materials and technical documents, the combination of images and text can help readers understand complex information and concepts more intuitively. Therefore, with the continuous advancement of AIGC technology, how to achieve a high degree of relevance between image and text content while ensuring the high quality of images has become an important topic in technology research and development and application promotion.

[0076] At present, the mainstream text-generated image technology solution is mainly based on the diffusion model of latent space, in which the noise scheduler mainly used is the scaled linear noise scheduler. However, this noise scheduler is mainly suitable for model training at a resolution of 512×512. When switching to a high resolution such as 1024×51024 for training, there will be a problem of incomplete noise addition, which will lead to a decrease in the quality of the image output and the correlation between the image and the text during image reasoning, and cannot meet the needs of practical applications. Based on this, the present disclosure proposes an image generation model training method that can solve this problem.

[0077] Figure 2 is a flowchart of an image generation model training method according to an exemplary embodiment. Figure 2 As shown, the image generation model training method is used in a server; it is understandable that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be but is not limited to various personal computers, laptops, smart phones and tablets, and the server can be implemented as an independent server or a server cluster composed of multiple servers. In this exemplary embodiment, the method includes the following steps:

[0078] In step S110, a first sample image of a first resolution, first text description information corresponding to the first sample image, and a first sample noisy image are obtained; the first sample noisy image is obtained by performing random noise processing on the first sample image using a first image noisy model.

[0079] The first resolution refers to a low resolution, such as 512×512. The first sample image refers to a low resolution image, such as a 512×512 image.

[0080] The first text description information corresponding to the first sample image refers to the detailed description information of the first sample image, specifically refers to an image description with an average number of characters of about 99, such as "There are a few white clouds in the sky, and there are mountains in the distance...", which can be generated by a multimodal large model for image description. The multimodal large model can refer to a CogVLM multimodal large model, a GPT4v multimodal large model, an InternVL multimodal large model, etc.

[0081] Among them, the first sample noisy image refers to the image obtained after the first sample image is randomly denoised by the first image noisy model, specifically refers to the image noisy to a random number of steps, such as the image noisy to the 8th step, the image noisy to the 18th step, the image noisy to the 28th step, etc.

[0082] The first image noise model refers to a model that randomly adds noise to an image, such as a scaled linear noise scheduler. During the model training process, by adding noise to the first sample image through the first image noise model, the image generation model can learn to recover the original image from the noise. It should be noted that, assuming that the total number of noise steps of the first image noise model is 1000, then after multiple rounds of training, the first sample noise image can cover all steps.

[0083] Exemplarily, the server obtains a first sample image of a first resolution from a database or a network, and inputs the first sample image into the multimodal large model for image description generation processing to obtain corresponding image description information as first text description information corresponding to the first sample image; at the same time, the first sample image is randomly denoised by a first image denoising model to obtain a corresponding noisy image as the first sample denoised image corresponding to the first sample image.

[0084] For example, the first image noise model is a scaled linear noise scheduler, and the formula of the noise scheduler is as follows:

[0085]

[0086] α t =1-β t; (Formula 2)

[0087]

[0088] Among them, x0 represents the initial image without adding noise, x t represents the image added with noise to the tth step, ε represents the added random noise, t∈[1,1000] represents the number of steps of adding noise, β t Controls the noise intensity at time t.

[0089] That is, by inputting x0 and t into the scaled linear noise scheduler, the image x added to the tth step can be obtained. t .

[0090] In step S120, the image generation model to be trained is iteratively trained according to the first sample noisy image and the first text description information to obtain an initial image generation model.

[0091] The image generation model to be trained refers to a diffusion model based on latent space, specifically a UNet model.

[0092] The initial image generation model refers to a trained image generation model, which can generate corresponding low-resolution images based on text description information.

[0093] Exemplarily, the server iteratively trains the image generation model to be trained based on the first sample noisy image and the first text description information until a first training number is reached or the first loss value of the trained image generation model is less than a first preset threshold, then stops training, and uses the trained image generation model as the initial image generation model.

[0094] In step S130, a second sample image of a second resolution, second text description information corresponding to the second sample image, and a second sample noisy image are obtained; the second resolution is higher than the first resolution; the second sample noisy image is obtained by randomly noisy processing the second sample image by a second image noisy model; the total number of noisy steps of the second image noisy model is greater than the total number of noisy steps of the first image noisy model.

[0095] The second resolution refers to a high resolution, such as 1024×1024. The second sample image refers to a high resolution image, such as a 1024×1024 image.

[0096] Among them, the second text description information corresponding to the second sample image refers to the detailed description information of the second sample image, specifically refers to the image description with an average number of approximately 99 characters, such as "There are many flowers in the park, and there is a lotus pond in the distance...", and the image description can also be generated through a multimodal large model.

[0097] Among them, the second sample noisy image refers to the image obtained after randomly adding noise to the second sample image through the second image noisy model, specifically refers to the image added with noise to a random number of steps, such as the image added with noise to the 6th step, the image added with noise to the 16th step, the image added with noise to the 26th step, etc.

[0098] The second image noise model refers to a model that randomly adds noise to an image, such as a delayed noise scheduler. During the model training process, by adding noise to the second sample image through the second image noise model, the image generation model can learn to recover the original image from the noise. It should be noted that, assuming that the total number of noise steps of the second image noise model is 1100, then after multiple rounds of training, the second sample noise image can cover all steps.

[0099] It should be noted that the total number of denoising steps for the second image denoising model (for example, 1100) is greater than the total number of denoising steps for the first image denoising model (for example, 1000) in order to avoid incomplete denoising of the model at high resolution, resulting in inconsistency in the initial state during model training and inference. Specifically, after completing the first stage of training, the model has the ability to depict general concepts, and then switches to high-resolution training to further improve the model's output quality; for the traditional scaled linear noise scheduler, the denoising effect of training at different resolutions is as follows Figure 2 As shown in the figure, it can be found that in the first stage at a resolution of 512×512, the model was close to pure noise when the noise was added to the 1000th step; while in the second stage at a resolution of 1024×1024, when the model was noised to the 1000th step, the noisy image still retained some low-frequency features of the original image, such as contours, colors, etc., rather than pure noise; and the model's reasoning starts with pure noise, so the use of a scaled linear noise scheduler at high resolution will lead to inconsistent training and reasoning, resulting in problems with the text-image correlation and image quality of the generated image. Therefore, in the second stage of high-resolution training, we designed a delayed noise scheduler, which extended the original 1000-step noise addition time to 1100 steps, and adjusted the β t The calculation method of equal noise intensity has been adjusted. Then, under the action of the delayed noise scheduler, the 1024×1024 high-resolution image will be closer to a pure noise image when it is denoised to the 1100th step, thus solving the problem of incomplete denoising when training at high resolution with the traditional scaled linear noise scheduler.

[0100] Exemplarily, the server obtains a second sample image of a second resolution from a database or a network, and inputs the second sample image into the multimodal large model for image description generation processing to obtain corresponding image description information as second text description information corresponding to the second sample image; at the same time, the second sample image is randomly noised through the second image noisy model to obtain a corresponding noisy image as the second sample noisy image corresponding to the second sample image.

[0101] For example, the second image noise adding model is a delayed noise scheduler, and the formula of the noise scheduler is as follows:

[0102]

[0103] α t =1-β t ; (Formula 6)

[0104]

[0105] Among them, x0 represents the initial image without adding noise, x t represents the image added with noise to the tth step, ε represents the added random noise, t∈[1,1000] represents the number of steps of adding noise, β t Controls the noise intensity at time t.

[0106] That is, by inputting x0 and t into the delayed noise scheduler, the image x added to the tth step can be obtained. t .

[0107] In step S140, the initial image generation model is iteratively trained according to the second sample noisy image and the second text description information to obtain a target image generation model.

[0108] Among them, the target image generation model refers to the trained initial image generation model, which is used to generate corresponding high-resolution images based on text description information.

[0109] Exemplarily, the server iteratively trains the initial image generation model based on the second sample noisy image and the second text description information until the second training times is reached or the second loss value of the trained initial image generation model is less than a second preset threshold, then stops the training and uses the trained initial image generation model as the target image generation model.

[0110] In the above-mentioned image generation model training method, a first sample image of a first resolution, a first text description information corresponding to the first sample image, and a first sample noisy image are first obtained; the first sample noisy image is obtained by randomly noising the first sample image through a first image noisy model; then, according to the first sample noisy image and the first text description information, the image generation model to be trained is iteratively trained to obtain an initial image generation model; then, a second sample image of a second resolution, and a second text description information and a second sample noisy image corresponding to the second sample image are obtained; the second resolution is higher than the first resolution; the second sample noisy image is obtained by randomly noising the second sample image through a second image noisy model; the total number of noisy steps of the second image noisy model is greater than the total number of noisy steps of the first image noisy model; finally, according to the second sample noisy image and the second text description information, the initial image generation model is iteratively trained to obtain a target image generation model. Thus, when training the model, a two-stage training scheme is designed. In the first stage, the image generation model to be trained is iteratively trained according to the first text description information and the first sample noisy image corresponding to the first sample image of the first resolution, so as to learn the knowledge of the text-image in the low-resolution training stage, thereby improving the image-text consistency of the image generated by the model; in the second stage, the initial image generation model trained in the first stage is iteratively trained according to the second text description information and the second sample noisy image corresponding to the second sample image of the second resolution, so as to learn the image quality improvement in the high-resolution training stage, thereby improving the image quality corresponding to the image generated by the model; and, in the second training stage, the introduction of the second image denoising model with a total denoising step greater than the first image denoising model used in the first training stage is conducive to extending the denoising time of the model, so that the initial state of the model training and reasoning is consistent, thereby improving the image-text relevance and image quality of the image generated by the training model, thereby improving the image generation accuracy, and at the same time avoiding the incomplete denoising of the model at high resolution, resulting in the inconsistency of the initial state during model training and reasoning, thereby causing the image-text relevance and image quality of the image generated by the training model to decrease, thereby causing the defect of low image generation accuracy.

[0111] In an exemplary embodiment, if Figure 3 As shown, the above step S120, according to the first sample noisy image and the first text description information, iteratively trains the image generation model to be trained to obtain the initial image generation model, which can be specifically implemented by the following steps:

[0112] In step S310, the first sample noisy image and the first text description information are input into the image generation model to be trained for denoising, so as to obtain a first predicted denoised image corresponding to the first sample noisy image.

[0113] In step S320, a first target denoised image corresponding to the first sample denoised image is obtained, and a first loss value is determined according to a difference between the first target denoised image and the first predicted denoised image.

[0114] In step S330, the image generation model to be trained is iteratively trained according to the first loss value to obtain a trained image generation model, and the trained image generation model is used as the initial image generation model.

[0115] The first predicted denoised image refers to the predicted denoised image corresponding to the first sample denoised image, which is specifically generated by the image generation model to be trained. The first target denoised image refers to the actual denoised image corresponding to the first sample denoised image, which is specifically processed by the first image denoising model.

[0116] The first loss value refers to a loss value obtained based on a difference between the first target denoised image and the first predicted denoised image, such as a mean square error.

[0117] Exemplarily, the server first inputs the first sample noisy image and the first text description information into the image generation model to be trained for denoising, and obtains the first predicted denoised image corresponding to the first sample noisy image; then, through the first image denoising model, obtains the first target denoised image corresponding to the first sample noisy image; according to the difference between the first target denoised image and the first predicted denoised image, and in combination with the mean square error loss function, calculates the first loss value; according to the first loss value, adjusts the model parameters of the image generation model to be trained to obtain the adjusted image generation model, and repeats the above process until the first loss value of the adjusted image generation model is less than the first preset threshold, then stops training, and uses the adjusted image generation model as the trained image generation model, and finally uses the trained image generation model as the initial image generation model.

[0118] For example, the first loss value is calculated by the following formula:

[0119] Loss1 = MSE_Loss(UNet(x_t) - x_{t-1} ); (Formula 9)

[0120] Among them, Loss1 represents the first loss value, MSE_Loss represents the mean square error function; x_t and x_{t-1} represent the first sample denoised image and the first target denoised image, respectively, specifically representing the images denoised to the t-th and (t-1)-th steps obtained by the scaled linear noise scheduler; UNet(x_t) represents the first predicted denoised image, specifically representing the predicted denoised image obtained after the image x_t denoised to the t-th step obtained by the scaled linear noise scheduler is input into UNet (i.e., the image generation model to be trained).

[0121] It should be noted that through the above-mentioned first-stage training method, the model can derive x_{t-1} from (x_t) (such as x_1000), derive x_{t-2} from x_{t-1}, and so on, until x_0 is derived.

[0122] In this embodiment, in the process of training the initial image generation model, the first sample noisy image, the first text description information and the first target denoised image are comprehensively considered, so that the model can learn the knowledge of text-generated images in the low-resolution training stage, thereby improving the model's image-text alignment ability, and further improving the image-text consistency of the image generated by the model.

[0123] In an exemplary embodiment, the above step S320, obtaining the first target denoised image corresponding to the first sample noisy image, specifically includes the following contents: obtaining the first denoising step number corresponding to the first sample noisy image; performing a decrementing process on the first denoising step number to obtain the processed first denoising step number; inputting the processed first denoising step number and the first sample image into the first image denoising model for denoising process to obtain the first processed image, and using the first processed image as the first target denoised image; the first processed image is used to represent the image obtained after denoising the first sample image to the processed first denoising step number.

[0124] The first number of noise adding steps refers to the number of noise adding steps corresponding to the first sample noise adding image. The decrement processing refers to the subtraction operation, for example, if the first number of noise adding steps is 10, then the first number of noise adding steps after processing is 9.

[0125] The first processed image (ie, the first target denoised image) refers to an image obtained by denoising the first sample image to a first number of denoising steps after processing using a first image denoising model.

[0126] Exemplarily, the server obtains a first denoising step number corresponding to the first sample noisy image, and decrements the first denoising step number to obtain the processed first denoising step number, and finally inputs the processed first denoising step number and the first sample image into the first image denoising model for denoising processing to obtain an image obtained by denoising the first sample image to the processed first denoising step number as the first target denoised image corresponding to the first sample noisy image.

[0127] For example, assuming that the first denoising step number corresponding to the first sample noisy image is 9, that is, the first sample noisy image is an image noisy to the 9th step, the server substitutes t=8 and the first sample image x0 into formulas (1) to (4) to obtain the image noisy to the 8th step as the first target denoised image corresponding to the image noisy to the 9th step.

[0128] It should be noted that, in the above first stage training process, the image of the previous step before the current step is added with noise, and is used as the first target denoised image of the image added with noise to the current step.

[0129] In this embodiment, the first denoising step number and the first sample image after the first denoising step number corresponding to the first sample denoised image are input into the first image denoising model for denoising processing to obtain a first processed image, and the first processed image is used as the first target denoised image of the first sample denoised image. This allows the initial image generation model trained based on the first sample denoised image, the first text description information and the first target denoised image to gradually derive a low-resolution image corresponding to the text description information from the pure noise image, thereby improving the image-text consistency of the image output by the image generation model.

[0130] In an exemplary embodiment, if Figure 4 As shown, the above step S140, according to the second sample noisy image and the second text description information, iteratively trains the initial image generation model to obtain the target image generation model, which can be specifically implemented by the following steps:

[0131] In step S410, the second sample noisy image and the second text description information are input into the initial image generation model for denoising, so as to obtain a second predicted denoised image corresponding to the second sample noisy image.

[0132] In step S420, a second target denoised image corresponding to the second sample denoised image is obtained, and a second loss value is determined according to a difference between the second target denoised image and the second predicted denoised image.

[0133] In step S430, the initial image generation model is iteratively trained according to the second loss value to obtain a trained initial image generation model, and the trained initial image generation model is used as the target image generation model.

[0134] The second predicted denoised image refers to the predicted denoised image corresponding to the second sample denoised image, which is specifically generated by the initial image generation model. The second target denoised image refers to the actual denoised image corresponding to the second sample denoised image, which is specifically processed by the second image denoising model.

[0135] The second loss value refers to a loss value obtained based on a difference between the second target denoised image and the second predicted denoised image, such as a mean square error.

[0136] Exemplarily, the server first inputs the second sample noisy image and the second text description information into the initial image generation model for denoising, and obtains a second predicted denoised image corresponding to the second sample noisy image; then, through the second image denoising model, a second target denoised image corresponding to the second sample noisy image is obtained; according to the difference between the second target denoised image and the second predicted denoised image, and in combination with the mean square error loss function, a second loss value is calculated; according to the second loss value, the model parameters of the initial image generation model are adjusted to obtain an adjusted initial image generation model, and the above process is repeated until the second loss value of the adjusted initial image generation model is less than a second preset threshold, then the training is stopped, and the adjusted initial image generation model is used as the trained initial image generation model, and finally the trained initial image generation model is used as the target image generation model.

[0137] For example, the second loss value is calculated by the following formula:

[0138] Loss2 = MSE_Loss (UNet (x_t) - x_{t-1}); (Formula 10)

[0139] Among them, Loss2 represents the second loss value, MSE_Loss represents the mean square error function; x_t and x_{t-1} represent the second sample denoised image and the second target denoised image, respectively, specifically representing the images denoised to the t-th and (t-1)-th steps obtained by the delayed noise scheduler; UNet(x_t) represents the second predicted denoised image, specifically representing the predicted denoised image obtained after the image x_t denoised to the t-th step obtained by the delayed noise scheduler is input into UNet (i.e., the initial image generation model).

[0140] It should be noted that through the above-mentioned second-stage training method, the model can derive x_{t-1} from (x_t) (such as x_1100), derive x_{t-2} from x_{t-1}, and so on, until x_0 is derived.

[0141] In this embodiment, in the process of training the target image generation model, the second sample noisy image, the second text description information and the second target denoised image are comprehensively considered, so that the model can learn to improve image quality in the high-resolution training stage, thereby improving the image quality corresponding to the image generated by the model.

[0142] In an exemplary embodiment, the above step S420, obtaining the second target denoised image corresponding to the second sample noisy image, specifically includes the following contents: obtaining the second denoising step number corresponding to the second sample noisy image; performing a decrementing process on the second denoising step number to obtain the processed second denoising step number; inputting the processed second denoising step number and the second sample image into the second image denoising model for denoising process to obtain the second processed image, and using the second processed image as the second target denoised image; the second processed image is used to represent the image obtained after denoising the second sample image to the processed second denoising step number.

[0143] The second number of noise adding steps refers to the number of noise adding steps corresponding to the second sample noise adding image. The decrement processing refers to the subtraction operation, for example, if the second number of noise adding steps is 20, then the second number of noise adding steps after processing is 19.

[0144] The second processed image (ie, the first target denoised image) refers to an image obtained after the second sample image is denoised by the second image denoising model to a second number of denoising steps.

[0145] Exemplarily, the server obtains the second noisy step number corresponding to the second sample noisy image, and decrements the second noisy step number to obtain the processed second noisy step number, and finally inputs the processed second noisy step number and the second sample image into the second image noisy model for noise processing to obtain the image obtained after noisy the second sample image to the processed second noisy step number as the second target denoised image corresponding to the second sample noisy image.

[0146] For example, assuming that the first denoising step number corresponding to the second sample noisy image is 19, that is, the second sample noisy image is an image noisy to the 19th step, the server substitutes t=18 and the second sample image x0 into formulas (5) to (8) to obtain the image noisy to the 18th step as the second target denoised image corresponding to the image noisy to the 19th step.

[0147] It should be noted that, in the second stage training process, the image of the previous step before the current step is used as the second target denoised image of the image of the current step before the current step.

[0148] In this embodiment, the second sample image with the second number of processed noisy steps corresponding to the second sample noisy image and the second sample image are input into the second image noisy model for noise processing to obtain a second processed image, and the second processed image is used as the second target denoised image of the second sample noisy image. This allows the target image generation model trained based on the second sample noisy image, the second text description information and the second target denoised image to gradually derive a high-resolution image corresponding to the text description information from the pure noise image, thereby improving the image quality corresponding to the image output by the image generation model.

[0149] In an exemplary embodiment, if Figure 5 As shown, the above step S130, obtaining the second sample image with the second resolution, can be specifically implemented by the following steps:

[0150] In step S510, an original image of a second resolution and image scores of the original image under a plurality of preset indicators are obtained.

[0151] In step S520, the image scores of the original image under multiple preset indicators are fused to obtain a fusion score of the original image.

[0152] In step S530, candidate images whose fusion scores satisfy a first preset score condition are screened out from the original images.

[0153] In step S540, the annotation scores of the candidate images are obtained, and a target image whose annotation score meets a second preset score condition is screened out from the candidate images, and the target image is used as the second sample image.

[0154] Among them, the preset indicators refer to preset scoring dimensions, such as OCR (Optical Character Recognition) detection results, watermark detection results, aesthetic scores, image-text relevance scores, clarity scores, etc.

[0155] The fusion score of the original image refers to the total score obtained after fusion processing of the image scores of the original image under multiple preset indicators.

[0156] Among them, the first preset score condition is used to represent the first preset score threshold; the candidate image refers to an image in the original image whose fusion score meets the first preset score condition, specifically refers to an image in the original image whose fusion score is greater than the first preset score threshold.

[0157] The annotation score of the candidate image refers to the score obtained through manual annotation.

[0158] Among them, the second preset score condition is used to represent the second preset score threshold; the target image refers to the image in the candidate image whose annotation score meets the second preset score condition, specifically refers to the image in the candidate image whose annotation score is greater than the second preset score threshold.

[0159] The second sample image refers to a target image selected from the candidate images and having a labeling score that satisfies a second preset score condition.

[0160] Exemplarily, the server obtains an original image of a second resolution, and calculates the image score of the original image under multiple preset indicators through a scoring model of multiple preset indicators; then, a weighted sum process is performed on the image score of the original image under multiple preset indicators to obtain a fusion score of the original image; then, from the original images, images whose fusion scores meet the first preset score condition are screened out as candidate images; then, the annotation scores of the candidate images are obtained, and images whose annotation scores meet the second preset score condition are screened out from the candidate images as target images, and finally the target image is used as the second sample image.

[0161] For example, based on funnels such as image resolution, OCR detection results, watermark detection results, aesthetic scores, image-text relevance scores, and clarity scores, a batch of high-quality image and text datasets can be screened out; on the basis of the screened image and text datasets, the image and text datasets are further manually annotated, and only data that the annotators believe has photographic-level aesthetics is retained, ultimately forming a high-quality image and text dataset for the second stage of model training.

[0162] In this embodiment, an original image of a second resolution and an image score of the original image under a plurality of preset indicators are first obtained, and the image scores of the original image under the plurality of preset indicators are fused to obtain a fusion score of the original image, and then candidate images whose fusion scores satisfy a first preset score condition are screened out from the original image, and finally the annotation scores of the candidate images are obtained, and target images whose annotation scores satisfy a second preset score condition are screened out from the candidate images, thereby obtaining a second sample image; in this way, obtaining the second sample image after layers of screening is beneficial to improving the image quality of the second sample image, thereby improving the image quality corresponding to the image generated by the target image generation model trained based on the second sample image.

[0163] Figure 6 is a flowchart of an image generating method according to an exemplary embodiment. Figure 6 As shown, the image generation method is used in a server and specifically includes the following steps:

[0164] In step S610, the text description information to be processed and the target noisy image are obtained; the target noisy image is used to represent the image obtained after the second sample image is noisy to the total number of noisy steps of the second image noisy model.

[0165] In step S620, the text description information to be processed and the target noisy image are input into the target image generation model for multiple denoising processes to obtain a second resolution image corresponding to the text description information to be processed; the target image generation model is trained by the image generation model training method.

[0166] The text description information to be processed refers to the description information of the image to be generated, and may be text description information input or selected by a user.

[0167] The target noisy image refers to an image obtained by adding noise to the second sample image to the total number of noisy steps of the second image noisy model, for example, an image obtained by adding noise to the second sample image to the 1100th step.

[0168] The second resolution image refers to a high-resolution image corresponding to the text description information to be processed, such as a 1024×1024 image.

[0169] Among them, multiple denoising processes mean that the target image generation model needs to undergo multiple processes before it can output the second resolution image corresponding to the text description information to be processed. For example, input the text description information to be processed and the image denoised to the 1100th step to obtain the image denoised to the 1099th step; input the text description information to be processed and the image denoised to the 1099th step to obtain the image denoised to the 1098th step, and so on, and finally output the image denoised to the 0th step, that is, the second resolution image corresponding to the text description information to be processed.

[0170] Exemplarily, the server obtains text description information to be processed and a second sample image, and inputs the second sample image and the total number of noisy steps of the second image noisy model into the second image noisy model for noisy processing, and obtains the image obtained after noisy the second sample image to the total number of noisy steps of the second image noisy model, as the target noisy image, and finally inputs the text description information to be processed and the target noisy image into the target image generation model for multiple denoising processes to obtain a second resolution image corresponding to the text description information to be processed.

[0171] In the above-mentioned image generation method, the text description information to be processed and the target noisy image are first obtained; the target noisy image is used to represent the image obtained after the second sample image is noisy to the total number of noisy steps of the second image noisy model, and then the text description information to be processed and the target noisy image are input into the target image generation model for multiple denoising processes to obtain a second resolution image corresponding to the text description information to be processed; in this way, by using the trained target image generation model and combining the text description information to be processed and the target noisy image, an image with high text-image consistency and image quality can be obtained, thereby improving the image generation accuracy.

[0172] In an exemplary embodiment, the above-mentioned step S620, inputting the text description information to be processed and the target noisy image into the target image generation model for multiple denoising processes to obtain a second resolution image corresponding to the text description information to be processed, specifically includes the following contents: inputting the text description information to be processed and the target noisy image into the target image generation model for denoising process to obtain a processed image corresponding to the text description information to be processed; using the processed image corresponding to the text description information to be processed as the target noisy image, and jumping to the step of inputting the text description information to be processed and the target noisy image into the target image generation model for denoising process to obtain the processed image corresponding to the text description information to be processed, until the cumulative number of denoising processes of the target image generation model reaches the total number of denoising steps of the second image denoising model; using the processed image obtained after the target image generation model performs the last denoising process as the second resolution image.

[0173] Among them, each time the denoising process is performed, a processed image corresponding to the text description information to be processed can be obtained; therefore, each time a processed image corresponding to the text description information to be processed is obtained, the cumulative number of denoising processes of the target image generation model is +1.

[0174] Assuming that the total number of denoising steps of the second image denoising model is n, when the cumulative number of denoising processes of the target image generation model reaches n, the input of the text description information to be processed and the target denoised image into the target image generation model is stopped.

[0175] Exemplarily, the server inputs the text description information to be processed and the target noisy image into the target image generation model for denoising, and obtains a processed image corresponding to the text description information to be processed; the processed image corresponding to the text description information to be processed is used as the target noisy image, and the above process is repeated until the cumulative number of denoising processes of the target image generation model is equal to the total number of denoising steps of the second image denoising model; when the cumulative number of denoising processes of the target image generation model is equal to the total number of denoising steps of the second image denoising model, the processed image obtained after the last denoising process of the target image generation model is used as the second resolution image.

[0176] For example, assuming that the total number of denoising steps of the second image denoising model is 1100, when the cumulative number of denoising processes of the target image generation model reaches 1100, the text description information to be processed and the target denoised image are stopped from being input into the target image generation model, and the processed image obtained after the 1100th denoising process of the target image generation model is used as the second resolution image. For example, the text description information to be processed and the image denoised to the 1100th step are first input into the target image generation model to obtain the image denoised to the 1099th step; then the text description information to be processed and the image denoised to the 1099th step are input into the target image generation model to obtain the image denoised to the 1098th step, and so on, and finally the image denoised to the 0th step is output, that is, the second resolution image corresponding to the text description information to be processed.

[0177] In this embodiment, the text description information to be processed and the target noisy image are input into the target image generation model for denoising processing to obtain a processed image corresponding to the text description information to be processed; the processed image corresponding to the text description information to be processed is used as the target noisy image, and the above process is repeated until the cumulative denoising processing times of the target image generation model reach the total denoising steps of the second image denoising model; in this way, after multiple denoising processing, it is beneficial to improve the image-text consistency and image quality of the obtained image, thereby improving the image generation accuracy.

[0178] Figure 7 is a flowchart of another image generation method according to an exemplary embodiment. Figure 7 As shown, the image generation method is used in a server and specifically includes the following steps:

[0179] In step S701, a first sample image of a first resolution, first text description information corresponding to the first sample image, and a first sample noisy image are obtained; the first sample noisy image is obtained by performing random noise processing on the first sample image using a first image noisy model.

[0180] In step S702, the first sample noisy image and the first text description information are input into the image generation model to be trained for denoising, so as to obtain a first predicted denoised image corresponding to the first sample noisy image.

[0181] In step S703, a first denoising step number corresponding to the first sample noisy image is obtained; the first denoising step number is decremented to obtain a processed first denoising step number; the processed first denoising step number and the first sample image are input into a first image denoising model for denoising to obtain a first processed image, and the first processed image is used as a first target denoising image corresponding to the first sample noisy image; the first processed image is used to represent an image obtained by denoising the first sample image to the processed first denoising step number.

[0182] In step S704, a first loss value is determined according to the difference between the first target denoised image and the first predicted denoised image; according to the first loss value, the image generation model to be trained is iteratively trained to obtain a trained image generation model, and the trained image generation model is used as the initial image generation model.

[0183] In step S705, the original image of the second resolution and the image scores of the original image under multiple preset indicators are obtained; the image scores of the original image under multiple preset indicators are fused to obtain the fusion scores of the original image; the second resolution is higher than the first resolution.

[0184] In step S706, candidate images whose fusion scores satisfy the first preset score condition are screened out from the original images; the annotation scores of the candidate images are obtained, and target images whose annotation scores satisfy the second preset score condition are screened out from the candidate images, and the target images are used as the second sample images of the second resolution.

[0185] In step S707, the second text description information and the second sample noisy image corresponding to the second sample image are obtained; the second sample noisy image is obtained by randomly noisy processing the second sample image through a second image noisy model; the total noisy step number of the second image noisy model is greater than the total noisy step number of the first image noisy model.

[0186] In step S708, the second sample noisy image and the second text description information are input into the initial image generation model for denoising, so as to obtain a second predicted denoised image corresponding to the second sample noisy image.

[0187] In step S709, the second denoising step number corresponding to the second sample noisy image is obtained; the second denoising step number is decremented to obtain the processed second denoising step number; the processed second denoising step number and the second sample image are input into the second image denoising model for denoising to obtain a second processed image, and the second processed image is used as the second target denoised image corresponding to the second sample noisy image; the second processed image is used to represent the image obtained after denoising the second sample image to the processed second denoising step number.

[0188] In step S710, a second loss value is determined based on the difference between the second target denoised image and the second predicted denoised image; based on the second loss value, the initial image generation model is iteratively trained to obtain a trained initial image generation model, and the trained initial image generation model is used as the target image generation model.

[0189] In step S711, the text description information to be processed and the target noisy image are obtained; the target noisy image is used to represent the image obtained after the second sample image is noisy to the total number of noisy steps of the second image noisy model.

[0190] In step S712, the text description information to be processed and the target noisy image are input into the target image generation model for denoising to obtain a processed image corresponding to the text description information to be processed.

[0191] In step S713, the processed image corresponding to the text description information to be processed is used as the target denoised image, and the process jumps to step S712 until the cumulative denoising processing times of the target image generation model reaches the total denoising steps of the second image denoising model.

[0192] In step S714, the processed image obtained after the target image generation model performs the last denoising process is used as the second resolution image corresponding to the text description information to be processed.

[0193] In the above-mentioned image generation method, a two-stage training scheme is designed when performing model training. In the first stage, the image generation model to be trained is iteratively trained according to the first text description information and the first sample noisy image corresponding to the first sample image of the first resolution, so as to learn the knowledge of the text image in the low-resolution training stage, thereby improving the image-text consistency of the image generated by the model; in the second stage, the initial image generation model trained in the first stage is iteratively trained according to the second text description information and the second sample noisy image corresponding to the second sample image of the second resolution, so as to learn to improve the image quality in the high-resolution training stage, thereby improving the image quality of the image generated by the model. The image quality corresponding to the image generated by the model; moreover, in the second training stage, introducing a second image denoising model with a total denoising step number greater than the first image denoising model used in the first training stage is conducive to extending the denoising time of the model, making the initial states of the model training and reasoning consistent, thereby improving the image-text relevance and image quality of the images generated by the trained model, thereby improving the image generation accuracy, and at the same time avoiding incomplete denoising of the model at high resolution, resulting in inconsistent initial states during model training and reasoning, thereby resulting in decreased image-text relevance and image quality of the images generated by the trained model, thereby resulting in the defect of low image generation accuracy.

[0194] In order to more clearly illustrate the image generation model training method provided by the embodiment of the present disclosure, the image generation model training method is specifically described below with a specific embodiment. In an exemplary embodiment, the present disclosure also provides an extended model training method for high-resolution image generation to solve the problem of decreased image quality and image-text relevance when the diffusion model generates images at high resolution, specifically by redesigning a noise scheduler adapted to high-resolution generation, improving the problem of inconsistent initial states of model training and reasoning at high resolution, thereby improving the model's generated image quality and image-text relevance at 1024×1024 resolution; for example, a two-stage training scheme of low-resolution knowledge learning and high-resolution quality improvement is designed, and a "delayed noise scheduler" is introduced during high-resolution training to extend the model's noise addition time and solve the problem of incomplete noise addition of the model at high resolution.

[0195] For example, in order to improve the image-text relevance and image quality of the images generated by the diffusion model, a two-stage training scheme is designed. The first stage is a low-resolution training stage for knowledge learning, and the second stage is a high-resolution training stage for improving the output quality. In the knowledge learning stage of the model, the image data is first described in detail to improve the image-text alignment ability of the model; the CogVLM multimodal large model is selected to generate image descriptions, and an image description with an average number of characters of about 99 is obtained; the training of this stage is carried out at a low resolution of 512×512, using the classic scaled linear noise scheduler; the formula of the scaled linear noise scheduler is shown in formulas (1) to (4). In the second stage of high-resolution training, a delayed noise scheduler is designed. The delayed noise scheduler is extended to 1100 steps based on the original 1000-step noise addition time, and the calculation method of the noise addition intensity is adjusted. Its formula is shown in formulas (5) to (8). Under the action of the delayed noise scheduler, the 1024×1024 high-resolution image will be closer to a pure noise image when it is denoised to the 1100th step, thus solving the problem of incomplete denoising when training at high resolution with the traditional scaled linear noise scheduler. At the same time, based on funnels such as image resolution, OCR detection results, watermark detection results, aesthetic scores, image-text relevance scores, and clarity scores, a batch of high-quality image-text datasets were screened. Based on the screened data, the data was further manually annotated, and only the data that the annotators believed had photographic aesthetics was retained, which ultimately constituted a high-quality dataset for the second stage of model training. The model was trained under this image-text dataset and the delayed noise scheduler, and the image-text relevance and image quality of the high-resolution images generated by the final model were significantly improved.

[0196] The above extended model training method for high-resolution image generation significantly improves the problem of image quality degradation in the traditional text-image diffusion model when generating high-resolution images; by optimizing the algorithm and model structure, the overall quality of the generated images and their relevance to the text description are effectively improved. In addition, this method can also significantly improve user participation and willingness to use, further promoting the improvement of creation efficiency and creative expression.

[0197] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0198] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can refer to each other, and each embodiment focuses on the differences from other embodiments. For related points, please refer to the description of other method embodiments.

[0199] Based on the same inventive concept, the embodiment of the present disclosure also provides an image generation model training device for implementing the above-mentioned image generation model training method.

[0200] Figure 8 is a block diagram of an image generation model training device according to an exemplary embodiment. Figure 8 The device includes a first acquisition unit 810, a first training unit 820, a second acquisition unit 830 and a second training unit 840.

[0201] The first acquisition unit 810 is configured to execute acquisition of a first sample image of a first resolution, first text description information corresponding to the first sample image, and a first sample noisy image; the first sample noisy image is obtained by performing random noise processing on the first sample image by a first image noisy model.

[0202] The first training unit 820 is configured to perform iterative training on the image generation model to be trained according to the first sample noisy image and the first text description information to obtain an initial image generation model.

[0203] The second acquisition unit 830 is configured to execute acquisition of a second sample image of a second resolution, second text description information corresponding to the second sample image, and a second sample noisy image; the second resolution is higher than the first resolution; the second sample noisy image is obtained by randomly noisy processing the second sample image by a second image noisy model; the total number of noisy steps of the second image noisy model is greater than the total number of noisy steps of the first image noisy model.

[0204] The second training unit 840 is configured to perform iterative training on the initial image generation model according to the second sample noisy image and the second text description information to obtain a target image generation model.

[0205] In an exemplary embodiment, the first training unit 820 is further configured to perform denoising processing on the first sample noisy image and the first text description information, input the first sample noisy image and the first text description information into the image generation model to be trained, and obtain a first predicted denoised image corresponding to the first sample noisy image; obtain a first target denoised image corresponding to the first sample noisy image, and determine a first loss value according to the difference between the first target denoised image and the first predicted denoised image; according to the first loss value, iteratively train the image generation model to be trained to obtain a trained image generation model, and use the trained image generation model as the initial image generation model.

[0206] In an exemplary embodiment, the first training unit 820 is further configured to execute the steps of obtaining a first denoising step corresponding to a first sample denoised image; performing a decrement process on the first denoising step to obtain a first denoising step after processing; inputting the first denoising step after processing and the first sample image into a first image denoising model for denoising to obtain a first processed image, and using the first processed image as a first target denoised image; the first processed image is used to represent an image obtained after denoising the first sample image to the first denoising step after processing.

[0207] In an exemplary embodiment, the second training unit 840 is further configured to perform denoising processing on the second sample noisy image and the second text description information, input the second sample noisy image and the second text description information into the initial image generation model to obtain a second predicted denoised image corresponding to the second sample noisy image; obtain a second target denoised image corresponding to the second sample noisy image, and determine a second loss value based on the difference between the second target denoised image and the second predicted denoised image; iteratively train the initial image generation model based on the second loss value to obtain a trained initial image generation model, and use the trained initial image generation model as the target image generation model.

[0208] In an exemplary embodiment, the second training unit 840 is further configured to execute the steps of obtaining a second denoising step corresponding to a second sample denoised image; performing a decrement process on the second denoising step to obtain a processed second denoising step; inputting the processed second denoising step and the second sample image into a second image denoising model for denoising to obtain a second processed image, and using the second processed image as a second target denoised image; the second processed image is used to represent an image obtained after denoising the second sample image to the processed second denoising step.

[0209] In an exemplary embodiment, the second acquisition unit 830 is further configured to execute acquisition of an original image of a second resolution, and an image score of the original image under a plurality of preset indicators; perform fusion processing on the image score of the original image under a plurality of preset indicators to obtain a fusion score of the original image; screen out candidate images whose fusion scores satisfy a first preset score condition from the original image; acquire annotation scores of the candidate images, screen out target images whose annotation scores satisfy a second preset score condition from the candidate images, and use the target images as the second sample image.

[0210] Fig. 9 is a block diagram of an image generating device according to an exemplary embodiment. Fig. 9 , the device includes a data acquisition unit 910 and an image generation unit 920.

[0211] The data acquisition unit 910 is configured to execute acquisition of the text description information to be processed and the target noisy image; the target noisy image is used to represent the image obtained after the second sample image is noisy to the total number of noisy steps of the second image noisy model.

[0212] The image generation unit 920 is configured to execute multiple denoising processes on the text description information to be processed and the target noisy image, and obtain a second resolution image corresponding to the text description information to be processed; the target image generation model is trained by the internal image generation model training method in any one of the embodiments of the first aspect.

[0213] In an exemplary embodiment, the image generation unit 920 is further configured to execute the step of inputting the text description information to be processed and the target noisy image into the target image generation model for denoising processing to obtain a processed image corresponding to the text description information to be processed; using the processed image corresponding to the text description information to be processed as the target noisy image, and jumping to the step of inputting the text description information to be processed and the target noisy image into the target image generation model for denoising processing to obtain a processed image corresponding to the text description information to be processed, until the cumulative number of denoising processing times of the target image generation model reaches the total number of denoising steps of the second image denoising model; and using the processed image obtained after the target image generation model performs the last denoising processing as the second resolution image.

[0214] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0215] Each module in the above-mentioned image generation model training device or image generation device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0216] Fig.10 1 is a block diagram of an electronic device 1000 for implementing an image generation model training method or an image generation method according to an exemplary embodiment. For example, the electronic device 1000 may be a server. Fig.10 , the electronic device 1000 includes a processing component 1020, which further includes one or more processors, and a memory resource represented by a memory 1022 for storing instructions executable by the processing component 1020, such as an application. The application stored in the memory 1022 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1020 is configured to execute the instructions to perform the above method.

[0217] The electronic device 1000 may further include a power supply component 1024 configured to perform power management of the electronic device 1000, a wired or wireless network interface 1026 configured to connect the electronic device 1000 to a network, and an input / output (I / O) interface 1028. The electronic device 1000 may operate based on an operating system stored in the memory 1022, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

[0218] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1022 including instructions, and the instructions can be executed by a processor of the electronic device 1000 to complete the above method. The storage medium can be a computer-readable storage medium, for example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0219] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions. The instructions can be executed by a processor of the electronic device 1000 to complete the above method.

[0220] It should be noted that the above-mentioned devices, electronic devices, computer-readable storage media, computer program products, etc. may also include other implementation methods according to the description of the method embodiments. The specific implementation methods can refer to the description of the relevant method embodiments, which will not be described one by one here.

[0221] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0222] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training an image generation model, characterized in that: include: Acquire a first sample image of a first resolution, as well as first text description information and a first sample noisy image corresponding to the first sample image; The first sample noisy image is obtained by performing random noise processing on the first sample image using a first image noisy model; Iteratively training the image generation model to be trained according to the first sample noisy image and the first text description information to obtain an initial image generation model; Acquire a second sample image of a second resolution, and second text description information and a second sample noisy image corresponding to the second sample image; the second resolution is higher than the first resolution; the second sample noisy image is obtained by performing random noisy processing on the second sample image by a second image noisy model; the total number of noisy steps of the second image noisy model is greater than the total number of noisy steps of the first image noisy model; The initial image generation model is iteratively trained according to the second sample noisy image and the second text description information to obtain a target image generation model.

2. The method according to claim 1, characterized in that The iterative training of the image generation model to be trained according to the first sample noisy image and the first text description information to obtain an initial image generation model includes: Inputting the first sample noisy image and the first text description information into the image generation model to be trained for denoising, thereby obtaining a first predicted denoised image corresponding to the first sample noisy image; Acquire a first target denoised image corresponding to the first sample denoised image, and determine a first loss value according to a difference between the first target denoised image and the first predicted denoised image; According to the first loss value, the image generation model to be trained is iteratively trained to obtain a trained image generation model, and the trained image generation model is used as the initial image generation model.

3. The method according to claim 2, characterized in that The obtaining of a first target denoised image corresponding to the first sample denoised image includes: Obtaining a first number of noise adding steps corresponding to the first sample noise adding image; Performing a decrement process on the first noise adding step number to obtain a processed first noise adding step number; The first processed denoising step number and the first sample image are input into the first image denoising model for denoising processing to obtain a first processed image, and the first processed image is used as the first target denoised image; the first processed image is used to represent the image obtained after denoising the first sample image to the first processed denoising step number.

4. The method according to claim 1, characterized in that: The iterative training of the initial image generation model according to the second sample noisy image and the second text description information to obtain a target image generation model includes: Inputting the second sample noisy image and the second text description information into the initial image generation model for denoising, to obtain a second predicted denoised image corresponding to the second sample noisy image; Acquire a second target denoised image corresponding to the second sample denoised image, and determine a second loss value according to a difference between the second target denoised image and the second predicted denoised image; According to the second loss value, the initial image generation model is iteratively trained to obtain a trained initial image generation model, and the trained initial image generation model is used as the target image generation model.

5. The method according to claim 4, characterized in that The obtaining of a second target denoised image corresponding to the second sample denoised image includes: Obtaining a second number of noise adding steps corresponding to the second sample noise adding image; Performing a decrement process on the second noise adding step number to obtain a processed second noise adding step number; The processed second denoising step number and the second sample image are input into the second image denoising model for denoising processing to obtain a second processed image, and the second processed image is used as the second target denoised image; the second processed image is used to represent the image obtained after denoising the second sample image to the processed second denoising step number.

6. The method according to any one of claims 1 to 5, characterized in that: The step of acquiring a second sample image with a second resolution includes: Acquire an original image at a second resolution, and image scores of the original image under a plurality of preset indicators; Performing fusion processing on the image scores of the original image under multiple preset indicators to obtain a fusion score of the original image; Filtering out candidate images whose fusion scores satisfy a first preset score condition from the original images; The annotation scores of the candidate images are obtained, a target image whose annotation score meets a second preset score condition is screened out from the candidate images, and the target image is used as the second sample image.

7. An image generation method, characterized in that: include: Obtain the text description information to be processed and the target noisy image; The target noisy image is used to represent an image obtained by adding noise to the second sample image to the total number of noisy steps of the second image noisy model; The text description information to be processed and the target noisy image are input into a target image generation model for multiple denoising processes to obtain a second resolution image corresponding to the text description information to be processed; the target image generation model is trained by the image generation model training method described in any one of claims 1 to 6.

8. The method according to claim 7, characterized in that The step of inputting the to-be-processed text description information and the target noisy image into a target image generation model for multiple denoising processes to obtain a second resolution image corresponding to the to-be-processed text description information includes: Input the to-be-processed text description information and the target noisy image into a target image generation model for denoising, thereby obtaining a processed image corresponding to the to-be-processed text description information; The processed image corresponding to the text description information to be processed is used as the target noisy image, and the process jumps to the step of inputting the text description information to be processed and the target noisy image into the target image generation model for denoising to obtain the processed image corresponding to the text description information to be processed, until the cumulative number of denoising processes of the target image generation model reaches the total number of denoising steps of the second image denoising model; The processed image obtained after the target image generation model performs the last denoising process is used as the second resolution image.

9. An image generation model training device, characterized in that: include: A first acquisition unit is configured to acquire a first sample image of a first resolution, first text description information corresponding to the first sample image, and a first sample noisy image; the first sample noisy image is obtained by performing random noise processing on the first sample image by a first image noisy model; A first training unit is configured to perform iterative training on the image generation model to be trained according to the first sample noisy image and the first text description information to obtain an initial image generation model; A second acquisition unit is configured to acquire a second sample image of a second resolution, second text description information corresponding to the second sample image, and a second sample noisy image; the second resolution is higher than the first resolution; the second sample noisy image is obtained by randomly noisy processing the second sample image by a second image noisy model; the total number of noisy steps of the second image noisy model is greater than the total number of noisy steps of the first image noisy model; The second training unit is configured to perform iterative training on the initial image generation model according to the second sample noisy image and the second text description information to obtain a target image generation model.

10. An image generating device, characterized in that: include: A data acquisition unit, configured to acquire text description information to be processed and a target noisy image; The target noisy image is used to represent an image obtained by adding noise to the second sample image to the total number of noisy steps of the second image noisy model; An image generation unit is configured to execute multiple denoising processes by inputting the text description information to be processed and the target noisy image into a target image generation model to obtain a second resolution image corresponding to the text description information to be processed; the target image generation model is trained by the image generation model training method described in any one of claims 1 to 6.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image generation model training method as described in any one of claims 1 to 6, or the image generation method as described in any one of claims 7 to 8.

12. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image generation model training method as described in any one of claims 1 to 6, or the image generation method as described in any one of claims 7 to 8.

13. A computer program product, comprising instructions, characterized in that: When the instruction is executed by a processor of an electronic device, the electronic device is enabled to execute the image generation model training method as described in any one of claims 1 to 6, or the image generation method as described in any one of claims 7 to 8.

Images