Image generation method and device, electronic equipment and storage medium
By cropping noise images to align different resolutions in distillation training between literary and biographical graphics models, the problem of resolution alignment in the prior art is solved, and the quality and accuracy of image generation are improved.
Patent Information
- Application Number
- CN202411745851.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-03-18
AI Technical Summary
When distillation training between literary and biographical models of different architectures, it is difficult to effectively align noises with different resolutions, resulting in poor distillation effect and affecting the accuracy and quality of generated images.
By cropping noisy images, the resolution alignment problem between models at different levels is solved. The specific method is to crop the input noise image consistent with the first model resolution from the input noise image of the second model as the input noise image of the first model; and crop the output noise image consistent with the first model resolution from the output noise image of the second model as the supervised noise image of the first model.
By cropping noisy images, resolution alignment between different literary and artistic graphics models is achieved, the distillation effect is improved, and the quality and accuracy of generated images are improved.
Smart Images
Figure CN119648566A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to an image generation method, a training method for a deep learning model, a device, an electronic device, a storage medium, and a computer program product. Background Art
[0002] In the current field of text-based graph models, models with similar architectures are usually used for distillation training. When distilling between text-based graph models with different architectures, due to the different optimal resolutions supported, noises of different resolutions need to be aligned. Existing resolution alignment methods lead to poor distillation results, which seriously affects the accuracy and quality of generated images. Summary of the invention
[0003] In view of this, the present disclosure provides an image generation method, a deep learning model training method, an apparatus, an electronic device, a storage medium, and a computer program product.
[0004] One aspect of the present disclosure provides an image generation method, including: obtaining text information and a noise image to be processed; using a first model to denoise the noise image to be processed according to the text information to obtain a target image corresponding to the text information; wherein the first model is obtained by distillation using a second model, and during the distillation process, the input noise image and the supervised noise image of the first model are respectively cropped from the input noise image and the output noise image of the second model.
[0005] According to an embodiment of the present disclosure, a first model is used to process images of a first type, and a second model is used to process images of a second type, wherein the second type differs from the first type in at least resolution; during the distillation process, the input noise image and the supervisory noise image of the first model are obtained by the following operations: cropping the input noise image of the first type from the input noise image of the second type of the second model as the input noise image of the first model; cropping the output noise image of the first type from the output noise image of the second type of the second model as the supervisory noise image of the first model.
[0006] According to an embodiment of the present disclosure, a first type of output noise image is cropped from a second type of output noise image of a second model as a supervisory noise image of the first model, including: obtaining a cropping position of the first type of input noise image in the second type of input noise image; determining a target position in the second type of output noise image corresponding to the cropping position; and cropping the first type of output noise image at the target position as the supervisory noise image of the first model.
[0007] According to an embodiment of the present disclosure, the distillation process includes: determining the output noise image of the first model based on the input noise image of the first model; obtaining the distillation loss based on the difference between the supervised noise image and the output noise image of the first model, and obtaining the task loss based on the difference between the input noise image of the first model and the output noise image of the first model; adjusting the parameters of the first model based on the overall loss determined by the distillation loss and the task loss.
[0008] According to an embodiment of the present disclosure, an output noise image of the first model is determined based on an input noise image of the first model, including: adding noise to a first sample image using the input noise image of the first model to obtain an intermediate noise image; and denoising the intermediate noise image based on the first sample text to obtain the output noise image of the first model.
[0009] Another aspect of the present disclosure also provides a training method for a deep learning model, including: inputting a first sample text, a first sample image, and a first input noise image into a first model to obtain a first output noise image; inputting a second sample text, a second sample image, and a second input noise image into a second model to obtain a second output noise image; wherein the first input noise image is cropped from the second input noise image; determining a distillation loss based on the difference between the supervisory noise image and the first output noise image; and determining a task loss based on the difference between the first input noise image and the first output noise image; wherein the supervisory noise image is cropped from the second output noise image; and adjusting the parameters of the first model based on the distillation loss and the task loss.
[0010] According to an embodiment of the present disclosure, the method further includes: transforming the original image into a first sample image of the first type and a second sample image of the second type.
[0011] According to an embodiment of the present disclosure, the first model is used to process text of a first language type, and the second model is used to process text of a second language type; the method further includes: determining a first sample text of the first language type and a second sample text of the second language type, wherein the second sample text is semantically consistent with the first sample text.
[0012] Another aspect of the present disclosure also provides an image generating device, including: a receiving module, used to obtain text information and a noise image to be processed; and an image generating module, used to use a first model to denoise the noise image to be processed according to the text information, to obtain a target image corresponding to the text information; wherein the first model is obtained by distillation using the second model, and during the distillation process, the input noise image and the supervised noise image of the first model are respectively cropped from the input noise image and the output noise image of the second model.
[0013] Another aspect of the present disclosure also provides a training device for a deep learning model, including: a first noise image generation module, used to input a first sample text, a first sample image, and a first input noise image into a first model to obtain a first output noise image; a second noise image generation module, used to input a second sample text, a second sample image, and a second input noise image into a second model to obtain a second output noise image; wherein the first input noise image is cropped from the second input noise image; a distillation loss determination module, used to determine the distillation loss based on the difference between the supervisory noise image and the first output noise image; and a task loss determination module, used to determine the task loss based on the difference between the first input noise image and the first output noise image; wherein the supervisory noise image is cropped from the second output noise image; a parameter adjustment module, used to adjust the parameters of the first model based on the distillation loss and the task loss.
[0014] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above method.
[0015] Another aspect of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to perform the above method.
[0016] Another aspect of the present disclosure further provides a computer program product, including a computer program, which implements the above method when executed by a processor.
[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:
[0019] Figure 1 The flowchart of the image generation method according to the embodiment of the present disclosure is schematically shown;
[0020] Figure 2 A schematic diagram of image cropping according to an embodiment of the present disclosure is schematically shown;
[0021] Figure 3 A schematic diagram schematically shows an image generation method according to an embodiment of the present disclosure;
[0022] Figure 4 A flowchart of a deep learning model training method according to an embodiment of the present disclosure is schematically shown;
[0023] Figure 5 A schematic diagram schematically illustrates a method for training a deep learning model according to an embodiment of the present disclosure;
[0024] Figure 6 The structure block diagram of the image generating device according to the embodiment of the present disclosure is schematically shown;
[0025] Figure 7 A structural block diagram schematically shows a deep learning model training device according to an embodiment of the present disclosure; and
[0026] Figure 8 A schematic block diagram of an electronic device that can be used to implement the image generation method and the deep learning model training method of the embodiments of the present disclosure is schematically shown. DETAILED DESCRIPTION
[0027] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to facilitate understanding, and they should be considered as merely exemplary. Therefore, it should be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Likewise, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0028] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of the data involved (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.
[0029] When selecting different architectures of Wensheng graph models for distillation, you can choose the SDXL model (StableDiffusion Extra Large, XL-level stable diffusion model) as the teacher model to guide the learning of the SD (Stable Diffusion Model) 1.5-level student model, thereby optimizing the performance of the student model. Among them, the best resolution of the SDXL model can be 1024*1024, and the best resolution of SD1.5 can be 512*512. Therefore, it is necessary to align the resolutions of different models before distilling heterogeneous Wensheng graph models.
[0030] In one example, the resolution alignment between models can be achieved by changing the resolution of the image output by the teacher model. For example, the SDXL model is required to output a 512*512 resolution image to distill the SD1.5 model. However, since the SDXL model is mainly trained on high-resolution images, when it is set to generate 512*512 images, the quality of the generated images will be greatly reduced, and it is difficult to achieve the ideal teaching effect when used as a teacher model for training.
[0031] In another example, the resolution alignment is achieved by reducing the resolution of the noise image to the required size through pooling or downsampling technology on the noise image input to the model. In the diffusion model, the input noise image must meet certain conditions of Gaussian distribution to ensure the normality and normalization of the data. However, the noise image processed by pooling or downsampling does not conform to the original Gaussian distribution characteristics. Inputting the processed noise image into the diffusion model will affect the quality of the image generated by the model.
[0032] Based on the above problems, the present disclosure provides an image generation method and a deep learning model training method, which aims to solve the resolution alignment problem between models of different levels by cropping noisy images and improve the distillation effect between Wensheng graph models of different architectures.
[0033] The technical solution provided by the present disclosure will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0034] Figure 1 The flowchart of the image generating method according to the embodiment of the present disclosure is schematically shown.
[0035] like Figure 1 As shown, the image generating method of this embodiment includes operations S110-S120.
[0036] In operation S110 , text information and a noise image to be processed are obtained.
[0037] In the embodiments of the present disclosure, text information refers to text describing the content of a generated image, and a noise image to be processed refers to a randomly generated noise image used to be input into a text graph model.
[0038] In operation S120, the first model is used to denoise the noisy image to be processed according to the text information to obtain a target image corresponding to the text information.
[0039] The first model may be a text image model, which can process the processed noisy image under the instruction of the text information and output a target image, and the resolution, size and other properties of the target image are the same as those of the processed noisy image. Denoising refers to the process of reducing noise in an image, and the processed noisy image can be gradually converted into a target image.
[0040] In an embodiment of the present disclosure, the first model is obtained by distilling the second model. During the distillation process, the input noise image and the supervised noise image of the first model are respectively cropped from the input noise image and the output noise image of the second model.
[0041] The first model may refer to a student model, and the second model may refer to a teacher model. Usually, the second model has higher resolution and performance than the first model. During the model training process, the first model is distilled using the second model to guide the learning of the first model, thereby optimizing the performance of the first model.
[0042] The input noise image refers to the noise image input to the first model or the second model. The supervised noise image is obtained by cropping the output noise image of the second model. The supervised noise image can be used to perform distillation training on the first model, and the loss is calculated by the supervised noise image and the input noise image of the first model, so as to optimize the first model.
[0043] Since the resolutions of the first model and the second model are different, in order to implement the distillation training of the first model and the second model, the input noise image input to the second model is cropped to obtain an image with the same resolution as the first model, which is used as the input noise image of the first model. The output noise image of the second model is cropped to obtain an image with the same resolution as the first model, which is the so-called supervised noise image of the first model.
[0044] For example, the first model is an SD1.5 model, which supports input noise images with an input resolution of 64*64. The second model is an SDXL model, which supports input noise images with an input resolution of 128*128. During model training, a noise image A with a resolution of 128*128 is generated as the input noise image of the second model. The noise image A is then cropped to obtain a noise image A' with a resolution of 64*64, which is used as the input noise image of the first model. Based on the noise image A, the second model outputs an output noise image B with a resolution of 128*128. Based on the noise image A', the first model outputs an output noise image C with a resolution of 64*64. The output noise image B is cropped to obtain a noise image B' with a resolution of 64*64, which is used as the supervised noise image of the first model. The first model can be optimized based on the output noise image C and the supervised noise image B'.
[0045] According to the embodiments of the present disclosure, the problem of resolution alignment when different viz-image models are distilled is solved by cropping the noise image. The noise image obtained by cropping only changes the resolution and still conforms to the Gaussian distribution that meets certain conditions. The correspondence between the teacher model and the student model is established without affecting the quality of the image generated by the model, and the performance of the student model is improved by knowledge distillation.
[0046] According to an embodiment of the present disclosure, a first model is used to process an image of a first type, and a second model is used to process an image of a second type, wherein the second type differs from the first type in at least resolution; during the distillation training process, the input noise image and the supervisory noise image of the first model are obtained by the following operations: cropping the input noise image of the first type from the input noise image of the second type of the second model as the input noise image of the first model; cropping the output noise image of the first type from the output noise image of the second type of the second model as the supervisory noise image of the first model.
[0047] Next, combine Figure 2 The image cropping of the present disclosure is further explained.
[0048] Figure 2 A schematic diagram of image cropping according to an embodiment of the present disclosure is schematically shown.
[0049] The first type refers to the type of the input noise image of the first model, and the second type refers to the type of the input noise image of the second model. Figure 2 As shown, image 201 may be an image of the first type, and image 202 may be an image of the second type. The resolution of the first type of image is different from that of the second type of image. During the distillation training process, the second type of noise image is used as the input noise image of the second model, and the input noise image of the second model is cropped to obtain the first type of noise image as the input noise image of the first model.
[0050] In the embodiment of the present disclosure, a second type of noise image with a mean of 0 and a variance of 1 is first generated as the input noise image of the second model. Then, two random integers (integers less than or equal to the resolution of the second type of image) can be randomly generated, and a point P is determined from the second type of noise image according to the two random integers as a vertex of the cropped image. According to P(x, y), a first type of image is cropped from the second type of noise image as the input noise image of the first model. The first type of image also meets the Gaussian distribution characteristics of 0 and variance 1.
[0051] For example, the resolution of the second type of image is 128*128. Two random integers less than or equal to 128 are randomly generated, namely 45 and 10, respectively, so that a point 45 away from the left edge of the second type of image and 10 away from the upper edge can be determined as a vertex P of the cropped image. According to P(45,10), cropping is performed on the second type of image to obtain a first type of image with a resolution of 64*64.
[0052] Similarly, for the second type of output noise image output by the second model, the same cropping operation can be performed to obtain the first type of supervised noise image.
[0053] According to an embodiment of the present disclosure, a first type of output noise image is cropped from a second type of output noise image of a second model as a supervisory noise image of the first model, including: obtaining a cropping position of the first type of input noise image in the second type of input noise image; determining a target position in the second type of output noise image corresponding to the cropping position; and cropping the first type of output noise image at the target position as the supervisory noise image of the first model.
[0054] In order to achieve alignment between the input image and the output image, when cropping the second type of output noise image, it is necessary to refer to the cropping position of the input noise image.
[0055] In an embodiment of the present disclosure, the cropping position can be determined on the second type of image by randomly generating numbers. After cropping, the second type and the first type of input noise images are respectively input into their respective models, and the output noise images of the second type and the first type can be respectively obtained. Based on the cropping position, the corresponding target position can be determined on the second type of output noise image for cropping to obtain the first type of image as the supervised noise image of the first model.
[0056] For example, in the second type of input noisy image, Figure 2 The first type of input noise image is obtained by cropping in the manner shown. After obtaining the second type of output noise image, the same position of the output noise image is cropped as the target position according to the cropping position of the input noise image to obtain the first type of supervised noise image. The first type of supervised noise image obtained by this method corresponds to the first type of input noise image, and a corresponding relationship between the second model and the first model can be established, the first model is optimized, and the performance of the first model is improved.
[0057] According to an embodiment of the present disclosure, the distillation process includes: determining the output noise image of the first model based on the input noise image of the first model; obtaining the distillation loss based on the difference between the supervised noise image and the output noise image of the first model, and obtaining the task loss based on the difference between the input noise image of the first model and the output noise image of the first model; adjusting the parameters of the first model based on the overall loss determined by the distillation loss and the task loss.
[0058] Next, combine Figure 3 The image generating method of this embodiment is further described.
[0059] Figure 3 A schematic diagram of an image generating method according to an embodiment of the present disclosure is schematically shown.
[0060] like Figure 3 As shown, the input noise image 312 of the second model 320 is input into the second model 320 to obtain the output noise image 322 of the second model 320. The input noise image 311 of the first model 310 obtained by cropping the input noise image 312 of the second model 320 is input into the first model 310 to obtain the output noise image 312 of the first model 310. The output noise image 322 of the second model 320 is cropped to obtain a supervisory noise image 322'. The supervisory noise image 322' and the input noise image 311 of the first model 310 are compared with the output noise image 312 of the first model 310 to determine the overall loss. The distillation loss is determined according to the difference between the supervisory noise image 322' and the output noise image 312 of the first model 310, and the task loss is determined according to the difference between the input noise image 311 of the first model 310 and the output noise image 312 of the first model 310.
[0061] The overall loss can be obtained by calculating the sum, product or weight value of the distillation loss and the task loss. Through the overall loss, the parameters of the first model 310 are adjusted, so as to achieve the distillation of the first model 310 by the second model 320.
[0062] According to an embodiment of the present disclosure, an output noise image of the first model is determined based on an input noise image of the first model, including: adding noise to a first sample image using the input noise image of the first model to obtain an intermediate noise image; and denoising the intermediate noise image based on the first sample text to obtain the output noise image of the first model.
[0063] Noising refers to the process of gradually adding Gaussian noise to the original image until the image data becomes random noise. Denoising is the reverse process of noisy image. By denoising the noisy image, we can get the generated image with content entity and the removed noise image.
[0064] In an embodiment of the present disclosure, the first sample image refers to an image containing a content entity and input into the first model as a training sample. The first sample text is used to describe the first sample image and instruct the first model to generate an image corresponding to the first sample text. The input noise image, the first sample image and the first sample text are input into the first model, and the first model is used to add noise to the first sample image according to the input noise image, and the first sample image data is converted into random noise to obtain an intermediate noise image. The first model is then used to denoise the intermediate noise image according to the first sample text to obtain an output image corresponding to the first sample text and an output noise image.
[0065] According to the embodiments of the present disclosure, by solving the problem of alignment of different resolutions, the knowledge distillation method can be used to transfer the knowledge contained in the high-performance teacher model to the student model, thereby effectively improving the performance of the student model and improving the quality of the generated image.
[0066] Figure 4 A flowchart of a deep learning model training method according to an embodiment of the present disclosure is schematically shown.
[0067] like Figure 4 As shown, the training method of the deep learning model of this embodiment includes operations S410 to S450.
[0068] In operation S410, a first sample text, a first sample image, and a first input noise image are input into a first model to obtain a first output noise image.
[0069] The first sample text may be a sentence or word describing the content of the image. The first sample image corresponds to the content of the first sample text. The first input noise image may be a randomly generated noise image. The first model may be a text graph model, such as a latent diffusion model, a denoising probability diffusion model, a stable diffusion model, etc. The first sample text, the first sample image and the first input noise image are input into the first model to obtain a first output noise image output by the first model.
[0070] For example, the first sample text is "the plate is full of fruits". The content of the first sample image is a fruit plate and the fruits in the plate. The first input noise image is a randomly generated noise image. The first model adds noise to the first sample image according to the first input noise image, and then denoises the noisy image according to the first sample text to obtain an output image and a first output noise image.
[0071] In operation S420, the second sample text, the second sample image, and the second input noise image are input into the second model to obtain a second output noise image. The first input noise image is cropped from the second input noise image. The resolution of the input image and the output image of the second model is higher than that of the first model. The resolution of the first sample image and the second sample image are different, but the image content is consistent. The working principle of the second model is similar to that of the first model, and will not be repeated for simplicity.
[0072] Since the resolution of the second model is higher, the second input noise image of the second model is cropped to obtain a noise image matching the resolution of the first model as the first input noise image of the first model.
[0073] In operation S430, a distillation loss is determined based on a difference between the supervisory noise image and the first output noise image, wherein the supervisory noise image is cropped from the second output noise image.
[0074] By cropping the second output noise image of the second model, a noise image matching the resolution of the first model is obtained as the supervised noise image. By comparing the difference between the supervised noise image and the first output noise image, the distillation loss can be determined. The distillation loss can be obtained by the following formula (1):
[0075] (1)
[0076] in, is the distillation loss, To supervise the noisy image, is the first output noise image of the first model.
[0077] In operation S440, a task loss is determined according to a difference between the first input noise image and the first output noise image.
[0078] By comparing the first input noisy image and the first output noisy image of the first model, the task loss of the first model itself can be determined. The task loss can be obtained by the following formula (2):
[0079] (2)
[0080] in, For mission loss, is the first input noise image after cropping, is the first output noise image of the first model.
[0081] In operation S450, parameters of the first model are adjusted according to the distillation loss and the task loss.
[0082] The overall loss of the first model can be determined based on the distillation loss and the task loss. The overall loss can be determined according to the following formula (3):
[0083] (3)
[0084] in, For the overall loss, and are preset constants, which are the weight coefficients of distillation loss and task loss respectively.
[0085] For example, the weight coefficients of distillation loss and task loss are both 1.0, and the overall loss is the sum of distillation loss and task loss.
[0086] According to the embodiments of the present disclosure, the problem of aligning different resolutions is solved by randomly cropping the noise model, and the correspondence between the teacher model and the student model is established in the form of knowledge distillation. The knowledge contained in the teacher model is transferred to the student model, thereby effectively improving the student model's level of textual analysis.
[0087] Next, combine Figure 5 The training method of the deep learning model disclosed in the present invention is further explained.
[0088] Figure 5 A schematic diagram schematically shows a method for training a deep learning model according to an embodiment of the present disclosure.
[0089] like Figure 5 As shown, the first model 510 is a student model and the second model 520 is a teacher model. The optimal resolution of the first model 510 is smaller than that of the second model 520. A Gaussian noise image is randomly generated as a second input noise image 521 of the second model 520. The second input noise image is cropped to obtain the first input noise image 511 of the first model 510. The first sample image 513 and the second sample image 523 have the same content but different resolutions. The first sample text 514 and the second sample text 524 are used to describe the content of the sample image.
[0090] The first input noise image 511, the first sample image 513 and the first sample text 514 are input into the first model 510, and the output of the first model 510 is the first output noise image 512. The second input noise image 521, the second sample image 523 and the second sample text 524 are input into the second model 520, and the output of the second model 520 is the second output noise image 522.
[0091] The second output noise image 522 of the second model 520 is cropped to obtain a supervisory noise image 522' having a resolution matching that of the first model 510. The supervisory noise image 522' is compared with the first output noise image 512 to determine the distillation loss 502. The first input noise image 511 is compared with the first output noise image 512 to determine the task loss 501. Based on the task loss 501 and the distillation loss 502, the parameters of the first model 510 are adjusted to optimize the image generation performance of the first model 510.
[0092] According to an embodiment of the present disclosure, the method further includes: transforming the original image into a first sample image of the first type and a second sample image of the second type.
[0093] The size of the original image sample01 is converted into resolutions adapted to the first model 510 and the second model 520 , respectively, to obtain a first sample image 513 and a second sample image 523 .
[0094] For example, the resolution of the original image is 1920*1920. The optimal resolution of the first model 510 is 512*512, and the optimal resolution of the second model 520 is 1024*1024. The original image is resized to obtain a first sample image 513 with a resolution of 512*512 and a second sample image 523 with a resolution of 1024*1024.
[0095] According to an embodiment of the present disclosure, the first model is used to process text of a first language type, and the second model is used to process text of a second language type; the method further includes: determining a first sample text of the first language type and a second sample text of the second language type, wherein the second sample text is semantically consistent with the first sample text.
[0096] In the embodiments of the present disclosure, the first model and the second model can process texts in different languages, such as Chinese, English, Japanese, etc. When different types of languages are used as sample texts for the first model and the second model, the semantics of the first sample text and the second sample text need to be kept consistent.
[0097] For example, the first model 510 can use the Chinese and English versions of the Taiyi model, and the second model 520 can use the playgroundv2 model. The first model 510 can process Chinese-type text, and the second model 520 is used to process English-type text. The text used to describe the original image can be determined, and the English text and Chinese text with consistent semantics can be obtained through the translation model. The Chinese text is input into the text encoder of the first model 510 to obtain the first sample text 514. The English text is input into the text encoder of the second model 520 to obtain the second sample text 524.
[0098] According to the embodiments of the present disclosure, by aligning the input text and image and establishing the correspondence between models of different architectures, it is possible to implement distillation training of low-performance models through high-performance models, thereby effectively improving the image generation level of low-performance models.
[0099] Figure 6 The structural block diagram of the image generating device according to the embodiment of the present disclosure is schematically shown.
[0100] like Figure 6 As shown, the image generating device 600 of this embodiment includes a receiving module 610 and an image generating module 620 .
[0101] The receiving module 610 is used to obtain text information and a noise image to be processed. In one embodiment, the receiving module 610 can be used to perform the operation S110 described above, which will not be described in detail here.
[0102] The image generation module 620 is used to use the first model to denoise the noisy image to be processed according to the text information to obtain a target image corresponding to the text information. In one embodiment, the receiving module 620 can be used to perform the operation S120 described above, which will not be repeated here.
[0103] According to an embodiment of the present disclosure, the first model is used to process images of a first type, and the second model is used to process images of a second type, and the second type of images is different from the first type of images in at least resolution. During the distillation process, the input noise image and the supervised noise image of the first model are obtained through the following submodules: an input image cropping submodule, which is used to crop the input noise image of the first type from the input noise image of the second type of the second model as the input noise image of the first model. A supervised image cropping submodule, which is used to crop the output noise image of the first type from the output noise image of the second type of the second model as the supervised noise image of the first model.
[0104] According to an embodiment of the present disclosure, the supervised image cropping submodule includes a cropping position determination unit, a target position determination unit and a cropping unit. The cropping position determination unit is used to obtain a cropping position of a first type of input noise image in a second type of input noise image. The target position determination unit is used to determine a target position corresponding to the cropping position in the second type of output noise image. The cropping unit is used to crop the first type of output noise image at the target position as a supervised noise image of the first model.
[0105] According to an embodiment of the present disclosure, the distillation process is performed by the following submodules: an image determination submodule, for determining an output noise image of the first model based on an input noise image of the first model. A loss determination submodule, for obtaining a distillation loss based on a difference between the supervised noise image and the output noise image of the first model, and a task loss based on a difference between the input noise image of the first model and the output noise image of the first model. A parameter adjustment submodule, for adjusting the parameters of the first model based on an overall loss determined by the distillation loss and the task loss.
[0106] According to an embodiment of the present disclosure, the image determination submodule includes a denoising unit and a denoising unit. The denoising unit is used to denoise the first sample image using the input noise image of the first model to obtain an intermediate noise image. The denoising unit is used to denoise the intermediate noise image according to the first sample text to obtain an output noise image of the first model.
[0107] Figure 7 The structural block diagram of a deep learning model training device according to an embodiment of the present disclosure is schematically shown.
[0108] like Figure 7 As shown, the training device 700 of the deep learning model of this embodiment includes a first noise image generating module 710, a second noise image generating module 720, a distillation loss determining module 730, a task loss determining module 740 and a parameter adjustment module 750.
[0109] The first noise image generation module 710 is used to input the first sample text, the first sample image, and the first input noise image into the first model to obtain a first output noise image. In one embodiment, it can be used to perform the operation S410 described above, which will not be repeated here.
[0110] The second noise image generation module 720 is used to input the second sample text, the second sample image, and the second input noise image into the second model to obtain a second output noise image. In one embodiment, the second noise image generation module 720 can be used to perform the operation S420 described above, which will not be repeated here.
[0111] The distillation loss determination module 730 is used to determine the distillation loss according to the difference between the supervision noise image and the first output noise image. In one embodiment, the distillation loss determination module 730 can be used to perform the operation S430 described above, which will not be repeated here.
[0112] The task loss determination module 740 is used to determine the task loss according to the difference between the first input noise image and the first output noise image. In one embodiment, the task loss determination module 740 can be used to perform the operation S440 described above, which will not be described in detail here.
[0113] The parameter adjustment module 750 is used to adjust the parameters of the first model according to the distillation loss and the task loss. In one embodiment, the parameter adjustment module 750 can be used to perform the operation S450 described above, which will not be described in detail here.
[0114] According to an embodiment of the present disclosure, the deep learning model training device 700 also includes a transformation module, which is used to transform the original image into a first sample image of a first type and a second sample image of a second type.
[0115] According to an embodiment of the present disclosure, the first model is used to process text of a first language type, and the second model is used to process text of a second language type. The training device 700 of the deep learning model also includes a text determination module, which is used to determine a first sample text of the first language type and a second sample text of the second language type, wherein the second sample text is semantically consistent with the first sample text.
[0116] Figure 8 A schematic block diagram of an electronic device that can be used to implement the image generation method and the deep learning model training method of the embodiments of the present disclosure is schematically shown.
[0117] like Figure 8 As shown, the electronic device 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 to a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include an onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.
[0118] In RAM 803, various programs and data required for the operation of electronic device 800 are stored. Processor 801, ROM 802 and RAM 803 are connected to each other via bus 804. Processor 801 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 802 and / or RAM 803. It should be noted that the program can also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 can also implement the method provided by the embodiment of the present disclosure by executing the program stored in the one or more memories.
[0119] According to an embodiment of the present disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to the bus 804. The electronic device 800 may further include one or more of the following components connected to the I / O interface 805: an input portion 806 including a keyboard, a mouse, etc.; an output portion 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 808 including a hard disk, etc.; and a communication portion 809 including a network interface card such as a LAN card, a modem, etc. The communication portion 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage portion 808 as needed.
[0120] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.
[0121] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.
[0122] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present disclosure.
[0123] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 801. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.
[0124] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0125] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.
[0126] It should be noted that the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals. In the technical solution of this disclosure, the user's authorization or consent is obtained before obtaining or collecting user personal information.
[0127] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).
[0128] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0129] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or coupled in a variety of ways, even if such combinations and / or couplings are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or coupled in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or couplings fall within the scope of the present disclosure.
[0130] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.
Claims
1. A method for generating an image, comprising: Obtain text information and noise images to be processed; Using the first model, denoising the noisy image to be processed according to the text information to obtain a target image corresponding to the text information; The first model is obtained by distilling the second model. During the distillation process, the input noise image and the supervision noise image of the first model are respectively cropped from the input noise image and the output noise image of the second model.
2. The method according to claim 1, wherein: The first model is used to process images of a first type, and the second model is used to process images of a second type, where the second type of images is different from the first type of images in at least resolution; In the distillation process, the input noise image and the supervision noise image of the first model are obtained by the following operations: Cropping an input noise image of a first type from an input noise image of a second type of the second model as an input noise image of the first model; An output noise image of the first type is cropped from the output noise image of the second type of the second model as a supervisory noise image of the first model.
3. The method according to claim 2, wherein: The step of cutting out the first type of output noise image from the second type of output noise image of the second model as the supervised noise image of the first model comprises: Obtaining a cropping position of the first type of input noisy image in the second type of input noisy image; determining a target position in the second type of output noisy image corresponding to the cropping position; and The first type of output noise image is cropped at the target position as a supervision noise image of the first model.
4. The method according to any one of claims 1 to 3, wherein: The distillation process comprises: Determining an output noise image of the first model according to an input noise image of the first model; Obtaining a distillation loss according to a difference between the supervised noise image and an output noise image of the first model, and obtaining a task loss according to a difference between an input noise image of the first model and an output noise image of the first model; The parameters of the first model are adjusted according to the overall loss determined by the distillation loss and the task loss.
5. The method according to claim 4, wherein: The step of determining the output noise image of the first model according to the input noise image of the first model comprises: Noising the first sample image using the input noise image of the first model to obtain an intermediate noise image; and The intermediate noisy image is denoised according to the first sample text to obtain an output noisy image of the first model.
6. A method for training a deep learning model, comprising: Inputting a first sample text, a first sample image, and a first input noise image into a first model to obtain a first output noise image; Inputting the second sample text, the second sample image, and the second input noise image into the second model to obtain a second output noise image; The first input noise image is cropped from the second input noise image; determining a distillation loss based on a difference between the supervision noise image and the first output noise image; and determining a task loss based on a difference between the first input noisy image and the first output noisy image; Wherein, the supervised noise image is cropped from the second output noise image; Adjusting parameters of the first model according to the distillation loss and the task loss.
7. The method according to claim 6, further comprising: The original image is transformed into a first sample image of a first type and a second sample image of a second type.
8. The method according to claim 6, wherein: The first model is used to process text of a first language type, and the second model is used to process text of a second language type; the method further includes: A first sample text of a first language type and a second sample text of a second language type are determined, wherein the second sample text is semantically consistent with the first sample text.
9. An image generating device, comprising: A receiving module, used for obtaining text information and a noise image to be processed; as well as An image generation module, configured to use the first model to denoise the noisy image to be processed according to the text information to obtain a target image corresponding to the text information; The first model is obtained by distilling the second model. During the distillation process, the input noise image and the supervision noise image of the first model are respectively cropped from the input noise image and the output noise image of the second model.
10. A training device for a deep training model, comprising: A first noise image generating module, used for inputting a first sample text, a first sample image, and a first input noise image into a first model to obtain a first output noise image; A second noise image generating module, used for inputting a second sample text, a second sample image, and a second input noise image into a second model to obtain a second output noise image; The first input noise image is cropped from the second input noise image; a distillation loss determination module, configured to determine a distillation loss according to a difference between a supervision noise image and the first output noise image; and a task loss determination module, configured to determine a task loss according to a difference between the first input noisy image and the first output noisy image; Wherein, the supervised noise image is cropped from the second output noise image; A parameter adjustment module is used to adjust the parameters of the first model according to the distillation loss and the task loss.