Image generation method and device, model training method and device and computer equipment

Abstract

The invention provides an image generation method and device, a model training method and device, and computer equipment, and relates to the field of image processing, and the implementation scheme is that based on a sample prompt text and a first noise image, a first output representation is obtained through a reference image generation model; obtaining a down-sampled output characterization; obtaining a second noise image corresponding to the first noise image; executing an image generation process including multi-step de-noising reasoning through the target image generation model to obtain a second output representation; determining a loss value based on the second output characterization and the downsampled output characterization; and adjusting parameters of the target image generation model based on the determined loss value to obtain the trained target image generation model, so that the image generation model for executing multi-step denoising reasoning under low resolution can be obtained, and therefore, the calculation amount and bandwidth pressure can be remarkably reduced, and meanwhile, high generation quality and style consistency can be kept.

Description

Technical Field

[0001] This disclosure relates to image processing, and more particularly to image generation methods and apparatus, model training methods and apparatus, and computer equipment. Background Technology

[0002] The application of diffusion models in image generation tasks is becoming increasingly widespread. However, its inference process usually requires multiple denoising iterations, and each step involves convolution, attention, and multi-branch operator computation on high-resolution latent space features, which brings significant computational and storage bandwidth pressure, making it difficult to meet the real-time or low-latency deployment requirements on edge devices or dedicated chips with limited bandwidth. Summary of the Invention

[0003] This disclosure provides an image generation method and apparatus, a model training method and apparatus, and a computer device.

[0004] According to one aspect of this disclosure, a method for training an image generation model is provided, the method comprising: obtaining sample prompt text; obtaining a first noise image corresponding to the dimension of a target image to be generated; performing an image generation process including multi-step denoising inference using a trained reference image generation model based on the sample prompt text and the first noise image to obtain a first output representation; downsampling the first output representation based on a predetermined spatial scaling ratio to obtain a downsampled output representation; obtaining a second noise image corresponding to the first noise image such that the size of the second noise image is smaller than the size of the first noise image, and the size of the second noise image and the size of the first noise image follow the predetermined spatial scaling ratio; obtaining a target image generation model; performing an image generation process including multi-step denoising inference using the target image generation model based on the sample prompt text and the second noise image to obtain a second output representation; determining a loss value based on the second output representation and the downsampled output representation; and adjusting the parameters of the target image generation model based on the determined loss value to obtain a trained target image generation model such that the trained target image generation model can perform multi-step denoising inference at a spatial resolution lower than that of the reference image generation model.

[0005] In some embodiments, the first output representation is a latent space representation obtained by performing multi-step denoising inference on the first noisy image, and the second output representation is a latent space representation obtained by performing multi-step denoising inference on the second noisy image.

[0006] In some embodiments, the image generation process further includes a decoding operation, wherein the first output representation is a pixel space representation obtained by performing multi-step denoising inference and decoding on the first noisy image, and the second output representation is a pixel space representation obtained by performing multi-step denoising inference and decoding on the second noisy image.

[0007] In some embodiments, the reference image generation model includes a first inference denoising module, and the target image generation model includes a second inference denoising module, each of the first inference denoising module and the second inference denoising module being selected from the group consisting of: UNet network, Transformer skeleton network.

[0008] In some embodiments, obtaining a second noise image corresponding to the first noise image includes downsampling the first noise image based on the predetermined spatial scaling ratio to obtain the second noise image.

[0009] In some embodiments, the first noise image and the second noise image correspond to at least one of the same noise schedule and the same random seed.

[0010] In some embodiments, the multi-step denoising inference performed by the reference image generation model and the multi-step denoising inference performed by the target image generation model are performed under the same control conditions.

[0011] In some embodiments, downsampling the first output representation based on a predetermined spatial scaling ratio includes downsampling the first output representation by a factor of 2 in spatial dimensions.

[0012] According to another aspect of this disclosure, an image generation method is provided, the method comprising: obtaining a prompt text describing an image to be generated; performing an image generation process including multi-step denoising inference based on a latent space noise image and the prompt text, using a trained target image generation model obtained according to any of the foregoing embodiments, to obtain a low-resolution generation result with a resolution lower than the target resolution; and enlarging and reconstructing the low-resolution generation result to the target resolution using a super-resolution reconstruction module, thereby obtaining an output image.

[0013] In some embodiments, the latent space noise image is obtained by determining the resolution of the latent space noise image based on the target resolution and the scaling ratio of the super-resolution reconstruction module to generate the latent space noise image.

[0014] According to another aspect of this disclosure, a training apparatus for an image generation model is provided, the apparatus comprising: a prompt text acquisition unit configured to acquire sample prompt text; a first noise image acquisition unit configured to acquire a first noise image corresponding to the dimension of a target image to be generated; a first output representation acquisition unit configured to perform an image generation process including multi-step denoising inference using a trained reference image generation model based on the sample prompt text and the first noise image, to obtain a first output representation; a downsampling unit configured to downsample the first output representation based on a predetermined spatial scaling ratio to obtain a downsampled output representation; and a second noise image acquisition unit configured to acquire a second noise image corresponding to the first noise image, such that the size of the second noise image is smaller than the size of the first noise image, and Furthermore, the size of the second noise image and the size of the first noise image follow the predetermined spatial scaling ratio; a target image generation model obtaining unit is configured to obtain a target image generation model; a second output representation obtaining unit is configured to perform an image generation process including multi-step denoising inference through the target image generation model based on the sample prompt text and the second noise image to obtain a second output representation; a loss determination unit is configured to determine a loss value based on the second output representation and the downsampled output representation; and a training unit is configured to adjust the parameters of the target image generation model based on the determined loss value to obtain a trained target image generation model, such that the trained target image generation model can perform multi-step denoising inference at a spatial resolution lower than that of the reference image generation model.

[0015] According to another aspect of this disclosure, an image generation apparatus is provided, the apparatus comprising: a text acquisition unit configured to acquire prompt text describing an image to be generated; a low-resolution result acquisition unit configured to perform an image generation process including multi-step denoising inference based on a latent space noise image and the prompt text, through a trained target image generation model obtained according to any of the foregoing embodiments, to obtain a low-resolution generation result with a resolution lower than the target resolution; and an output image acquisition unit configured to enlarge and reconstruct the low-resolution generation result to the target resolution through a super-resolution reconstruction module, thereby obtaining an output image.

[0016] According to another aspect of this disclosure, a computer device is provided, the computer device comprising: at least one processor; and a memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method according to any of the foregoing embodiments.

[0017] According to another aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, causes the processor to perform the method according to any of the foregoing embodiments.

[0018] According to another aspect of this disclosure, a computer program product is provided, the computer program product including a computer program that, when executed by a processor, causes the processor to perform the method according to any of the foregoing embodiments.

[0019] According to one or more embodiments of this disclosure, computational load and bandwidth pressure can be significantly reduced while maintaining high generation quality and style consistency.

[0020] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0021] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0022] Figure 1 A schematic diagram of a training method for an image generation model according to an exemplary embodiment of the present disclosure is shown.

[0023] Figure 2 A schematic diagram of an image generation model according to an exemplary embodiment of the present disclosure is shown.

[0024] Figure 3 A schematic diagram of an image generation method according to an exemplary embodiment of the present disclosure is shown.

[0025] Figure 4 A schematic diagram of a training apparatus for an image generation model according to an exemplary embodiment of the present disclosure is shown.

[0026] Figure 5 A schematic diagram of an image generation apparatus according to an exemplary embodiment of the present disclosure is shown.

[0027] Figure 6 A schematic diagram of a computer device according to an exemplary embodiment of the present disclosure is shown. Detailed Implementation

[0028] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0029] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0030] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0031] Figure 1 A training method 100 for an image generation model according to an embodiment of the present disclosure is illustrated.

[0032] In step 101, sample prompt text is obtained.

[0033] In step 102, a first noise image corresponding to the dimension of the target image to be generated is obtained.

[0034] At step 103, based on the sample prompt text and the first noisy image, an image generation process including multi-step denoising inference is performed by a trained reference image generation model to obtain a first output representation.

[0035] At step 104, the first output representation is downsampled based on a predetermined spatial scaling ratio to obtain a downsampled output representation.

[0036] At step 105, a second noise image corresponding to the first noise image is obtained such that the size of the second noise image is smaller than the size of the first noise image, and the size of the second noise image and the size of the first noise image follow the predetermined spatial scaling ratio.

[0037] In step 106, the target image generation model is obtained.

[0038] At step 107, based on the sample prompt text and the second noisy image, an image generation process including multi-step denoising inference is performed by the target image generation model to obtain a second output representation.

[0039] At step 108, a loss value is determined based on the second output representation and the downsampled output representation.

[0040] In step 109, the parameters of the target image generation model are adjusted based on the determined loss value to obtain a trained target image generation model, which enables the trained target image generation model to perform multi-step denoising inference at a spatial resolution lower than that of the reference image generation model.

[0041] The methods described above, provided by the embodiments of this disclosure, can significantly reduce computational load and bandwidth pressure while maintaining high generation quality and style consistency.

[0042] The following will describe in detail non-limiting embodiments of this disclosure.

[0043] According to some embodiments, the first output representation is a latent space representation obtained by performing multi-step denoising inference on the first noisy image, and the second output representation is a latent space representation obtained by performing multi-step denoising inference on the second noisy image.

[0044] According to some embodiments, the image generation process further includes a decoding operation, wherein the first output representation is a pixel space representation obtained by performing multi-step denoising inference and decoding on the first noisy image, and the second output representation is a pixel space representation obtained by performing multi-step denoising inference and decoding on the second noisy image.

[0045] It is understandable that the obtained reference image generation model can be an image generation model that includes a diffusion model. The reference image generation model can be trained to generate images that conform to the description of the prompt text based on the prompt text and noise features.

[0046] It is understood, and will become clear from the further explanation below, that the dimensional correspondence between the first noisy image and the target image to be generated can mean that the dimension of the first noisy image can be obtained from the corresponding image generation network to obtain the dimension of the target image with the target dimension or target resolution. For example, the first noisy image can correspond to the input dimension of the reference image generation model.

[0047] According to some embodiments, the reference image generation model includes a first inference denoising module, and the target image generation model includes a second inference denoising module, each of the first inference denoising module and the second inference denoising module being selected from the group consisting of: UNet network, Transformer skeleton network.

[0048] For example, the reference image generation model may include a first inference denoising module and a first decoder module, and the target image generation model may include a second inference denoising module and a second decoder module. In such an embodiment, obtaining a first output image through the target image generation model may include: performing multi-step denoising inference through the first inference denoising module of the target image generation model based on sample prompt text and a first noise image to obtain a first inference feature; and obtaining the first output image through the first decoder module of the target image generation model based on the first inference feature. In such an embodiment, obtaining a second output image through the target image generation model may include: performing multi-step denoising inference through the second inference denoising module of the target image generation model based on sample prompt text and a second noise image to obtain a second inference feature; and obtaining the second output image through the second decoder module of the target image generation model based on the second inference feature.

[0049] Figure 2 A schematic diagram of an image generation model according to an exemplary embodiment of the present disclosure is shown. Figure 2 As shown, the architecture can contain two parallel generation paths: the lower reference image generation model path (i.e., the teacher model path) and the upper target image generation model path (i.e., the student model path).

[0050] For example, in Figure 2 In the reference path below, the system receives high-resolution noise of size 128×128 and a prompt text. This noise is first input to the base model, undergoing multiple denoising steps to output 128×128 unrefined latent variables. Subsequently, these latent variables are fed into the refiner model for further denoising and feature refinement, outputting 128×128 refined latent variables. Finally, a variational autoencoder (VAE) decoder decodes the refined latent variables into a pixel-space image, with a resolution exemplarily 1024×1024.

[0051] For example, in Figure 2 In the target path above, the system receives low-resolution noise of size 64×64 along with the same prompt text.

[0052] In diffusion inference, the initial input is typically a latent space noise image or a latent space noise tensor. Its spatial resolution is related to the target image resolution. In the embodiments of this disclosure, the initial noise (latent variable) is downsampled by 2 times in the spatial dimension to obtain low-resolution noise. Subsequently, a full or partial number of denoising iterations are performed in the low-resolution latent space to obtain the low-resolution latent variables. .

[0053] Similar to the reference path, the noise is processed sequentially by the base model and the refined model. Notably, because the spatial dimension of the input noise is halved (from 128 to 64), the base model and the refined model in the target path are computed in a low-resolution latent space, outputting 64×64 unrefined and refined latent variables, respectively. This step offers significant advantages: First, the computational cost per step is reduced because the smaller size of the low-resolution feature map significantly reduces the computational scale of operators such as convolution and attention in the spatial dimension; second, intermediate activations and memory access are reduced, as the volume of intermediate tensors in each iteration is reduced, significantly lowering memory usage and bandwidth pressure; finally, hardware friendliness is improved, as smaller features can improve cache hit rates and reduce data transfer bottlenecks, especially suitable for bandwidth-constrained NPUs / GPUs / ASICs / FPGAs.

[0054] Subsequently, the low-resolution refined latent variables are decoded by the VAE decoder to generate an intermediate image with a resolution of 512×512.

[0055] To ensure high quality generated at low resolution Figure 2 This demonstrates the role of the distillation module. The 1024×1024 image generated from the reference path is processed by the downsampling module to generate a 512×512 image (as a soft label) that matches the output size of the target path. The distillation module calculates a loss value based on this downsampled image and the 512×512 image generated from the target path, and uses this loss value to optimize the parameters of the model in the target path, thereby achieving knowledge distillation.

[0056] During the inference phase or when generating the final result, the 512×512 image generated by the target path is further input into the super-resolution model. The super-resolution model enlarges and reconstructs this image, ultimately outputting a final image with a resolution of 1024×1024. Through this combination of "low-resolution denoising + distillation + super-resolution," it is possible to obtain a high-quality image with the same resolution as the reference model while significantly reducing intermediate computational overhead.

[0057] It is understood that the above are merely examples, and the methods according to various embodiments of this disclosure can be applied to image generation models with other structures and images of other resolutions, as long as the model includes an image generation step based on denoising iteration.

[0058] According to one or more embodiments of this disclosure, a large-scale text-to-image model, such as SDXL, can be used as at least one of the reference image generation model and the target image generation model.

[0059] According to one or more embodiments of this disclosure, a teacher-supervised distillation strategy is provided to avoid image quality degradation caused by low-resolution denoising. Under the same conditions (e.g., text cue words, negative cue words, control conditions, random seeds, or noise scheduling, etc.), a teacher output image or latent variable result can be generated using an original reference image generation model (as the teacher model), and this result can be used as the ground truth (GT) label to fine-tune the target image generation model.

[0060] Through the distillation training described above, the target image generation model (student model) can more closely approximate the output of the reference image generation model (teacher model) when performing low-resolution denoising, thereby maintaining acceptable image quality while significantly reducing computational scale.

[0061] It is understood that the target image generation model may be, for example, a pre-trained model, and adjusting the parameters of the target image generation model based on the determined loss value may include fine-tuning the parameters of the pre-trained target image generation model so that the generation distribution of the low-resolution student model approximates the distribution of the teacher model at low resolution.

[0062] According to some embodiments, obtaining a second noise image corresponding to the first noise image includes downsampling the first noise image based on the predetermined spatial scaling ratio to obtain the second noise image.

[0063] According to some embodiments, the first noise image and the second noise image have a consistent spatial distribution.

[0064] According to some embodiments, the first noise image and the second noise image correspond to at least one of the same noise schedule and the same random seed. Exemplarily, obtaining the second noise image corresponding to the first noise image includes generating the second noise image with a size following the predetermined spatial scaling ratio, based on the same random seed as the first noise image.

[0065] For example, the sample prompt text may include text prompt words and negative prompt words, thereby enabling two processes to contain the same text prompt words and negative prompt words.

[0066] According to some embodiments, the multi-step denoising inference performed by the reference image generation model and the multi-step denoising inference performed by the target image generation model are performed under the same control conditions.

[0067] According to some embodiments, downsampling the first output representation based on a predetermined spatial scaling ratio includes downsampling the first output representation by a factor of 2 in spatial dimensions.

[0068] The image generation model training method described in the embodiments of this disclosure can achieve the following beneficial technical effects:

[0069] This paper addresses the issues of image quality degradation and semantic drift caused by low-resolution generation: existing techniques that directly reduce inference resolution or the number of network layers often result in loss of image details, blurred textures, or semantic inconsistencies. The training method disclosed here employs a teacher-supervised distillation strategy, using the output representation (as soft label / Ground Truth) generated by a trained reference model (teacher model) under the same cue words and noise conditions to supervise the training of the target model (student model). This method forces the target model to learn and approximate the high-dimensional distribution features of the reference model in a low-resolution latent space, thereby significantly reducing computational dimensionality while maximizing the preservation of the original SDXL model's high-quality generation capabilities and style consistency.

[0070] This approach achieves source-level compression of computational scale and memory bandwidth: by explicitly constructing low-resolution noisy features during the training phase (e.g., downsampling the spatial dimension by 2x), the target model trained by this method can naturally adapt to low-resolution latent space inputs. This means that in each iteration of model inference, the size of the intermediate feature map processed by the convolution and attention operators is significantly reduced (approximately 1 / 4 of the original number of pixels). This trained model structure fundamentally reduces the memory usage and data transfer bandwidth requirements of edge NPU / GPU, thereby overcoming the memory wall bottleneck on edge devices.

[0071] A consistent "training-inference" optimization scheme is provided: unlike schemes that only optimize operators or perform quantization during the inference stage, the training method disclosed herein ensures that the model parameters are specifically optimized for low-resolution feature distributions. By internalizing "downsampling denoising" as an inherent capability of the model, the problem of accuracy loss requiring complex calibration, common in traditional post-processing acceleration schemes, is avoided. This allows the final trained model to achieve efficient inference on hardware without additional complex adaptations.

[0072] The following is combined with Figure 3 This describes an image generation method 300 exemplified according to an example form of this disclosure.

[0073] At step 301, a prompt text describing the image to be generated is obtained.

[0074] At step 302, based on the latent space noise image and the prompt text, an image generation process including multi-step denoising inference is performed by a trained target image generation model obtained according to the model training method of this disclosure to obtain a low-resolution generation result with a resolution lower than the target resolution.

[0075] In step 303, the low-resolution generated result is enlarged and reconstructed to the target resolution using a super-resolution reconstruction module to obtain the output image.

[0076] The image generation method according to the embodiments of this disclosure can significantly reduce inference latency and computational power consumption. In the inference phase, the method of this disclosure shifts the originally computationally intensive diffusion denoising process to a low-resolution latent space. For example, when the spatial dimension is reduced to half of its original value, the number of pixels can be reduced to approximately one-quarter. This directly leads to a significant reduction in the computational load of core operators such as convolution and attention in the spatial dimension. This significantly reduces the number of floating-point operations (FLOPs) required for each denoising iteration, thereby greatly shortening the total inference latency of the generated image under the same hardware conditions and reducing the power consumption of the device.

[0077] The image generation method described in this disclosure can alleviate storage bandwidth pressure and is suitable for edge device deployment: due to the reduced size of intermediate feature maps and activation tensors during the denoising process, this method significantly reduces the peak memory usage during inference. More importantly, this reduces the amount of data transfer between the processor (such as NPU / GPU) and memory, alleviating the bandwidth bottleneck caused by the "memory wall". The smaller feature size also helps improve the hardware cache hit rate, making this method particularly suitable for mobile terminals and edge computing devices with limited video memory or bandwidth.

[0078] The image generation method according to the embodiments of this disclosure can balance generation speed and high-frequency detail reconstruction by introducing a super-resolution reconstruction module as the back-end processing of the generation process. After the "skeleton" is constructed quickly through low-resolution denoising, the super-resolution module is specifically responsible for restoring the high-frequency details, edge sharpness, and texture consistency of the image. This division of labor and cooperation mechanism effectively compensates for the loss of details or blurring that may be caused by low-resolution generation, and avoids the jagged effect caused by direct upsampling, thereby outputting a final image with visual quality comparable to the original high-resolution model at extremely low inference cost.

[0079] It is understandable that low-resolution generated results can be either latent space generated results or pixel space generated results.

[0080] In the example where the low-resolution generation result is the latent space generation result, the low-resolution generation result can be a low-resolution latent variable obtained after multi-step denoising inference. In this case, the super-resolution reconstruction in step 303 can include various implementation methods: as one example, the low-resolution latent variable can first be decoded into a low-resolution image in pixel space by a decoder (e.g., a variational autoencoder VAE decoder), and then the low-resolution image can be enlarged and reconstructed using a super-resolution reconstruction module to obtain an output image. As another example, the super-resolution reconstruction module can also be configured to directly super-resolution the latent space features, that is, first convert the low-resolution latent variable into a high-resolution latent variable corresponding to the target resolution, and then decode the high-resolution latent variable to obtain the final output image.

[0081] In the example where the low-resolution generation result is a pixel-space generation result, the low-resolution generation result can be a low-resolution image, and the output image can be obtained by upscaling and reconstructing the low-resolution image to the target resolution of the output image through a super-resolution reconstruction module. In such an example, the trained target image generation model can include or be connected to a decoder module (e.g., a variational autoencoder, VAE decoder) at the output end. After multi-step denoising inference in the low-resolution latent space is completed, the generated low-resolution latent space features can be mapped to the pixel space by the decoder module to form the low-resolution image (e.g., a 512×512 resolution image). Subsequently, this low-resolution image is fed as input to the super-resolution reconstruction module. The super-resolution reconstruction module (e.g., a lightweight super-resolution network based on CNN or Transformer) uses pre-trained prior knowledge to recover the high-frequency details and textures of the image, upscaling the spatial dimension of the image to the target resolution (e.g., from 512×512 to 1024×1024), thereby compensating for the loss of sharpness that may be caused by low-resolution denoising while maintaining the overall low cost and low memory usage of the inference chain, ensuring the visual quality of the final output image.

[0082] Furthermore, it is understood that the image generation process, which includes multi-step denoising inference, performed by a trained target image generation model obtained according to the model training method of this disclosure based on the latent space noise image and the prompt text, may include inputting the latent space noise image and the prompt text for image generation into the target image generation model to obtain an image generation result corresponding to the prompt text.

[0083] In some embodiments, the latent space noise image may be obtained by determining the resolution of the latent space noise image based on the target resolution and the scaling ratio of the super-resolution reconstruction module to generate the latent space noise image.

[0084] In some embodiments, determining the resolution of the latent space noise image to be generated can be achieved through the following steps: First, determining the target size of the low-resolution generation result. This may include: obtaining the resolution transformation rate of the low-resolution image generation model (i.e., the ratio of the output image resolution to the input noise resolution); obtaining the corresponding high noise resolution based on the expected resolution of the image to be generated and the resolution transformation rate; and calculating the low noise resolution based on the high noise resolution and the magnification and reconstruction ratio of the super-resolution reconstruction module, and using it as the resolution of the latent space noise image to be generated.

[0085] Understandably, the super-resolution module can be a super-resolution network model built on a convolutional neural network (CNN) or Transformer architecture. To adapt to the resource constraints of edge devices or edge computing devices, the super-resolution module can be a lightweight super-resolution model. Furthermore, the super-resolution module can also be an operator-accelerated version implemented for specific hardware (such as an NPU, GPU, or ASIC) to minimize inference latency and computational overhead while ensuring the restoration of high-frequency details in the image.

[0086] In existing technologies, reducing inference costs typically involves reducing the number of denoising steps, using low-precision quantization, or pruning structures. However, simply reducing the number of steps can easily lead to loss of detail and semantic drift, while direct pruning or quantization can cause a significant quality degradation and requires complex calibration. Furthermore, if the main denoising computation is still performed at the target high resolution, the spatial feature map size remains large, and the read / write operations of intermediate activation and attention-related caches still consume a significant amount of bandwidth, making the edge-side bottleneck still obvious. Therefore, there is an urgent need for a technical solution that can reduce the computational and data transport overhead of the backbone network while maintaining consistency between the generated distribution and the image.

[0087] According to one or more embodiments of this disclosure, the following beneficial effects can be achieved: Significantly reduces computational and bandwidth pressure: By performing denoising iterations in a low-resolution latent space sampled in the spatial dimension, the feature map size is significantly reduced, thereby reducing the computational scale of convolution and attention operators in the spatial dimension, and reducing the volume of intermediate tensors and memory usage, alleviating data transfer bottlenecks, especially suitable for bandwidth-constrained NPUs or edge computing devices.

[0088] Maintaining high generation quality and style consistency: A teacher-supervised distillation strategy is adopted, using the output of the original reference model (teacher model) as a supervision signal to fine-tune the low-resolution target model, so that the target model can approximate the generation distribution of the teacher model in the low-resolution domain, avoiding image quality collapse or style deviation caused by direct downsampling inference.

[0089] Balancing inference speed and detail reconstruction: It combines the efficiency of low-resolution denoising with the detail recovery capability of super-resolution reconstruction. While significantly compressing latency through low-resolution inference, it effectively restores high-frequency details and textures of the image using the super-resolution module, solving the blurring problem that is easily caused by low-resolution generation, and achieving a unity of "computational scale compression" and "visual quality preservation".

[0090] Figure 4 A training apparatus 400 for an image generation model according to an exemplary embodiment of the present disclosure is shown. The apparatus 400 may include a prompt text acquisition unit 401, a first noisy image acquisition unit 402, a first output representation acquisition unit 403, a downsampling unit 404, a second noisy image acquisition unit 405, a target image generation model acquisition unit 406, a second output representation acquisition unit 407, a loss determination unit 408, and a training unit 409. The prompt text acquisition unit 401 may be configured to acquire sample prompt text. The first noisy image acquisition unit 402 may be configured to acquire a first noisy image corresponding to the dimension of the target image to be generated. The first output representation acquisition unit 403 may be configured to perform an image generation process including multi-step denoising inference using a trained reference image generation model based on the sample prompt text and the first noisy image to obtain a first output representation. The downsampling unit 404 may be configured to downsample the first output representation based on a predetermined spatial scaling ratio to obtain a downsampled output representation. The second noisy image acquisition unit 405 can be configured to acquire a second noisy image corresponding to the first noisy image, such that the size of the second noisy image is smaller than the size of the first noisy image, and the size of the second noisy image and the size of the first noisy image follow the predetermined spatial scaling ratio. The target image generation model acquisition unit 406 can be configured to acquire a target image generation model. The second output representation acquisition unit 407 can be configured to perform an image generation process including multi-step denoising inference using the target image generation model based on the sample prompt text and the second noisy image, to acquire a second output representation. The loss determination unit 408 can be configured to determine a loss value based on the second output representation and the downsampled output representation. The training unit 409 can be configured to adjust the parameters of the target image generation model based on the determined loss value to acquire a trained target image generation model, such that the trained target image generation model can perform multi-step denoising inference at a spatial resolution lower than that of the reference image generation model.

[0091] Figure 5An image generation apparatus 500 according to an exemplary embodiment of the present disclosure is shown. The apparatus 500 may include a text acquisition unit 501, a low-resolution result acquisition unit 502, and an output image acquisition unit 503. The text acquisition unit 501 may be configured to acquire prompt text describing the image to be generated. The low-resolution result acquisition unit 502 may be configured to perform an image generation process including multi-step denoising inference based on a latent space noise image and the prompt text, using a trained target image generation model obtained according to the model training method of the present disclosure, to obtain a low-resolution generated result with a resolution lower than the target resolution. The output image acquisition unit 503 may be configured to enlarge and reconstruct the low-resolution generated result to the target resolution using a super-resolution reconstruction module, thereby obtaining an output image.

[0092] It is understood that the various modifications and benefits described in connection with the method embodiments in this disclosure are equally applicable to the corresponding device embodiments, and will not be repeated here.

[0093] According to embodiments of the present disclosure, a computer device is also provided, the computer device comprising: at least one processor; and a memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method described according to embodiments of the present disclosure.

[0094] According to embodiments of the present disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, causes the processor to perform the method described according to embodiments of the present disclosure.

[0095] According to embodiments of the present disclosure, a computer program product is also provided, the computer program product including a computer program that, when executed by a processor, causes the processor to perform the method described according to embodiments of the present disclosure.

[0096] Figure 6 An example computer device 600 is shown in which any of the embodiments described herein may be implemented. The computer device 600 may be used to implement one or more components of the systems and methods described above. The computer device 600 may include a bus 602 or other communication mechanism for communicating information, and one or more processors 604 coupled to the bus 602 for processing information. The processor 604 may be, for example, one or more general-purpose microprocessors.

[0097] Computer device 600 may also include main memory 606, such as random access memory (RAM), cache, and / or other dynamic storage devices, coupled to bus 602, for storing information and instructions to be executed by processor 604. Main memory 606 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 604. Such instructions, when stored in a storage medium accessible to processor 604, can make computer device 600 a special-purpose machine customized to perform the operations specified in the instructions. Main memory 606 may include non-volatile media and / or volatile media. Non-volatile media may include, for example, optical discs or magnetic disks. Volatile media may include dynamic memory. Common media formats may include, for example, floppy disks, collapsible disks, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage media, CD-ROMs (read-only optical disc drives), any other optical data storage media, any physical media with a perforated arrangement, RAM (random access memory), DRAM (dynamic random access memory), PROM (programmable read-only memory) and EPROM (erasable programmable read-only memory), FLASH-EPROM (fast erase programmable read-only memory), NVRAM (non-volatile random access memory), any other memory chips or tape cartridges, or network versions of the above.

[0098] Computer device 600 may implement the techniques described herein using custom hardwired logic, one or more ASICs (Application-Specific Integrated Circuits) or FPGAs (Field-Programmable Gate Arrays), firmware, and / or program logic, which, when combined with computer device 600, enable computer device 600 to become a special-purpose machine or to be programmed therein. According to one embodiment, the techniques herein are executed by computer device 600 in response to processor 604 executing one or more sequences of one or more instructions contained in main memory 606. Such instructions may be read into main memory 606 from another storage medium, such as storage device 608. Executing the sequence of instructions contained in main memory 606 causes processor 604 to perform the processing steps described herein. For example, the processes / methods disclosed herein may be implemented by computer program instructions stored in main memory 606. When these instructions are executed by processor 604, they may perform the steps shown in the corresponding figures and as described above. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0099] Computer device 600 also includes a network interface 610 coupled to bus 602. Network interface 610 can provide bidirectional data communication coupled to one or more network links connected to one or more networks. As another example, network interface 610 can be a local area network (LAN) card to provide data communication connectivity with a compatible LAN (or a WAN component communicating with a WAN (wide area network)). Wireless links can also be implemented.

[0100] The performance of certain operations can be distributed across processors, not just residing within a single machine, but deployed across many machines. In some exemplary embodiments, the processor or the processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other exemplary embodiments, the processor or the processor-implemented engine may be distributed across many geographic locations.

[0101] Each process, method, and algorithm described in the preceding sections can be embodied in a code module executed by one or more computer systems or computer processors including computer hardware, and can be fully or partially automated by them. These processes and algorithms can be implemented, in part or in whole, in a specific application circuit.

[0102] When the functions disclosed herein are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, can be embodied in the form of a software product. This software product can be stored in a storage medium and includes instructions to cause a computer device (which may be a personal computer, server, network device, etc.) to perform all or part of the steps of the methods described in the embodiments of this application. The storage medium may include a flash drive, a portable hard drive, ROM, RAM, a magnetic disk, an optical disk, another medium suitable for storing program code, or any combination thereof.

[0103] The embodiments disclosed herein can be implemented via a cloud platform, server, or group of servers that interact with a client. The client can be a terminal device or a client registered by a user on the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can install platform applications.

[0104] The various features and processes described above can be used independently or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Furthermore, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in a non-specifically disclosed order, or multiple blocks or states may be combined in a single block or state. Exemplary blocks or states may be executed serially, in parallel, or otherwise. Blocks or states may be added to or removed from the disclosed exemplary embodiments. The exemplary systems and components described herein may be configured differently from those described. For example, elements may be added, removed, or rearranged compared to the disclosed exemplary embodiments.

[0105] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. An algorithm may consist of program code or instructions stored in memory (such as the non-transitory computer-readable storage medium described above). Such an algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not be explicitly programmed into the computer to perform the function, but may learn from training data to obtain a predictive model for performing that function.

[0106] The various operations of the exemplary methods described herein can be performed at least in part by one or more processors, which are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute the engine of a processor implementation whose operation is to perform one or more of the operations or functions described herein.

[0107] Similarly, the methods described herein can be implemented at least partially by a processor, where a specific processor or one or more processors are examples of hardware. For example, at least some operations of the methods can be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors can also run in a “cloud computing” environment or as “Software as a Service” (SaaS) to support the execution of the relevant operations. For example, at least some operations can be performed by a group of computers (as an example of a machine including processors), which can be accessed via a network (e.g., the Internet) and through one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0108] The performance of certain operations can be distributed across processors, not just residing within a single machine, but deployed across many machines. In some exemplary embodiments, the processor or the processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other exemplary embodiments, the processor or the processor-implemented engine may be distributed across many geographic locations.

[0109] In this specification, multiple instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are described and illustrated as independent operations, one or more individual operations may be performed concurrently, and these operations are not required to be performed in the order shown. Structures and functionalities presented as independent components in the example configuration may be implemented as combined structures or components. Similarly, structures and functionalities presented as individual components may be implemented as independent components. These and other variations, modifications, additions, and improvements are all within the scope of this document.

[0110] As used herein, “or” is inclusive rather than exclusive unless explicitly stated or indicated by context. Furthermore, “and” is both common and individual unless explicitly stated or indicated by context. Moreover, multiple instances may be provided for the resources, operations, or structures described herein as a single example. Furthermore, the boundaries between various resources, operations, engines, and data stores are somewhat arbitrary, and specific operations are illustrated within the context of a particular illustrative configuration. The allocation of other functionalities is conceivable and may fall within the scope of various embodiments of this disclosure. Generally, structures and functionalities presented as independent resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functionalities presented as individual resources may be implemented as independent resources. These and other variations, modifications, additions, and improvements are all within the scope of embodiments of this disclosure. Therefore, this specification and accompanying drawings should be viewed in an illustrative rather than restrictive sense.

[0111] The terms “comprising” or “including” are used to indicate the presence of a subsequently stated feature, but do not preclude the addition of other features. Conditional language, in particular, such as “may,” “can,” or “may,” unless specifically stated or otherwise understood in the context of use, is generally intended to express that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Therefore, such conditional language generally does not imply that a feature, element, and / or step is necessary in any way for one or more embodiments, or that one or more embodiments must include logic that, with or without user input or prompting, determines whether such features, elements, and / or steps are included in any particular embodiment, or whether they are to be performed in any particular embodiment.

Claims

1. A training method for an image generation model, characterized in that, The method includes: Obtain sample prompt text; Obtain a first noise image corresponding to the dimension of the target image to be generated; Based on the sample prompt text and the first noisy image, an image generation process including multi-step denoising inference is performed through a trained reference image generation model to obtain a first output representation; The first output representation is downsampled based on a predetermined spatial scaling ratio to obtain a downsampled output representation. A second noise image corresponding to the first noise image is obtained such that the size of the second noise image is smaller than the size of the first noise image, and the size of the second noise image and the size of the first noise image follow the predetermined spatial scaling ratio; Obtain the target image generation model; Based on the sample prompt text and the second noisy image, an image generation process including multi-step denoising inference is performed by the target image generation model to obtain a second output representation; The loss value is determined based on the second output representation and the downsampled output representation; and By adjusting the parameters of the target image generation model based on the determined loss value, a trained target image generation model is obtained, enabling the trained target image generation model to perform multi-step denoising inference at a spatial resolution lower than that of the reference image generation model.

2. The method according to claim 1, characterized in that, The first output representation is a latent space representation obtained by performing multi-step denoising inference on the first noisy image, and the second output representation is a latent space representation obtained by performing multi-step denoising inference on the second noisy image.

3. The method according to claim 1, characterized in that, The image generation process further includes a decoding operation. The first output representation is a pixel space representation obtained by performing multi-step denoising inference and decoding on the first noisy image, and the second output representation is a pixel space representation obtained by performing multi-step denoising inference and decoding on the second noisy image.

4. The method according to any one of claims 1-3, characterized in that, The reference image generation model includes a first inference denoising module, and the target image generation model includes a second inference denoising module, each of the first inference denoising module and the second inference denoising module being selected from the group consisting of: UNet network, Transformer skeleton network.

5. The method according to any one of claims 1-3, characterized in that, Obtaining a second noise image corresponding to the first noise image includes downsampling the first noise image based on the predetermined spatial scaling ratio to obtain the second noise image.

6. The method according to any one of claims 1-3, characterized in that, The first noise image and the second noise image correspond to at least one of the same noise schedule and the same random seed.

7. The method according to any one of claims 1-3, characterized in that, The multi-step denoising inference performed by the reference image generation model and the multi-step denoising inference performed by the target image generation model are carried out under the same control conditions.

8. The method according to any one of claims 1-3, characterized in that, Downsampling the first output representation based on a predetermined spatial scaling ratio includes downsampling the first output representation by a factor of 2 in spatial dimension.

9. An image generation method, characterized in that, The method includes: Obtain the prompt text describing the image to be generated; Based on the latent space noise image and the prompt text, an image generation process including multi-step denoising inference is performed using a target image generation model trained according to any one of claims 1-8 to obtain a low-resolution generation result with a resolution lower than the target resolution; and The low-resolution generated result is enlarged and reconstructed to the target resolution using a super-resolution reconstruction module, thereby obtaining the output image.

10. The method according to claim 9, characterized in that, The latent space noise image is obtained by determining the resolution of the latent space noise image based on the target resolution and the scaling ratio of the super-resolution reconstruction module to generate the latent space noise image.

11. A training device for an image generation model, characterized in that, The device includes: The prompt text acquisition unit is configured to acquire sample prompt text; The first noise image acquisition unit is configured to acquire a first noise image corresponding to the dimension of the target image to be generated; The first output representation acquisition unit is configured to perform an image generation process including multi-step denoising inference through a trained reference image generation model based on the sample prompt text and the first noisy image to obtain the first output representation. The downsampling unit is configured to downsample the first output representation based on a predetermined spatial scaling ratio to obtain a downsampled output representation. The second noise image acquisition unit is configured to acquire a second noise image corresponding to the first noise image, such that the size of the second noise image is smaller than the size of the first noise image, and the size of the second noise image and the size of the first noise image follow the predetermined spatial scaling ratio; The target image generation model acquisition unit is configured to acquire the target image generation model; The second output representation acquisition unit is configured to perform an image generation process including multi-step denoising inference through the target image generation model based on the sample prompt text and the second noise image to obtain the second output representation. The loss determination unit is configured to determine a loss value based on the second output representation and the downsampled output representation; and The training unit is configured to obtain a trained target image generation model by adjusting the parameters of the target image generation model based on a determined loss value, such that the trained target image generation model can perform multi-step denoising inference at a spatial resolution lower than that of the reference image generation model.

12. An image generation apparatus, characterized in that, The device includes: The text acquisition unit is configured to acquire prompt text describing the image to be generated; A low-resolution result acquisition unit is configured to perform an image generation process including multi-step denoising inference using a target image generation model trained according to any one of claims 1-8, based on a latent space noise image and the prompt text, to obtain a low-resolution generation result with a resolution lower than the target resolution; and The output image acquisition unit is configured to enlarge and reconstruct the low-resolution generation result to the target resolution through a super-resolution reconstruction module, thereby obtaining an output image.

13. A computer device, characterized in that, The computer device includes: At least one processor; A memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method of any one of claims 1-10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method according to any one of claims 1-10.

15. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-10.

Images