Image generation method and device, model training method and device, storage medium and computing device

Abstract

The invention provides an image generation method and device, a model training method and device, a storage medium and a computing device. The image generation method comprises the following steps: acquiring a feature map of an input image; the feature map and a coordinate map are spliced, a splicing result is obtained, and the coordinate map is used for providing position information of feature points in the feature map; based on the splicing result, utilizing a machine learning model to predict noise in the feature map; de-noising the feature map based on the predicted noise to obtain a de-noised feature map; and generating a target image based on the de-noised feature map, thereby generating a variable-size image of which the resolution and the aspect ratio are not limited by flexibly providing coordinate information for the feature map, and improving the effect of the generated variable-size image.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology. More specifically, the present disclosure relates to an image generation method and device, a training method and device for a machine learning model, a machine learning model, a computer-readable storage medium, and a computing device. Background Art

[0002] Current image generation methods can be divided into two categories, one is based on generative adversarial networks (GANs); the other is based on diffusion models. Models based on generative adversarial networks show less image diversity and are difficult to train. Diffusion models have made significant progress in recent years, especially pure transformer architectures that show good scalability. However, they struggle to generate images of arbitrary resolution with good extrapolation capabilities. For example, the diffusion transformer has good scalability but can only generate images with 512×512 and 256×256 resolutions. Diffusion model-based methods can use NaViT or Bucket to support dynamic sizes. However, for generating images larger than those seen during training, the model cannot implement extrapolation during inference time. Summary of the invention

[0003] The exemplary embodiments of the present disclosure provide an image generation method and device, a training method and device for a machine learning model, a machine learning model, a computer-readable storage medium, and a computing device to improve the extrapolation capability of the machine learning model and realize the generation of variable-size images with unrestricted resolution and aspect ratio.

[0004] According to an exemplary embodiment of the present disclosure, there is provided an image generation method, comprising: acquiring a feature map of an input image; splicing the feature map with a coordinate map to obtain a splicing result, wherein the coordinate map is used to provide position information of feature points in the feature map; based on the splicing result, predicting the noise in the feature map using a machine learning model; denoising the feature map based on the predicted noise to obtain a denoised feature map; and generating a target image based on the denoised feature map.

[0005] Optionally, predicting the noise in the feature map using a machine learning model based on the splicing result may include: determining at least one word-gram of the feature map based on the splicing result; obtaining at least one output word-gram by performing attention calculation on the at least one word-gram; and predicting the noise in the feature map by inputting the at least one output word-gram into the machine learning model.

[0006] Optionally, obtaining at least one output word-gram by performing attention calculation on the at least one word-gram may include: mapping each word-gram in the at least one word-gram to a query, a key, and a value respectively; performing attention calculation on the at least one word-gram based on the query, the key, and the value to obtain an attention calculation result word-gram; and mapping the attention calculation result word-gram to the at least one output word-gram respectively.

[0007] Optionally, predicting the noise in the feature map by inputting the at least one output word into the machine learning model may include: predicting a first noise in the feature map based on the at least one output word; determining a first denoised feature map based on the first noise and the feature map; enlarging the first denoised feature map to a predetermined size to obtain a second denoised feature map; adding real noise to the second denoised feature map to obtain a new feature map; using the first denoised feature map as a guide to determine a second noise in the new feature map, and using the second noise as the noise in the feature map.

[0008] Optionally, generating a target image based on the denoised feature map may include: decoding the denoised feature map into the target image through a variational decoder.

[0009] According to an exemplary embodiment of the present disclosure, a training method for a machine learning model is provided, comprising: obtaining a noisy feature map of a training image in a training image set, wherein the noisy feature map is obtained by adding real noise to the feature map of the training image; splicing the noisy feature map with a coordinate map to obtain a splicing result, wherein the coordinate map is used to provide position information of feature points in the noisy feature map; based on the splicing result, predicting the noise in the noisy feature map using the machine learning model; and adjusting parameters of the machine learning model based on the predicted noise and the real noise until the model converges.

[0010] Optionally, predicting the noise in the noisy feature map using the machine learning model based on the splicing result may include: determining at least one word-gram of the noisy feature map based on the splicing result; obtaining at least one output word-gram by performing attention calculation on the at least one word-gram; and predicting the noise in the noisy feature map by inputting the at least one output word-gram into the machine learning model.

[0011] Optionally, obtaining at least one output word-gram by performing attention calculation on the at least one word-gram may include: mapping each word-gram in the at least one word-gram to a query, a key, and a value respectively; performing attention calculation on the at least one word-gram based on the query, the key, and the value to obtain an attention calculation result word-gram; and mapping the attention calculation result word-gram to the at least one output word-gram respectively.

[0012] Optionally, predicting the noise in the noisy feature map by inputting the at least one output word into the machine learning model may include: predicting a first noise in the noisy feature map based on the at least one output word; determining a first denoised feature map based on the first noise and the noisy feature map; enlarging the first denoised feature map to a predetermined size to obtain a second denoised feature map; adding real noise to the second denoised feature map to obtain a new noisy feature map; using the first denoised feature map as a guide to determine a second noise in the new noisy feature map, and using the second noise as the noise in the noisy feature map.

[0013] Optionally, obtaining the noisy feature map of the training images in the training image set may include: in each iteration of the machine learning model training, selecting a training image group from the training image set, wherein the training image set includes a first group of training images and a second group of training images; sampling the images in the training image group to obtain the training images, wherein the aspect ratio of the first group of training images is within a first range, the aspect ratio of the second group of training images is within a second range, and the first range is greater than the second range.

[0014] Optionally, obtaining the noisy feature map of the training image in the training image set may include: processing the training image; extracting the noisy feature map based on the processed training image, wherein the processing includes: under a first probability, center-cropping the training image into a block of a predetermined size; under a second probability, while maintaining the aspect ratio, re-adjusting the height of the training image to a predetermined length, based on the minimum width among the widths of the training images, center-cropping other training images in the training image whose widths are not the minimum width along the width direction, and adjusting the size of the training image to be less than or equal to a threshold.

[0015] According to an exemplary embodiment of the present disclosure, there is provided an image generating apparatus, comprising: a feature acquiring unit, configured to acquire a feature map of an input image; a coordinate stitching unit, configured to stitch the feature map with a coordinate map to obtain a stitching result, wherein the coordinate map is used to provide position information of feature points in the feature map; a noise predicting unit, configured to predict the noise in the feature map using a machine learning model based on the stitching result; a feature denoising unit, configured to denoise the feature map based on the predicted noise to obtain a denoised feature map; and an image generating unit, configured to generate a target image based on the denoised feature map.

[0016] Optionally, the noise prediction unit can be configured to: determine at least one word-gram of the feature graph based on the splicing result; obtain at least one output word-gram by performing attention calculation on the at least one word-gram; and predict the noise in the feature graph by inputting the at least one output word-gram into the machine learning model.

[0017] Optionally, the noise prediction unit can be configured to: map each word in the at least one word-gram to a query, a key and a value respectively; perform attention calculation on the at least one word-gram based on the query, the key and the value to obtain an attention calculation result word-gram; and map the attention calculation result word-gram to the at least one output word-gram respectively.

[0018] Optionally, the noise prediction unit can be configured to: predict a first noise in the feature map based on the at least one output word; determine a first denoised feature map based on the first noise and the feature map; enlarge the first denoised feature map to a predetermined size to obtain a second denoised feature map; add real noise to the second denoised feature map to obtain a new feature map; use the first denoised feature map as a guide to determine the second noise in the new feature map, and use the second noise as the noise in the feature map.

[0019] Optionally, the image generating unit may be configured to: decode the denoised feature map into the target image through a variational decoder.

[0020] According to an exemplary embodiment of the present disclosure, there is provided a training device for a machine learning model, comprising: a noisy feature acquisition unit, configured to acquire a noisy feature map of a training image in a training image set, wherein the noisy feature map is obtained by adding real noise to the feature map of the training image; a coordinate stitching unit, configured to stitch the noisy feature map with a coordinate map to obtain a stitching result, wherein the coordinate map is used to provide position information of feature points in the noisy feature map; a noise prediction unit, configured to predict the noise in the noisy feature map using the machine learning model based on the stitching result; and a parameter adjustment unit, configured to adjust the parameters of the machine learning model based on the predicted noise and the real noise until the model converges.

[0021] Optionally, the noise prediction unit can be configured to: determine at least one word-gram of the noisy feature map based on the splicing result; obtain at least one output word-gram by performing attention calculation on the at least one word-gram; and predict the noise in the noisy feature map by inputting the at least one output word-gram into the machine learning model.

[0022] Optionally, the noise prediction unit can be configured to: map each word in the at least one word-gram to a query, a key and a value respectively; perform attention calculation on the at least one word-gram based on the query, the key and the value to obtain an attention calculation result word-gram; and map the attention calculation result word-gram to the at least one output word-gram respectively.

[0023] Optionally, the noise prediction unit can be configured to: predict the first noise in the noisy feature map based on the at least one output word; determine a first denoised feature map based on the first noise and the noisy feature map; enlarge the first denoised feature map to a predetermined size to obtain a second denoised feature map; add real noise to the second denoised feature map to obtain a new noisy feature map; use the first denoised feature map as a guide to determine the second noise in the new noisy feature map, and use the second noise as the noise in the noisy feature map.

[0024] Optionally, the noisy feature acquisition unit can be configured to: in each iteration of the machine learning model training, select a training image group from the training image set, wherein the training image set includes a first group of training images and a second group of training images; sample the images in the training image group to obtain the training images, wherein the aspect ratio of the first group of training images is in a first range, the aspect ratio of the second group of training images is in a second range, and the first range is greater than the second range.

[0025] Optionally, the noisy feature acquisition unit can be configured to: process the training image; extract the noisy feature map based on the processed training image, wherein the processing includes: under a first probability, center-cropping the training image into a block of a predetermined size; under a second probability, while maintaining the aspect ratio, re-adjusting the height of the training image to a predetermined length, based on the minimum width among the widths of the training images, center-cropping other training images in the training image whose width is not the minimum width along the width direction, and adjusting the size of the training image to be less than or equal to a threshold.

[0026] According to an exemplary embodiment of the present disclosure, there is provided an image generation device model, comprising: an input embedding module and a diffusion transformer module, wherein the input embedding module and the diffusion transformer module are connected, wherein the input embedding module is configured to obtain a feature map of an input image, splice the feature map with a coordinate map to obtain a splicing result, wherein the coordinate map is used to provide position information of feature points in the feature map, and the diffusion transformer module is configured to predict noise in the feature map based on the splicing result using a machine learning model, denoise the feature map based on the predicted noise to obtain a denoised feature map, and generate a target image based on the denoised feature map.

[0027] Optionally, the diffusion transformer module can be configured to: determine at least one word-gram of the feature map based on the splicing result; obtain at least one output word-gram by performing attention calculation on the at least one word-gram; and predict the noise in the feature map by inputting the at least one output word-gram into the machine learning model.

[0028] Optionally, the diffusion transformer module can be configured to: map each word in the at least one word to a query, a key and a value respectively; perform attention calculation on the at least one word based on the query, the key and the value to obtain an attention calculation result word; and map the attention calculation result word to the at least one output word respectively.

[0029] Optionally, the diffusion transformer module can be configured to: predict a first noise in the feature map based on the at least one output word; determine a first denoised feature map based on the first noise and the feature map; enlarge the first denoised feature map to a predetermined size to obtain a second denoised feature map; add real noise to the second denoised feature map to obtain a new feature map; and use the first noise as a guide to determine the second noise in the new feature map as the noise in the feature map.

[0030] Optionally, the diffusion transformer module may be configured to: decode the denoised feature map into the target image via a variational decoder.

[0031] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, an image generation method and a training method of a machine learning model according to an exemplary embodiment of the present disclosure are implemented.

[0032] According to an exemplary embodiment of the present disclosure, there is provided a computing device, comprising: at least one processor; and at least one memory storing a computer program, wherein when the computer program is executed by the at least one processor, an image generation method and a machine learning model training method according to an exemplary embodiment of the present disclosure are implemented.

[0033] According to an exemplary embodiment of the present disclosure, a computer program product is provided, and instructions in the computer program product can be executed by a processor of a computer device to complete an image generation method and a training method of a machine learning model according to an exemplary embodiment of the present disclosure.

[0034] According to the image generation method and device and the machine learning model of the exemplary embodiments of the present disclosure, a feature map of an input image is obtained, and the feature map is spliced ​​with a coordinate map to obtain a splicing result, wherein the coordinate map is used to provide position information of feature points in the feature map. Based on the splicing result, the noise in the feature map is predicted using a machine learning model, and the feature map is denoised based on the predicted noise to obtain a denoised feature map. A target image is generated based on the denoised feature map, thereby generating a variable-size image with unrestricted resolution and aspect ratio by flexibly providing coordinate information for the feature map, thereby improving the effect of the generated variable-size image.

[0035] According to the training method and device of the machine learning model of the exemplary embodiment of the present disclosure, a noisy feature map of a training image in a training image set is obtained, wherein the noisy feature map is obtained by adding real noise to the feature map of the training image, and the noisy feature map is spliced ​​with a coordinate map to obtain a splicing result, wherein the coordinate map is used to provide position information of feature points in the noisy feature map, and based on the splicing result, the machine learning model is used to predict the noise in the noisy feature map, and the parameters of the machine learning model are adjusted based on the predicted noise and the real noise until the model converges, thereby improving the extrapolation ability of the machine learning model by flexibly providing coordinate information for the noisy feature map, achieving the generation of variable-size images with unrestricted resolution and aspect ratio by the machine learning model, and improving the effect of variable-size images generated by the machine learning model.

[0036] Additional aspects and / or advantages of the present general inventive concept will be set forth in part in the following description and in part will be apparent from the description or may be learned through practice of the present general inventive concept. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and other objects and features of the exemplary embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings which exemplarily illustrate the embodiments, in which:

[0038] Figure 1 A flowchart showing an image generating method according to an exemplary embodiment of the present disclosure;

[0039] Figure 2 A schematic diagram showing a two-stage denoising process according to an exemplary embodiment of the present disclosure;

[0040] Figure 3 A flowchart showing a method for training a machine learning model according to an exemplary embodiment of the present disclosure;

[0041] Figure 4A schematic diagram showing a machine learning model according to an exemplary embodiment of the present disclosure;

[0042] Figure 5 A schematic diagram showing an input embedding module 41 according to an exemplary embodiment of the present disclosure;

[0043] Figure 6 A schematic diagram showing a self-attention module in a diffusion transformer module 42 according to an exemplary embodiment of the present disclosure;

[0044] Figure 7 A block diagram showing an image generating apparatus according to an exemplary embodiment of the present disclosure;

[0045] Figure 8 A block diagram illustrating a training apparatus for a machine learning model according to an exemplary embodiment of the present disclosure; and

[0046] Fig. 9 A schematic diagram illustrating a computing device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] Reference will now be made in detail to exemplary embodiments of the present disclosure, examples of which are shown in the accompanying drawings, wherein like reference numerals refer to like parts throughout. The embodiments will be described below with reference to the drawings in order to explain the present disclosure.

[0048] Generating high-resolution images with dynamic sizes is popular in many applications. However, using high-resolution training data to train models is expensive. (1) It is difficult to collect a large number of high-resolution images; (2) high-resolution images cause the transformer to need to calculate a large number of labels, which is computationally expensive or even impossible.

[0049] The Diffusion Transformer (DiT) model is a model for image and video generation. It gradually introduces noise and trains a neural network to reverse the process of noise increase. Combined with the Transformer model, it realizes the generation and transformation of images or videos. The workflow of the DiT model includes the following steps:

[0050] Data preprocessing: Convert the input image or video data into a format that the model can process, such as dividing the image into small patches of fixed size and then converting these patches into feature vectors;

[0051] Noise introduction: gradually introduce noise into the preprocessed feature vector to form a diffusion process of noise increase;

[0052] Model training: The Diffusion Transformer model is trained using the feature vectors with noise introduced. The goal of the model is to learn how to reverse the process of noise increase, that is, to recover the original data from the noisy data.

[0053] Image or video generation: After the model training is completed, you can generate new images or videos by inputting noise data (or randomly generated noise) into the model and processing it.

[0054] In this disclosure, a transformer-based diffusion model with better extrapolation capability is proposed as an improved diffusion transformer model (iDiT) for generating resizable images. The transformer-based diffusion model in this disclosure has the following characteristics:

[0055] 1) The improved diffusion transformer model has better extrapolation capability and can generate variable-size images with unrestricted resolution and aspect ratio;

[0056] 2) Ability to achieve better extrapolation at inference time, generating images with higher resolution than the training data (for example, but not limited to, generating images with 4 times higher resolution than the training data).

[0057] Therefore, the transformer-based diffusion model proposed in the present disclosure does not need to use high-resolution images as training images to train the model, and low-resolution images can be used as training images to train the model. In addition, the transformer-based diffusion model proposed in the present disclosure can also use high-resolution images as training images to train the model, so as to achieve better extrapolation during inference, generate images with higher resolution than the training data, and thus improve the resolution of the image that can be generated.

[0058] Figure 1 A flowchart showing an image generating method according to an exemplary embodiment of the present disclosure is shown. Figure 2 A schematic diagram showing a two-stage denoising process according to an exemplary embodiment of the present disclosure.

[0059] Reference Figure 1 In step S101, a feature map of an input image is obtained. Here, the feature map may be a potential feature map.

[0060] In step S102, the feature map and the coordinate map are spliced ​​to obtain a splicing result. Here, the coordinate map is used to provide position information of feature points in the feature map. The size of the coordinate map can be the same as the size of the noisy feature map. In addition, the coordinate map can be a normalized coordinate map to facilitate coordinate calculation.

[0061] In the present disclosure, a coordinate map of the same size as the feature map is designed, and the coordinates in the coordinate map can be normalized to [0, 1], for example but not limited to. Then, the feature map and the coordinate map are connected and spliced. Since the coordinate map corresponds to the feature map, the coordinate map can flexibly provide position (coordinate) information for the feature points in the feature map, so that the resolution of the generated image is not limited. That is, the image generation method in the present disclosure can be used to flexibly generate images with high resolution.

[0062] In step S103, based on the splicing result, the noise in the feature map is predicted using a machine learning model. Here, the machine learning model can be trained by the machine learning model training method in the present disclosure.

[0063] In an exemplary embodiment of the present disclosure, based on the splicing result, predicting the noise in the feature map using a machine learning model may include: determining at least one word-gram of the feature map based on the splicing result; obtaining at least one output word-gram by performing attention calculation on the at least one word-gram; predicting the noise in the feature map by inputting the at least one output word-gram into the machine learning model, thereby avoiding embedding each word-gram separately and improving the speed of obtaining the word-gram. In the present disclosure, the attention calculation may be a self-attention calculation.

[0064] In an exemplary embodiment of the present disclosure, obtaining at least one output word-gram by performing attention calculation on the at least one word-gram may include: mapping each word-gram in the at least one word-gram to a query, a key, and a value, respectively; performing attention calculation on the at least one word-gram based on the query, the key, and the value to obtain an attention calculation result word-gram; and mapping the attention calculation result word-gram to the at least one output word-gram, respectively.

[0065] In an exemplary embodiment of the present disclosure, predicting the noise in the feature map by inputting the at least one output word-gram into the machine learning model may include: predicting a first noise in the feature map based on the at least one output word-gram; determining a first denoised feature map based on the first noise and the feature map; enlarging the first denoised feature map to a predetermined size to obtain a second denoised feature map; adding real noise to the second denoised feature map to obtain a new feature map; using the first denoised feature map as a guide to determine a second noise in the new feature map, and using the second noise as the noise in the feature map.

[0066] In step S104, the feature map is denoised based on the predicted noise to obtain a denoised feature map.

[0067] In step S105, a target image is generated based on the denoising feature map.

[0068] In an exemplary embodiment of the present disclosure, generating a target image based on the denoised feature map may include: decoding the denoised feature map into the target image through a variational decoder.

[0069] As an example, Figure 2 As shown, 512 2 For example, image generation is given by training hw≤256 2 The entire example flow of the image generation method using a machine learning model to perform a two-stage denoising process:

[0070] 1) Given a requested image size (h, w), calculate the size of the feature map (e.g., potential feature map) as (H2, W2) = ceil(h / 8, w / 8), where h represents the height of the image, w represents the width of the image, H2 represents the height of the feature map (e.g., potential feature map), and W2 represents the width of the feature map (e.g., potential feature map);

[0071] 2) Reduce the feature map size (H2, W2) to (H 1 ,W 1 ), where H 1 W 1 ≤256 2 , (H 1 ,W 1 )=(16,16)), where H1 represents the height of the feature map (e.g., the potential feature map) after being reduced, and W1 represents the width of the feature map (e.g., the potential feature map) after being reduced;

[0072] 3) If Figure 2 As shown, the whole denoising step is divided into two stages. In the first stage (T1 step), the white noise x is defined T ~N(0,1), size H 1 ×W 1 ×4;

[0073] 4) For t=T,...T 1 , based on being trained to hw≤256 2 A machine learning model (e.g., a trained machine learning model or an improved diffusion transformer model in the machine learning model) predicts the noise, and denoises the feature map according to the predicted noise to obtain a denoised feature map x t-1 (That is, the denoised feature maps are obtained in turn ), until the guided feature map

[0074] 5) From the guided feature map Prediction guide feature map (For example, the diffusion transformer model can be described in the same way as in the related diffusion transformer model calculate ), which will serve as the second stage (T 2 Steps) basis;

[0075] 6) In the second stage, the guide feature map is upsampled by the nearest neighbor The size is adjusted to (H 2 ,W 2 ), and obtain the feature map

[0076] 7) Add noise to the feature map (T2+T1=T), the noisy feature map of the required size is obtained (For example, the noisy feature map can be calculated in the same way as in the correlated diffusion transformer model );

[0077] 8) Use the same trained for hw≤256 2 The machine learning model is Perform denoising, here, guide feature map is used as a conditional guide to avoid uncontrolled pattern generation, for t = T2, ..., 1, 8.1) predicting feature maps 8.2) Calculate y t The mean μ(y t ) and variance σ, 8.3) calculation To replace y t The mean μ(y t ), 8.4) by formula y t-1 ~N(μ(y t ),σ) to determine the feature map y t-1 (For example, y can be determined in the same way as in the related diffusion transformer model t-1 );

[0078] 9) Get the enlarged size H 2 ×W 2 The feature map (latent feature map) y 0 ;

[0079] 10) The feature map (latent feature map) y is transformed through the variational decoder 0 Decode into a target image (for example, but not limited to, an RGB image).

[0080] According to the image generation method of the exemplary embodiment of the present disclosure, by flexibly providing coordinate information for the feature map, the extrapolation capability of the machine learning model used to generate the image can be improved, thereby realizing the generation of variable-size images with unrestricted resolution and aspect ratio. Figure 3A flowchart of a method for training a machine learning model according to an exemplary embodiment of the present disclosure is shown. Figure 3 The machine learning model in may be a model including an improved diffusion transformer model for generating variable-size images with high resolution and unrestricted resolution and aspect ratio.

[0081] Reference Figure 3 In step S301, a noisy feature map of a training image in a training image set is obtained. Here, the noisy feature map is obtained by adding real noise to the feature map of the training image. In the present disclosure, the noisy feature map may be a noisy potential feature map. In the present disclosure, the real noise may be ε~(0,1).

[0082] In an exemplary embodiment of the present disclosure, obtaining a noisy feature map of a training image in a training image set may include: in each iteration of the machine learning model training, selecting a training image group from the training image set, wherein the training image set includes a first group of training images and a second group of training images; sampling images in the training image group to obtain the training images, wherein the aspect ratio (h / w) of the first group of training images is in a first range, the aspect ratio of the second group of training images is in a second range, and the first range is greater than the second range, so that the machine learning model can support the generation of images of different sizes. In this way, the machine learning model can support the generation of images of different aspect ratios or images with unlimited aspect ratios, for example.

[0083] In an exemplary embodiment of the present disclosure, obtaining a noisy feature map of a training image in a training image set may include: processing the training image; extracting the noisy feature map based on the processed training image, wherein the processing includes: center cropping the training image into a block of a predetermined size under a first probability; under a second probability, while maintaining the aspect ratio, re-adjusting the height (e.g., the long side) of the training image to a predetermined length, center cropping other training images in the training image whose width is not the minimum width along the width direction based on the minimum width among the widths (e.g., the short sides) of the training image, and adjusting the size of the training image to be less than or equal to a threshold. In this way, the machine learning model can support the generation of images of different sizes. When the size of the training image is adjusted to be less than or equal to a threshold, for example, but not limited to, the height-width product of the training image is adjusted to be less than or equal to the height-width product threshold. Here, the first probability and the second probability refer to the probabilities of executing the first processing method and the second processing method, respectively. For example, the first probability may be about 30%, such as, but not limited to 30%, 25%, 20%, etc., and the first probability may be about 70%, such as, but not limited to 70%, 75%, 80%, etc. For example, when processing the training images, there is a 30% probability of center cropping each training image in the training images into a block of a predetermined size; there is a 70% probability of re-adjusting the height of each training image in the training images to a predetermined length while maintaining the aspect ratio, center cropping other training images in the training images whose width is not the minimum width along the width direction based on the minimum width of all the training images in the training images, and adjusting the size of each training image in the training images to be less than or equal to a threshold.

[0084] As an example, in order to enable the machine learning model to support the generation of images of different sizes, the following training image loading strategy is designed in this disclosure:

[0085] 1) The entire training image set is divided into two groups: the first group of training images (h / w ≥ 1) and the second group of training images (h / w < 1), where h represents the height of the training image and w represents the width of the training image;

[0086] 2) In each iteration of the machine learning model training, a group of training images is randomly selected from the first group of training images and the second group of training images as a sampling group for sampling;

[0087] 3) Randomly select N images from the sampling group, where N is determined based on a predetermined batch size;

[0088] 4) With a probability of, for example but not limited to, 30%, the centers of the N images are cropped into blocks of a predetermined size (for example, but not limited to, a square);

[0089] 5) with a probability of, for example, but not limited to, 70%, resizing the heights (for example, the long sides) of the N images to, for example, but not limited to, 512 while maintaining the aspect ratio;

[0090] 6) Determine the image with the smallest width among the N images, and perform center cropping on the other images among the N images except the image with the smallest width along their width direction;

[0091] 7) Adjust the size of the N images to be less than or equal to a predetermined size threshold. For example, but not limited to, adjust the height-width product of the N images to hw≤256 2 .

[0092] In step S302, the noisy feature map is spliced ​​with the coordinate map to obtain a splicing result. Here, the coordinate map is used to provide position information of feature points in the noisy feature map. The size of the coordinate map can be the same as the size of the noisy feature map. In addition, the coordinate map can be a normalized coordinate map.

[0093] In the present disclosure, a coordinate map of the same size as the noisy feature map is designed, and the coordinates in the coordinate map can be normalized to [0,1]. Then, the noisy feature map and the coordinate map are connected and spliced. Since the coordinate map corresponds to the noisy feature map, the coordinate map can flexibly provide position (coordinate) information for the feature points in the noisy feature map, so that the resolution of the generated image is not limited. In other words, the machine learning model in the present disclosure can be used to flexibly generate images with high resolution.

[0094] In step S303, based on the splicing result, the machine learning model is used to predict the noise in the noisy feature map.

[0095] In an exemplary embodiment of the present disclosure, based on the splicing result, predicting the noise in the noisy feature map using the machine learning model may include: determining at least one token of the noisy feature map based on the splicing result; obtaining at least one output token by performing attention calculation on the at least one token; predicting the noise in the noisy feature map by inputting the at least one output token into the machine learning model, thereby avoiding embedding each token separately and improving the speed of obtaining tokens. In the present disclosure, the attention calculation may be a self-attention calculation.

[0096] In an exemplary embodiment of the present disclosure, obtaining at least one output word-gram by performing attention calculation on the at least one word-gram may include: mapping each word-gram in the at least one word-gram to a query, a key, and a value, respectively; performing attention calculation on the at least one word-gram based on the query, the key, and the value to obtain an attention calculation result word-gram; and mapping the attention calculation result word-gram to the at least one output word-gram, respectively.

[0097] The original diffusion transformer DiT model uses linear projection to calculate queries, keys, and values. In the machine learning model of the present disclosure (i.e., the improved diffusion transformer model), a convolutional layer with padding is used to replace the point-to-point linear projection used by the multi-head self-attention in the original diffusion transformer DiT model to incorporate neighborhood information. Such a design can enhance the extrapolation capability of the machine learning model and improve the ability of the machine learning model to generate higher resolution images. In order to reduce the size of parameters, a deep separable convolutional layer can be used in the machine learning model of the present disclosure. Specifically, for example, the at least one word is first reshaped into a two-dimensional word graph, and then the at least one word is mapped to a query, key, and value using a convolutional mapping layer, and then attention calculation is performed, and finally the attention calculation result is reshaped into a two-dimensional shape, and another convolutional mapping layer is used to obtain the at least one output word.

[0098] The attention mechanism is a model that controls the distribution of attention to make the query vector pay more attention to specific elements in the input vector sequence, thereby better processing information. The example steps of attention calculation can be as follows:

[0099] Input processing: Convert the input feature matrix (X) into a (Q, K, V) matrix through a fully connected layer;

[0100] Matrix transformation: Calculate the dot product of (Q) and (K) to get the attention score matrix;

[0101] Attention score calculation: perform softmax operation on the score to get the attention weight;

[0102] Output generation: Weighted summation (V) matrix according to attention weights to get the final output.

[0103] In an exemplary embodiment of the present disclosure, predicting the noise in the noisy feature map by inputting the at least one output word-gram into the machine learning model may include: predicting a first noise in the noisy feature map based on the at least one output word-gram; determining a first denoised feature map based on the first noise and the noisy feature map; enlarging the first denoised feature map to a predetermined size to obtain a second denoised feature map; adding real noise to the second denoised feature map to obtain a new noisy feature map; using the first denoised feature map as a guide to determine a second noise in the new noisy feature map, and using the second noise as the noise in the noisy feature map.

[0104] As an example, the training process of the present disclosure is described in more detail below, enabling the machine learning model to generate higher resolution images.

[0105] like Figure 2 As shown, the whole denoising step is divided into two stages. In the first stage (T 1 Step), use the machine learning model to generate a size (H 1 ×W 1 ) Specifically, initialize a white noise x T ~N(0,1), for t=T,...T 1 , based on the machine learning model to predict the noise, the feature map is denoised according to the predicted noise to obtain the denoised feature map x t-1 (That is, the denoised feature maps are obtained in turn ), until the guided feature map After T 1 After step denoising, the obtained guided feature map Will contain semantic information. From the guided feature map Prediction guide feature map (For example, the diffusion transformer model can be described in the same way as in the related diffusion transformer model calculate ), which will serve as the second stage (T 2 In the second stage, the feature map is guided (for example, but not limited to, by nearest neighbor upsampling) Enlarge to the desired feature map size (e.g., H 2 ×W 2 ), and obtain the feature map And add noise of the same size (for example, Gaussian noise, white noise) to obtain a noisy feature map of the required size (For example, the noisy feature map can be calculated in the same way as in the correlated diffusion transformer model ). Then, the machine learning model is used to extract the noisy feature map Start denoising until t = 1, where the guide feature map is used as a conditional guide to avoid uncontrolled pattern generation. For t = T 2 ,...,1,First predict the feature map Then calculate y t The mean μ(y t ) and variance σ, calculate To replace y t The mean μ(y t ), and finally through the formula y t-1 ~Ν(μ(y t ),σ) to determine the feature map y t- (For example, y can be determined in the same way as in the related diffusion transformer model t-1 ). The final enlarged size is H 2 ×W 2 The final feature map (final latent feature map) y 0 Here, H 2 Represents the final feature map y 0 Height, W 2 Represents the final feature map y 0 Here, s represents the parameters of the machine learning model, represents the noise of the current prediction, is the guide feature map, The role of is based on the difference between the current predicted feature map and the guided feature map To adjust y t The mean μ(y t ), Indicates the difference The gradient of the currently predicted feature map, that is, based on the gradient descent method, makes the currently predicted feature map close to the guide feature map, for example, makes the currently predicted feature map more similar to the guide feature map.

[0106] In step S304, the parameters of the machine learning model are adjusted based on the predicted noise and the actual noise until the model converges.

[0107] As an example, given a batch of training images, hw≤256 2 ; A variational auto-encoder (VAE) may be used to extract a feature map (e.g., a latent feature map), and the size of the feature map (e.g., a latent feature map) is hw≤32 2 ; Add noise ε~(0,1) to the feature map (e.g., potential feature map); Use the cluster learning model (iDiT model) in the present disclosure to predict the noise; 5) Based on The gradient descent method is used to train the model until convergence and obtain the model parameters. Here, ∈ represents white noise, represents the noise predicted by the machine learning model, θ represents the parameters of the machine learning model, represents the gradient of the parameter θ.

[0108] According to the training method of the machine learning model of the exemplary embodiment of the present disclosure, by flexibly providing coordinate information for the noisy feature map, the extrapolation capability of the machine learning model is improved, and the generation of variable-size images with unrestricted resolution and aspect ratio is achieved.

[0109] Figure 4 A schematic diagram showing a machine learning model according to an exemplary embodiment of the present disclosure. Figure 5 A schematic diagram of an input embedding module 41 according to an exemplary embodiment of the present disclosure is shown. Figure 6 A schematic diagram showing a self-attention module in the diffusion transformer module 42 according to an exemplary embodiment of the present disclosure.

[0110] Reference Figure 4 , the machine learning model includes an input embedding module 41 and a diffusion transformer module 42, and the input embedding module 41 and the diffusion transformer module 42 are connected.

[0111] The input embedding module 41 is configured to obtain a feature map of an input image, and to splice the feature map with a coordinate map to obtain a splicing result, wherein the coordinate map is used to provide position information of feature points in the feature map. Figure 5 As shown, a convolution layer with padding can be used in the input embedding module 41 to avoid embedding each word separately. The padding operation in the convolution layer with padding can provide relative position information and encode context information, thereby improving the extrapolation ability of the machine learning model and improving the ability to generate higher resolution images.

[0112] The diffusion transformer module 42 is configured to predict the noise in the feature map based on the stitching result using a machine learning model, denoise the feature map based on the predicted noise to obtain a denoised feature map, and generate a target image based on the denoised feature map.

[0113] In an exemplary embodiment of the present disclosure, the diffusion transformer module 42 may be configured to: determine at least one word-element of the feature graph based on the concatenation result; obtain at least one output word-element by performing attention calculation on the at least one word-element; and predict the noise in the feature graph by inputting the at least one output word-element into the machine learning model. Figure 6The self-attention module shown first reshapes the at least one token into a two-dimensional token graph, and then uses a convolutional mapping layer to map the at least one token into a query, a key, and a value, respectively, and then performs attention calculation, and finally reshapes the attention calculation result into a two-dimensional shape, and uses another convolutional mapping layer to obtain the at least one output token.

[0114] In an exemplary embodiment of the present disclosure, the diffusion transformer module 42 can be configured to: map each word in the at least one word-gram to a query, a key and a value respectively; perform attention calculation on the at least one word-gram based on the query, the key and the value to obtain an attention calculation result word-gram; and map the attention calculation result word-gram to the at least one output word-gram respectively.

[0115] In an exemplary embodiment of the present disclosure, the diffusion transformer module 42 can be configured to: predict the first noise in the feature map based on the at least one output word; determine a first denoised feature map based on the first noise and the feature map; enlarge the first denoised feature map to a predetermined size to obtain a second denoised feature map; add real noise to the second denoised feature map to obtain a new feature map; use the first denoised feature map as a guide to determine the second noise in the new feature map, and use the second noise as the noise in the feature map.

[0116] In an exemplary embodiment of the present disclosure, the diffusion transformer module 42 may be configured to decode the denoised feature map into the target image through a variational decoder.

[0117] The above has been combined Figures 1 to 6 The training method of the machine learning model, the image generation method, and the machine learning model according to the exemplary embodiment of the present disclosure are described. Figure 7 and Figure 8 A training device and units of a machine learning model, and an image generating device and units thereof according to an exemplary embodiment of the present disclosure are described.

[0118] Figure 7 A block diagram of a training apparatus for a machine learning model according to an exemplary embodiment of the present disclosure is shown.

[0119] Reference Figure 7 The training device of the machine learning model includes a noisy feature acquisition unit 71, a coordinate splicing unit 72, a noise prediction unit 73 and a parameter adjustment unit 74.

[0120] The noisy feature acquisition unit 71 is configured to acquire a noisy feature map of a training image in a training image set, wherein the noisy feature map is obtained by adding real noise to the feature map of the training image.

[0121] In an exemplary embodiment of the present disclosure, the noisy feature acquisition unit 71 may be configured to: in each iteration of the machine learning model training, select a training image group from the training image set, wherein the training image set includes a first group of training images and a second group of training images; sample the images in the training image group to obtain the training images, wherein the aspect ratio of the first group of training images is in a first range, the aspect ratio of the second group of training images is in a second range, and the first range is greater than the second range.

[0122] In an exemplary embodiment of the present disclosure, the noisy feature acquisition unit 71 may be configured to: process the training image; extract the noisy feature map based on the processed training image, wherein the processing includes: under a first probability, center-cropping the training image into a block of a predetermined size; under a second probability, while maintaining the aspect ratio, re-adjusting the height of the training image to a predetermined length, center-cropping other training images in the training image whose width is not the minimum width along the width direction based on the minimum width of the widths of the training images, and adjusting the size of the training image to be less than or equal to a threshold. When adjusting the size of the training image to be less than or equal to the threshold, for example, but not limited to, adjusting the height-width product of the training image to be less than or equal to the height-width product threshold.

[0123] The coordinate stitching unit 72 is configured to stitch the noisy feature map with the coordinate map to obtain a stitching result, wherein the coordinate map is used to provide position information of feature points in the noisy feature map.

[0124] The noise prediction unit 73 is configured to predict the noise in the noisy feature map based on the splicing result using the machine learning model.

[0125] In an exemplary embodiment of the present disclosure, the noise prediction unit 73 may be configured to: determine at least one word-gram of the noisy feature map based on the splicing result; obtain at least one output word-gram by performing attention calculation on the at least one word-gram; and predict the noise in the noisy feature map by inputting the at least one output word-gram into the machine learning model.

[0126] In an exemplary embodiment of the present disclosure, the noise prediction unit 73 may be configured to: map each word-gram in the at least one word-gram to a query, a key, and a value, respectively; perform attention calculation on the at least one word-gram based on the query, the key, and the value to obtain an attention calculation result word-gram; and map the attention calculation result word-gram to the at least one output word-gram, respectively.

[0127] In an exemplary embodiment of the present disclosure, the noise prediction unit 73 may be configured to: predict the first noise in the noisy feature map based on the at least one output word; determine a first denoised feature map based on the first noise and the noisy feature map; enlarge the first denoised feature map to a predetermined size to obtain a second denoised feature map; add real noise to the second denoised feature map to obtain a new noisy feature map; use the first denoised feature map as a guide to determine the second noise in the new noisy feature map, and use the second noise as the noise in the noisy feature map.

[0128] The parameter adjustment unit 74 is configured to adjust the parameters of the machine learning model based on the predicted noise and the actual noise until the model converges.

[0129] Figure 8 A block diagram illustrating an image generating apparatus according to an exemplary embodiment of the present disclosure.

[0130] Reference Figure 8 The image generating device includes a feature acquiring unit 81, a coordinate stitching unit 82, a noise predicting unit 83, a feature denoising unit 84 and an image generating unit 85.

[0131] The feature acquisition unit 81 is configured to acquire a feature map of an input image.

[0132] The coordinate stitching unit 82 is configured to stitch the feature map with the coordinate map to obtain a stitching result, wherein the coordinate map is used to provide position information of feature points in the feature map.

[0133] The noise prediction unit 83 is configured to predict the noise in the feature map using a machine learning model based on the splicing result.

[0134] In an exemplary embodiment of the present disclosure, the noise prediction unit 83 may be configured to: determine at least one word-gram of the feature graph based on the splicing result; obtain at least one output word-gram by performing attention calculation on the at least one word-gram; and predict the noise in the feature graph by inputting the at least one output word-gram into the machine learning model.

[0135] In an exemplary embodiment of the present disclosure, the noise prediction unit 83 may be configured to: map each word-gram in the at least one word-gram to a query, a key, and a value, respectively; perform attention calculation on the at least one word-gram based on the query, the key, and the value to obtain an attention calculation result word-gram; and map the attention calculation result word-gram to the at least one output word-gram, respectively.

[0136] In an exemplary embodiment of the present disclosure, the noise prediction unit 83 may be configured to: predict a first noise in the feature map based on the at least one output word; determine a first denoised feature map based on the first noise and the feature map; enlarge the first denoised feature map to a predetermined size to obtain a second denoised feature map; add real noise to the second denoised feature map to obtain a new feature map; use the first denoised feature map as a guide to determine the second noise in the new feature map, and use the second noise as the noise in the feature map.

[0137] The feature denoising unit 84 is configured to denoise the feature map based on the predicted noise to obtain a denoised feature map.

[0138] The image generating unit 85 is configured to generate a target image based on the denoised feature map.

[0139] In an exemplary embodiment of the present disclosure, the image generation unit 85 may be configured to decode the denoised feature map into the target image through a variational decoder.

[0140] In addition, according to an exemplary embodiment of the present disclosure, there is also provided a computer-readable storage medium on which a computer program is stored. When the computer program is executed, the image generating method according to the exemplary embodiment of the present disclosure is implemented.

[0141] In an exemplary embodiment of the present disclosure, the computer-readable storage medium may carry one or more programs, and when the computer program is executed, the following steps can be implemented: obtaining a noisy feature map of a training image in a training image set, wherein the noisy feature map is obtained by adding real noise to the feature map of the training image; splicing the noisy feature map with a coordinate map to obtain a splicing result, wherein the coordinate map is used to provide position information of feature points in the noisy feature map; based on the splicing result, using the machine learning model to predict the noise in the noisy feature map; and adjusting the parameters of the machine learning model based on the predicted noise and the real noise until the model converges.

[0142] In an exemplary embodiment of the present disclosure, the computer-readable storage medium may carry one or more programs, and when the computer program is executed, the following steps can be implemented: obtaining a feature map of an input image; splicing the feature map with a coordinate map to obtain a splicing result, wherein the coordinate map is used to provide position information of feature points in the feature map; based on the splicing result, predicting the noise in the feature map using a machine learning model; denoising the feature map based on the predicted noise to obtain a denoised feature map; and generating a target image based on the denoised feature map.

[0143] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In an embodiment of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a computer program that may be used by or in conjunction with an instruction execution system, device or device. The computer program contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof. The computer-readable storage medium may be contained in any device; it may also exist alone without being assembled into the device.

[0144] In addition, according to an exemplary embodiment of the present disclosure, a computer program product is also provided, and instructions in the computer program product can be executed by a processor of a computer device to complete the image generation method according to the exemplary embodiment of the present disclosure.

[0145] The above has been combined Figure 7 and Figure 8 The image generating apparatus according to the exemplary embodiment of the present disclosure is described. Fig. 9 A computing device according to an exemplary embodiment of the present disclosure is described.

[0146] Fig. 9 A schematic diagram illustrating a computing device according to an exemplary embodiment of the present disclosure.

[0147] Reference Fig. 9 According to an exemplary embodiment of the present disclosure, a computing device 9 includes a memory 91 and a processor 92. The memory 91 stores a computer program. When the computer program is executed by the processor 92, the image generating method according to the exemplary embodiment of the present disclosure is implemented.

[0148] In an exemplary embodiment of the present disclosure, when the computer program is executed by the processor 92, the following steps can be implemented: obtaining a noisy feature map of a training image in a training image set, wherein the noisy feature map is obtained by adding real noise to the feature map of the training image; splicing the noisy feature map with a coordinate map to obtain a splicing result, wherein the coordinate map is used to provide position information of feature points in the noisy feature map; based on the splicing result, predicting the noise in the noisy feature map using the machine learning model; and adjusting the parameters of the machine learning model based on the predicted noise and the real noise until the model converges.

[0149] In an exemplary embodiment of the present disclosure, when the computer program is executed by the processor 92, the following steps can be implemented: obtaining a feature map of an input image; splicing the feature map with a coordinate map to obtain a splicing result, wherein the coordinate map is used to provide position information of feature points in the feature map; based on the splicing result, predicting the noise in the feature map using a machine learning model; denoising the feature map based on the predicted noise to obtain a denoised feature map; and generating a target image based on the denoised feature map.

[0150] The computing device in the embodiments of the present disclosure may include, but is not limited to, devices such as mobile phones, laptop computers, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, and the like. Fig. 9 The computing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0151] The above has been referred to Figures 1 to 9 The training method and apparatus of the machine learning model, the image generation method and apparatus, and the machine learning model according to the exemplary embodiments of the present disclosure are described. However, it should be understood that: Figure 7 and Figure 8 The training device and units of the machine learning model, the image generation device and units thereof shown in the figure may be respectively configured as software, hardware, firmware or any combination of the above items to perform specific functions. Fig. 9 The computing device shown in is not limited to including the components shown above, but some components may be added or deleted as needed, and the above components may also be combined.

[0152] According to the image generation method and device and the machine learning model of the exemplary embodiments of the present disclosure, a feature map of an input image is obtained, and the feature map is spliced ​​with a coordinate map to obtain a splicing result, wherein the coordinate map is used to provide position information of feature points in the feature map. Based on the splicing result, the noise in the feature map is predicted using a machine learning model, and the feature map is denoised based on the predicted noise to obtain a denoised feature map. A target image is generated based on the denoised feature map, thereby generating a variable-size image with unrestricted resolution and aspect ratio by flexibly providing coordinate information for the feature map, thereby improving the effect of the generated variable-size image.

[0153] According to the training method and device of the machine learning model of the exemplary embodiment of the present disclosure, a noisy feature map of a training image in a training image set is obtained, wherein the noisy feature map is obtained by adding real noise to the feature map of the training image, and the noisy feature map is spliced ​​with a coordinate map to obtain a splicing result, wherein the coordinate map is used to provide position information of feature points in the noisy feature map, and based on the splicing result, the machine learning model is used to predict the noise in the noisy feature map, and the parameters of the machine learning model are adjusted based on the predicted noise and the real noise until the model converges, thereby improving the extrapolation ability of the machine learning model by flexibly providing coordinate information for the noisy feature map, achieving the generation of variable-size images with unrestricted resolution and aspect ratio by the machine learning model, and improving the effect of variable-size images generated by the machine learning model.

[0154] While the present disclosure has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the following claims.

Claims

1. A method for generating an image, comprising: Get the feature map of the input image; Splicing the feature map with the coordinate map to obtain a splicing result, wherein the coordinate map is used to provide position information of feature points in the feature map; Based on the splicing result, using a machine learning model to predict the noise in the feature map; Denoising the feature map based on the predicted noise to obtain a denoised feature map; and A target image is generated based on the denoised feature map.

2. The method according to claim 1, wherein: The method of predicting the noise in the feature map based on the splicing result by using a machine learning model includes: Determine at least one word element of the feature graph based on the splicing result; Obtain at least one output word-unit by performing attention calculation on the at least one word-unit; Noise in the feature map is predicted by inputting the at least one output word-gram into the machine learning model.

3. The method according to claim 2, wherein: The step of obtaining at least one output word-unit by performing attention calculation on the at least one word-unit comprises: Mapping each word-gram of the at least one word-gram to a query, a key, and a value, respectively; Performing attention calculation on the at least one word-element based on the query, the key, and the value to obtain an attention calculation result word-element; The attention calculation result word-grams are respectively mapped to the at least one output word-gram.

4. The method according to claim 2, wherein: The predicting the noise in the feature map by inputting the at least one output word into the machine learning model comprises: predicting a first noise in the feature map based on the at least one output word-gram; Determining a first denoising feature map based on the first noise and the feature map; Enlarging the first denoised feature map to a predetermined size to obtain a second denoised feature map; Add real noise to the second denoised feature map to obtain a new feature map; The first denoised feature map is used as a guide to determine the second noise in the new feature map, and the second noise is used as the noise in the feature map.

5. The method according to claim 1, wherein: The generating a target image based on the denoising feature map comprises: The denoised feature map is decoded into the target image through a variational decoder.

6. A method for training a machine learning model, comprising: Obtaining a noisy feature map of a training image in a training image set, wherein the noisy feature map is obtained by adding real noise to the feature map of the training image; Splicing the noisy feature map with a coordinate map to obtain a splicing result, wherein the coordinate map is used to provide position information of feature points in the noisy feature map; Based on the splicing result, using the machine learning model to predict the noise in the noisy feature map; and The parameters of the machine learning model are adjusted based on the predicted noise and the actual noise until the model converges.

7. The method according to claim 6, wherein: The method of predicting the noise in the noisy feature graph by using the machine learning model based on the splicing result includes: Determine at least one word element of the noisy feature graph based on the splicing result; Obtain at least one output word-unit by performing attention calculation on the at least one word-unit; Noise in the noisy feature map is predicted by inputting the at least one output word-gram into the machine learning model.

8. The method according to claim 7, wherein: The step of obtaining at least one output word-unit by performing attention calculation on the at least one word-unit comprises: Mapping each word-gram of the at least one word-gram to a query, a key, and a value, respectively; Performing attention calculation on the at least one word-element based on the query, the key, and the value to obtain an attention calculation result word-element; The attention calculation result word-grams are respectively mapped to the at least one output word-gram.

9. The method according to claim 7, wherein: The step of predicting the noise in the noisy feature graph by inputting the at least one output word into the machine learning model comprises: predicting a first noise in the noisy feature graph based on the at least one output word-gram; Determine a first denoised feature map based on the first noise and the noisy feature map; Enlarging the first denoised feature map to a predetermined size to obtain a second denoised feature map; Add real noise to the second denoised feature map to obtain a new noisy feature map; The first denoised feature map is used as a guide to determine the second noise in the new noisy feature map, and the second noise is used as the noise in the noisy feature map.

10. The method according to claim 6, wherein: The step of obtaining a noisy feature map of a training image in a training image set includes: In each iteration of the machine learning model training, selecting a training image group from the training image set, wherein the training image set includes a first group of training images and a second group of training images; Sampling the images in the training image group to obtain the training images, The aspect ratio of the first group of training images is within a first range, the aspect ratio of the second group of training images is within a second range, and the first range is greater than the second range.

11. The method according to claim 6, wherein: The step of obtaining a noisy feature map of a training image in a training image set includes: Processing the training image; Extracting the noisy feature map based on the processed training image, The processing includes: under a first probability, center-cropping the training image into a block of a predetermined size; under a second probability, while maintaining the aspect ratio, re-adjusting the height of the training image to a predetermined length, based on the minimum width among the widths of the training images, center-cropping other training images in the training image whose width is not the minimum width along the width direction, and adjusting the size of the training image to be less than or equal to a threshold.

12. An image generating device, comprising: A feature acquisition unit, configured to acquire a feature map of an input image; A coordinate splicing unit is configured to splice the feature map with the coordinate map to obtain a splicing result, wherein the coordinate map is used to provide position information of feature points in the feature map; A noise prediction unit, configured to predict the noise in the feature map using a machine learning model based on the splicing result; a feature denoising unit, configured to denoise the feature map based on the predicted noise to obtain a denoised feature map; and An image generating unit is configured to generate a target image based on the denoising feature map.

13. A machine learning model comprising: An input embedding module and a diffusion transformer module, wherein the input embedding module and the diffusion transformer module are connected, wherein: The input embedding module is configured to obtain a feature map of an input image, and to splice the feature map with a coordinate map to obtain a splicing result, wherein the coordinate map is used to provide position information of feature points in the feature map, and The diffusion transformer module is configured to predict the noise in the feature map based on the splicing result using a machine learning model, denoise the feature map based on the predicted noise to obtain a denoised feature map, and generate a target image based on the denoised feature map.

14. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

15. A computing device comprising: at least one processor; At least one memory stores a computer program, and when the computer program is executed by the at least one processor, the method according to any one of claims 1 to 11 is implemented.

Images