Method and device for generating image according to input information, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a method and device for generating an image according to input information, electronic equipment and a storage medium. The method comprises the steps of receiving a request for generating an image according to input information, generating an image sequence with continuously increased picture details according to the input information, and displaying a target image when the target image in the image sequence is sequentially obtained, so that the image with fewer details is added and generated in the image generation process, and the image generation efficiency is improved. According to the method, the image with fewer details is displayed to the user from the image with fewer details to the final image with more details, and the user gradually sees the effect that the generated image becomes clear step by step from the beginning fuzzy contour in the waiting time of the image generation process, so that the experience interest of the user is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for generating an image based on input information, an apparatus for generating an image based on input information, an electronic device, and a readable storage medium. Background Technology

[0002] With the development of artificial intelligence technology, the technology of generating images from inputs (such as text, images, and speech) is becoming increasingly mature. However, because image generation models typically involve large computational scales, users often experience long waiting times during the image generation process, especially for generating clearer images with more detail, which can easily reduce user engagement and interest. Summary of the Invention

[0003] The technical problem to be solved by the embodiments of the present invention is to provide a method, apparatus, electronic device, and readable storage medium for generating images based on input information, so as to solve the technical problem that the long waiting time for users in the image generation process can easily reduce the user's interest in the experience.

[0004] To address the above problems, the present invention provides a method for generating an image based on input information, the method comprising:

[0005] Receive a request to generate an image based on the input information;

[0006] Based on the input information, an image sequence with continuously increasing image detail is generated;

[0007] When a target image is obtained sequentially from the image sequence, the target image is displayed.

[0008] Optionally, the image sequence with progressively increasing detail includes at least one intermediate image and a final image, wherein generating at least one intermediate image based on the input information includes:

[0009] The input information is input into the image generation model;

[0010] Obtain at least one intermediate latent vector of the image generation model during the denoising process;

[0011] Based on the at least one intermediate hidden vector, the corresponding at least one intermediate image is generated.

[0012] Optionally, the image generation model includes a first decoder and a second decoder, the second decoder being used to generate the final image, the parameter size of the first decoder being smaller than the parameter size of the second decoder, and the step of generating the corresponding at least one intermediate image based on the at least one intermediate latent vector includes:

[0013] For each intermediate latent vector, the intermediate latent vector is input into the first decoder for decoding to obtain the corresponding intermediate image.

[0014] Optionally, before receiving a request to generate an image based on the input information, the method further includes:

[0015] Receive training data, which includes input information samples, at least one corresponding intermediate image label, and a final image label;

[0016] The input information sample is input into the image generation model to obtain at least one intermediate image and one final image.

[0017] For each output intermediate image, the parameters in the first decoder are updated based on the difference between the output intermediate image and the corresponding intermediate image label;

[0018] Based on the difference between the output final image and the final image label, the parameters in the image generation model, excluding the first decoder, are updated to obtain the trained image generation model.

[0019] Optionally, updating the parameters in the first decoder based on the difference between the output intermediate image and the corresponding intermediate image label includes:

[0020] Based on the output intermediate image and the corresponding intermediate image label, calculate the weighted sum of reconstruction loss, perceptual loss and probability distribution difference loss to obtain the total loss;

[0021] Update the parameters in the first decoder based on the total loss.

[0022] Optionally, obtaining at least one intermediate latent vector of the image generation model in the denoising process includes:

[0023] The input information is subjected to feature extraction processing to obtain the latent features of the image;

[0024] The latent features of the image are refined;

[0025] Output at least one intermediate latent vector obtained from a preset denoising process, wherein the feature extraction and refinement processes include multiple denoising processes.

[0026] The present invention also provides an apparatus for generating an image based on input information, the apparatus comprising:

[0027] The request receiving module is used to receive requests to generate images based on input information;

[0028] The sequence generation module is used to generate an image sequence with continuously increasing image details based on the input information;

[0029] The image display module is used to display the target image when a target image is obtained sequentially from the image sequence.

[0030] Optionally, the image sequence with progressively increasing detail includes at least one intermediate image and a final image, wherein the sequence generation module includes:

[0031] An information input submodule is used to input the input information into the image generation model;

[0032] The latent vector acquisition submodule is used to acquire at least one intermediate latent vector of the image generation model in the denoising process;

[0033] An image generation submodule is used to generate the corresponding at least one intermediate image based on the at least one intermediate latent vector.

[0034] Optionally, the image generation model includes a first decoder and a second decoder, the second decoder being used to generate the final image, the parameter size of the first decoder being smaller than the parameter size of the second decoder, and the image generation submodule including:

[0035] The decoding unit is used to input the intermediate latent vector into the first decoder for decoding each intermediate latent vector to obtain the corresponding intermediate image.

[0036] Optionally, the device further includes:

[0037] The data receiving module is configured to receive training data before receiving a request to generate an image based on input information. The training data includes input information samples, at least one corresponding intermediate image label, and a final image label.

[0038] The sample input module is used to input the input information sample into the image generation model to obtain at least one intermediate image and one final image.

[0039] The first update module is used to update the parameters in the first decoder for each output intermediate image based on the difference between the output intermediate image and the corresponding intermediate image label.

[0040] The second update module is used to update the parameters in the image generation model other than the first decoder based on the difference between the output final image and the final image label, thereby obtaining the trained image generation model.

[0041] Optionally, the first update module includes:

[0042] The total loss calculation submodule is used to calculate the weighted sum of reconstruction loss, perceptual loss and probability distribution difference loss based on the output intermediate image and the corresponding intermediate image label, and obtain the total loss.

[0043] The first update submodule is used to update the parameters in the first decoder based on the total loss.

[0044] Optionally, the latent vector acquisition submodule includes:

[0045] The feature extraction unit is used to perform feature extraction processing on the input information to obtain potential image features;

[0046] A fine-tuning unit is used to perform fine-tuning on the latent features of the image;

[0047] The latent vector output unit is used to output at least one intermediate latent vector obtained by a preset number of denoising processes, wherein the feature extraction process and the refinement process include multiple denoising processes.

[0048] This invention also discloses an electronic device, characterized in that it includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0049] Memory, used to store computer programs;

[0050] The processor, when executing a program stored in memory, implements the method described above.

[0051] This invention also discloses a readable storage medium that, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform the method described above.

[0052] According to an embodiment of the present invention, by receiving a request to generate an image based on input information, an image sequence with continuously increasing detail is generated based on the input information. When a target image is obtained in the image sequence, the target image is displayed. This allows for the generation of images with less detail during the image generation process, starting with images with less detail and progressing to images with more detail. During the waiting time of the image generation process, the user can gradually see the generated image gradually become clearer from the initial blurry outline, thereby enhancing the user's experience and interest. Attached Figure Description

[0053] Figure 1 A flowchart illustrating the steps of a method for generating an image based on input information according to an embodiment of the present invention is shown.

[0054] Figure 2The image shown is an intermediate image obtained using Gaussian filtering;

[0055] Figure 3 The image shown is an intermediate image obtained using median filtering;

[0056] Figure 4 A sequence of images is shown;

[0057] Figure 5 Another image sequence is shown;

[0058] Figure 6 A flowchart illustrating the steps of a method for generating an image based on input information, according to another embodiment of the present invention, is shown.

[0059] Figure 7 A schematic diagram of the SDXL model is shown;

[0060] Figure 8 A schematic diagram of a scaled-down VAE-structured decoder is shown;

[0061] Figure 9 A schematic diagram illustrating the principle of the image generation model is shown;

[0062] Figure 10 An image sequence is shown;

[0063] Figure 11 A structural block diagram of an embodiment of an apparatus for generating an image based on input information is shown in one embodiment of the present invention;

[0064] Figure 12 A structural block diagram of an electronic device for generating an image based on input information is shown according to an exemplary embodiment. Detailed Implementation

[0065] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] The method for generating images based on input information in this embodiment of the invention can be executed by electronic devices such as smart vehicle systems and mobile terminals. For example, this method for generating images based on input information can be applied to smart vehicle systems or any other applicable hardware and software. This embodiment of the invention does not limit this.

[0067] Reference Figure 1 The diagram illustrates a flowchart of a method for generating an image based on input information according to an embodiment of the present invention, which may specifically include the following steps:

[0068] Step 101: Receive a request to generate an image based on the input information.

[0069] In this embodiment of the invention, the input information includes any form of information such as input text, images, and voice, and may specifically include any applicable information; this embodiment of the invention does not limit this.

[0070] In this embodiment of the invention, the request to generate an image based on input information refers to submitting input information and initiating an image generation request. After receiving the request to generate an image based on the input information, the electronic device can generate and display the image, or it can forward the request to a remote server, which will then generate the image and return it to the electronic device for display.

[0071] Step 102: Based on the input information, generate an image sequence with continuously increasing image details.

[0072] In this embodiment of the invention, the image generated based on the input information can be generated using a method based on artificial intelligence technology. For example, image generation based on AIGC (Artificial Intelligence Generated Content) refers to generating new images through computer algorithms and models. These images may be completely fictional, artistic creations, or modified and enhanced versions of existing images.

[0073] For example, the SDXL (Stable Diffusion XL, a model based on the "Latent Diffusion Model") algorithm is an algorithm that introduces text conditions (text features) into U-NET (a U-shaped deep learning-based convolutional neural network) to generate images based on text. Compared to the LDM (Latent Diffusion Model) algorithm, it adds a refiner module, which can refine the obtained latent features to obtain higher resolution images. Finally, the decoder of the VAE (Variational Auto-Encoder) transforms and decodes the vectors under the latent into images. VAE is a deep generative model based on an autoencoder structure. Autoencoders are widely used in dimensionality reduction and feature extraction. Their basic structure involves mapping samples to latent vectors in a low-dimensional space through an encoding process, and then restoring the latent vectors to reconstruct the samples through a decoding process.

[0074] In this embodiment of the invention, an image sequence with continuously increasing detail refers to a sequence of at least two images whose detail increases sequentially. The image sequence includes at least one intermediate image and a final image. The final image is the last image in the image sequence, i.e., the image with the most detail.

[0075] In this embodiment of the invention, the final image generation method can employ any applicable artificial intelligence-based image generation method. The process of generating the final image based on artificial intelligence technology involves continuously increasing the details in the image. Therefore, in the process of generating the final image based on artificial intelligence technology, at least one intermediate image is generated using the intermediate results of the artificial intelligence algorithm. Thus, the details of the image continuously increase from the intermediate image to the final image. Similarly, when multiple intermediate images are generated sequentially, the details of each intermediate image also continuously increase sequentially.

[0076] In the embodiments of the present invention, there are various specific ways to generate intermediate images, and the embodiments of the present invention do not limit them.

[0077] For example, in the process of generating the final image using artificial intelligence technology, image feature data, i.e., intermediate vectors, is acquired. An image is then generated based on this image feature data, and mathematical methods are used to process the image. Typically, an n×n kernel is defined, and mathematical formulas are used to assign values ​​to the parameters within the kernel. The n×n kernel is then used to filter each region of the image, for example, using Gaussian filtering and median filtering, thus obtaining an intermediate image. Figure 2 The intermediate image shown is obtained using Gaussian filtering, as follows: Figure 3 The image shown is an intermediate image obtained using median filtering.

[0078] For example, a deep learning-based method learns the latent mapping from a noisy image to a clean image, resulting in a decoder. Image feature data, i.e., intermediate vectors, are obtained during the process of generating the final image using artificial intelligence technology. This image feature data is then input into the decoder to obtain the intermediate image generated by the decoder.

[0079] Step 103: When a target image is obtained sequentially from the image sequence, the target image is displayed.

[0080] In this embodiment of the invention, target images in an image sequence are generated sequentially, and the generated target images are displayed sequentially by an electronic device, thereby realizing the process of the generated image gradually changing from having less detail to having more detail, so that the user can gradually see the generated image gradually becoming clearer from the initial blurry outline.

[0081] In this embodiment of the invention, each time a target image is obtained, the target image is displayed, thereby sequentially displaying each image in the image sequence.

[0082] For example, such as Figure 4 and Figure 5 The image sequence shown is displayed sequentially from left to right, with later images covering earlier ones.

[0083] According to an embodiment of the present invention, by receiving a request to generate an image based on input information, an image sequence with continuously increasing detail is generated based on the input information. When a target image is obtained in the image sequence, the target image is displayed. This allows for the generation of images with less detail during the image generation process, starting with images with less detail and progressing to images with more detail. During the waiting time of the image generation process, the user can gradually see the generated image gradually become clearer from the initial blurry outline, thereby enhancing the user's experience and interest.

[0084] Reference Figure 6 The diagram illustrates a flowchart of a method for generating an image based on input information according to another embodiment of the present invention, which may specifically include the following steps:

[0085] Step 201: Receive a request to generate an image based on the input information.

[0086] In this embodiment of the invention, the specific implementation of this step can be found in the description of the foregoing embodiments, and will not be repeated here.

[0087] Step 202: Input the input information into the image generation model.

[0088] In this embodiment of the invention, the image sequence with continuously increasing detail includes at least one intermediate image and a final image.

[0089] In this embodiment of the invention, the image generation model refers to a generative model capable of generating a sequence of images with continuously increasing detail based on input information. Compared to generative models that only generate the final image, this image generation model adds the generation of intermediate images.

[0090] In this embodiment of the invention, input information is input into the image generation model to begin the process of generating an image sequence based on the input information.

[0091] Step 203: Obtain at least one intermediate latent vector of the image generation model in the denoising process.

[0092] In this embodiment of the invention, the image generation model encodes the input information to obtain corresponding image feature data, also known as intermediate latent vectors. For example, the intermediate latent vectors in the SDXL model during the denoising process.

[0093] In this embodiment of the invention, the encoding includes multiple denoising processes. After a preset number of denoising processes, intermediate latent vectors are obtained. The image generation model then continues to generate the final image based on the intermediate latent vectors.

[0094] In an optional embodiment of the present invention, a specific implementation of obtaining at least one intermediate latent vector of the image generation model in the denoising process may include: performing feature extraction processing on the input information to obtain latent image features, performing fine processing on the latent image features, and outputting at least one intermediate latent vector obtained by a preset number of denoising processes, wherein the feature extraction processing and fine processing include multiple denoising processes.

[0095] The image generation model first performs feature extraction on the input information to obtain latent image features. Then, it refines these latent features to obtain the final latent image features used to generate the final image.

[0096] For example, such as Figure 7 The diagram illustrates the principle of the SDXL model, which includes key components such as the Base model, Refiner model, and VAE decoder. The Base model is the first stage of the SDXL model, primarily responsible for extracting and generating latent features from the image using modules like U-NET and VAE. This process can be understood as transforming the input text description into latent image features, providing a foundation for subsequent image generation. Inputting the Prompt (a prompt word) and Noise (a 128×128 pixel noise image) into the Base model yields the output Unrefined Latent. The Refiner model is the second stage of the SDXL model, its main task being to refine the latent image features generated by the Base model. Through modules like U-NET and VAE, the Refiner model further optimizes and adjusts the image generated by the Base model, making the generated image more consistent with the input text description. Essentially, the Refiner model performs image-to-image generation, continuously optimizing and adjusting to achieve higher quality and more realistic images. The UnrefinedLatent is input into the Refiner model for refinement, thus obtaining the Refined Latent (refined image latent features).

[0097] The aforementioned feature extraction and refinement processes include multiple denoising processes. The intermediate latent vectors obtained from a preset number of denoising processes are then output. The preset number of denoising processes can be selected according to actual needs, and this embodiment of the invention does not impose any limitations on this selection.

[0098] For example, during normal inference, the image generation model repeats the denoising process in U-NET 30 times. After the last denoising process, the latent image features are fed into the VAE-Decoder for decoding, thereby generating the final image. During these 30 denoising processes, intermediate latent vectors obtained from 3, 12, 18, or 28 denoising processes can be used.

[0099] Step 204: Generate the corresponding at least one intermediate image based on the at least one intermediate latent vector.

[0100] In this embodiment of the invention, a corresponding intermediate image can be generated based on each intermediate latent vector. Any applicable implementation method can be used, and this embodiment of the invention does not impose any limitations on this. For example, the intermediate latent vectors can be fed into the VAE decoder to generate the intermediate image.

[0101] In an optional embodiment of the present invention, the image generation model includes a first decoder and a second decoder, the second decoder being used to generate the final image, the parameter size of the first decoder being smaller than the parameter size of the second decoder, and a specific implementation of generating the corresponding at least one intermediate image based on the at least one intermediate latent vector may include: for each intermediate latent vector, inputting the intermediate latent vector into the first decoder for decoding to obtain the corresponding intermediate image.

[0102] The image generation model includes a first decoder and a second decoder.

[0103] The second decoder is used to generate the final image, for example, the decoder of a typical VAE.

[0104] The first decoder is used to generate intermediate images, and the parameter size of the first decoder is smaller than that of the second decoder, for example, a VAE decoder with reduced parameter size.

[0105] Since the second decoder of the image generation model (such as the decoder of VAE) takes a certain amount of time to decode, this method of generating intermediate images at intervals of a certain number of steps will significantly increase the time required to generate the final image, which is obviously not worth the user experience.

[0106] Since intermediate images do not necessarily require high resolution, this invention uses a first decoder (which is still a VAE decoder) with reduced parameter scale to generate intermediate images.

[0107] For example, a scaled-down VAE structure decoder such as Figure 8As shown, both the number of layers and the size of the convolutional kernels are reduced. The VAE-structured decoder consists of eight convolutional layers, with each layer having a significantly smaller number of parameters, resulting in rapid generation. During inference, due to the small number of parameters in the network structure itself, the time to generate an intermediate image is only 3 milliseconds per image, a 1000-fold reduction compared to the 3 seconds of the original VAE's second decoder. This optimizes the user experience while waiting, with virtually no increase in network latency.

[0108] For each intermediate latent vector, the intermediate latent vector is input into the first decoder for decoding, thereby obtaining the corresponding intermediate image. Since the number of parameters in the first decoder is small, the time spent generating the intermediate image is reduced, so the total time spent generating the image is almost not increased.

[0109] For example, such as Figure 9 The diagram illustrates the principle of the image generation model. A noisy image of size 128×128 is input into U-NET. U-NET performs 30 denoising processes. The latent features (128×128 pixels) obtained from the 3rd, 12th, 18th, 28th, and 30th denoising processes are then input into the VAE-Decoder to obtain the Image (intermediate and final images). Figure 4 , Figure 5 and Figure 10 The image sequence shown is an intermediate image generated from the intermediate latent vectors obtained in the 3rd, 12th, 18th, and 28th denoising processes. The main subject in the image can be clearly distinguished, and the image details are gradually increasing, which is in line with the expectations for the generation effect.

[0110] In an optional embodiment of the present invention, before receiving the request to generate an image based on input information, the method may further include: receiving training data, the training data including input information samples, at least one corresponding intermediate image label, and a final image label; inputting the input information samples into the image generation model to obtain at least one output intermediate image and a final image; for each output intermediate image, updating the parameters in the first decoder based on the difference between the output intermediate image and the corresponding intermediate image label; and updating the parameters in the image generation model other than the first decoder based on the difference between the output final image and the final image label, thereby obtaining a trained image generation model.

[0111] Compared to generative models that only generate the final image, the image generation model of this invention adds at least one intermediate image label to the training process. Thus, the training data needs to include input information samples, at least one corresponding intermediate image label, and a final image label.

[0112] The intermediate image labels are image samples with less detail than the final image labels.

[0113] During training, input information samples are fed into the image generation model to be trained, which then outputs at least one intermediate image and one final image. The parameters of the image generation model are then updated based on the output of at least one intermediate image and one final image, along with their corresponding labels. This process is iterated until a trained image generation model is obtained.

[0114] Similar to generative models that only generate the final image, after each output of the final image, the parameters in the image generation model, excluding the first decoder, are updated based on the difference between the final image and the final image label.

[0115] This invention updates the parameters in the first decoder after each intermediate image output, based on the difference between the output intermediate image and its corresponding intermediate image label. Specifically, the gradient can be calculated based on the difference between the output intermediate image and its corresponding intermediate image label, and then backpropagated to adjust the parameters in the first decoder.

[0116] After multiple iterations, a well-trained image generation model is obtained.

[0117] In an optional embodiment of the present invention, a specific implementation of updating the parameters in the first decoder based on the difference between the output intermediate image and the corresponding intermediate image label may include: calculating a weighted sum of reconstruction loss, perceptual loss and probability distribution difference loss based on the output intermediate image and the corresponding intermediate image label to obtain a total loss; and updating the parameters in the first decoder based on the total loss.

[0118] Intermediate image labels are used as accurate labels or data for training and evaluating deep learning models. They serve as the reference standard, or ground truth, for machine learning algorithms. The loss between the output intermediate images and their corresponding intermediate image labels is calculated.

[0119] Specifically, the reconstruction loss, perceptual loss, and probability distribution difference loss can be calculated based on the output intermediate image and its corresponding label. Then, the reconstruction loss, perceptual loss, and probability distribution difference loss are weighted and summed to obtain the total loss.

[0120] For example, the formula for total loss is:

[0121] loss=mse_loss+α×lpips_loss+β×kl_loss

[0122] Where α is the weight of lpips_loss, which is set to 0.1 during training, and β is the weight of kl_loss, which is set to 0.3 during training.

[0123] In this formula, the reconstruction loss (mse_loss) is the L1 loss calculated between the output of the VAE and the intermediate image labels. mse_loss is used to control the intermediate images output by the trained model so that they do not excessively alter the intermediate image labels.

[0124] The perceptual loss (lpips_loss) measures the difference between two images. This metric learns the inverse mapping from the generated intermediate image to the intermediate image label, forcing the generator to learn the inverse mapping to reconstruct the real image from the fake image, and prioritizing the perceptual similarity between them. lpips_loss and mse_loss control the similarity between images in the output image sequence of the VAE model from different directions, thereby ensuring that the images in the image sequence are as similar as possible.

[0125] Among them, probability distribution difference loss (kl_loss) is a method to measure the difference between probability distributions. During training, kl_loss is used to regularize the latent space vector (the spatial vector of latent features of an image). This regularization helps to align the encoder's output latent distribution with the prior distribution, which helps the model learn to generate a wide range of diverse representations of the data. At the same time, by limiting the complexity of the latent space, KL divergence prevents the model from overfitting to specific features of the training data, thereby improving the generalization ability of the generated results.

[0126] Step 205: Obtain the final image generated by the image generation model.

[0127] In this embodiment of the invention, after obtaining the intermediate image, it is also necessary to obtain the final image generated by the image generation model.

[0128] Step 206: When a target image is obtained sequentially from the image sequence, the target image is displayed.

[0129] In this embodiment of the invention, the specific implementation of this step can be found in the description of the foregoing embodiments, and will not be repeated here.

[0130] According to an embodiment of the present invention, by receiving a request to generate an image based on input information, the input information is input into an image generation model, at least one intermediate latent vector of the image generation model in the denoising process is obtained, at least one intermediate image is generated based on the at least one intermediate latent vector, the final image generated by the image generation model is obtained, and when a target image is obtained in the image sequence, the target image is displayed. This allows for the generation of images with less detail during the image generation process, starting with images with less detail and progressing to images with more detail. During the waiting time of the image generation process, the user can gradually see the generated image gradually become clearer from the initial blurry outline, thereby enhancing the user's experience and interest.

[0131] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0132] Reference Figure 11 The diagram illustrates a structural block diagram of an embodiment of a device for generating images based on input information, according to another embodiment of the present invention. Specifically, it may include the following modules:

[0133] The request receiving module 301 is used to receive a request to generate an image based on the input information;

[0134] The sequence generation module 302 is used to generate an image sequence with continuously increasing image details based on the input information;

[0135] The image display module 303 is used to display the target image when a target image is obtained sequentially from the image sequence.

[0136] Optionally, the image sequence with progressively increasing detail includes at least one intermediate image and a final image, wherein the sequence generation module includes:

[0137] An information input submodule is used to input the input information into the image generation model;

[0138] The latent vector acquisition submodule is used to acquire at least one intermediate latent vector of the image generation model in the denoising process;

[0139] An image generation submodule is used to generate the corresponding at least one intermediate image based on the at least one intermediate latent vector.

[0140] Optionally, the image generation model includes a first decoder and a second decoder, the second decoder being used to generate the final image, the parameter size of the first decoder being smaller than the parameter size of the second decoder, and the image generation submodule including:

[0141] The decoding unit is used to input the intermediate latent vector into the first decoder for decoding each intermediate latent vector to obtain the corresponding intermediate image.

[0142] Optionally, the device further includes:

[0143] The data receiving module is configured to receive training data before receiving a request to generate an image based on input information. The training data includes input information samples, at least one corresponding intermediate image label, and a final image label.

[0144] The sample input module is used to input the input information sample into the image generation model to obtain at least one intermediate image and one final image.

[0145] The first update module is used to update the parameters in the first decoder for each output intermediate image based on the difference between the output intermediate image and the corresponding intermediate image label.

[0146] The second update module is used to update the parameters in the image generation model other than the first decoder based on the difference between the output final image and the final image label, thereby obtaining the trained image generation model.

[0147] Optionally, the first update module includes:

[0148] The total loss calculation submodule is used to calculate the weighted sum of reconstruction loss, perceptual loss and probability distribution difference loss based on the output intermediate image and the corresponding intermediate image label, and obtain the total loss.

[0149] The first update submodule is used to update the parameters in the first decoder based on the total loss.

[0150] Optionally, the latent vector acquisition submodule includes:

[0151] The feature extraction unit is used to perform feature extraction processing on the input information to obtain potential image features;

[0152] A fine-tuning unit is used to perform fine-tuning on the latent features of the image;

[0153] The latent vector output unit is used to output at least one intermediate latent vector obtained by a preset number of denoising processes, wherein the feature extraction process and the refinement process include multiple denoising processes.

[0154] According to an embodiment of the present invention, by receiving a request to generate an image based on input information, an image sequence with continuously increasing detail is generated based on the input information. When a target image is obtained in the image sequence, the target image is displayed. This allows for the generation of images with less detail during the image generation process, starting with images with less detail and progressing to images with more detail. During the waiting time of the image generation process, the user can gradually see the generated image gradually become clearer from the initial blurry outline, thereby enhancing the user's experience and interest.

[0155] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0156] Figure 12 This is a structural block diagram illustrating an electronic device 700 for generating images based on input information, according to an exemplary embodiment. For example, the electronic device 700 may be a domain controller, an in-vehicle computer, an in-vehicle controller, a battery management system (BMS), a fully self-driving system (FSD), etc.

[0157] Reference Figure 12 The electronic device 700 may include one or more of the following components: a processing component 702, a memory 704, a power supply component 706, a multimedia component 708, an audio component 710, an input / output (I / O) interface 712, a sensor component 714, and a communication component 716.

[0158] Processing component 702 typically controls the overall operation of electronic device 700, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 702 may include one or more modules to facilitate interaction between processing component 702 and other components. For example, processing component 702 may include a multimedia module to facilitate interaction between multimedia component 708 and processing component 702.

[0159] Memory 704 is configured to store various types of data to support the operation of device 700. Examples of this data include instructions for any application or method operating on electronic device 700, contact data, phonebook data, messages, pictures, videos, etc. Memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0160] Power supply component 706 provides power to various components of electronic device 700. Power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 700.

[0161] Multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 708 includes a front-facing camera and / or a rear-facing camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0162] Audio component 710 is configured to output and / or input audio signals. For example, audio component 710 includes a microphone (MIC) configured to receive external audio signals when electronic device 700 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 704 or transmitted via communication component 716. In some embodiments, audio component 710 also includes a speaker for outputting audio signals.

[0163] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0164] Sensor assembly 714 includes one or more sensors for providing state assessments of various aspects of electronic device 700. For example, sensor assembly 714 may detect the on / off state of device 700, the relative positioning of components such as the display and keypad of electronic device 700, changes in position of electronic device 700 or a component of electronic device 700, the presence or absence of user contact with electronic device 700, orientation or acceleration / deceleration of electronic device 700, and temperature changes of electronic device 700. Sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 714 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0165] Communication component 716 is configured to facilitate wired or wireless communication between electronic device 700 and other devices. Electronic device 700 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 716 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 716 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0166] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0167] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, which can be executed by a processor 720 of an electronic device 700 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0168] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a terminal's processor, enables the terminal to perform a method for generating an image based on input information, the method comprising:

[0169] Receive a request to generate an image based on the input information;

[0170] Based on the input information, an image sequence with continuously increasing image detail is generated;

[0171] When a target image is obtained sequentially from the image sequence, the target image is displayed.

[0172] Optionally, the image sequence with progressively increasing detail includes at least one intermediate image and a final image, wherein generating at least one intermediate image based on the input information includes:

[0173] The input information is input into the image generation model;

[0174] Obtain at least one intermediate latent vector of the image generation model during the denoising process;

[0175] Based on the at least one intermediate latent vector, the corresponding at least one intermediate image is generated.

[0176] Optionally, the image generation model includes a first decoder and a second decoder, the second decoder being used to generate the final image, the parameter size of the first decoder being smaller than the parameter size of the second decoder, and the step of generating the corresponding at least one intermediate image based on the at least one intermediate latent vector includes:

[0177] For each intermediate latent vector, the intermediate latent vector is input into the first decoder for decoding to obtain the corresponding intermediate image.

[0178] Optionally, before receiving a request to generate an image based on the input information, the method further includes:

[0179] Receive training data, which includes input information samples, at least one corresponding intermediate image label, and a final image label;

[0180] The input information sample is input into the image generation model to obtain at least one intermediate image and one final image.

[0181] For each output intermediate image, the parameters in the first decoder are updated based on the difference between the output intermediate image and the corresponding intermediate image label;

[0182] Based on the difference between the output final image and the final image label, the parameters in the image generation model, excluding the first decoder, are updated to obtain the trained image generation model.

[0183] Optionally, updating the parameters in the first decoder based on the difference between the output intermediate image and the corresponding intermediate image label includes:

[0184] Based on the output intermediate image and the corresponding intermediate image label, calculate the weighted sum of reconstruction loss, perceptual loss and probability distribution difference loss to obtain the total loss;

[0185] Update the parameters in the first decoder based on the total loss.

[0186] Optionally, obtaining at least one intermediate latent vector of the image generation model in the denoising process includes:

[0187] The input information is subjected to feature extraction processing to obtain the latent features of the image;

[0188] The latent features of the image are refined;

[0189] Output at least one intermediate latent vector obtained from a preset denoising process, wherein the feature extraction and refinement processes include multiple denoising processes.

[0190] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0191] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0192] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0193] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0194] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0195] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0196] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0197] The foregoing has provided a detailed description of a method for generating an image based on input information, an apparatus for generating an image based on input information, an electronic device, and a readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for generating an image based on input information, characterized in that, The method includes: Receive a request to generate an image based on the input information; Based on the input information, an image sequence with continuously increasing image detail is generated; When a target image is obtained sequentially from the image sequence, the target image is displayed.

2. The method according to claim 1, characterized in that, The image sequence with continuously increasing detail includes at least one intermediate image and a final image, wherein generating at least one intermediate image based on the input information includes: The input information is input into the image generation model; Obtain at least one intermediate latent vector of the image generation model during the denoising process; Based on the at least one intermediate latent vector, the corresponding at least one intermediate image is generated.

3. The method according to claim 2, characterized in that, The image generation model includes a first decoder and a second decoder. The second decoder is used to generate the final image. The parameter size of the first decoder is smaller than that of the second decoder. Generating the corresponding at least one intermediate image based on the at least one intermediate latent vector includes: For each intermediate latent vector, the intermediate latent vector is input into the first decoder for decoding to obtain the corresponding intermediate image.

4. The method according to claim 3, characterized in that, Before receiving the request to generate an image based on the input information, the method further includes: Receive training data, which includes input information samples, at least one corresponding intermediate image label, and a final image label; The input information sample is input into the image generation model to obtain at least one intermediate image and one final image. For each output intermediate image, the parameters in the first decoder are updated based on the difference between the output intermediate image and the corresponding intermediate image label; Based on the difference between the output final image and the final image label, the parameters in the image generation model, excluding the first decoder, are updated to obtain the trained image generation model.

5. The method according to claim 4, characterized in that, The step of updating the parameters in the first decoder based on the difference between the output intermediate image and the corresponding intermediate image label includes: Based on the output intermediate image and the corresponding intermediate image label, calculate the weighted sum of reconstruction loss, perceptual loss and probability distribution difference loss to obtain the total loss; Update the parameters in the first decoder based on the total loss.

6. The method according to claim 2, characterized in that, Obtaining at least one intermediate latent vector of the image generation model during the denoising process includes: The input information is subjected to feature extraction processing to obtain the latent features of the image; The latent features of the image are refined; Output at least one intermediate latent vector obtained from a preset denoising process, wherein the feature extraction and refinement processes include multiple denoising processes.

7. An apparatus for generating an image based on input information, characterized in that, The device includes: The request receiving module is used to receive requests to generate images based on input information; The sequence generation module is used to generate an image sequence with continuously increasing image details based on the input information; The image display module is used to display the target image when a target image is obtained sequentially from the image sequence.

8. The apparatus according to claim 7, characterized in that, The image sequence with continuously increasing detail includes at least one intermediate image and a final image, wherein the sequence generation module includes: An information input submodule is used to input the input information into the image generation model; The latent vector acquisition submodule is used to acquire at least one intermediate latent vector of the image generation model in the denoising process; An image generation submodule is used to generate the corresponding at least one intermediate image based on the at least one intermediate latent vector.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method as described in any one of claims 1-6.

10. A readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-6.

Images