Image generation method, device, equipment, readable storage medium and program product

Abstract

The application discloses an image generation method, device, equipment, readable storage medium and program product, which are applied to scenes such as text-to-image and image-to-image. Specifically, prompt information for guiding image generation is acquired; an image generation model is acquired, the image generation model is obtained based on a predicted noise information map of a sample noise image and an uncertainty map corresponding to the predicted noise information map, the uncertainty map includes uncertainty values of pixel points in the predicted noise information map, and the uncertainty values represent the reliability of noise prediction of the pixel points in the predicted noise information map; and image generation processing is performed based on the prompt information through the image generation model to obtain a target image. The application can process local details of an image in a targeted manner and improve image quality.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to an image generation method, apparatus, device, readable storage medium, and program product. Background Technology

[0002] In graph generation tasks, with the continuous increase in training data and the expansion of AI (Artificial Intelligence) model scale, the problem of large-area generation distortions in generated images has been significantly improved. However, many small distortions still exist in the generated images. Therefore, how to improve the quality of generated images and ensure the accuracy of image details remains a significant challenge for current graph generation technology.

[0003] In related technologies, the common approach is to optimize the entire image. However, this approach is difficult to accurately and effectively identify and correct local detail defects in the image, such as distortions or texture abnormalities in local objects, which makes it difficult for the quality of the generated image to meet expectations. Summary of the Invention

[0004] This application provides an image generation method, apparatus, device, readable storage medium, and program product. The implementation of this application enables the image generation model to process local details in a targeted manner, effectively solving the problem of insufficient identification and correction of local defects in related technologies, improving detail accuracy, and enhancing image quality.

[0005] This application provides an image generation method, including:

[0006] Obtain prompts to guide the generation of the graph; An image generation model is obtained, which is trained based on a predicted noise information map of a sample noisy image and an uncertainty map corresponding to the predicted noise information map. The uncertainty map includes the uncertainty value of a pixel in the predicted noise information map, and the uncertainty value characterizes the reliability of the noise prediction for the corresponding pixel. The image generation model is used to generate the target image based on the prompt information.

[0007] Accordingly, embodiments of this application provide an image generation apparatus, including: The information acquisition module is used to acquire prompts that guide the generation of the diagram; The model acquisition module is used to acquire an image generation model. The image generation model is trained based on the predicted noise information map of the sample noisy image and the uncertainty map corresponding to the predicted noise information map. The uncertainty map includes the uncertainty value of the pixel in the predicted noise information map, and the uncertainty value characterizes the reliability of the noise prediction of the corresponding pixel. The processing module is used to perform image generation processing based on the prompt information using the image generation model to obtain the target image.

[0008] Optionally, in some embodiments of this application, the apparatus further includes: The data acquisition module is used to acquire training data, which includes sample images and sample prompt information used to guide the generation of the image; The noise-adding processing module is used to add noise to the sample image to obtain a noisy sample image; The noise prediction module is used to perform noise prediction processing on the sample noisy image based on the sample prompt information through the image generation model to obtain a predicted noise information map corresponding to the sample noisy image. The pixels in the predicted noise information map represent the noise information corresponding to the pixels at the corresponding positions in the sample noisy image. The calculation and processing module is used to calculate the prediction reliability of the noise information corresponding to the pixel in the prediction noise information map, and obtain the uncertainty map corresponding to the prediction noise information map. The parameter adjustment module is used to adjust the parameters of the image generation model based on the uncertainty map and the prediction noise information map to obtain the trained image generation model.

[0009] Optionally, in some embodiments of this application, when the parameter adjustment module is used to perform parameter adjustment on the image generation model based on the uncertainty map and the prediction noise information map to obtain the trained image generation model, it is specifically used for: Based on the uncertainty value corresponding to each pixel in the uncertainty map, the uncertainty type corresponding to each pixel in the uncertainty map is determined. Based on the uncertainty type, determine the weight information corresponding to each pixel in the uncertainty map; Based on the predicted noise information map and the real noise corresponding to the sample noisy image, the initial loss information corresponding to the predicted noise information map is determined. The initial loss information is updated based on the weight information to obtain the noise prediction loss information corresponding to the prediction noise information map; Based on the noise prediction loss information corresponding to the predicted noise information map, the parameters of the image generation model are adjusted to obtain the trained image generation model.

[0010] Optionally, in some embodiments of this application, when the noise prediction module is used to perform noise prediction processing on the sample noisy image based on the sample cue information using the image generation model to obtain the predicted noise information map corresponding to the sample noisy image, it is specifically used for: Using the image generation model, noise prediction processing is performed on the sample noisy image based on the sample prompt information to obtain the predicted noise information map of the sample noisy image in the current iteration round; Based on the predicted noise information map, the sample noisy image is denoised to obtain a denoised image; The denoised image is then identified as a new sample image with added noise. Return to the step of performing noise prediction processing on the sample noisy image based on the sample prompt information using the image generation model, until the number of iterations meets the preset condition, and obtain the predicted noise information map corresponding to each iteration.

[0011] Optionally, in some embodiments of this application, when the calculation processing module performs calculation processing on the prediction reliability of noise information corresponding to pixels in the prediction noise information map to obtain the uncertainty map corresponding to the prediction noise information map, it is specifically used for: Determine the target iteration round that satisfies the preset conditions; The prediction reliability of pixels in the prediction noise information map of the target iteration round is calculated to obtain the uncertainty map corresponding to the target iteration round.

[0012] Optionally, in some embodiments of this application, when the parameter adjustment module is used to perform parameter adjustment on the image generation model based on the uncertainty map and the prediction noise information map to obtain the trained image generation model, it is specifically used for: For the target iteration round, based on the uncertainty map and the corresponding prediction noise information map corresponding to the target iteration round, the noise prediction loss information corresponding to the target iteration round is determined; For non-target iteration rounds, based on the prediction noise information map corresponding to the non-target iteration round, the noise prediction loss information corresponding to the non-target iteration round is determined; The total noise prediction loss information is determined based on the noise prediction loss information corresponding to each target iteration round and the noise prediction loss information corresponding to each non-target iteration round. Based on the total noise prediction loss information, the parameters of the image generation model are adjusted to obtain the trained image generation model.

[0013] Optionally, in some embodiments of this application, when the calculation processing module performs calculation processing on the prediction reliability of noise information corresponding to pixels in the prediction noise information map to obtain the uncertainty map corresponding to the prediction noise information map, it is specifically used for: Obtain the posterior distribution information of the original parameters of the image generation model under multiple preset training data; Based on the posterior distribution information, the observation information matrix corresponding to the parameter changes of the image generation model is calculated; Based on the observation information matrix, calculate the parameter uncertainty value of the image generation model; Based on the uncertainty value of the parameters, the prediction reliability of the noise information corresponding to the pixels in the prediction noise information map is calculated to obtain the uncertainty map corresponding to the prediction noise information map.

[0014] Optionally, in some embodiments of this application, when the computation processing module is used to obtain the posterior distribution information corresponding to the original parameters of the image generation model under multiple preset training data, it is specifically used for: Based on a preset prior distribution function, the original parameters of the image generation model are processed to obtain the prior distribution information corresponding to the original parameters; Obtain the likelihood statistics of the original parameters of the image generation model under multiple preset training data; Based on the prior distribution information and the likelihood statistics, the posterior distribution information corresponding to the original parameters of the image generation model is calculated.

[0015] Optionally, in some embodiments of this application, when the computation processing module is used to perform the calculation of the observation information matrix of the parameter changes corresponding to the image generation model based on the posterior distribution information, it is specifically used for: Obtain the noise prediction loss information corresponding to the image generation model under the multiple preset training data; Based on the noise prediction loss information corresponding to each preset training data, the target training data is determined from the plurality of preset training data; The image generation model is updated with parameters based on the noise prediction loss information corresponding to the target training data to obtain the target parameters. Based on the posterior distribution information and the target parameters, the observation information matrix corresponding to the parameter changes of the image generation model is calculated.

[0016] Optionally, in some embodiments of this application, when the parameter adjustment module is used to determine the weight information corresponding to each pixel in the uncertainty map according to the uncertainty type, it is specifically used for: Determine at least one target pixel that satisfies the preset type condition for the uncertainty type; For the target pixel, the uncertainty region in the uncertainty map is determined based on the uncertainty type of the neighboring pixels of the target pixel; Based on the uncertainty value of the pixels in the uncertainty region, set the weight information of the pixels in the uncertainty region; For pixels in the uncertainty map that do not belong to the uncertainty region, the uncertainty value of the pixels is suppressed to determine the weight information of the pixels in the uncertainty map that do not belong to the uncertainty region.

[0017] Optionally, in some embodiments of this application, when the parameter adjustment module is used to set the weight information of pixels in the uncertainty region based on the uncertainty value of the pixels in the uncertainty region, it is specifically used for: Based on the uncertainty values ​​of the pixels in the uncertainty region, calculate the local uncertainty value of the uncertainty region; Calculate the global uncertainty value of the uncertainty map based on the uncertainty values ​​of the pixels in the uncertainty map; Based on the global uncertainty value and the local uncertainty value of the uncertainty region, the weight information of the pixels in the uncertainty region is set.

[0018] Optionally, in some embodiments of this application, when the parameter adjustment module is used to determine the uncertainty region in the uncertainty map based on the uncertainty type of the neighboring pixels of the target pixel, it is specifically used for: The uncertainty type of the neighboring pixels of the target pixel is detected, so as to select the target neighboring pixels whose uncertainty type meets the preset type condition from the neighboring pixels; Based on the target's neighboring pixels and the target pixel, the uncertainty region in the uncertainty map is determined.

[0019] Optionally, in some embodiments of this application, when the parameter adjustment module is used to set the weight information of pixels in the uncertainty region based on the global uncertainty value and the local uncertainty value of the uncertainty region, it is specifically used for: When the local uncertainty value of the uncertainty region is greater than the global uncertainty value, the weight information of the pixels in the uncertainty region is set based on the local uncertainty value of the uncertainty region. When the local uncertainty value of the uncertainty region is not greater than the global uncertainty value, the local uncertainty value of the uncertainty region is suppressed to obtain the suppressed uncertainty value, and the weight information of the pixels in the uncertainty region is set according to the suppressed uncertainty value.

[0020] Optionally, in some embodiments of this application, the noise prediction module is further configured to: before determining the denoised image as a new sample noisy image. Based on the denoised image and the expected denoised image corresponding to the sample denoised image, determine the reward feedback information for the current iteration round; Based on the reward feedback information, the image generation model is updated to obtain the updated image generation model.

[0021] Optionally, in some embodiments of this application, when the noise prediction module is used to update the image generation model based on the reward feedback information to obtain an updated image generation model, it is specifically used for: Determine the uncertainty map corresponding to the prediction noise information map of the current iteration round; Based on the uncertainty graph, the reward feedback information is updated to obtain the updated reward feedback information; Based on the updated reward feedback information, the image generation model is updated to obtain the updated image generation model.

[0022] Optionally, in some embodiments of this application, when the noise prediction module is used to update the image generation model based on the reward feedback information to obtain an updated image generation model, it is specifically used for: Based on the reward feedback information, the image generation model is updated to obtain a preliminary updated image generation model; Select the parameters to be reset from the parameters of the initially updated image generation model; The parameters to be reset are reset to obtain the updated image generation model.

[0023] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0024] An electronic device provided in this application includes a processor and a memory. The memory stores a computer program, and the processor is used to run the computer program to execute the method provided in this application.

[0025] This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the method provided in this application.

[0026] Furthermore, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0027] This application provides an image generation method, apparatus, device, readable storage medium, and program product. Specifically, after obtaining prompt information for guiding the generation of an image, an image generation model can be used to perform image generation processing based on the prompt information to obtain a target image. The image generation model is trained based on a predicted noise information map of a sample noisy image and an uncertainty map corresponding to the predicted noise information map. The uncertainty map includes the uncertainty value of a pixel in the predicted noise information map, and the uncertainty value characterizes the reliability of the noise prediction corresponding to the pixel in the predicted noise information map.

[0028] In this application, the uncertainty map can reflect the difference in the reliability of noise prediction corresponding to different pixels in the prediction noise information map. The pixel-level fine-grained supervision based on the uncertainty map enables the model to specifically correct local details with high uncertainty during training, suppress local defects, and effectively improve the problem of insufficient identification and correction of local defects in related technologies. Thus, in the image generation process, the trained image generation model can achieve accurate processing of local image details, improve the accuracy of image details, and improve the quality of generated images. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the architecture of an image generation method provided in an embodiment of this application; Figure 2 This is a flowchart of an image generation method provided in an embodiment of this application; Figure 3 This is a flowchart of another image generation method provided in the embodiments of this application; Figure 4 This is a schematic diagram of an interface provided in an embodiment of this application; Figure 5 This is another schematic diagram of the interface provided in the embodiment of this application; Figure 6 This is a schematic diagram of a sample noisy image, a predicted noise information map, and an uncertainty map provided in an embodiment of this application; Figure 7This is a schematic diagram of an uncertainty region analysis provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an image generation model provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of an image generation device provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] This application provides an image generation method, apparatus, device, readable storage medium, and program product. Specifically, the image generation apparatus can be integrated into an electronic device, such as a terminal or server.

[0033] It is understood that the image generation method of this embodiment can be executed on a terminal, on a server, or jointly by a terminal and a server. The above examples should not be construed as limiting this application.

[0034] like Figure 1 As shown, taking the joint execution of an image generation method by a terminal and a server as an example, the image generation system provided in this application embodiment includes a terminal 110 and a server 120, etc.; the terminal 110 and the server 120 are connected via a network, such as a wired or wireless network, etc., wherein the image generation device can be integrated into the server.

[0035] Server 120 can be used to: obtain prompt information for guiding the generation of the image, and perform image generation processing based on the prompt information through an image generation model to obtain the target image.

[0036] The server 120 can also be used to train an image generation model, including: acquiring training data, which includes sample images and sample prompts for guiding image generation; adding noise to the sample images to obtain noisy sample images; using the image generation model, performing noise prediction processing on the noisy sample images based on the sample prompts to obtain a predicted noise information map corresponding to the noisy sample images, where pixels in the predicted noise information map represent noise information corresponding to pixels at corresponding positions in the noisy sample images; calculating the prediction reliability of the noise information corresponding to pixels in the predicted noise information map to obtain an uncertainty map corresponding to the predicted noise information map; and adjusting the parameters of the image generation model based on the uncertainty map and the predicted noise information map to obtain the trained image generation model.

[0037] Among them, server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0038] The terminal 110 can be used to: send prompts to the server 120 to guide the generation of an image, so that the server 110 can generate a target image based on the prompts; the terminal 110 can also receive the target image sent by the server 120. The terminal 110 can include a mobile phone, tablet computer, laptop computer, desktop computer, smart TV, etc. A client can also be set on the terminal 110, which can be an application client or a browser client, etc.

[0039] In a feasible embodiment, the image generation method provided in this application can be applied to scenarios such as text-to-image and image-to-image generation. Optionally, the prompting information used to guide the generation of the image includes at least one of text information or image information.

[0040] For example, such as Figure 1 As shown, in the text-to-image scenario, the user can input text information to guide the generation of the image through the terminal 110 (such as a laptop computer), such as "generate a picture of a smiley face". The terminal 110 sends the obtained text information to the server 120. The server 120 calls the image generation model, generates the target image based on the text information, and sends the generated target image to the terminal 110.

[0041] For example, such as Figure 1 As shown, in the graph-to-graph scenario, the object being manipulated can input image information (such as...) to guide the generation of the graph via terminal 110. Figure 1 The gesture diagram shown is used by the terminal 110 to send the acquired image information to the server 120. The server 120 then calls the image generation model to generate the target image (such as...) based on the image information. Figure 1(The gesture diagram shown), and send the generated target image to terminal 110.

[0042] For example, the user can also input text and image information through terminal 110. After terminal 110 sends a prompt message including text and image information to server 120, server 120 can call an image generation model to generate a target image based on the text and image information, and send the generated target image to terminal 110.

[0043] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0044] This embodiment will be described from the perspective of an image generating device, which can be integrated into an electronic device, such as a server or a terminal.

[0045] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0046] The image generation method provided in this application embodiment can be implemented using an AI (Artificial Intelligence) model, such as the image generation model provided in this application embodiment.

[0047] Optionally, the image generation model is used to generate new images based on input prompts, and is a generative model based on AI technology. For example, the image generation model can be a diffusion model, which includes a diffusion process and a reverse diffusion process. The diffusion process is a gradual addition of noise to an image. Taking an image as an example, starting with a clear original image, noise (such as Gaussian noise) is continuously added to the image at multiple time steps. As time steps progress, the image is gradually filled with noise, eventually forming a purely noisy image or an image close to purely noisy. It can be understood that the diffusion process adopts the form of a Markov chain, such as the image state at the current time step depending on the image state at the previous time step. The reverse diffusion process is the inverse of the diffusion process. The goal of the reverse diffusion process is to gradually remove noise from the purely noisy image to recover the clear original image. The diffusion model achieves image generation by learning the reverse diffusion process, such as starting with random noise and gradually generating the desired image by performing iterative denoising steps.

[0048] Taking the diffusion model as an example, the network structure of the image generation model provided in this application embodiment will be described. The image generation model may include VAE (Variational Autoencoder), encoder, and U-Net.

[0049] Visual Image Generative Models (VAEs) are deep generative models based on autoencoder structures that enhance data generation capabilities by introducing probabilistic generative models. VAEs encode input data into a low-dimensional latent space representation (such as latent variables). In image generation models, VAEs can encode and map input sample images to a latent feature space (such as...). Figure 8 The encoder shown Furthermore, the features obtained after iterative denoising can be decoded to generate the image (e.g. Figure 8 The decoder shown is D).

[0050] The encoder is used to convert input data (such as prompts used to guide the generation of a graph) into a vector representation (e.g., ...). Figure 8 shown This vector can be called the context vector or encoder hidden state. In image generation models, the encoder is used to encode the cue information. For example, if the cue information is text, the text information can be processed to obtain text features. .

[0051] U-Net (U-shaped Convolutional Neural Network) employs a U-shaped encoder-decoder structure, in which the encoder's feature map is directly passed to the decoder via skip connections. In image generation models, U-Net can be used for iterative denoising. In each iteration, U-Net predicts the noise corresponding to each pixel in the noisy image, gradually removing the predicted noise through iteration to obtain the predicted image. Finally, the predicted image is decoded by a VAE to obtain the target image.

[0052] like Figure 2 As shown, the image generation method provided in this application embodiment specifically includes S101 to S103: S101. Obtain prompt information used to guide the generation of the graph.

[0053] S102. Obtain the image generation model. The image generation model is trained based on the predicted noise information map and the uncertainty map corresponding to the predicted noise information map of the sample noisy image. The uncertainty map includes the uncertainty value of the pixel in the predicted noise information map. The uncertainty value represents the reliability of the noise prediction of the corresponding pixel.

[0054] S103. Using an image generation model, image generation processing is performed based on the prompt information to obtain the target image.

[0055] Optionally, the prompts may include text information (such as descriptive text) or image information (such as reference images, sketches, etc.), or a combination of text and image information, to provide semantic or structural prompts for the image generation model to generate the image.

[0056] For example, the text information included in the prompt message can be directly input by the user, determined based on the voice information input by the user, or randomly selected from a preset prompt message library or selected according to requirements. In one example, such as... Figure 5 As shown, the text information can be "a yellow puppy spinning in circles" entered by the user in the "Input Text" input box. This information serves as a prompt to guide the generation of the image. After entering the text, the corresponding "Click to Generate" control can be triggered, thereby generating the corresponding image, which will then be displayed on the animation sound image creation page.

[0057] For example, the image information included in the prompt message can be input by the user or at least one frame selected from a preset animation library. Figure 5 The animated text-to-image creation page shown can be used to select at least one frame from historically generated images (such as images included in the preset animation library) by the object using the "Select Animation" control.

[0058] Optionally, in this embodiment, the step "to obtain the target image by performing image generation processing based on the prompt information using an image generation model" may include: Noise feature information is generated by sampling from a preset noise distribution; The image generation model is used to denoise the noise features based on the prompt information to obtain the target image.

[0059] Optionally, the image generation model is trained based on the noise prediction loss information of the sample noisy image. The sample noisy image can be an image obtained by adding noise to the original sample image. The noise prediction loss information is determined based on the predicted noise information map of the sample noisy image and the uncertainty map corresponding to the predicted noise information map. The predicted noise information map can reflect the noise prediction situation at each position in the sample noisy image. The noise prediction loss information is used as an indicator to measure the difference between the model prediction result and the actual result during model training. In this embodiment, the noise prediction loss information is determined based on the uncertainty map corresponding to the predicted noise information map and the predicted noise information map, which can reflect the accuracy of the model's noise information estimation during the noise prediction process. The uncertainty map includes the uncertainty value of the pixel in the predicted noise information map. The uncertainty value characterizes the reliability of the noise prediction corresponding to the pixel in the predicted noise information map (such as indicating whether the predicted noise information represented by the pixel is accurate). It can be understood that the higher the uncertainty value, the less reliable the pixel in the predicted noise information map. For example, the uncertainty value can be used to determine the weight value of the noise information, and the error in the predicted noise information map is weighted to obtain the noise prediction loss information.

[0060] Optionally, when generating a target image based on prompts using an image generation model, the acquired prompts can be input into a trained image generation model. The model then processes the prompts using its internal neural network structure to generate a target image that matches the prompt description. During the generation process, the image generation model gradually constructs the target image by combining the knowledge learned during training with the prompts. For example, if the model learns during training that there is high uncertainty in processing the feather region of the image, it can specifically optimize the details of the feather region, such as the feather texture and edges, improving the accuracy of local details in the generated image and thus enhancing the quality of the generated image.

[0061] For example, such as Figure 4 As shown, the user can use the "Text-to-Video" function through the client running on the terminal. They can also enter text information describing the content and dynamic process of the video in the input box as a prompt, so that the target image related to the video can be generated through the image generation model and processed by "Text-to-Video".

[0062] For example, you can use the "Text-to-Video" function to input novel text and then export the corresponding script video.

[0063] In the embodiments of this application, the provided image generation method can reflect the reliability differences of different pixels in the predicted noise information map through the uncertainty map. The model can accurately locate each local region in the image, so that the model can process local details in a targeted manner during training. This effectively improves the problem of insufficient identification and correction of local defects in related technologies, and achieves accurate processing of local details in the image generation process, improving the accuracy of image details (such as effectively reducing defects such as local distortion and structural distortion) and the quality of the generated image.

[0064] Optionally, the implementation of the image generation method can not only reduce large-area generation distortion problems, but also effectively improve small distortion cases, which is conducive to improving the accuracy of image details. Furthermore, by accurately identifying and correcting local detail defects in the image, such as local object distortion and texture abnormalities, it is beneficial to improve the quality of the generated image and meet the demand for high-quality images in various application scenarios.

[0065] In one example, the image generation model can be trained based on a collection of image-text pairs at the frame level of an animation. The trained image generation model can generate images related to the animation when given text information (such as information with animated characters or architectural scenes), which can be used as post-production gameplay or wallpapers for the animation.

[0066] In one feasible embodiment, such as Figure 3 As shown, before executing S101 to obtain the prompt information used to guide the generation of the diagram, S001 to S005 are also included: S001. Obtain training data, which includes sample images and sample prompts used to guide the generation of the graph.

[0067] S002. Add noise to the sample image to obtain a noisy sample image.

[0068] S003. Using an image generation model, noise prediction processing is performed on the sample noisy image based on the sample prompt information to obtain the predicted noise information map corresponding to the sample noisy image. The pixels in the predicted noise information map represent the noise information corresponding to the pixels at the corresponding positions in the sample noisy image.

[0069] S004. Calculate the prediction reliability of the noise information corresponding to the pixels in the prediction noise information map to obtain the uncertainty map corresponding to the prediction noise information map.

[0070] S005. Based on the uncertainty map and the prediction noise information map, the parameters of the image generation model are adjusted to obtain the trained image generation model.

[0071] Optionally, the model training involved in S001 to S005 can be fine-tuning the pre-trained model using new business data. For example, for animation A, image-text sample pairs of animation A are collected (such as training data including sample images and sample text). The pre-trained image generation model is then fine-tuned using image-text sample pairs of animation A to obtain an image generation model suitable for animation A. This allows the image generation model to generate target images related to animation A and consistent with the description of the prompt information based on the input prompt information in practical applications.

[0072] Optionally, the image generation model includes VAE, encoder, and U-net. The network parameters of VAE and encoder can be fixed, and the network parameters of U-net can be updated by training the image generation model with training data.

[0073] Optionally, the training data is a dataset used to train the image generation model, and may include sample images and sample prompts to guide the generation of images. Sample images may be real-world image instances, and sample prompts may be at least one of sample text information or sample image information. Sample images and sample prompts form sample pairs. Optionally, the training data may be data collected for a target scenario. For example, for an animal behavior analysis scenario, sample images related to animal behavior may be collected, and corresponding sample prompts may be configured for each sample image. For instance, if the collected sample image is about an orange cat, the sample prompt may be the text information "an orange cat".

[0074] Optionally, the sample noisy image is the image obtained by adding noise to the sample image. For example, such as... Figure 8 As shown, the sample image is input to the model as the input image X, and the sample image is processed by the encoder. The process involves processing the image to obtain the latent vector Z, followed by a diffusion process (adding noise over time step T) to gradually transform the sample image into a pure noise image or a near-pure noise image, resulting in the latent vector with added noise. (Such as feature information of the sample noisy image), as input for the subsequent denoising process.

[0075] Optionally, the predicted noise information map is an image obtained by the image generation model after performing noise prediction processing on the sample noisy image based on the sample prompt information. The predicted noise information map can reflect the noise information corresponding to each pixel in the sample noisy image predicted by the model, and can be a matrix describing the noise distribution during the image generation process. For example, by inputting the sample prompt information and the sample noisy image into the image generation model, the model can predict the noise corresponding to each pixel in the sample noisy image based on the prompt information and output the predicted noise information map. For example, based on the prompt information "an orange cat", the model can predict which parts of the sample noisy image are features that originally belong to the orange cat but are masked by noise and the distribution of noise.

[0076] Optionally, the uncertainty map is used to represent the reliability of the noise prediction corresponding to each pixel in the prediction noise information map. It can be a matrix that reflects the uncertainty of each pixel during the image generation process. The higher the uncertainty value, the less reliable the model's prediction of the noise information represented by the pixel; the lower the uncertainty value, the more reliable the model's prediction of the noise information represented by the pixel.

[0077] Optionally, the uncertainty values ​​of the pixels in the prediction noise information map included in the uncertainty map can be calculated using various methods. For example, a Bayesian method can be used to estimate the posterior distribution of the model's predictions, and the uncertainty value can be determined by calculating the variance or entropy of the predictions. Alternatively, an ensemble learning method can be used to train multiple models for noise prediction and calculate the difference in prediction results among the models as the uncertainty value. For example, for each pixel in the prediction noise information map, the variance or entropy of the predicted noise information is calculated. A higher variance or entropy results in a higher uncertainty value, and vice versa. After calculating the uncertainty values ​​of each pixel in the prediction noise information map, an uncertainty map is obtained, such as... Figure 6 As shown, the positions of each pixel in the sample noisy image, the predicted noise information map, and the uncertainty map correspond one-to-one.

[0078] Optionally, after calculating the uncertainty value of the pixel in the predicted noise information map (such as when obtaining the uncertainty map), the uncertainty value can be subjected to minimum-maximum normalization to scale the value range of the uncertainty value between [0,1], so as to facilitate the subsequent calculation of threshold and weight values ​​and simplify the calculation complexity.

[0079] Optionally, each pixel in the prediction noise information map can be assigned a corresponding weight value based on the uncertainty map, and the parameters of the image generation model can be updated using an optimization algorithm (such as stochastic gradient descent) according to the loss function. After multiple iterations, the trained image generation model is obtained. For example, larger weight values ​​can be assigned to pixels with high uncertainty (such as defective regions), so that the model pays more attention to the inaccurate prediction regions during the optimization process.

[0080] In this embodiment, the reliability of prediction is calculated for the noise information corresponding to the pixels in the prediction noise information map to obtain the uncertainty map corresponding to the prediction noise information map. The pixels in the uncertainty map represent the uncertainty value of the corresponding pixels in the prediction noise information map. Based on this, the image generation model can be updated by combining the uncertainty map and the prediction noise information map to introduce pixel-level uncertainty values. Different learning weight values ​​can be assigned to regions with different uncertainty values ​​in the image, so that the image generation model can repair local details in a targeted manner, thereby generating images with richer details and more natural appearance.

[0081] In a feasible embodiment, in step S003, an image generation model performs noise prediction processing on the noisy sample image based on sample cue information to obtain a predicted noise information map of the noisy sample image, including: Using an image generation model, the sample noisy image is downsampled and upsampled at multiple scales based on the sample prompt information to obtain a predicted noise information map of the sample noisy image.

[0082] Optional, combined Figure 8 The neural network structure shown is explained below: Noise prediction processing can be achieved using the denoising U-Net structure in the image generation model. The U-Net includes multiple downsampling layers, multiple upsampling layers, and skip connections. Figure 8 (The arrows in the text indicate the connections between different layers). During the process of inputting the sample noisy image into the denoising U-Net, downsampling and upsampling processes at multiple scales are performed. Specifically, based on the sample noisy image at the current time step and information from previous time steps (such as intermediate results obtained from previous time steps), the noise information corresponding to each pixel in the sample noisy image at the current time step can be predicted, and a predicted noise information map can be output.

[0083] Optionally, using an image generation model, based on sample prompts, the noisy sample image is downsampled and upsampled at multiple scales to obtain a predicted noise information map of the noisy sample image, including S003a to S003d (not shown in the figure): S003a. Using an image generation model, attention features are extracted from the noisy sample image based on the sample prompt information to obtain attention features.

[0084] S003b. Perform downsampling on the attention features at multiple scales to obtain the target features.

[0085] S003c. Perform upsampling on the target features at multiple scales to obtain the upsampled target features.

[0086] S003d. Based on the sample prompt information, attention processing is performed on the upsampled target features to obtain the predicted noise information map of the sample noisy image.

[0087] Optionally, attention mechanisms are used to make the model focus on the importance of different parts of the input data. Attention features are features extracted from the input data (such as noisy sample images) through attention mechanisms, and these features can reflect the degree of attention the model pays to different regions of the input data.

[0088] Optionally, the sample prompt information and the noisy sample image can be used as input to an image generation model (such as U-Net). The model can then perform weighted processing on different regions of the noisy sample image based on the requirements expressed by the sample prompt information, focusing on regions related to the prompt information and extracting features with attention information (such as attention features). Through the extraction of attention features, the model can focus on key regions in the noisy sample image that are related to the sample prompt information, while ignoring irrelevant information.

[0089] Optionally, the target feature is the feature obtained by downsampling the attention feature at multiple scales. Downsampling reduces the feature resolution, decreases the number of features, and extracts feature information at different scales. The upsampled target feature is the feature obtained by upsampling the target feature at multiple scales. Upsampling increases the feature resolution and restores the feature's detailed information.

[0090] Optionally, downsampling methods (such as max pooling, average pooling, etc.) can be used to perform multiple downsampling operations on the attention features to obtain the target features. Multi-scale downsampling allows the model to capture the structure and features of the noisy sample image at different scales, increasing feature diversity and richness. Subsequently, upsampling methods (such as transposed convolution, interpolation, etc.) can be used to perform multiple upsampling operations on the target features. Through multi-scale upsampling, the model can recover detailed information in the target features, making the features more refined and complete. This helps the model more accurately represent the structure and content of the noisy sample image, providing a clearer feature representation for subsequent attention processing and the generation of predicted noise information maps.

[0091] Optionally, attention processing can be applied to the upsampled target features based on the sample prompts. This attention processing can further focus on noise-related feature regions. The features can be weighted and adjusted according to the prompts to highlight the distribution and features of noise. The processed features can then be converted into a prediction noise information map representing the predicted noise information.

[0092] In a feasible embodiment, in S005, the parameters of the image generation model are adjusted based on the uncertainty map and the prediction noise information map to obtain the trained image generation model, including S005a to S005e (not shown): S005a. Based on the uncertainty value corresponding to each pixel in the uncertainty map, determine the uncertainty type corresponding to each pixel in the uncertainty map.

[0093] S005b. Based on the type of uncertainty, determine the weight information corresponding to each pixel in the uncertainty graph.

[0094] S005c. Based on the predicted noise information map and the real noise corresponding to the sample noisy image, determine the initial loss information corresponding to the predicted noise information map.

[0095] S005d. Based on the weight information, update the initial loss information to obtain the noise prediction loss information corresponding to the prediction noise information map.

[0096] S005e. Based on the noise prediction loss information corresponding to the predicted noise information map, the parameters of the image generation model are adjusted to obtain the trained image generation model.

[0097] This embodiment can generate a pixel-level reliability evaluation of the generated image by generating an uncertainty map and performing reliability quantification calculation. Then, it can generate local detail block-level weights through the uncertainty reliability evaluation and finally combine these weights to train the generation loss, thereby strengthening the learning of high uncertainty regions and suppressing local defects in model updates.

[0098] Optionally, the uncertainty type includes a first uncertainty type or a second uncertainty type, where the uncertainty value of a pixel corresponding to the first uncertainty type is higher than the uncertainty value of a pixel corresponding to the second uncertainty type. The first uncertainty type can also be called a high uncertainty type, and the second uncertainty type can also be called a low uncertainty type.

[0099] Optionally, each pixel in the uncertainty map can be categorized based on its corresponding uncertainty value. This categorization can be achieved using thresholding methods (such as using a fixed threshold or a dynamic threshold) or clustering methods (such as the K-means algorithm). Determining the uncertainty type clarifies which pixels require special attention (such as edges, complex texture regions, etc.), providing a reference for subsequent weight allocation.

[0100] For example, uncertainty graphs can be used. (t indicates the median of the uncertainty value corresponding to the pixel in the denoising step) As an adaptive dynamic threshold, it iterates through each pixel in the uncertainty map. The position of each pixel can be represented as (i,j), where i represents the row position and j represents the column position. If it is determined that the uncertainty value corresponding to a certain pixel is greater than the median... If the uncertainty value of a pixel is not greater than the median, then the pixel can be identified as a high-uncertainty pixel, that is, classified as the first uncertainty type; if the uncertainty value of a pixel is determined to be no greater than the median... If so, the pixel can be identified as a low-uncertainty pixel, that is, classified as the second uncertainty type.

[0101] Optionally, when using the threshold method to determine the uncertainty type corresponding to each pixel in the uncertainty map, the dynamic threshold used can be flexibly adjusted according to task requirements. For example, it can be the median as shown in the example above, or it can be the mean, quartiles, or percentiles. The median, rather than the mean, is used in the example above because it better reflects the general level of uncertainty when insensitive to outliers (such as a very small number of pixels with extremely high uncertainty), thus improving the effectiveness of model optimization based on pixel-level uncertainty.

[0102] Optionally, when determining the weight information corresponding to each pixel in the uncertainty map based on the uncertainty type, a fine-grained weight mapping can be generated. This is used to weight the loss function. Considering that pixel regions with higher uncertainty values ​​may have greater potential problems, they should have higher weights in the loss function for weighted learning. Conversely, discrete points with excessively high uncertainty need to be suppressed to avoid noise amplification. Based on this, appropriate weight values ​​can be assigned by combining the uncertainty type and the pixel distribution. For example, higher weights are assigned to regions corresponding to the first uncertainty type to strengthen the model's optimization of that region; lower weights are assigned to regions corresponding to the second uncertainty type to reduce overfitting of the model to that stable region. Furthermore, to avoid the influence of discrete points... Optionally, the real noise is the actual noise value added to the sample noisy image, which can be used as a supervisory signal for calculating the loss function.

[0103] Optionally, in step S002, the sample image is subjected to noise processing to obtain a noisy sample image, including: iteratively adding noise to the sample image to obtain a noisy sample image. Based on this, the method provided in this application embodiment further includes: for the noisy sample image corresponding to each round of iterative noise processing, determining the real noise information corresponding to the noisy sample image based on the noisy sample image and the sample image.

[0104] Optionally, noise can be gradually added to the original sample image through a forward process, eventually transforming it into an image with pure noise or near-pure noise. Specifically, in each round, Gaussian noise is added to the current image according to a preset noise schedule, and after multiple iterations, a noisy sample image is obtained. In each iteration, the real noise is the actual noise component of the current noisy image relative to the original sample image.

[0105] Optionally, during model training, the training objective is to make the noise predicted by the model as close as possible to the real noise. When calculating the loss, mean squared error loss can be used. By minimizing the loss, the model can learn to predict and remove noise from any noisy image, thus achieving image generation.

[0106] Optionally, the initial loss information is the loss value (such as L2 loss or mean squared error MSE) calculated directly from the predicted noise information map and the real noise. It can be used to quantify the deviation between the model's predicted noise (such as the noise information corresponding to each pixel in the predicted noise information map) and the real noise.

[0107] Optionally, the weight information can be weight values ​​assigned to each pixel in the predicted noise information map based on the uncertainty type, which can be used to adjust the contribution of different pixels in the loss function. For example, combining the weight information with the initial loss information can yield a weighted loss (the noise prediction loss information corresponding to the predicted noise information map obtained after updating the initial loss information). This weighted loss can highlight the loss contribution of high uncertainty regions and suppress the interference of low uncertainty regions, making the optimization process more efficient.

[0108] Optionally, based on weighted loss, the parameters of the image generation model are updated through backpropagation, and iterative optimization is performed until the loss converges or the preset number of iterations is reached. This can result in a trained image generation model, which improves the quality of regions with high uncertainty (such as edges and complex textures) while maintaining the overall image stability.

[0109] Optional, weighted loss Lweighted The calculation is shown in the following formula (1): (1) Formula (1) above shows the weighted loss of the noise predicted by the image generation model and the real noise for denoising step t (performing T rounds of iterative denoising, where t is one round in T). Here, H is the height of the image, W is the width of the image, and i and j represent the pixel positions in the image. Wt ( i , j ) is for location ( i , jThe pixel weight value, ( i , j ()( xt , t ) is the noise reduction step t for the input Xt Predicted noise, ( i , j ) is real noise. Xt It is the image input at step t.

[0110] Optionally, by calculating the new MSE (Mean Squared Error Loss) after weighting, the image generation model can be updated to achieve fine-grained uncertainty reinforcement learning.

[0111] In a feasible embodiment, step S005b determines the weight information corresponding to each pixel in the uncertainty map according to the uncertainty type, including steps A1 to A4 (not shown in the figure): Step A1: Determine at least one target pixel whose uncertainty type satisfies the preset type conditions.

[0112] Optionally, the preset type conditions may include an uncertainty type of a first uncertainty type or an uncertainty type of a second uncertainty type. Based on this, at least one target pixel belonging to the first uncertainty type or at least one target pixel belonging to the second uncertainty type can be identified. That is, through the implementation of step A1, pixels with high uncertainty type and pixels with low uncertainty type can be distinguished. By filtering target pixels that meet the preset type conditions, regions with different uncertainties in the image can be quickly located, improving processing efficiency.

[0113] Step A2: For the target pixel, determine the uncertainty region in the uncertainty map based on the uncertainty type of the neighboring pixels of the target pixel.

[0114] Optionally, adjacent pixels can be pixels in the image that are adjacent to the target pixel. For example, when using the four-neighbor method, adjacent pixels can include pixels in the four directions of up, down, left, and right; when using the eight-neighbor method, adjacent pixels can also include pixels in the diagonal direction, that is, pixels in the up, down, left, right, and four diagonal directions.

[0115] Optionally, pixels within an uncertainty region (also known as a connected component, which can be an image region obtained through connected component analysis) have similar or identical uncertainty values. For example, for each target pixel, the uncertainty type of its neighboring pixels is obtained. If the uncertainty type of a neighboring pixel is the same as the uncertainty type of the target pixel, the corresponding neighboring pixels are grouped into a region. By continuously expanding this process, all interconnected target pixels that meet the preset type conditions and their neighboring pixels can be combined into an uncertainty region.

[0116] Step A3: Based on the uncertainty value of the pixels in the uncertainty region, set the weight information of the pixels in the uncertainty region.

[0117] Optionally, based on the uncertainty value of the pixels in the uncertainty region, a preset function or rule can be used to set corresponding weight values. For example, the weight value can be set to be proportional to the uncertainty value, such as a higher weight value for a higher uncertainty value; alternatively, a more complex functional relationship, such as an exponential function or a logarithmic function, can be used to adjust the relationship between the weight and the uncertainty value as needed.

[0118] Step A4: For pixels that do not belong to the uncertainty region in the uncertainty map, perform uncertainty suppression processing on the pixel values ​​to determine the weight information of pixels that do not belong to the uncertainty region in the uncertainty map.

[0119] Optionally, there may be some discrete pixels that do not belong to any uncertain region, for example, such as Figure 7 As shown, the pixel located at row 1, column 9, marked as 1, is not associated with any shaded area; this pixel is a discrete pixel. For pixels not belonging to uncertain regions, their uncertainty values ​​can be suppressed in various ways, such as multiplying the uncertainty value by a coefficient less than 1, or setting the uncertainty value to a low fixed value. In this case, the pixel's weight information can be set based on the suppressed uncertainty value. By suppressing the uncertainty values ​​of discrete pixels, the influence of discrete points can be reduced, which helps improve the quality of the generated image.

[0120] Optionally, in step A2, the uncertainty region in the uncertainty map is determined based on the uncertainty type of the neighboring pixels of the target pixel, including steps A21 to A22 (not shown in the figure): Step A21: Detect the uncertainty type of the neighboring pixels of the target pixel, so as to select the target neighboring pixels whose uncertainty type meets the preset type conditions.

[0121] Step A22: Based on the target's neighboring pixels and the target pixel, determine the uncertainty region in the uncertainty map.

[0122] Optionally, connectivity analysis can be performed on the target pixel and its neighboring pixels to group connected pixels into a region. This connectivity analysis can be implemented using algorithms such as depth-first search or breadth-first search to find all pixels connected to the target pixel and its neighboring pixels, thus identifying uncertain regions.

[0123] For example, such as Figure 7 As shown, pixels belonging to the first uncertainty type are marked as 1, and pixels belonging to the second uncertainty type are marked as 0. When determining the uncertainty region for pixels belonging to the first uncertainty type, a traversal approach can be used, sequentially employing an eight-neighbor method to obtain adjacent pixels that also belong to the first uncertainty type, forming... Figure 7 The image shows multiple uncertainty regions indicated by diagonal shading. Correspondingly, based on the same method, uncertainty regions belonging to the second type of uncertainty can be obtained, forming... Figure 7 The image shows multiple uncertain regions represented by the shaded array of points. It should be noted that the unconnected uncertain regions are independent of each other.

[0124] Optionally, in step A3, weight information for pixels in the uncertainty region is set based on the uncertainty value of the pixels in the uncertainty region, including steps A31 to A33 (not shown in the figure): Step A31: Calculate the local uncertainty value of the uncertainty region based on the uncertainty value of the pixels in the uncertainty region.

[0125] Step A32: Calculate the global uncertainty value of the uncertainty map based on the uncertainty values ​​of the pixels in the uncertainty map.

[0126] Step A33: Based on the global uncertainty value and the local uncertainty value of the uncertainty region, set the weight information of the pixels in the uncertainty region.

[0127] Optionally, local uncertainty values ​​are determined on a per-region basis, while global uncertainty values ​​are determined on a per-overall basis of the uncertainty map. Local and global uncertainty values ​​can be determined using methods such as simple averaging, weighted averaging, or the median.

[0128] For example, for local uncertainty values, the mean value of pixels in the uncertainty region can be used as the local uncertainty value of the corresponding uncertainty region, and its calculation is shown in the following formula (2): (2) in, For the region The local uncertainty value; For the region The total number of pixels; For position The uncertainty value of a pixel.

[0129] For example, for global uncertainty values, a threshold used to classify uncertainty types can be used, such as the example above, where the median of pixels in the uncertainty graph can be used as the global uncertainty value.

[0130] Optionally, the global uncertainty value can also be obtained by calculating the mean or weighted average of the local uncertainty values ​​of each uncertainty region after determining the local uncertainty values ​​of each uncertainty region.

[0131] Optionally, the weight information can be determined by calculating a preset function or model, such as setting a linear function and introducing adjustable parameters. By adjusting the size of the adjustable parameters, the influence of local uncertainty and total uncertainty on the weight can be controlled.

[0132] In this embodiment, by determining local and global uncertainty values ​​and setting pixel weights based on these values, the weights can be adaptively allocated according to the degree of uncertainty in the region where the pixel is located and the overall uncertainty of the image. For example, pixels in regions with higher uncertainty can be assigned higher weights, while pixels in regions with lower uncertainty can be assigned lower weights. This allows the model to selectively repair local details, thereby generating images with richer details and a more natural appearance. Furthermore, analysis of uncertain regions is provided, which can reduce resource waste caused by global uniform optimization, thereby improving training efficiency.

[0133] Optionally, in step A33, weight information for pixels in the uncertainty region is set based on the global uncertainty value and the local uncertainty value of the uncertainty region, including steps A331 to A332 (not shown in the figure): Step A331: When the local uncertainty value of the uncertainty region is greater than the global uncertainty value, set the weight information of the pixels in the uncertainty region based on the local uncertainty value of the uncertainty region.

[0134] Step A332: When the local uncertainty value of the uncertainty region is not greater than the global uncertainty value, the local uncertainty value of the uncertainty region is suppressed to obtain the suppressed uncertainty value, and the weight information of the pixels in the uncertainty region is set according to the suppressed uncertainty value.

[0135] Exemplarily, the local uncertainty value is the mean of each pixel in the corresponding uncertainty region , and the overall uncertainty value is the median of each pixel in the uncertainty map . For example: If > , it indicates that the average uncertainty value of the corresponding uncertainty region is higher than the global uncertainty value. The weights of the pixel points in the corresponding uncertainty region Wt ( i , j ) can be set to . If ≤ , the uncertainty values of the pixel points in the corresponding uncertainty region are suppressed, and the corresponding weight values are set based on the suppressed uncertainty values. Optionally, the suppression process can be multiplying the local uncertainty value by a value less than 1 or dividing it by a value greater than 1 to reduce the weight values corresponding to the low-uncertainty regions, so that the model can reduce its attention to the non-critical regions (regions with lower uncertainty). Exemplarily, when performing the suppression process, the weight Wt ( i , j ) can be set to / 4.

[0136] Optionally, in step A332, when the local uncertainty value of the uncertainty region is not greater than the global uncertainty value, the local uncertainty value of the uncertainty region is suppressed to obtain the suppressed uncertainty value, including: when the local uncertainty value of the uncertainty region is not greater than the global uncertainty value, the local uncertainty value of the uncertainty region is suppressed based on a preset first suppression ratio to obtain the suppressed uncertainty value.

[0137] Exemplarily, when the suppression process is dividing by a certain value, the first suppression ratio a1 can be set to 0 < a1 ≤ 10. For example, if a1 is assigned a value of 4, the local uncertainty value can be divided by 4 to obtain the suppressed uncertainty value.

[0138] In the embodiments of the present application, by suppressing the local uncertainty values of the low-uncertainty regions, the weights of the low-uncertainty regions in subsequent calculations can be reduced, avoiding excessive influence on the results.

[0139] Optionally, in step A4, for pixels that do not belong to the uncertainty region in the uncertainty map, the uncertainty value of the pixels is suppressed to determine the weight information of the pixels that do not belong to the uncertainty region in the uncertainty map. This includes: for pixels that do not belong to the uncertainty region in the uncertainty map, the uncertainty value of the pixels is suppressed based on a preset second suppression ratio to determine the weight information of the pixels that do not belong to the uncertainty region in the uncertainty map. The preset second suppression ratio is greater than the preset first suppression ratio.

[0140] Optionally, the second suppression weight a2 is used to suppress the uncertainty value corresponding to the pixel in the non-uncertain region, and satisfies a2>a1.

[0141] Optionally, referring to the example of dividing the uncertainty region in the above embodiments, the uncertainty map can be divided into two types of pixels: pixels in the uncertainty region and pixels in the non-uncertainty region. The uncertainty value corresponding to the pixels in the non-uncertainty region can be adjusted according to the second suppression weight a2. For example, for discrete points (such as isolated pixels that fail to form an effective uncertainty region), their weight values ​​can be adjusted. Wt ( i , j ) set to / 8.

[0142] In this embodiment, the weights of discrete points are set to be lower, and the weights of pixels in uncertain regions are set to be even lower. This can effectively suppress the influence of these discrete points that may be noise and improve the overall performance of the model.

[0143] Optionally, in step A2, for the target pixel, based on the uncertainty type of the neighboring pixels, the uncertainty region in the uncertainty map is determined, including: For the target pixel, an initial uncertainty region is constructed based on the uncertainty type of the target pixel's neighboring pixels; Based on the number of pixels contained in the initial uncertainty region, an uncertainty region is selected from the initial uncertainty region.

[0144] Optionally, considering the existence of noise points with small and discrete uncertainty values, and that these points may originate from transient noise rather than structural generation defects, the embodiments of this application further screen the uncertainty region.

[0145] For example, after constructing an initial uncertainty region based on the target pixel and its adjacent pixels, the initial uncertainty region can be filtered to determine the region containing more than or equal to a preset number of pixels as the uncertainty region, and the remaining regions (regions containing fewer than the preset number of pixels) as the non-uncertain regions.

[0146] For example, such as Figure 7 Three adjacent pixels located in rows 14 to 16 and column 16 (all marked as 0) constitute the non-uncertain region in the initial uncertainty region when the preset number is 4.

[0147] Optionally, the method also includes: Based on preset weights, set the weight information of pixels in the non-uncertain regions within the initial uncertainty region.

[0148] Alternatively, the weight information of the pixels in the non-uncertain regions of the initial uncertainty region can be determined based on the uncertainty values ​​of the pixels in the non-uncertain regions of the initial uncertainty region.

[0149] Optionally, for pixels that do not belong to any uncertainty region and are non-discrete (the initial uncertainty region is a non-uncertain region, mostly consisting of relatively deterministic background or stable regions), the weight value of the corresponding pixel can be set to a preset threshold, or the original uncertainty value (the unnormalized uncertainty value) can be used as the weight value. Furthermore, to match the weight values ​​of other pixels in the uncertainty map, the weight values ​​of pixels that do not belong to any uncertainty region and are non-discrete can be scaled after determining the weight values ​​of other pixels. For example, if the minimum weight value of other pixels in the uncertainty map is 0.3 and the maximum is 0.7, then the scale of the weight values ​​of pixels that do not belong to any uncertainty region and are non-discrete can be scaled to between 0.3 and 0.7.

[0150] In a feasible embodiment, in step S003, an image generation model is used to perform noise prediction processing on the sample noisy image based on the sample prompt information to obtain the predicted noise information map corresponding to the sample noisy image, including steps B1 to B4 (not shown): Step B1: Using an image generation model, perform noise prediction processing on the sample noisy image based on the sample prompt information to obtain the predicted noise information map of the sample noisy image in the current iteration round.

[0151] Optionally, in the current iteration, using the sample noisy image (output of the previous iteration) and sample cue information as input, the image generation model uses the sample cue information as a condition to predict noise in the input image, and outputs a predicted noise information map of the same size as the input image (such as the noise distribution in the current image predicted by the image generation model). Figure 6 As shown.

[0152] Step B2: Based on the predicted noise information map, the sample noisy image is denoised to obtain the denoised image.

[0153] Optionally, the predicted noise information map represents the distribution of noise in the current image predicted by the image generation model. Based on this, in the current iteration, the predicted noise can be subtracted from the current sample noisy image based on the predicted noise information map to obtain the denoised image.

[0154] Step B3: Determine the denoised image as the new sample image with added noise.

[0155] Optionally, the denoised image can be used as input for the next iteration, i.e., a new sample image with added noise. The iteration is performed by gradually reducing the noise in the image, transitioning from a high-noise state (close to pure noise) to a low-noise state (close to a real image). During the iteration, the image generation model is guided by sample prompts to generate image content, such as gradually forming image content corresponding to the sample prompts from random noise.

[0156] Step B4: Return to the step of performing noise prediction processing on the sample noisy image based on the sample prompt information through the image generation model until the number of iterations meets the preset condition, and obtain the predicted noise information map corresponding to each iteration.

[0157] Optionally, steps B1 to B3 can be repeated until the number of iterations meets the preset condition (e.g., the number of iterations reaches T times, where T is greater than 1), and the prediction noise information map corresponding to each round of iteration can be obtained.

[0158] Optionally, the termination condition for iteration can also be that the noise level is below a threshold, or dynamic stopping can be achieved by detecting the variance of the predicted noise or an image quality metric.

[0159] In the embodiments of this application, by iteratively predicting and denoising noise, combined with sample prompts, it is possible to generate high-quality images from noise, which is beneficial to improving the performance of the image generation model.

[0160] In a feasible embodiment, step S004 involves calculating the prediction reliability of the noise information corresponding to the pixels in the prediction noise information map to obtain an uncertainty map corresponding to the prediction noise information map. This includes: determining a target iteration round that satisfies a preset condition; and calculating the prediction reliability of the pixels in the prediction noise information map of the target iteration round to obtain an uncertainty map corresponding to the target iteration round.

[0161] Optionally, considering that the image content is unstable in the initial stage of iteration, it is not meaningful and computationally expensive to evaluate uncertainty too early. Therefore, this embodiment sets a target iteration round to reduce computational costs and improve processing efficiency. Specifically, the optimization of the uncertainty map can be intervened in the middle stage of the denoising process (such as when the image structure has been roughly formed). For example, if the total number of iterations is set to T rounds, the target iteration round is the rounds after 75%T; if the total number of iterations is set to 20 rounds, then starting from step t=15 (and in each subsequent round), the prediction reliability of the pixels in the prediction noise information map is calculated to obtain the uncertainty map corresponding to the target iteration round.

[0162] In the embodiments of this application, pixel-level uncertainty assessment is performed on the intermediate image (the iterative sample noisy image) in the key steps of the denoising process (such as step 15). This can achieve dynamic guidance of the generation process, automatically identify high uncertainty regions with higher generation detail defects, and construct a fine loss weight mask accordingly. This guides the loss function to selectively optimize the fine parts of the generated image, which is beneficial to suppress defects in the early stage of formation, improve the stability of the generation process and the final image quality.

[0163] Optionally, in step S005, the parameters of the image generation model are adjusted based on the uncertainty map and the prediction noise information map to obtain the trained image generation model, including steps C1 to C4 (not shown in the figure): Step C1: For the target iteration round, based on the uncertainty map and the corresponding prediction noise information map corresponding to the target iteration round, determine the noise prediction loss information corresponding to the target iteration round.

[0164] Optionally, for each target iteration, the above weighted loss calculation formula can be used to calculate the noise prediction loss information corresponding to that target iteration based on the uncertainty map, predicted noise information map, and real noise information map corresponding to that iteration (such as pre-calculated through the forward process of the diffusion model or obtained from the data), and the weights can be determined based on the uncertainty map.

[0165] Step C2: For non-target iteration rounds, determine the noise prediction loss information corresponding to the non-target iteration rounds based on the prediction noise information map corresponding to the non-target iteration rounds.

[0166] Optionally, for each non-target iteration round, the standard MSE loss (without uncertainty weights) can be used to calculate the corresponding noise prediction loss information, which can constrain the noise prediction of non-target iteration rounds and prevent the model from deviating too much from the reasonable noise distribution, such as reducing the possibility of generating unnatural smooth regions.

[0167] Step C3: Determine the total noise prediction loss information based on the noise prediction loss information corresponding to each target iteration round and the noise prediction loss information corresponding to each non-target iteration round.

[0168] Optionally, the losses from the target iteration rounds and non-target iteration rounds can be weighted and summed to obtain the total noise prediction loss. For example, the weights of each iteration round can be dynamically adjusted according to the training phase; for instance, the initial phase may focus on stabilizing training in non-target rounds, while later phases may focus on optimizing details in target iteration rounds. In this embodiment, the total noise prediction loss can balance the non-target iteration rounds (model stability) and the target iteration rounds (detail optimization capability), avoiding training instability due to too many target rounds, and also preventing excessively high weights on non-target iteration rounds that could limit the model's expressive power.

[0169] Step C4: Based on the total noise prediction loss information, adjust the parameters of the image generation model to obtain the trained image generation model.

[0170] Optionally, the total noise prediction loss information can be backpropagated to the image generation model to calculate the gradient, and the model parameters can be updated based on the gradient. Then, steps C1 to C4 can be repeated until the model converges (e.g., the total loss stabilizes or a preset number of iterations is reached).

[0171] In the embodiments of this application, by using uncertainty-weighted loss and multi-round collaborative training, the adaptability of the image generation model to complex scenes can be improved, enabling the model to dynamically focus on high uncertainty regions, balance local and global optimization, and enhance the robustness of the model.

[0172] In a feasible embodiment, step S004 calculates the prediction reliability of the noise information corresponding to the pixels in the prediction noise information map to obtain an uncertainty map corresponding to the prediction noise information map, including steps D1 to D4 (not shown in the figure): Step D1: Obtain the posterior distribution information of the original parameters of the image generation model under multiple preset training data.

[0173] Optionally, posterior distribution information of model parameters can be obtained to quantify the uncertainty of model parameters (such as the range of parameter fluctuations reflected by posterior covariance), providing a data basis for calculating uncertainty.

[0174] Optionally, step D1 involves obtaining the posterior distribution information corresponding to the original parameters of the image generation model under multiple preset training data, including steps D11 to D13 (not shown in the figure): Step D11: Based on the preset prior distribution function, process the original parameters of the image generation model to obtain the prior distribution information corresponding to the original parameters.

[0175] Optionally, a prior distribution can be set to apply a Gaussian prior to the output layer parameters in the image generation model (such as U-Net). ,in, Let be the prior covariance matrix.

[0176] Step D12: Obtain the likelihood statistics of the original parameters of the image generation model under multiple preset training data.

[0177] Optionally, model parameters can be calculated based on multiple preset training data (such as images with different noise levels). likelihood function Where D is the training dataset. Likelihood statistics specifically characterize the parameters... The following observations reveal the reasonableness of the training data, which serves as both the loss function used to drive parameter optimization during training and the data evidence of the posterior distribution in Bayesian inference.

[0178] Step D13: Based on prior distribution information and likelihood statistics, calculate the posterior distribution information corresponding to the original parameters of the image generation model.

[0179] Alternatively, Bayes' theorem can be used. By combining the prior and likelihood, the posterior distribution of the parameters is obtained.

[0180] For example, the Laplace approximation (LLLA) can be used, employing a Gaussian distribution. Approximate posterior, where H is the Hessian matrix (such as the second derivative matrix). These are the parameters estimated by the maximum a posteriori. The Laplace approximation specifically approximates a complex probability distribution as a Gaussian distribution.

[0181] Step D2: Based on the posterior distribution information, calculate the observation information matrix of the parameter changes corresponding to the image generation model.

[0182] Optionally, based on posterior distribution information, an observation information matrix, such as the Hessian matrix, can be calculated.

[0183] Optionally, in step D2, based on the posterior distribution information, the observation information matrix of the parameter changes corresponding to the image generation model is calculated, including steps D21 to D24 (not shown in the figure): Step D21: Obtain the noise prediction loss information corresponding to the image generation model under multiple preset training data.

[0184] Step D22: Based on the noise prediction loss information corresponding to each preset training data, determine the target training data from multiple preset training data.

[0185] Step D23: Update the parameters of the image generation model based on the noise prediction loss information corresponding to the target training data to obtain the target parameters.

[0186] Step D24: Based on the posterior distribution information and target parameters, calculate the observation information matrix of the corresponding parameter changes of the image generation model.

[0187] Optionally, the prediction loss of the image generation model to noise, such as the MSE loss, can be calculated on multiple preset training datasets. Training data with higher losses (such as high-noise or complex-texture samples) can be selected as target training data for calculating the observation information matrix. The parameters of the image generation model can be updated using gradient descent based on the loss of the target training data to obtain the optimized parameters, i.e., the target parameters. Optionally, the target parameter can be... The observation information matrix for calculating the loss function is derived; this matrix is ​​specifically the Hessian matrix, denoted as H. The inverse of the observation information matrix... Approximate posterior covariance, related parameter uncertainty. By selecting target data, we can focus on samples that are difficult for the model to predict, thus improving the targeting of uncertainty estimation.

[0188] Step D3: Calculate the parameter uncertainty value of the image generation model based on the observation information matrix.

[0189] Optionally, parameter uncertainty values ​​can quantify the range of fluctuation of model parameters, reflecting the model's confidence in different parameters.

[0190] Optionally, in step D3, the parameter uncertainty values ​​of the image generation model are calculated based on the observation information matrix, including steps D31 to D32 (not shown in the figure): Step D31: Calculate the variance of parameter changes in the image generation model based on the observation information matrix.

[0191] Step D32: Determine the parameter uncertainty value of the image generation model based on the parameter variation variance.

[0192] For example, it can be derived from the inverse of the Hessian matrix. Extract the diagonal elements to obtain the variance of each parameter, and aggregate the parameter variances into parameter uncertainty values.

[0193] The inverse of the Hessian matrix in the embodiments of this application can reflect the range of motion of the parameters.

[0194] Step D4: Based on the parameter uncertainty value, calculate the prediction reliability of the noise information corresponding to the pixel in the prediction noise information map to obtain the uncertainty map corresponding to the prediction noise information map.

[0195] Optionally, step D4 can generate a pixel-level uncertainty map with the same spatial size as the prediction noise information map.

[0196] Optionally, in step D4, based on the parameter uncertainty value, the prediction reliability of the noise information corresponding to the pixel in the prediction noise information map is calculated to obtain the uncertainty map corresponding to the prediction noise information map, including steps D41 to D42 (not shown in the figure): Step D41: Determine the amplification factor information between the parameter changes of the image generation model and the noise prediction changes.

[0197] Step D42: Based on the magnification information and parameter uncertainty value, calculate the prediction reliability of the noise information corresponding to the pixel in the prediction noise information map to obtain the uncertainty map corresponding to the prediction noise information map.

[0198] Optionally, the amplification factor information between changes in model parameters and changes in noise prediction can be calculated, such as the Jacobian matrix, which reflects the amplification effect of parameter perturbations on noise prediction. Then, the noise prediction variance for each pixel can be calculated by combining the parameter uncertainty value and the amplification factor.

[0199] In this embodiment, the LLLA (Last-Layer Laplace Approximation) method can be applied to the denoising process of the image generation model to achieve pixel-level uncertainty estimation. Furthermore, it can map parameter uncertainty to the noise prediction space, solving the problem of uncertainty propagation at different scales. In the training of the image generation model, the provided uncertainty-aware scheme can improve image generation quality while maintaining high efficiency.

[0200] Optionally, the image generation model includes a first network layer module and a second network layer module.

[0201] For example, an image generation model (such as U-Net) has N layers. The first network layer module can be the first N-1 layers; the second network layer module can be the Nth layer, which is the last layer. In the embodiments of this application, considering that the first N-1 layers of the image generation model have been fully trained and have fixed and reliable parameters, while the parameters of the last layer (output layer) are uncertain, Bayesian inference can be performed based on the parameters of the last layer.

[0202] In one specific embodiment, Bayesian Diffusion (BayesDiff) is the estimation of uncertainty in a diffusion model based on Bayesian inference. This involves estimating the "parameter uncertainty" (specifically encoded in...) The model propagates to the output space through its local linear structure (J), thereby obtaining the confidence interval of each pixel's predicted value—the higher the uncertainty of a pixel, the wider the "parameter range that can explain the data" in the parameter space.

[0203] Specifically, this embodiment can employ Bayesian inference on the parameters of the last layer, avoiding the expensive Bayesian inference on the entire neural network. The parameters of the last layer are constrained by a Gaussian prior, and the Hessian matrix of the optimal parameter points is obtained by maximizing the posterior probability distribution; the inverse Hessian matrix... This is an approximation of the covariance of the posterior distribution, representing parameter uncertainty; then, the output z of the sample noisy image is obtained after the first N-1 layers of processing by the image generation model. , z Specifically, this refers to the input of the last layer (Nth layer) of the image generation model. Finally, the uncertainty in the parameter space of the Nth layer is expressed through the Jacobian matrix J(z). The linear propagation is applied to the prediction space to obtain a pixel-level uncertainty estimate.

[0204] Optionally, an image generation model is used to perform noise prediction processing on the sample noisy image based on the sample prompt information to obtain a predicted noise information map corresponding to the sample noisy image. This includes: extracting features from the sample noisy image based on the sample prompt information through a first network layer module to obtain feature information; and performing noise prediction processing on the feature information through a second network layer module to obtain a predicted noise information map.

[0205] The first network layer module has been fully trained and its parameters are fixed, and it can be used to extract stable image features, such as edges and textures. Optionally, the first network layer module can be used to extract features from the noisy sample image and sample prompt information to generate multi-scale feature maps.

[0206] The parameters of the second network layer module are uncertain. Optionally, the second network layer module can be used to predict noise in the feature map and output a predicted noise information map. However, the uncertainty of the output layer parameters affects the reliability of the noise prediction.

[0207] In this application, the impact of Bayesian inference on noise prediction can be quantified, i.e., uncertainty estimation of output layer parameters can be performed, which can effectively reduce computational complexity and retain the feature extraction capability of the first network layer module. By separating feature extraction and noise prediction, the dimensionality of Bayesian inference can be reduced, avoiding high-dimensional posterior distribution calculations for the entire model, thus improving training efficiency.

[0208] Optionally, the posterior distribution information corresponding to the original parameters of the image generation model under multiple preset training data is obtained, including: obtaining the posterior distribution information corresponding to the original parameters of the second network layer module under multiple preset training data.

[0209] Optionally, the posterior distribution information obtained in step D1 above can be the original parameters corresponding to the second network layer module, that is, obtained by processing the output layer parameters.

[0210] Optionally, based on the posterior distribution information, the observation information matrix of the parameter changes corresponding to the image generation model is calculated, including: based on the posterior distribution information, the observation information matrix of the parameter changes corresponding to the second network layer module is calculated.

[0211] Optional, output layer parameters posterior distribution As input, the loss function L of the output layer can be computed on the training data. , The observation information matrix H is used to calculate the loss function.

[0212] In a feasible embodiment, the image generation model provided in this application has compatibility, versatility, and scalability in training. For example, it can be integrated into various diffusion models of related technologies and can also be applied to reinforcement learning without changing the main structure of the model. The introduction of the reinforcement learning framework enables the model to dynamically adjust parameters based on the generation quality (measured by reward feedback). For example, the model predicts noise and generates a denoised image. Then, a reward signal can be generated by comparing the denoised image with the desired image (real data or a high-confidence reference image). The reward signal is adjusted using the uncertainty map of the noise prediction to avoid the model overfitting to high uncertainty regions. Based on this, some parameters can be adjusted to prevent local optima.

[0213] For example, reinforcement learning frameworks could be DPO (Direct Preference Optimization, an algorithm that optimizes based on preferences), RLHF (Reinforcement Learning from Human Feedback), TDPO-R (a reinforcement learning algorithm that uses a fine-grained reward mechanism), etc.

[0214] Before step B3 determines the denoised image as the new sample noisy image, it also includes steps B01 to B02 (not shown in the figure): Step B01: Determine the reward feedback information for the current iteration based on the denoised image and the expected denoised image corresponding to the sample denoised image.

[0215] Step B02: Based on the reward feedback information, update the image generation model to obtain the updated image generation model.

[0216] This embodiment can introduce a temporal differential reward mechanism, focusing on the intermediate steps of the inference process. Specifically, it provides fine-grained reward feedback information for each step of the model's inference (i.e., each iteration), continuously adjusting the strategy based on the reward feedback signals during the generation process. This ensures that each step of the learning process is guided, rather than solely relying on the correctness of the final answer to evaluate the model's performance. Through this fine-grained control, the model can continuously optimize its inference path, effectively addressing long-range dependency issues in complex tasks and ensuring that the model maintains logical consistency during inference.

[0217] Specifically, this embodiment utilizes the real-time feedback provided by the time-difference reward mechanism at each time step, allowing for immediate policy updates at each step of the diffusion model's denoising process without waiting for all steps to complete. This real-time update enables the model to promptly correct deviations that occur during denoising, preventing problems from accumulating in subsequent steps. Simultaneously, because updates are performed at each step in real time, the model can achieve better optimization results in a shorter time, reducing latency and unnecessary computational overhead associated with traditional global update methods, and improving training sample efficiency.

[0218] Optionally, in iterative denoising, for the current iteration, the denoised image (generated by the model) and the expected denoised image (real data or reference image) corresponding to the sample denoised image can be used as inputs. Reward functions, such as pixel-level error MSE or SSIM (Structural Similarity Index Measure), are used to calculate reward feedback information. The model is then updated based on the reward feedback information to obtain the updated image generation model.

[0219] Optionally, the calculated reward feedback information can be normalized to ensure that the reward scale is consistent across different batches.

[0220] In this embodiment, the reward signal reflects the difference between the generated image and the desired image, guiding the direction of model optimization. Furthermore, normalized rewards ensure the rationality of gradient updates and improve the stability of model training.

[0221] Optionally, in step B02, the image generation model is updated based on the reward feedback information to obtain an updated image generation model, including steps B021 to B023 (not shown in the figure): Step B021: Determine the uncertainty map corresponding to the prediction noise information map of the current iteration round.

[0222] Step B022: Based on the uncertainty graph, update the reward feedback information to obtain the updated reward feedback information.

[0223] Step B023: Update the image generation model based on the updated reward feedback information to obtain the updated image generation model.

[0224] Optionally, after calculating the uncertainty graph based on the above embodiments, the original reward can be weighted by uncertainty to generate an updated reward. Then, the updated reward can be converted into a gradient update signal (e.g., through a policy gradient method) to update the model.

[0225] In this embodiment, the uncertainty map of noise prediction can be used to adjust the reward feedback information (such as the reward signal). On the one hand, the uncertainty map can identify high uncertainty regions and weight the reward feedback information; on the other hand, it can reduce the model's over-optimization of high uncertainty regions, guide the model to stabilize low uncertainty regions, and improve generation stability; furthermore, the model can adjust parameters according to the updated reward feedback information, gradually improving the image generation quality, and uncertainty weighting can avoid the model overfitting noise, which is beneficial to enhancing robustness.

[0226] Optionally, step B02 updates the image generation model based on the reward feedback information to obtain an updated image generation model, including steps B02a to B02c (not shown in the figure): Step B02a: Based on the reward feedback information, update the image generation model to obtain a preliminary updated image generation model.

[0227] Step B02b: Select the parameters to be reset from the parameters of the initially updated image generation model.

[0228] Step B02c: Reset the parameters to be reset to obtain the updated image generation model.

[0229] Optionally, the model parameters can be updated using gradient descent with an initial reward. Then, parameters to be reset can be selected from the initially updated model parameters, such as through random selection or selection based on uncertainty. For example, in random selection, a subset of parameters (such as output layer weights) can be randomly selected according to a preset proportion (e.g., 10%). In uncertainty-based selection, parameters with high uncertainty can be reset first. Subsequently, the parameters to be reset can be reset, such as reinitializing them to small random values, and an updated image generation model can be obtained based on the reset parameters.

[0230] This application embodiment can enhance the model's exploration ability and avoid local optima by randomly resetting some parameters. On the one hand, the initial optimization of model parameters through initial reward feedback information can improve the quality of image generation; on the other hand, random resetting can prevent the model from over-exploring, and the guidance of uncertainty can make the model focus on parameters with high uncertainty; furthermore, resetting parameters can help the model escape local optima and accelerate convergence.

[0231] In this embodiment, by combining reinforcement learning and diffusion models, dynamic optimization can be achieved. By guiding the model to focus on pixel-level uncertainties rather than global reward scores, the model can be encouraged to learn to generate more robust features that better match the distribution of real data, effectively alleviating the problem of reward over-optimization and enhancing the model's generalization ability. In addition, parameter resetting can prevent local optima, accelerate convergence, and improve the model's ability to generate images.

[0232] The implementation process of the image generation model provided in the embodiments of this application is illustrated below.

[0233] refer to Figure 8 The network architecture shown, taking a text-to-image scenario as an example, works as follows: For the original image, it is first encoded (embedding, which can be mapped to the latent feature space using VAE encoding) and noise is added. Then, a diffusion process is performed to obtain the latent space representation at time T. Next, through T denoising operations (e.g., using U-Net), the features of the target image (i.e., the original image features without noise) are predicted. The original image features are subtracted from the original input of U-Net to obtain the noise prediction. Then, the features are decoded by VAE to finally obtain the target image. For the text part, the text features (such as text embedding) are first obtained through the CLIP (Contrastive Language-Image Pre-training) text branch, and then controlled by the QKV mechanism of U-Net. Diffusion sampling is used to map the features of the noisy image after VAE encoding to the latent space representation at time T. The subsequent image denoising process learns to fit the noise representation, so that the original image representation minus the noise representation is obtained to get the real image representation, and then the decoder D is used to obtain the final real image Y.

[0234] The training process of the image generation model is illustrated below.

[0235] Training setup: The entire training data (e.g., N image-text pairs) is used for M rounds (e.g., 10 rounds) of iterative training. In each iteration, the entire image data is processed in batches of 1s (bs), and the model is updated once for each batch. All batches (N / bs in total) are trained once, meaning all samples are trained on the model at least once; this is called one iteration. Training uses image-text pairs as samples, and the specific process is as follows: Step 1: Initialize parameters before the first batch of training in the first round: Use pre-trained open-source models (such as stable-diffusion models) for VAE, text_encoder, and U-Net. The U-Net parameters can be updated during the training of this application, while other parameters remain unchanged. A learning rate of 0.0004 can be used during initialization. After each 5 rounds of training, the learning rate will be 0.1 times the original rate. A total of M (10) rounds of training will be conducted.

[0236] Step 2: Extract b image-text pairs and input them into the model: For each image, first generate a latent space representation Ei through the encoder, then randomly generate b seeds, and generate corresponding b noise maps (with the same Z dimension) based on these seeds. Superimpose the noise map with the representation Ei to form Z, and then perform a diffusion process to generate... (As the raw input for subsequent U-Net denoising).

[0237] Step 3, Text Information Processing and U-Net Denoising: The text information is processed through CLIP to obtain a text representation, which is then input into the image generation model. The text representation can serve as key and value input information. T noise reduction forward calculations are performed under KV constraints. After the first forward calculation, the result is obtained. Finally, after T iterations, the U-Net output yields the predicted value. Then, image decoding is performed to obtain the predicted image.

[0238] Step 4: Calculate the batch loss: Calculate the generation loss (which can be the MSE loss of the predicted image and the supervised image under the mask or the weighted MSE), and calculate the total loss for the batch of samples. For example, the original MSE loss is used in the first 75% of steps (e.g., the first 1-15 steps of 20 steps); the weighted MSE loss is used in the last 25% of steps.

[0239] Step 5, Parameter Update: Using the SGD (Stochastic Gradient Descent) method, the loss is backpropagated to the image generation model to obtain the gradient of the model parameters (such as U-Net) and update the parameters.

[0240] Based on steps 1 to 5 above, complete all batch training. After completing all N / bs batch training, end one iteration.

[0241] To better implement the above methods, this application also provides an image generation apparatus 900, such as... Figure 9 As shown, the image generation device 900 may include: an information acquisition module 901, used to acquire prompt information for guiding the generation of an image; a model acquisition module 902, used to acquire an image generation model, which is trained based on a predicted noise information map of a sample noisy image and an uncertainty map corresponding to the predicted noise information map, wherein the uncertainty map includes the uncertainty value of a pixel in the predicted noise information map, and the uncertainty value characterizes the reliability of the noise prediction of the corresponding pixel; and a processing module 903, used to perform image generation processing based on the prompt information through the image generation model to obtain a target image.

[0242] Optionally, in some embodiments of this application, the image generation apparatus 900 further includes: The data acquisition module is used to acquire training data, which includes sample images and sample prompts used to guide the generation of the graph. The noise-adding processing module is used to add noise to the sample image to obtain a noisy sample image; The noise prediction module is used to perform noise prediction processing on the sample noisy image based on the sample prompt information through the image generation model, and obtain the predicted noise information map corresponding to the sample noisy image. The pixels in the predicted noise information map represent the noise information corresponding to the pixels at the corresponding positions in the sample noisy image. The calculation and processing module is used to calculate the prediction reliability of the noise information corresponding to the pixel in the prediction noise information map, and obtain the uncertainty map corresponding to the prediction noise information map. The parameter adjustment module is used to adjust the parameters of the image generation model based on the uncertainty map and the prediction noise information map to obtain the trained image generation model.

[0243] Optionally, in some embodiments of this application, when the parameter adjustment module is used to perform parameter adjustment on the image generation model based on the uncertainty map and the prediction noise information map to obtain the trained image generation model, it is specifically used for: Based on the uncertainty value corresponding to each pixel in the uncertainty map, determine the uncertainty type corresponding to each pixel in the uncertainty map. Based on the type of uncertainty, determine the weight information corresponding to each pixel in the uncertainty graph; Based on the predicted noise information map and the real noise corresponding to the sample noisy image, the initial loss information corresponding to the predicted noise information map is determined. Based on the weight information, the initial loss information is updated to obtain the noise prediction loss information corresponding to the prediction noise information map; Based on the noise prediction loss information corresponding to the predicted noise information map, the parameters of the image generation model are adjusted to obtain the trained image generation model.

[0244] Optionally, in some embodiments of this application, when the noise prediction module is used to perform noise prediction processing on the sample noisy image based on sample cue information using an image generation model to obtain a predicted noise information map corresponding to the sample noisy image, it is specifically used for: By using an image generation model, noise prediction processing is performed on the sample noisy image based on the sample prompt information to obtain the predicted noise information map of the sample noisy image in the current iteration round; Based on the predicted noise information map, the sample noisy image is denoised to obtain the denoised image. The denoised image is identified as the new sample image with added noise. Return to the step of performing noise prediction processing on the sample noisy image based on the sample prompt information through the image generation model, until the number of iterations meets the preset condition, and obtain the predicted noise information map corresponding to each iteration.

[0245] Optionally, in some embodiments of this application, when the calculation processing module performs the calculation of the prediction reliability of the noise information corresponding to the pixel in the prediction noise information map to obtain the uncertainty map corresponding to the prediction noise information map, it is specifically used for: Determine the target iteration round that satisfies the preset conditions; The prediction reliability of pixels in the prediction noise information map of the target iteration round is calculated and processed to obtain the uncertainty map corresponding to the target iteration round.

[0246] Optionally, in some embodiments of this application, when the parameter adjustment module is used to perform parameter adjustment on the image generation model based on the uncertainty map and the prediction noise information map to obtain the trained image generation model, it is specifically used for: For the target iteration round, based on the uncertainty map and the corresponding prediction noise information map corresponding to the target iteration round, the noise prediction loss information corresponding to the target iteration round is determined; For non-target iteration rounds, the noise prediction loss information corresponding to the non-target iteration rounds is determined based on the prediction noise information map corresponding to the non-target iteration rounds. The total noise prediction loss information is determined based on the noise prediction loss information corresponding to each target iteration round and the noise prediction loss information corresponding to each non-target iteration round. Based on the total noise prediction loss information, the parameters of the image generation model are adjusted to obtain the trained image generation model.

[0247] Optionally, in some embodiments of this application, when the calculation processing module performs the calculation of the prediction reliability of the noise information corresponding to the pixel in the prediction noise information map to obtain the uncertainty map corresponding to the prediction noise information map, it is specifically used for: Obtain the posterior distribution information of the original parameters of the image generation model under multiple preset training data; Based on the posterior distribution information, the observation information matrix corresponding to the parameter changes of the image generation model is calculated; Based on the observation information matrix, calculate the parameter uncertainty values ​​of the image generation model; Based on the parameter uncertainty value, the prediction reliability of the noise information corresponding to the pixel in the prediction noise information map is calculated and processed to obtain the uncertainty map corresponding to the prediction noise information map.

[0248] Optionally, in some embodiments of this application, when the computation processing module is used to obtain the posterior distribution information corresponding to the original parameters of the image generation model under multiple preset training data, it is specifically used for: Based on a preset prior distribution function, the original parameters of the image generation model are processed to obtain the prior distribution information corresponding to the original parameters; Obtain the likelihood statistics of the original parameters of the image generation model under multiple preset training data; Based on prior distribution information and likelihood statistics, the posterior distribution information corresponding to the original parameters of the image generation model is calculated.

[0249] Optionally, in some embodiments of this application, when the computation processing module is used to perform the calculation of the observation information matrix corresponding to the parameter changes of the image generation model based on posterior distribution information, it is specifically used for: Obtain noise prediction loss information for image generation models under multiple preset training data; Based on the noise prediction loss information corresponding to each preset training data, the target training data is determined from multiple preset training data. The image generation model is updated with parameters based on the noise prediction loss information corresponding to the target training data to obtain the target parameters. Based on the posterior distribution information and target parameters, the observation information matrix corresponding to the parameter changes of the image generation model is calculated.

[0250] Optionally, in some embodiments of this application, when the parameter adjustment module is used to determine the weight information corresponding to each pixel in the uncertainty map according to the uncertainty type, it is specifically used for: Identify at least one target pixel whose uncertainty type satisfies a preset type condition; For a target pixel, the uncertainty region in the uncertainty map is determined based on the uncertainty type of the target pixel's neighboring pixels; Based on the uncertainty value of the pixel in the uncertainty region, set the weight information of the pixel in the uncertainty region; For pixels that do not belong to the uncertainty region in the uncertainty map, the uncertainty value of the pixels is suppressed in order to determine the weight information of the pixels that do not belong to the uncertainty region in the uncertainty map.

[0251] Optionally, in some embodiments of this application, when the parameter adjustment module is used to set the weight information of pixels in the uncertainty region based on the uncertainty value of pixels in the uncertainty region, it is specifically used for: Calculate the local uncertainty value of the uncertainty region based on the uncertainty value of the pixels in the uncertainty region; Calculate the global uncertainty value of the uncertainty map based on the uncertainty values ​​of the pixels in the uncertainty map; Based on the global uncertainty value and the local uncertainty value of the uncertainty region, the weight information of the pixels in the uncertainty region is set.

[0252] Optionally, in some embodiments of this application, when the parameter adjustment module is used to determine the uncertainty region in the uncertainty map based on the uncertainty type of the target pixel's neighboring pixels, it is specifically used for: The uncertainty type of the target pixel's neighboring pixels is detected, so as to select the target neighboring pixels whose uncertainty type meets the preset type condition; Based on the target's neighboring pixels and the target pixel, the uncertainty region in the uncertainty map is determined.

[0253] Optionally, in some embodiments of this application, when the parameter adjustment module is used to set the weight information of pixels in the uncertainty region based on the global uncertainty value and the local uncertainty value of the uncertainty region, it is specifically used for: When the local uncertainty value of an uncertain region is greater than the global uncertainty value, the weight information of the pixels in the uncertain region is set based on the local uncertainty value. When the local uncertainty value of the uncertainty region is not greater than the global uncertainty value, the local uncertainty value of the uncertainty region is suppressed to obtain the suppressed uncertainty value, and the weight information of the pixels in the uncertainty region is set according to the suppressed uncertainty value.

[0254] Optionally, in some embodiments of this application, the noise prediction module is further configured to: before determining the denoised image as a new sample noisy image. Based on the denoised image and the expected denoised image corresponding to the sample denoised image, determine the reward feedback information for the current iteration round; Based on the reward feedback information, the image generation model is updated to obtain the updated image generation model.

[0255] Optionally, in some embodiments of this application, when the noise prediction module is used to update the image generation model based on reward feedback information to obtain the updated image generation model, it is specifically used for: Determine the uncertainty map corresponding to the prediction noise information map of the current iteration round; Based on the uncertainty graph, the reward feedback information is updated to obtain the updated reward feedback information; Based on the updated reward feedback information, the image generation model is updated to obtain the updated image generation model.

[0256] Optionally, in some embodiments of this application, when the noise prediction module is used to update the image generation model based on reward feedback information to obtain the updated image generation model, it is specifically used for: Based on the reward feedback information, the image generation model is updated to obtain a preliminary updated image generation model; Select the parameters to be reset from the parameters of the initially updated image generation model; The parameters to be reset are reset to obtain the updated image generation model.

[0257] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0258] As can be seen from the above, in this embodiment, the information acquisition module 901 can acquire the prompt information used to guide the generation of the image; the model acquisition module 902 can acquire the image generation model, which is trained based on the predicted noise information map of the sample noisy image and the uncertainty map corresponding to the predicted noise information map. The uncertainty map includes the uncertainty value of the pixel in the predicted noise information map, and the uncertainty value represents the reliability of the noise prediction of the corresponding pixel; the processing module 903 performs image generation processing based on the prompt information through the image generation model to obtain the target image.

[0259] The implementation of this application can reflect the difference in the reliability of noise prediction corresponding to different pixels in the prediction noise information map through the uncertainty map. The model can accurately locate each local region in the image, so that the model can process local details in a targeted manner during training. This effectively improves the problem of insufficient identification and correction of local defects in related technologies, and achieves accurate processing of local details in the image generation process, thereby improving the accuracy of image details and the quality of the generated image.

[0260] This application also provides an electronic device, such as... Figure 10 As shown, it includes a memory 402 and a processor 401. The memory 402 stores a computer program, and the processor 401 is used to run the computer program in the memory 402 to perform the steps in the image generation method described above in the embodiments of this application.

[0261] Optional, Figure 10 The diagram shows a schematic representation of the electronic device involved in an embodiment of this application. This electronic device may be a terminal or a server, etc. Specifically: The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 10 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes software programs and / or modules stored in the memory 402, and calls data stored in the memory 402, to perform various functions and process data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0262] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0263] The electronic device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0264] The electronic device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0265] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to execute the steps in any of the image generation methods provided in the embodiments of this application.

[0266] As can be seen from the above, the embodiments of this application provide an image generation method, apparatus, device, readable storage medium, and program product. Specifically, after obtaining prompt information for guiding the generation of an image, an image generation model can be used to perform image generation processing based on the prompt information to obtain a target image. The image generation model is trained based on a predicted noise information map of a sample noisy image and an uncertainty map corresponding to the predicted noise information map. The uncertainty map includes the uncertainty values ​​of pixels in the predicted noise information map, and the uncertainty values ​​characterize the reliability of the noise prediction for the corresponding pixel. In this application, the uncertainty map can reflect the differences in the reliability of noise prediction for different pixels in the predicted noise information map. Pixel-level fine-grained supervision based on the uncertainty map allows the model to specifically correct high-uncertainty local details during training, suppressing local defects and effectively improving the problem of insufficient identification and correction of local defects in related technologies. Therefore, during the image generation process, the trained image generation model can achieve accurate processing of local image details, improving the accuracy of image details and the quality of the generated image.

[0267] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0268] Therefore, embodiments of this application provide a computer-readable storage medium storing a computer program that is adapted to be loaded by a processor to perform the steps in any of the image generation methods provided in embodiments of this application.

[0269] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0270] Since the computer program stored in the computer-readable storage medium can execute the steps of any of the image generation methods provided in the embodiments of this application, the beneficial effects that any of the image generation methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0271] According to one aspect of this application, a computer program product is provided, which includes a computer program that, when executed by a processor, can implement the steps in the image generation method described above in the embodiments of this application.

[0272] The processor of the electronic device reads a computer program from a computer-readable storage medium, executes the computer program, and causes the electronic device to perform the methods provided in the various alternative implementations of the above-described image generation aspects.

[0273] The foregoing has provided a detailed description of an image generation method, apparatus, device, readable storage medium, and program product provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image generation method, characterized in that, include: Obtain prompts to guide the generation of the graph; An image generation model is obtained, which is trained based on a predicted noise information map of a sample noisy image and an uncertainty map corresponding to the predicted noise information map. The uncertainty map includes the uncertainty value of a pixel in the predicted noise information map, and the uncertainty value characterizes the reliability of the noise prediction for the corresponding pixel. The image generation model is used to generate the target image based on the prompt information.

2. The method according to claim 1, characterized in that, Before obtaining the prompt information used to guide the generation of the graph, the process also includes: Acquire training data, which includes sample images and sample prompts used to guide the generation of the graph; The sample image is subjected to noise processing to obtain a noisy sample image; Using the image generation model, noise prediction processing is performed on the sample noisy image based on the sample prompt information to obtain the predicted noise information map corresponding to the sample noisy image. The pixels in the predicted noise information map represent the noise information corresponding to the pixels at the corresponding positions in the sample noisy image. The prediction reliability is calculated by processing the noise information corresponding to the pixels in the prediction noise information map to obtain the uncertainty map corresponding to the prediction noise information map. Based on the uncertainty map and the prediction noise information map, the parameters of the image generation model are adjusted to obtain the trained image generation model.

3. The method according to claim 2, characterized in that, The step of adjusting the parameters of the image generation model based on the uncertainty map and the predicted noise information map to obtain the trained image generation model includes: Based on the uncertainty value corresponding to each pixel in the uncertainty map, the uncertainty type corresponding to each pixel in the uncertainty map is determined. Based on the uncertainty type, determine the weight information corresponding to each pixel in the uncertainty map; Based on the predicted noise information map and the real noise corresponding to the sample noisy image, the initial loss information corresponding to the predicted noise information map is determined. The initial loss information is updated based on the weight information to obtain the noise prediction loss information corresponding to the prediction noise information map; Based on the noise prediction loss information corresponding to the predicted noise information map, the parameters of the image generation model are adjusted to obtain the trained image generation model.

4. The method according to claim 2, characterized in that, The step of performing noise prediction processing on the noisy sample image based on the sample prompt information using the image generation model to obtain the predicted noise information map corresponding to the noisy sample image includes: Using the image generation model, noise prediction processing is performed on the sample noisy image based on the sample prompt information to obtain the predicted noise information map of the sample noisy image in the current iteration round; Based on the predicted noise information map, the sample noisy image is denoised to obtain a denoised image; The denoised image is then identified as a new sample image with added noise. Return to the step of performing noise prediction processing on the sample noisy image based on the sample prompt information using the image generation model, until the number of iterations meets the preset condition, and obtain the predicted noise information map corresponding to each iteration.

5. The method according to claim 4, characterized in that, The step of calculating the prediction reliability of the noise information corresponding to the pixels in the predicted noise information map to obtain the uncertainty map corresponding to the predicted noise information map includes: Determine the target iteration round that satisfies the preset conditions; The prediction reliability of pixels in the prediction noise information map of the target iteration round is calculated to obtain the uncertainty map corresponding to the target iteration round.

6. The method according to claim 5, characterized in that, The step of adjusting the parameters of the image generation model based on the uncertainty map and the predicted noise information map to obtain the trained image generation model includes: For the target iteration round, based on the uncertainty map and the corresponding prediction noise information map corresponding to the target iteration round, the noise prediction loss information corresponding to the target iteration round is determined; For non-target iteration rounds, based on the prediction noise information map corresponding to the non-target iteration round, the noise prediction loss information corresponding to the non-target iteration round is determined; The total noise prediction loss information is determined based on the noise prediction loss information corresponding to each target iteration round and the noise prediction loss information corresponding to each non-target iteration round. Based on the total noise prediction loss information, the parameters of the image generation model are adjusted to obtain the trained image generation model.

7. The method according to claim 2, characterized in that, The step of calculating the prediction reliability of the noise information corresponding to the pixels in the predicted noise information map to obtain the uncertainty map corresponding to the predicted noise information map includes: Obtain the posterior distribution information of the original parameters of the image generation model under multiple preset training data; Based on the posterior distribution information, the observation information matrix corresponding to the parameter changes of the image generation model is calculated; Based on the observation information matrix, calculate the parameter uncertainty value of the image generation model; Based on the uncertainty value of the parameters, the prediction reliability of the noise information corresponding to the pixels in the prediction noise information map is calculated to obtain the uncertainty map corresponding to the prediction noise information map.

8. The method according to claim 7, characterized in that, The step of obtaining the posterior distribution information corresponding to the original parameters of the image generation model under multiple preset training data includes: Based on a preset prior distribution function, the original parameters of the image generation model are processed to obtain the prior distribution information corresponding to the original parameters; Obtain the likelihood statistics of the original parameters of the image generation model under multiple preset training data; Based on the prior distribution information and the likelihood statistics, the posterior distribution information corresponding to the original parameters of the image generation model is calculated.

9. The method according to claim 7, characterized in that, The step of calculating the observation information matrix corresponding to the parameter changes of the image generation model based on the posterior distribution information includes: Obtain the noise prediction loss information corresponding to the image generation model under the multiple preset training data; Based on the noise prediction loss information corresponding to each preset training data, the target training data is determined from the plurality of preset training data; The image generation model is updated with parameters based on the noise prediction loss information corresponding to the target training data to obtain the target parameters. Based on the posterior distribution information and the target parameters, the observation information matrix corresponding to the parameter changes of the image generation model is calculated.

10. The method according to claim 3, characterized in that, The step of determining the weight information corresponding to each pixel in the uncertainty map based on the uncertainty type includes: Determine at least one target pixel that satisfies the preset type condition for the uncertainty type; For the target pixel, the uncertainty region in the uncertainty map is determined based on the uncertainty type of the neighboring pixels of the target pixel; Based on the uncertainty value of the pixels in the uncertainty region, set the weight information of the pixels in the uncertainty region; For pixels in the uncertainty map that do not belong to the uncertainty region, the uncertainty value of the pixels is suppressed to determine the weight information of the pixels in the uncertainty map that do not belong to the uncertainty region.

11. The method according to claim 10, characterized in that, The step of setting weight information for pixels in the uncertainty region based on their uncertainty values ​​includes: Based on the uncertainty values ​​of the pixels in the uncertainty region, calculate the local uncertainty value of the uncertainty region; Calculate the global uncertainty value of the uncertainty map based on the uncertainty values ​​of the pixels in the uncertainty map; Based on the global uncertainty value and the local uncertainty value of the uncertainty region, the weight information of the pixels in the uncertainty region is set.

12. The method according to claim 10, characterized in that, The step of determining the uncertainty region in the uncertainty map based on the uncertainty type of the neighboring pixels of the target pixel includes: The uncertainty type of the neighboring pixels of the target pixel is detected, so as to select the target neighboring pixels whose uncertainty type meets the preset type condition from the neighboring pixels; Based on the target's neighboring pixels and the target pixel, the uncertainty region in the uncertainty map is determined.

13. The method according to claim 11, characterized in that, The step of setting the weight information of pixels in the uncertainty region based on the global uncertainty value and the local uncertainty value of the uncertainty region includes: When the local uncertainty value of the uncertainty region is greater than the global uncertainty value, the weight information of the pixels in the uncertainty region is set based on the local uncertainty value of the uncertainty region. When the local uncertainty value of the uncertainty region is not greater than the global uncertainty value, the local uncertainty value of the uncertainty region is suppressed to obtain the suppressed uncertainty value, and the weight information of the pixels in the uncertainty region is set according to the suppressed uncertainty value.

14. The method according to claim 4, characterized in that, Before determining the denoised image as the new sample noisy image, the method further includes: Based on the denoised image and the expected denoised image corresponding to the sample denoised image, determine the reward feedback information for the current iteration round; Based on the reward feedback information, the image generation model is updated to obtain the updated image generation model.

15. The method according to claim 14, characterized in that, The step of updating the image generation model based on the reward feedback information to obtain the updated image generation model includes: Determine the uncertainty map corresponding to the prediction noise information map of the current iteration round; Based on the uncertainty graph, the reward feedback information is updated to obtain the updated reward feedback information; Based on the updated reward feedback information, the image generation model is updated to obtain the updated image generation model.

16. The method according to claim 14, characterized in that, The step of updating the image generation model based on the reward feedback information to obtain the updated image generation model includes: Based on the reward feedback information, the image generation model is updated to obtain a preliminary updated image generation model; Select the parameters to be reset from the parameters of the initially updated image generation model; The parameters to be reset are reset to obtain the updated image generation model.

17. An image generation apparatus, characterized in that, include: The information acquisition module is used to acquire prompts that guide the generation of the diagram; The model acquisition module is used to acquire an image generation model. The image generation model is trained based on the predicted noise information map of the sample noisy image and the uncertainty map corresponding to the predicted noise information map. The uncertainty map includes the uncertainty value of the pixel in the predicted noise information map, and the uncertainty value characterizes the reliability of the noise prediction of the corresponding pixel. The processing module is used to perform image generation processing based on the prompt information using the image generation model to obtain the target image.

18. An electronic device, characterized in that, It includes a memory and a processor; the memory stores a computer program, and the processor is configured to run the computer program in the memory to perform the method according to any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to perform the method according to any one of claims 1 to 16.

20. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the method described in any one of claims 1 to 16.

Images