Image generation method and device, equipment, medium and product

Abstract

The invention discloses an image generation method and device, equipment, a medium and a product, and relates to the field of image processing. The method comprises the following steps: acquiring a first image and a gray level change graph, and acquiring a noise image; in the process of carrying out iterative noise reduction on the noise image by taking modification of the first image as a noise reduction condition, obtaining the ith noise image corresponding to the ith iteration noise reduction stage; when i = 1, performing noise reduction on the first noise image to obtain a first noise reduction stage image output in a first iteration noise reduction stage; when i is greater than 1, based on a fusion proportion indicated by a second pixel value in the gray level change graph, fusing the i-th noise image and the (i-1)-th noise reduction stage image output in the (i-1)-th iteration noise reduction stage to obtain an i-th fused image; and carrying out noise reduction on the ith fusion image to obtain an ith noise reduction stage image output in the ith iteration noise reduction stage until the iteration noise reduction requirement is met, and obtaining a generated image as a modification result of the first image. The method can improve the generation effect of the image modification result.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image generation method, device, equipment, medium and product. Background Art

[0002] Image generation technologies include text-to-image (T2I) and image-to-image (I2I). In the image-to-image function based on the diffusion model, the diffusion model converts the input original image into a target image based on the original image and the text prompt, thereby realizing image style transfer or image synthesis.

[0003] In the related art, if one wants to retain some details in the original image, it is necessary to implement it through an image inpainting function based on a diffusion model, that is, the user selects the area in the original image that needs to be redrawn using a brush tool, and obtains a mask image after adding a mask to the area that needs to be redrawn. The area in the mask image that needs to be redrawn is black, and the area that needs to be retained is the original pixel of the original image. After the mask image is input into the diffusion model, the diffusion model implements image inpainting of the black area in the mask image through model association, thereby obtaining the target image.

[0004] However, when generating a target image, the above solution may easily lead to an obvious sense of separation at the edge where the redrawn area and the original area meet, resulting in a poor image generation effect. Summary of the invention

[0005] The embodiment of the present application provides a method, device, equipment, medium and product for generating an image. The technical solution is as follows:

[0006] In one aspect, a method for generating an image is provided, the method comprising:

[0007] Acquire a first image and a grayscale change map, and acquire a noise image; wherein a first pixel position of the first image corresponds to a first pixel value, a second pixel position corresponding to the first pixel position in the grayscale change map corresponds to a second pixel value, and the second pixel value is used to indicate the change intensity of the first pixel position when generating a new image;

[0008] In the process of iteratively denoising the noise image by modifying the first image as a denoising condition, obtaining an i-th noise image corresponding to the i-th round of iterative denoising stage, i is a positive integer, when i=1, the noise image is an initial noise image, when i>1, the noise intensity of the i-th noise image corresponds to the i-th round of iterative denoising stage;

[0009] When i=1, the first noise image is denoised to obtain the first denoised stage image outputted by the first round of iterative denoising stage;

[0010] When i>1, based on the fusion ratio indicated by the second pixel value in the grayscale change map, the i-th noise image and the i-1th noise reduction stage image outputted from the i-1th iterative noise reduction stage are fused to obtain the i-th fused image;

[0011] Denoising is performed on the i-th fused image to obtain an i-th denoising stage image outputted in the i-th round of iterative denoising stage, until it meets the iterative denoising requirements, and a generated image is obtained as a modification result of the first image.

[0012] In another aspect, a device for generating an image includes:

[0013] A first acquisition module, used to acquire a first image and a grayscale change map, and to acquire a noise image; wherein a first pixel position of the first image corresponds to a first pixel value, a second pixel position corresponding to the first pixel position in the grayscale change map corresponds to a second pixel value, and the second pixel value is used to indicate a change intensity of the first pixel position when generating a new image;

[0014] A second acquisition module is used to acquire an i-th noise image corresponding to an i-th round of iterative noise reduction stage during iterative noise reduction of the noise image by modifying the first image as a noise reduction condition, where i is a positive integer, and when i=1, the noise image is an initial noise image, and when i>1, the noise intensity of the i-th noise image corresponds to the i-th round of iterative noise reduction stage;

[0015] A denoising module, used for denoising the first noise image when i=1, to obtain the first denoising stage image outputted by the first round of iterative denoising stage;

[0016] a fusion module, configured to fuse the i-th noise image and the i-1th noise reduction stage image outputted from the i-1th iterative noise reduction stage based on a fusion ratio indicated by a second pixel value in the grayscale change map when i>1, to obtain an i-th fused image;

[0017] The denoising module is further used to denoise the i-th fused image to obtain the i-th denoising stage image outputted by the i-th round of iterative denoising stage, until it meets the iterative denoising requirements, and obtain a generated image as a modification result of the first image.

[0018] In some optional embodiments, the fusion module includes:

[0019] A screening unit, used for converting the grayscale change map into an i-th mask corresponding to the i-th round of iterative denoising stage, wherein the i-th mask is used for indicating the regions corresponding to the i-th noise image and the i-1-th denoising stage image in the i-th round of iterative denoising stage when participating in fusion;

[0020] A fusion unit is used to fuse the i-th noise image and the i-1-th noise reduction stage image based on the i-th mask to obtain the i-th fused image.

[0021] In some optional embodiments, the fusion module further includes:

[0022] A first acquisition unit, used for acquiring an i-th pixel threshold corresponding to the i-th round of iterative noise reduction stage, wherein the i-th pixel threshold is used for selecting the second pixel position according to the second pixel value in the grayscale change map;

[0023] The screening unit is further configured to screen the second pixel value in the grayscale change map by using an i-th pixel threshold to obtain the i-th mask.

[0024] In some optional embodiments, the screening unit is further used to screen the second pixel positions in the grayscale change map where the second pixel value is greater than the i-th pixel threshold to obtain a first mask map;

[0025] The screening unit is further used to screen the first pixels whose second pixel values ​​in the grayscale change map are less than or equal to the i-th pixel threshold to obtain a second mask map;

[0026] The fusion unit is further used to superimpose the first mask image and the i-th noise image to obtain a first superimposed image;

[0027] The fusion unit is further used to superimpose the second mask image and the image in the (i-1)th denoising stage to obtain a second superimposed image;

[0028] The fusion unit is further configured to fuse the first superimposed image and the second superimposed image to obtain the i-th fused image.

[0029] In some optional embodiments, the device further comprises:

[0030] The noise adding module is used to add noise to the first image to obtain the noise image.

[0031] In some optional embodiments, the noise adding module includes:

[0032] A first sampling unit, used for sampling standard Gaussian noise to obtain the sampled noise;

[0033] A noise adding unit is used to add noise to the first image according to the sampling noise and the diffusion time to obtain the noise image, wherein the diffusion time is used to indicate the number of steps required to generate the generated image from the noise image during the iterative denoising process.

[0034] In some optional embodiments, the second acquisition module further includes:

[0035] A second sampling unit is used to sample the diffusion duration to obtain an i-th sampling moment corresponding to the i-th round of iterative noise reduction stage;

[0036] A noise reduction unit, used for, when i=1, using the noise image as the first noise image corresponding to the first sampling moment;

[0037] The noise reduction unit is further used to, when i>1, perform noise reduction on the noise image to obtain the i-th noise image corresponding to the i-th sampling moment; or, when i>1, perform noise reduction on the i-1th noise image corresponding to the i-1th sampling moment to obtain the i-th noise image corresponding to the i-th sampling moment.

[0038] In some optional embodiments, the device further comprises:

[0039] an encoding module, configured to input the first image into a first encoder of the diffusion model to obtain an image encoding representation;

[0040] The encoding module is further used to input the grayscale change map into a second encoder of the diffusion model to obtain a change map encoding representation;

[0041] The noise adding module is further used to input the image encoding representation into the forward diffusion network in the diffusion model to obtain the noise image feature representation corresponding to the noise image;

[0042] The denoising module is further used to input the noise image feature representation and the variation map encoding representation into a reverse diffusion network in the diffusion model to obtain a generated image feature representation;

[0043] A decoding module is used to input the generated image feature representation into a decoder of the diffusion model to obtain the generated image.

[0044] In some optional embodiments, the first acquisition module is further used to acquire a prompt word, where the prompt word is used to guide image changes when a new image is generated based on the first image through the diffusion model;

[0045] The encoding module is further used to input the prompt word into the third encoder of the diffusion model to obtain a text encoding representation;

[0046] The denoising module is further used to input the noise image feature representation, the change map encoding representation and the text encoding representation into the reverse diffusion network in the diffusion model to obtain the generated image feature representation.

[0047] In some optional embodiments, the prompt word includes at least one of the following contents:

[0048] Generate content prompt information, where the generated content prompt information is used to indicate the semantics of content that needs to be generated when generating a new image based on the first image;

[0049] generating style hint information, wherein the generated style hint information is used to indicate an image style of the new image generated based on the first image;

[0050] Image attribute prompt information, where the image attribute prompt information is used to indicate image attributes of the new image generated based on the first image.

[0051] In some optional embodiments, the first acquisition module includes:

[0052] an identification unit, configured to identify image content in the first image and determine a plurality of image subjects existing in the first image;

[0053] a segmentation unit, configured to segment the first image into subject areas corresponding to the plurality of image subjects respectively;

[0054] A second acquisition unit is used to acquire subject modification requirement information corresponding to each of the plurality of image subjects;

[0055] A processing unit is used to perform grayscale processing on the main body area of ​​the first image based on the main body modification requirement information to obtain the grayscale change map.

[0056] In some optional embodiments, the processing unit is further configured to perform grayscale processing on the subject area of ​​the first image based on the subject modification requirement information to obtain a grayscale image corresponding to the first image;

[0057] The second acquisition unit is further used to receive a region selection instruction for the grayscale image, the region selection instruction is used to indicate at least one target region, the target region is a region where the first pixel value of the first image is adjusted with a specified degree of change;

[0058] The processing unit is further used to adjust the pixel values ​​in the at least one target area in the grayscale image to the target grayscale value corresponding to the specified change degree based on the area selection instruction, so as to obtain the grayscale change map.

[0059] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement an image generation method as described in any of the above-mentioned embodiments of the present application.

[0060] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the storage medium, and the at least one computer program is loaded and executed by a processor to implement the image generation method as described in any of the above embodiments of the present application.

[0061] On the other hand, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the image generation method as described in any one of the above embodiments is implemented.

[0062] The technical solution provided by this application includes at least the following beneficial effects:

[0063] When a new image is generated based on the first image, the grayscale change map is used to control the area in the first image that needs to be modified or retained, and the modification intensity in the modification process is controlled. That is, in the process of iteratively denoising the acquired noisy image with the modification of the first image as the denoising condition, the second pixel value in the grayscale change map is used to control the fusion ratio between the original image content carried by the first image and the new image content generated from the noise, thereby achieving an image modification result with controllable generation intensity at the pixel level. Under the guidance of the grayscale change map, there are no longer only two generation options, namely, change and unchanged. The modification of the original image content can be more freely controllable. At the same time, there is a better transition effect for the junction of the change and the unchanged in the picture, thereby improving the generation effect of the image modification result. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0065] Figure 1 is a structural block diagram of a computer system provided by an exemplary embodiment of the present application;

[0066] Figure 2 is a flow chart of a method for generating an image provided by an exemplary embodiment of the present application;

[0067] Figure 3 is an example diagram of a grayscale change diagram provided by an exemplary embodiment of the present application;

[0068] Figure 4 is a schematic diagram of generating a grayscale change map provided by an exemplary embodiment of the present application;

[0069] Figure 5 It is a schematic diagram of the generation effect of generating a controllable variation range through a grayscale variation map provided by an exemplary embodiment of the present application;

[0070] Figure 6 is a flow chart of a method for generating an image provided by an exemplary embodiment of the present application;

[0071] Figure 7 is a schematic diagram of extracting a noise image and a previous round of noise reduction image by masking in multiple iterative noise reduction stages provided by an exemplary embodiment of the present application;

[0072] Figure 8 is a schematic diagram of an image generation process provided by an exemplary embodiment of the present application;

[0073] Fig. 9 is a schematic diagram of a diffusion model provided by an exemplary embodiment of the present application;

[0074] Fig.10 is a flow chart of a data processing process of a diffusion model provided by an exemplary embodiment of the present application;

[0075] Fig.11 It is a schematic diagram of the expression form of a grayscale change graph provided by an exemplary embodiment of the present application;

[0076] Fig.12 It is a schematic diagram of an interface for custom editing of a grayscale change graph provided by an exemplary embodiment of the present application;

[0077] Fig.13 is a schematic diagram of an image generation process provided by an exemplary embodiment of the present application;

[0078] Fig.14 is a structural block diagram of an image generation device provided by an exemplary embodiment of the present application;

[0079] Fig.15 is a structural block diagram of an image generation device provided by an exemplary embodiment of the present application;

[0080] Fig.16 It is a schematic diagram of the structure of a server provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solution and advantages of the present application clearer, the implementation mode of the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0082] In this application, the terms "first", "second", etc. are used to distinguish identical items or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first" and "second", nor are there any limitations on quantity and execution order.

[0083] First, a brief introduction is given to the terms involved in the embodiments of the present application.

[0084] Artificial Intelligence Generated Content (AIGC): refers to content generated by artificial intelligence (AI) technology. This content can include text, images, audio, video and other forms. AIGC technology uses machine learning and deep learning algorithms to automatically generate high-quality content, reducing the time and cost of manual creation.

[0085] The working principle of AIGC mainly relies on the following key technologies and steps:

[0086] 1. Data collection and preprocessing: First, a large amount of training data needs to be collected. The training data includes text, images, audio, video and other forms. The collected data needs to be cleaned, labeled and formatted to facilitate model training.

[0087] 2. Model selection and training: Select a suitable machine learning or deep learning model. Commonly used models include Generative Adversarial Networks (GAN), Variational Autoencoder (VAE), Transformer models, etc. Use preprocessed data to train the model. The training process continuously adjusts the model parameters to enable it to generate high-quality content.

[0088] 3. Generation and optimization: The trained model can generate content based on input conditions or random noise. For example, a text generation model can generate articles based on input keywords, and an image generation model can generate images based on input labels. The generated content may need further optimization and adjustment to improve quality and meet specific needs. This step can be achieved through post-processing techniques or further model fine-tuning.

[0089] 4. Evaluation and iteration: Evaluate the generated content to ensure its quality and compliance with expectations. Evaluation can be performed through automated metrics (e.g., the Bilingual Evaluation Understudy (BLEU) metric) or manual review. Based on the evaluation results, the model is iterated and optimized to continuously improve the quality of the generated content.

[0090] Diffusion Model: It is a generative model whose core principle is to simulate a forward diffusion process that gradually transitions from data distribution to simple distribution (usually Gaussian noise), and a reverse denoising process that restores from simple distribution to original data distribution. Structurally, a diffusion model usually consists of a forward diffusion process and a trainable reverse denoising process. The forward process gradually adds Gaussian noise to the image until pure noise is finally obtained; the reverse process is to train a neural network, starting from pure noise and gradually denoising until a real image is obtained. Diffusion models have a wide range of application scenarios, including but not limited to image super-resolution, image restoration, object detection, video generation, etc.

[0091] Stable Diffusion Model: It is a generation technology based on the diffusion model, which is used to generate high-quality images. Stable Diffusion can generate clear and detailed images through a stable diffusion process. Stable Diffusion models usually include the following key parts: forward diffusion process, reverse diffusion process, noise prediction network and sampling process.

[0092] The image-to-image (I2I) function of the stable diffusion model is a function implemented by image conversion technology, which realizes image style migration, style conversion or image synthesis by converting the input image into the target image. The stable diffusion I2I method uses the stability principle in the diffusion process to convert the input image into the target image in a reasonable way by analyzing the feature differences between the input image and the target image, as well as their contextual information and local features. This method is usually based on deep learning models, such as generative adversarial networks (GANs) or variational autoencoders (VAEs), and achieves high-quality image conversion effects by learning the mapping relationship between the input image and the target image. The stable diffusion I2I function can be applied to multiple fields such as image style migration, image conversion, image synthesis, etc., providing users with a rich variety of image processing and creation tools.

[0093] The image inpainting function of the stable diffusion model refers to a function implemented by image restoration technology, which restores the integrity of the image by filling in the missing or damaged parts of the image. The stable diffusion inpainting method uses the stability principle of the diffusion process. By analyzing the contextual information and local features of the image, it fills the missing areas in a reasonable way so that the repaired image remains visually smooth and continuous. This method usually performs repairs based on features such as image texture, color, and structure to maintain the appearance and content of the original image as much as possible.

[0094] In the related art, when realizing image generation through a diffusion model, if you want to retain some details in the original image, you need to use the image inpainting function based on the diffusion model to achieve this, that is, the user selects the area that needs to be redrawn in the original image through a brush tool, and obtains a mask image after adding a mask to the area that needs to be redrawn. The area that needs to be redrawn in the mask image is black, and the area that needs to be retained is the original pixel of the original image. After the mask image is input into the diffusion model, the diffusion model implements image inpainting of the black area in the mask image through model association, thereby obtaining the target image.

[0095] However, when the target image is generated through the Inpainting function, the generated intensity of the image content in the redrawn area is uniform, that is, the change intensity of the overall image can only be specified through Prompt, and the change intensity of different areas cannot be specified, which will cause an obvious sense of separation at the edge where the redrawn area and the original area meet, and the image generation effect is poor. In addition, this inpainting mode requires retraining and fine-tuning of the diffusion model, which has certain usage and training costs.

[0096] In the image generation method provided in the embodiment of the present application, when a new image is generated based on a first image, a grayscale change map is used to control the area in the first image that needs to be modified or retained, as well as the modification intensity during the modification process. That is, in the process of iteratively denoising the acquired noisy image with the modification of the first image as the denoising condition, the second pixel value in the grayscale change map is used to control the fusion ratio between the original image content carried by the first image and the new image content generated from the noise, thereby achieving an image modification result with controllable generation intensity at the pixel level. Under the guidance of the grayscale change map, there are no longer only two generation options, namely, change and unchanged. The modification of the original image content can be more freely controllable. At the same time, there is a better transition effect at the junction of the change and the unchanged in the picture, thereby improving the generation effect of the image modification result.

[0097] Figure 1 The structure block diagram of a computer system provided by an exemplary embodiment of the present application is shown. The computer system 100 includes: a terminal 110 and a server 120 .

[0098] The terminal 110 has installed and runs an application that provides an image generation function. The application may be at least one of the following: a game application, a social application, an audio and video application, a live broadcast application, an e-commerce application, an application store, a browser, and the like.

[0099] The device types of the terminal 110 include: game consoles, extended reality (XR) (including virtual reality (VR), augmented reality (AR), mixed reality (MR)) devices, desktop computers, smart phones, tablet computers, e-book readers, Moving Picture Experts Group Audio Layer III (MP3) players, Moving Picture Experts Group Audio Layer IV (MP4) players, smart home appliances, vehicle-mounted devices, laptop computers, etc. The following embodiments are illustrated by taking the terminal 110 as a desktop computer.

[0100] The terminal 110 is connected to the server 120 via a wireless network or a wired network.

[0101] Those skilled in the art will appreciate that the number of the above devices may be more or less. For example, the above device may be only one, or the above devices may be dozens or hundreds, or more. The embodiment of the present application does not limit the number and type of devices.

[0102] The server 120 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. In some embodiments, the server 120 may also be implemented as a node in a blockchain system.

[0103] The server 120 is used to provide background services for the image generation function. Optionally, the server 120 undertakes the main computing work and the terminal 110 undertakes the secondary computing work; or, the server 120 undertakes the secondary computing work and the terminal 110 undertakes the main computing work; or, the server 120 and the terminal 110 adopt a distributed computing architecture to perform collaborative computing.

[0104] It is worth noting that the above-mentioned server 120 can be implemented as a physical server or as a cloud server in the cloud.

[0105] Schematically, the user uploads the first image and the grayscale change map in the application of the terminal 110, and indicates to generate a new image based on the first image. The terminal 110 sends the first image and the grayscale change map to the server 120, and sends an image generation request to the server 120. After receiving the image generation request, the server 120 inputs the first image and the grayscale change map into the image generation service module, and the image generation service module inputs the first image and the grayscale change map into the pre-trained diffusion model, obtains the noise image through the diffusion model, and in the process of iteratively denoising the noise image with the modification of the first image as the denoising condition, obtains the i-th noise image corresponding to the i-th round of iterative denoising stage, and fuses the i-th noise image and the i-1th denoising stage image obtained by the i-1th round of iterative denoising based on the fusion ratio indicated by the second pixel value in the grayscale change map to obtain the i-th fused image, denoises the i-th fused image, and obtains the i-th denoising stage image output by the i-th round of iterative denoising stage, until the iterative denoising requirements are met, and the generated image is obtained as the modification result of the first image. The server 120 sends the obtained generated image to the terminal 110 , and the terminal 110 renders and displays the generated image on the screen.

[0106] It is worth noting that the data processing process of the image generation method provided in the embodiment of the present application can also be completed independently by the terminal 110, that is, the terminal 110 is equipped with an AI chip with a pre-trained diffusion model, and the process of generating an image based on the first image can be completed locally on the terminal 110 by calling the AI ​​chip.

[0107] In combination with the above-mentioned noun introduction and implementation environment, the image generation method provided by the present application is described. Schematically, the method is executed by a computer device. Optionally, the computer device can be at least one of a terminal or a server. In the embodiment of the present application, the computer device is a server as an example. Figure 2As shown, the method includes the following steps 210 to 250.

[0108] Step 210: Acquire a first image and a grayscale change map, and acquire a noise image.

[0109] The first image refers to a two-dimensional digital matrix composed of pixels. In the embodiment of the present application, the first image is an image to be modified. The first image is used to generate a new image, and the first pixel position of the first image corresponds to a first pixel value.

[0110] Optionally, the first image may be RGB image data, that is, each pixel corresponds to three color channels; the first image may also be image data obtained by channel merging and normalizing an RGB image to a floating point number between 0 and 1.

[0111] In the embodiment of the present application, a grayscale change map (Change Map) is a map containing grayscale information, and the pixel value in the grayscale change map is an intensity value used to represent the brightness of the pixel, and this intensity value usually ranges from 0 (black) to 255 (white). In the embodiment of the present application, the pixel value in the grayscale change map is used to indicate the intensity of the change of the pixel during the process of modifying the first image, that is, the second pixel position corresponding to the first pixel position in the grayscale change map corresponds to the second pixel value, and the second pixel value is used to indicate the intensity of the change of the first pixel position when generating a new image.

[0112] Optionally, the grayscale change map may be obtained by at least one of the following:

[0113] The first one is to select a grayscale change map from the candidate grayscale maps.

[0114] The candidate grayscale image is a grayscale image preset and provided by the system.

[0115] Optionally, the candidate grayscale image includes a gradient grayscale image or a pattern grayscale image. When the candidate grayscale image is a gradient grayscale image, the gradient type of the candidate grayscale image includes at least one of linear gradient, ray gradient, rectangular gradient, and path gradient, and the gradient direction includes at least one of diagonal direction, up-down direction, left-right direction, and central radiation direction. When the candidate grayscale image is a pattern grayscale image, the grayscale pattern in the candidate grayscale image includes at least one of rectangle, circle, star, triangle, and polygon.

[0116] In some embodiments, the terminal obtains and displays candidate grayscale images, receives a user's selection operation of selecting a target grayscale image from the candidate grayscale images, and indicates the target grayscale image selected by the user to the server, and the server uses the target grayscale image as a grayscale change image for the image generation process.

[0117] In other embodiments, modification requirement information of the first image when performing image modification is obtained, and a target grayscale image matching the modification requirement information is determined from the candidate grayscale images as a grayscale change image. Optionally, the modification requirement information includes a region to be modified in the first image that needs to be modified, and a target grayscale image whose grayscale change meets the requirements of the region to be modified is selected from the candidate grayscale images.

[0118] For example, Figure 3 As shown, it shows an example diagram of a grayscale change map provided by an exemplary embodiment of the present application, where the area 311 to be modified in the first image 310 is the area range of the left 1 / 3 of the first image, and the target grayscale map 321 is selected as the grayscale change map in the provided candidate grayscale map 320, wherein the target grayscale map 321 is a gradient grayscale map that gradually changes from left to right, and the pixel values ​​of the target grayscale map 321 gradually change from 0 (black) to 255 (white) from left to right.

[0119] The second method generates a grayscale change map from the first image.

[0120] Schematically, the first image is grayed to obtain a grayscale change map, that is, the grayscale change map is a grayscale map generated from the first image.

[0121] In some embodiments, based on the content and / or outline of the first image as the original image, an algorithm is used to perform recognition and generate a corresponding grayscale change map. Schematically, the image content in the first image is recognized to determine multiple image subjects in the first image; the first image is divided into subject areas corresponding to the multiple image subjects; and the multiple subject areas are grayed to obtain a grayscale change map corresponding to the first image.

[0122] In some embodiments, when grayscale processing is performed on multiple subject areas respectively, the grayscale processing is performed according to the subject modification requirement information corresponding to each image subject. Schematically, the image content in the first image is identified, and multiple image subjects existing in the first image are determined; the first image is divided into subject areas corresponding to the multiple image subjects respectively; the subject modification requirement information corresponding to the multiple image subjects is obtained; and the subject area of ​​the first image is grayscale processed based on the subject modification requirement information to obtain a grayscale change map.

[0123] Optionally, the above-mentioned subject modification requirement information may be indicated by the user through the terminal, for example, the user enters a prompt word (Prompt) to indicate the subject modification requirement. Wherein, when the subject area is grayed, the gray value of the subject area after graying is negatively correlated with the modification intensity indicated by the subject modification requirement information, that is, the greater the modification intensity of the subject area indicated by the subject modification requirement information, the lower the gray value of the subject area after graying (the closer to black).

[0124] like Figure 4 As shown, it shows a schematic diagram of generating a grayscale change map provided by an exemplary embodiment of the present application, where the first image 410, after content recognition and contour recognition, generates a grayscale change map 420 that meets the subject modification requirement information indicated, "modify the sky in the picture", wherein the grayscale value of the area belonging to the sky in the grayscale change map 420 is lower than that of other areas, so that when generating a new image, content generation is mainly performed on the sky area.

[0125] In some embodiments, when graying each main area, a grayscale gradient process is performed on the area boundary between adjacent main areas, so as to avoid the image segmentation problem caused by different modification intensities between different image subjects in the generated image.

[0126] In some embodiments, based on the grayscale image of the first image automatically generated by the system, the user can also select an area on the grayscale image and indicate a specified degree of change for the selected area. Schematically, the subject area of ​​the first image is grayscaled based on the subject modification requirement information to obtain a grayscale image corresponding to the first image; a region selection instruction for the grayscale image is received, wherein the region selection instruction is used to indicate at least one target area, and the target area is an area for adjusting the first pixel value of the first image with a specified degree of change; based on the region selection instruction, the pixel value in at least one target area in the grayscale image is adjusted to a target grayscale value corresponding to the specified degree of change to obtain a grayscale change map.

[0127] Optionally, the above-mentioned specified change degree can be a preset value or a user-defined value. In some embodiments, the target grayscale value of the target area is negatively correlated with the specified change degree, and the conversion relationship between the specified change degree and the target grayscale value in the grayscale change map is indicated by a preset conversion formula. In one example, the preset conversion formula is shown in Formula 1.

[0128] Formula 1: Specified change degree = (255-target grayscale value) / 255

[0129] For example, when the indicated specified degree of change is 0%, that is, indicating that the image content in the target area is not changed when the new image is generated, the target grayscale value corresponding to the target area is 255 (white); when the indicated specified degree of change is 50%, that is, indicating that the degree of change of the content in the target area is 50% when the new image is generated, the target grayscale value corresponding to the target area is 128; when the indicated specified degree of change is 100%, that is, indicating that the content in the target area is completely changed when the new image is generated, the target grayscale value corresponding to the target area is 0 (black).

[0130] Optionally, the target area determined by the user when performing area selection on the grayscale image can be an area selected based on multiple subject areas obtained after the system segmented multiple image subjects, or it can be an area selected by a custom graphic drawn on the first image using a selection tool provided by the terminal.

[0131] Optionally, the method of acquiring the noise image includes at least one of the following:

[0132] The first one is to directly obtain the noisy image.

[0133] Illustratively, while acquiring the first image and the grayscale change map, a preset noise image is acquired. Optionally, the noise type of the preset noise image can be implemented as Gaussian noise, uniform noise, exponential noise, impulse noise, salt and pepper noise, etc.

[0134] The second method is to add noise to the first image to obtain a noisy image.

[0135] Illustratively, the first image is subjected to noise processing to obtain a noise image. Optionally, the noise type added to the first image includes Gaussian noise, uniform noise, exponential noise, impulse noise, salt and pepper noise, etc.

[0136] In one example, standard Gaussian noise is sampled to obtain sampled noise; noise is added to the first image according to the sampled noise and the diffusion time to obtain a noise image. The diffusion time is used to indicate the number of steps required to generate the generated image from the noise image in the iterative denoising process, that is, the number of steps required for the model to gradually recover (generate) from the initial state (noise image) to the generated image. The diffusion time is a continuous time variable t∈[0, T]. The data distribution corresponding to time 0 can be expressed as x0~q(x0), and the data distribution corresponding to time T can be expressed as x T ~q(x T ).

[0137] When a noisy image is obtained by adding noise to the first image, the obtained noisy image can carry the image information of the first image, so that the noisy image can restore the original image content in the direct denoising process without additional prompt words.

[0138] Step 220 , in the process of iteratively denoising the noisy image by modifying the first image as a denoising condition, obtaining the i-th noisy image corresponding to the i-th round of iterative denoising stage.

[0139] Wherein, i is a positive integer, when i=1, the noise image is the initial noise image, when i>1, the noise intensity of the i-th noise image corresponds to the i-th round of iterative noise reduction stage. In some embodiments, as the number of iterative noise reduction stages increases, the noise intensity of the noise image decreases.

[0140] Optionally, a method for acquiring the i-th noise image corresponding to the i-th round of iterative noise reduction stage may be implemented as at least one of the following:

[0141] The first method is to obtain an initial noise image belonging to pure noise when acquiring a noise image. In the process of iteratively denoising the noise image by modifying the first image as a denoising condition, the noise image is denoised in each round of iterative denoising stage to obtain a noise image corresponding to the round of iterative denoising stage.

[0142] Schematically, in the i-th round of iterative noise reduction stage, the i-1th noise image corresponding to the i-1th round of iterative noise reduction stage is obtained, and the i-1th noise image is subjected to noise reduction to obtain the i-th noise image. Exemplarily, in the first round of iterative noise reduction stage, the initial noise image is obtained as the first noise image corresponding to the round of iterative noise reduction stage; in the second round of iterative noise reduction stage, the initial noise image is subjected to noise reduction to obtain the second noise image; in the third round of iterative noise reduction stage, the second noise image is subjected to noise reduction to obtain the third noise image…

[0143] The second method is to obtain noise images corresponding to multiple rounds of iterative noise reduction stages when obtaining the noise image, and directly obtain the noise image corresponding to the current iterative noise reduction stage during the process of iterative noise reduction of the noise image by modifying the first image as the noise reduction condition.

[0144] Schematically, noise images corresponding to n rounds of iterative denoising stages are obtained, and in the i-th round of iterative denoising stage, the i-th noise image corresponding to the i-th round of iterative denoising stage is obtained from the n noise images, where n is an integer. That is, the process includes n rounds of iterative denoising stages, wherein the noise intensity corresponding to the n noise images is negatively correlated with n, that is, the noise intensity of the noise image corresponding to the later iterative denoising stage is lower.

[0145] In some embodiments, when the noise image is obtained by adding noise to the first image, noise is added based on the diffusion duration during the noise addition process, and noise is reduced by sampling the diffusion duration during the noise reduction process.

[0146] In schematic form, the diffusion duration is sampled to obtain the i-th sampling moment corresponding to the i-th round of iterative noise reduction stage; when i=1, the noise image is used as the first noise image corresponding to the first sampling moment; when i>1, the noise image is denoised to obtain the i-th noise image corresponding to the i-th sampling moment; or, when i>1, the i-1-th noise image corresponding to the i-1-th sampling moment is denoised to obtain the i-th noise image corresponding to the i-th sampling moment. That is, the first image is denoised to obtain an initial noise image, which is used as the first noise image used in the first round of iterative noise reduction stage, and the sampling moment determined after sampling the diffusion duration is used as a noise reduction guide, and the first noise image is denoised to obtain a second noise image, which is used in the second round of iterative noise reduction stage, and the second noise image is denoised to obtain a third noise image, which is used in the third round of iterative noise reduction stage...

[0147] Optionally, the time is randomly and uniformly sampled within the diffusion duration to determine the sampling moment.

[0148] Optionally, the process of determining the sampling time can be expressed as t~U(T), where T is the diffusion time.

[0149] Optionally, during the denoising process, the computer device may sample the diffusion time once to obtain a single sampling moment; or may sample the diffusion time multiple times to obtain multiple sampling moments.

[0150] Schematically, the first image, the grayscale change image and the noise image are images of the same size.

[0151] Step 230, when i=1, denoise the first noisy image to obtain the first denoised stage image outputted by the first round of iterative denoising stage.

[0152] In an embodiment of the present application, when i=1, that is, the noise image and the grayscale change map are input into the first round of iterative denoising stage. Since there is no preceding denoising stage image in the first round of iterative denoising stage, in the first round of iterative denoising stage, the first denoising stage image outputted by the first round of iterative denoising stage is obtained by directly denoising the acquired initial noise image (the first noise image), so as to be used for generating a fused image in a subsequent iterative denoising stage.

[0153] In some embodiments, when the initial noise image is obtained by adding noise to the first image based on the diffusion duration, the diffusion duration is sampled to obtain the first sampling moment corresponding to the first round of iterative noise reduction stage, the initial noise image is denoised to obtain the second noise image corresponding to the first sampling moment, and the second noise image is used as the first noise reduction stage image outputted by the first round of iterative noise reduction stage.

[0154] Step 240, when i>1, based on the fusion ratio indicated by the second pixel value in the grayscale change map, the i-th noise image and the i-1th denoising stage image outputted from the i-1th iterative denoising stage are fused to obtain the i-th fused image.

[0155] In an embodiment of the present application, when i>1, in the i-th round of iterative noise reduction stage, the second pixel value in the grayscale change map is used as a fusion guide to fuse the noise image corresponding to the current iterative noise reduction stage and the noise reduction stage image obtained in the previous round of iterative noise reduction stage.

[0156] Optionally, when fusion is performed based on the grayscale change map, the fusion method may be implemented as at least one of the following:

[0157] The first one is to perform fusion based on the grayscale change map with pixels as the fusion scale.

[0158] Schematically, a first fusion coefficient corresponding to the i-th noise image and a second fusion coefficient corresponding to the i-1-th denoising stage image are determined based on the second pixel value in the grayscale change map, a first image feature representation of the i-th noise image is weighted based on the first fusion coefficient, and a second image feature representation of the i-1-th denoising stage image is weighted based on the second fusion coefficient, and the weighted first image feature representation and second image feature representation are fused to obtain a fused feature representation to indicate the i-th fused image.

[0159] In some embodiments, the first fusion coefficient and the second fusion coefficient are determined by a preset conversion function. In one example, the preset conversion function includes formula two and formula three.

[0160] Formula 2: First fusion coefficient = second pixel value / 255

[0161] Formula 3: Second fusion coefficient = 1-second pixel value / 255

[0162] The second method is to perform fusion based on the grayscale change map using the region as the fusion scale.

[0163] Illustratively, based on the second pixel value in the grayscale change map, when the ith fused image is obtained by fusion, the first region from the ith noise image and the second region from the (i-1)th denoising stage image are determined.

[0164] In some embodiments, a mask is generated to indicate the fusion ratio of the i-th noise image and the i-1th noise reduction stage image, and the fusion ratio indicates the ratios of the first area and the second area in the fused image. Schematically, the grayscale change map is converted into the i-th mask corresponding to the i-th round of iterative noise reduction stage, wherein the i-th mask is used to indicate the areas corresponding to the i-th noise image and the i-1th noise reduction stage image in the i-th round of iterative noise reduction stage when participating in the fusion; the i-th noise image and the i-1th noise reduction stage image are fused based on the i-th mask to obtain the i-th fused image.

[0165] In some embodiments, the second pixel value is screened by a pixel threshold to obtain an i-th mask. Schematically, the i-th pixel threshold corresponding to the i-th round of iterative noise reduction stage is obtained, wherein the i-th pixel threshold is used to screen the second pixel position according to the second pixel value in the grayscale change map, and the first pixel value in the grayscale change map is screened by the i-th pixel threshold to obtain the i-th mask.

[0166] Step 250, denoising the i-th fused image to obtain the i-th denoising stage image outputted by the i-th round of iterative denoising stage, until the iterative denoising requirement is met, and a generated image is obtained as a modification result of the first image.

[0167] In the embodiment of the present application, in the i-th round of iterative denoising stage, denoising is performed on the i-th fused image after fusion, thereby obtaining the i-th denoising stage image.

[0168] In an embodiment of the present application, when the image of the i-th denoising stage meets the iterative denoising requirements, the image of the i-th denoising stage is the generated image obtained by denoising; when the image of the i-th denoising stage does not meet the iterative denoising requirements, the image of the i-th denoising stage is input into the i+1th iterative denoising stage to participate in the iterative denoising process.

[0169] Optionally, the above-mentioned iterative noise reduction requirements include at least one of the following:

[0170] The first one is that the total number of rounds of iterative denoising reaches the target number threshold.

[0171] Illustratively, taking the target number threshold as m as an example, the lower m denoising stage images outputted from the mth round of iterative denoising stage are taken as the modification result of the first image, where m is a positive integer.

[0172] Optionally, the target number threshold may be implemented as a preset fixed value; or, when randomly and uniformly sampling time within the diffusion duration to determine the sampling moment, the total number of samplings is used as the target number threshold.

[0173] The second one is that the image clarity of the image in the i-th denoising stage reaches the target clarity threshold.

[0174] In an illustrative manner, the image clarity of the image at the i-th denoising stage is detected, and when the image clarity of the image at the i-th denoising stage reaches a target clarity threshold, the image at the i-th denoising stage is used as the modification result of the first image.

[0175] Optionally, the image clarity may be determined by at least one of a mean square error, a peak signal-to-noise ratio, and a structural similarity index (Structural Similarity Index, SSIM).

[0176] Please refer to Figure 5 , which shows a schematic diagram of the generation effect of generating a controllable change range through a grayscale change map provided by an exemplary embodiment of the present application. Image A511 obtains a generated image A513 under the control of the grayscale change map A512, wherein, since the grayscale change map A512 is a gradient grayscale map, the grayscale change map A512 indicates that when the image A511 is modified, the pixel change intensity gradually increases from left to right. It can be seen from the generated image A513 that in the generated image A513, the content on the right side is completely changed compared to the image A511, the content in the middle part is somewhat different from the image A511, but still not similar to the building, and the content on the left side is the same as the image A511.

[0177] Image B521 is controlled by the grayscale change map B522 to obtain the generated image B523, wherein the grayscale value in the grayscale change map B522 indicates that the sky part of image B521 is modified the most during the modification process, followed by the mountain and grassland parts, and the road part is not modified. Therefore, the sky part of the generated image B523 is most different from that of image B521, and there are certain changes in the mountain and grassland parts, but the original content semantics are still retained (that is, there are mountains and grasslands), while the road part is the same as image B521.

[0178] In some embodiments, when denoising the i-th fused image, the content of the denoising process is generated by guiding the prompt word. In an illustrative example, a prompt word is obtained, and the denoising process of the i-th fused image is guided by the prompt word, and the i-th denoising stage image outputted by the i-th round of iterative denoising stage is obtained, until the iterative denoising requirement is met, and the generated image is obtained as the modification result of the first image.

[0179] In an embodiment of the present application, since there is no preceding denoising stage image in the first round of iterative denoising stage, the fusion of the i-th noise image and the i-1-th denoising stage image starts from the second round of iterative denoising stage. In the first round of iterative denoising stage, the first denoising stage image output by the first round of iterative denoising stage is obtained by directly denoising the acquired initial noise image, so as to be used for the fusion process of the second noise image and the first denoising stage image in the second round of iterative denoising stage.

[0180] Specifically, in the first round of iterative denoising stage, the first noise image (initial noise image) is obtained, and the first noise image is denoised to obtain the first denoising stage image; in the second round of iterative denoising stage, the second noise image is obtained, the second noise image and the first denoising stage image are fused based on the grayscale change map to obtain the second fused image, and the second fused image is denoised to obtain the second denoising stage image; in the third round of iterative denoising stage, the third noise image is obtained, the third noise image and the second denoising stage image are fused based on the grayscale change map to obtain the third fused image, and the third fused image is denoised to obtain the third denoising stage image…

[0181] In summary, when a new image is generated based on a first image, the grayscale change map is used to control the area in the first image that needs to be modified or retained, as well as the modification intensity during the modification process. That is, in the process of iteratively denoising the acquired noisy image by modifying the first image as a denoising condition, the second pixel value in the grayscale change map is used to control the fusion ratio between the original image content carried by the first image and the new image content generated from the noise, thereby achieving an image modification result with controllable generation intensity at the pixel level. Under the guidance of the grayscale change map, there are no longer only two generation options, namely, change and unchanged. The modification of the original image content can be more freely controllable. At the same time, there is a better transition effect at the junction of the change and the unchanged in the picture, thereby improving the generation effect of the image modification result.

[0182] In some optional embodiments, in each round of iterative denoising stage, the second pixel value in the grayscale change map is screened according to the pixel threshold corresponding to the iterative denoising stage, so as to obtain a mask corresponding to the iterative denoising stage, and the mask is used to fuse the noise image corresponding to the current iterative denoising stage and the denoised stage image obtained in the previous round of iterative denoising stage. Please refer to Figure 6 , which shows a flowchart of an image generation method provided by an exemplary embodiment of the present application, step 240 includes steps 241 to 243.

[0183] Step 241, obtaining the i-th pixel threshold corresponding to the i-th round of iterative noise reduction stage.

[0184] Among them, the i-th pixel threshold is used to filter the second pixel position according to the second pixel value in the grayscale change map. In an embodiment of the present application, there is a corresponding pixel threshold in each iterative noise reduction stage to filter the second pixel value in the grayscale change map.

[0185] Optionally, the pixel threshold may be a preset threshold; or, the pixel threshold may be determined according to a round of the iterative noise reduction stage.

[0186] In some embodiments, the pixel threshold corresponding to each round of iterative noise reduction is determined based on the total number of rounds of iterative noise reduction. In an exemplary embodiment, the total number of rounds of iterative noise reduction, q, is determined, and q pixel thresholds corresponding to the number of rounds q are obtained by sampling within a preset grayscale range, where q is a positive integer.

[0187] Optionally, the above sampling process may be implemented as uniform sampling, or the above sampling process may be implemented as random sampling, which is not limited here.

[0188] In some embodiments, the pixel threshold is positively correlated with the iterative denoising stage. Exemplarily, when the pixel value range of the second pixel value of the grayscale change map is 0 (black) to 255 (white), the pixel threshold increases as the number of iterative denoising stages increases. For example, taking the number of rounds k=5 and uniform sampling as an example, the time step t of the iterative denoising process is 4 to 0, and the iterative threshold of each iterative denoising stage is (kt) / k. Therefore, the pixel threshold of each iterative denoising stage is: in the first round of iterative denoising stage, the first iterative threshold is 0.2, and its corresponding pixel threshold is 51, that is, the pixel position where the second pixel value in the grayscale change map is less than 51; in the second round of iterative denoising stage, the second iterative threshold is 0.4, and its corresponding pixel threshold is 102, that is, the second pixel value in the grayscale change map is less than 10 2; in the third round of iterative denoising stage, the third iteration threshold is 0.6, and its corresponding pixel threshold is 153, that is, the pixel position where the second pixel value in the grayscale change map is less than 153; in the fourth round of iterative denoising stage, the fourth iteration threshold is 0.8, and its corresponding pixel threshold is 204, that is, the pixel position where the second pixel value in the grayscale change map is less than 204; in the fifth round of iterative denoising stage, the fifth iteration threshold is 1.0, and its corresponding pixel threshold is 255, that is, the pixel position where the second pixel value in the grayscale change map is less than 255.

[0189] Step 242: Screen the second pixel value in the grayscale change map using the i-th pixel threshold to obtain the i-th mask.

[0190] A mask is a binary image of the same size as the grayscale change map, where the white (or 1) area in the mask represents the part of the original image that needs to be retained, and the black (or 0) area represents the part that needs to be hidden or deleted.

[0191] In some embodiments, the i-th mask includes a first mask image and a second mask image, wherein the first mask image is a mask for extracting a first area in the i-th noise image, and the second mask image is a mask for extracting a second area in the i-1th denoising stage image.

[0192] Schematically, the second pixel position whose second pixel value in the grayscale change map is greater than the i-th pixel threshold is screened to obtain the first mask map; the first pixel whose second pixel value in the grayscale change map is less than or equal to the i-th pixel threshold is screened to obtain the second mask map.

[0193] That is, in the first mask map, the third pixel position corresponding to the second pixel position corresponds to the first mask value, when the second pixel value of the second pixel position in the grayscale change map is greater than the i-th pixel threshold, the first mask value of the third pixel position in the first mask map is 1, when the second pixel value of the second pixel position in the grayscale change map is less than or equal to the i-th pixel threshold, the first mask value of the third pixel position in the first mask map is 0; in the second mask map, the fourth pixel position corresponding to the second pixel position corresponds to the second mask value, when the second pixel value of the second pixel position in the grayscale change map is less than or equal to the i-th pixel threshold, the second mask value of the fourth pixel position in the second mask map is 1, and when the second pixel value of the second pixel position in the grayscale change map is greater than the i-th pixel threshold, the second mask value of the fourth pixel position in the second mask map is 0.

[0194] Step 243: fuse the i-th noise image and the i-1-th denoising stage image based on the i-th mask to obtain the i-th fused image.

[0195] In some embodiments, when the i-th mask includes a first mask image and a second mask image, the first mask image and the i-th noise image are superimposed to obtain a first superimposed image; the second mask image and the i-1th noise reduction stage image are superimposed to obtain a second superimposed image; the first superimposed image and the second superimposed image are fused to obtain the i-th fused image. That is, the first image content corresponding to the first area in the i-th noise image is extracted by the first mask image, and the second image content corresponding to the second area in the i-1th noise reduction stage image is extracted by the second mask image, and the first image content and the second image content are fused to obtain the i-th fused image. That is, the first mask image and the second mask image are obtained by pixel threshold screening corresponding to the iterative noise reduction stage, the first mask image can extract the area on the noise image, and the second mask image can extract the area on the noise reduction stage image, so that the image areas extracted respectively are fused to obtain a fused image, and the first mask image and the second mask image obtained by pixel threshold screening improve the acquisition efficiency of the fused image.

[0196] In an embodiment of the present application, after obtaining the i-th fused image, denoising is performed on the i-th fused image to obtain the i-th denoising stage image outputted by the i-th round of iterative denoising stage, until the iterative denoising requirements are met, and a generated image is obtained as a modification result of the first image.

[0197] In some embodiments, when denoising the i-th fused image, the content of the denoising process is generated by guiding the prompt word. In an illustrative example, a prompt word is obtained, and the denoising process of the i-th fused image is guided by the prompt word, and the i-th denoising stage image outputted by the i-th round of iterative denoising stage is obtained, until the iterative denoising requirement is met, and the generated image is obtained as the modification result of the first image.

[0198] For example, please refer to Figure 7 , which shows a schematic diagram of extracting a noise image and a previous round of noise reduction image through a mask in multiple iterative noise reduction stages provided by an exemplary embodiment of the present application, wherein a first image 701 is generated under the guidance of a grayscale change map 702 and a prompt word 703 to obtain a generated image 704, wherein in each iterative noise reduction stage in the generation process, a mask corresponding to each iterative noise reduction stage is generated according to the grayscale change map 702, a first image content 710 is extracted from the noise image through the mask, and a second image content 720 is extracted from the noise reduction stage image obtained in the previous round of iterative noise reduction stage, and the first image content 710 and the second image content 720 are fused to obtain a fused image corresponding to the current iterative noise reduction stage, wherein the noise image is denoised with the first image 701 as a noise reduction guide in the iterative noise reduction process, and the fused image is denoised with the prompt word 703 as a guide to obtain a noise reduction stage image.

[0199] It can be seen that the upper area is the part copied from the noise image of the first image 701. As time goes by, the proportion of the first image 701 gradually decreases. The lower area is the generated area specified by the grayscale change graph 702, which is the generated content generated according to the prompt word 703. As the number of iteration steps increases, this part of the area gradually expands its influence range according to the guidance of the black area in the grayscale change graph 702. During the denoising process, the white unchanged area and the black changed area in the grayscale change graph 702 are continuously merged with the iteration of the reasoning steps. It can be seen that the method adopted by the embodiment of the present application allows different areas of the image to be edited to different degrees, thereby achieving more refined control.

[0200] For an example, see Figure 8 , which shows a schematic diagram of the image generation process provided by an exemplary embodiment of the present application. Schematically, taking 4 rounds of iterative denoising as an example, a first image 810, a grayscale change map 820 and a text prompt 801 for image generation are obtained. Under the guidance of the text prompt 801, a generated image 870 is obtained based on the first image 810 and the grayscale change map 820.

[0201] First, in the first iterative denoising stage, the first mask 831 is obtained based on the grayscale change map 820. Since the first iterative denoising stage is the first denoising stage, the first mask 831 indicates that the image to be denoised is the initial noise image 811 itself, that is, the initial noise image 811 is denoised to obtain the first denoising stage image 832.

[0202] In the second iterative denoising stage, the second mask 841 is obtained based on the grayscale change map 820, and the initial noise image 811 is denoised with the first image 810 as a guide to obtain the second noise image 842. The second noise image 842 and the first denoising stage image 832 are fused and denoised according to the second mask 841 to obtain the second denoising stage image 843, wherein the second mask 841 includes the first mask map and the second mask map corresponding to the iterative denoising stage. In the process of fusing the second noise image 842 and the first denoising stage image 832, the first mask map is superimposed on the second noise image 842 to extract the image area used for fusion in the second noise image 842, and the second mask map is superimposed on the first denoising stage image 832 to extract the image area used for fusion in the first denoising stage image 832.

[0203] In the third iterative denoising stage, the third mask 851 is obtained based on the grayscale change map 820, and the second noise image 842 is denoised with the first image 810 as a guide to obtain the third noise image 852. The third noise image 852 and the second noise reduction stage image 842 are fused and denoised according to the third mask 851 to obtain the third noise reduction stage image 853, wherein the third mask 851 includes the first mask map and the second mask map corresponding to the iterative denoising stage. In the process of fusing the third noise image 852 and the second noise reduction stage image 842, the first mask map is superimposed on the third noise image 852 to extract the image area used for fusion in the third noise image 852, and the second mask map is superimposed on the second noise reduction stage image 842 to extract the image area used for fusion in the second noise reduction stage image 842.

[0204] In the fourth iterative denoising stage, the fourth mask 861 is obtained based on the grayscale change map 820, and the third noise image 852 is denoised with the first image 810 as a guide to obtain the fourth noise image 862. The fourth noise image 862 and the third noise reduction stage image 853 are fused and denoised according to the fourth mask 861 to finally obtain the generated image 870, wherein the fourth mask 861 includes the first mask map and the second mask map corresponding to the iterative denoising stage. In the process of fusing the fourth noise image 862 and the third noise reduction stage image 853, the first mask map is superimposed on the fourth noise image 862 to extract the image area used for fusion in the fourth noise image 862, and the second mask map is superimposed on the third noise reduction stage image 853 to extract the image area used for fusion in the third noise reduction stage image 853.

[0205] In summary, when a new image is generated based on a first image, the grayscale change map is used to control the area in the first image that needs to be modified or retained, as well as the modification intensity during the modification process. That is, in the process of iteratively denoising the acquired noisy image by modifying the first image as a denoising condition, the second pixel value in the grayscale change map is used to control the fusion ratio between the original image content carried by the first image and the new image content generated from the noise, thereby achieving an image modification result with controllable generation intensity at the pixel level. Under the guidance of the grayscale change map, there are no longer only two generation options, namely, change and unchanged. The modification of the original image content can be more freely controllable. At the same time, there is a better transition effect at the junction of the change and the unchanged in the picture, thereby improving the generation effect of the image modification result.

[0206] In an embodiment of the present application, when generating an image through a diffusion model, a grayscale change map is input to achieve stable generation of image details with controllable degree of change of a single pixel, which greatly improves the user's ability to control the generation results during the image generation process of the diffusion model.

[0207] In some optional embodiments, the image generation method provided in the embodiments of the present application is performed by a diffusion model. The diffusion model can learn image features by adding noise and denoising the image, so the diffusion model needs to implement the adding noise and denoising of the image through a forward process and a reverse process, that is, the diffusion model adds noise to the image through a forward diffusion network, and denoises the noisy image through a reverse diffusion network.

[0208] Optionally, the process of adding noise to an image using a diffusion model can be expressed as Formula 4, and the process of denoising can be expressed as Formula 5.

[0209] Formula 4: dx t =f(t)x t dt+g(t)dw

[0210] Formula 5:

[0211] Where w represents the standard Wiener process (also known as Brownian motion), f(t) is x t Drift Coefficient, g(t) is x t The diffusion coefficient, x t is the image sample corresponding to time t, represents the standard Wiener process when time flows backward from T to 0, q(x t ) is x t The data distribution, Represents the data distribution score.

[0212] In the process of adding noise to the image, the diffusion time is a continuous time variable t∈[0, T]. The data distribution corresponding to time 0 can be expressed as x0~q(x0), and the data distribution corresponding to time T can be expressed as x T ~q(x T ). In order to learn the image features, the computer device needs to sample the real samples x0~q(x0) through the discretization inverse process.

[0213] Optionally, the diffusion model includes at least one of a stable diffusion model, a denoising diffusion probabilistic model (Denoising Diffusion Probabilistic Models, DDPM), a latent diffusion model (Latent Diffusion Models, LDM), etc.

[0214] Schematically, take the diffusion model implemented as a stable diffusion model as an example, Fig. 9As shown, the diffusion model 900 includes an encoder 910, a U-Net network structure 920 and a decoder 930, wherein the U-Net network structure 920 includes a forward diffusion network 921 and a reverse diffusion network 922.

[0215] The encoder 910 and the decoder 930 are implemented as a residual network (Residual Network, ResNet). The encoder 910 compresses the image representation into a low-resolution image (downsampling process), and the decoder 930 decodes the low-resolution back to a high-resolution image (upsampling process).

[0216] In order to prevent the U-Net network structure 920 from losing important information during downsampling, a direct connection is added between the downsampled ResNet of the encoder 910 and the upsampled ResNet of the decoder 930 to transfer information.

[0217] like Fig. 9 As shown, during the training process, the training sample image 901 is encoded and input into the diffusion model 900 to obtain the sample prediction image 902. The diffusion model 900 is trained based on the effect of the sample prediction image 902 until the loss function converges to the optimal point to obtain the optimized diffusion model 900.

[0218] Please refer to Fig.10 , which shows a flowchart of a data processing process of a diffusion model provided by an exemplary embodiment of the present application. Schematically, the process is executed by a computer device, and optionally, the computer device can be at least one of a terminal or a server. In the embodiment of the present application, the computer device is a server as an example for explanation, wherein the process includes steps 1011 to 1040.

[0219] Step 1011: input the first image into a first encoder of a diffusion model to obtain an image encoding representation.

[0220] like Fig. 9 As shown, the diffusion model 900 includes an encoder 910 , and the encoder 910 is used to encode data input into the diffusion model 900 .

[0221] In the embodiment of the present application, the input of the diffusion model 900 includes a first image 903 and a grayscale change map 904. In some embodiments, the encoder 910 includes a first encoder, and the first image 903 is encoded by the first encoder to obtain an image encoding representation, and the image encoding representation is input to the U-Net network structure 920. That is, the first image 903 is encoded by the encoder 910 to the latent space z init .

[0222] Step 1012: input the grayscale change map to the second encoder of the diffusion model to obtain a coded representation of the change map.

[0223] In some embodiments, the encoder 910 further includes a second encoder, and the grayscale change map 904 is encoded by the second encoder to obtain a change map encoding representation, and the change map encoding representation is input into the U-Net network structure 920. That is, the grayscale change map 904 is downsampled by the encoder 910 to a latent space z having the same dimension as the image encoding representation. init , so as to be consistent with the latent space z init The pixel positions of the first image 903 are aligned.

[0224] Optionally, the first encoder and the second encoder may be implemented as the same encoder or as different encoders.

[0225] In some embodiments, the input of the diffusion model 900 further includes a prompt 905 , which is used to guide image changes when the diffusion model generates a new image based on the first image.

[0226] In some embodiments, the encoder 910 further includes a third encoder, which schematically obtains the prompt word 905, and inputs the prompt word 905 into the third encoder of the diffusion model 900 to obtain a text encoding representation. That is, the prompt word 905 is encoded by the third encoder to obtain a text encoding representation, and the text encoding representation is input into the U-Net network structure 920.

[0227] Optionally, the third encoder and the first encoder / second encoder may be implemented as the same encoder or as different encoders.

[0228] Optionally, the prompt word 905 includes at least one of the following contents:

[0229] The first is to generate content prompt information, wherein the generated content prompt information is used to indicate the semantics of the content that needs to be generated when generating a new image based on the first image, that is, the generated content prompt information is used to guide the modification target of the content of the modified part in the first image. For example, when the generated content prompt information is "stars", stars are generated in the area to be modified indicated by the grayscale change diagram 904.

[0230] The second type is to generate style hint information, wherein the generated style hint information is used to indicate the image style of the new image generated based on the first image, that is, the generated style hint information is used to guide the style type of the content of the modified part in the first image. Optionally, the generated style hint information includes at least one of oil painting style, comic style, realistic style, photo style, digital art style, etc.

[0231] The third type is image attribute prompt information, wherein the image attribute prompt information is used to indicate the image attributes of the new image generated based on the first image, that is, the image attribute prompt information is used to control the image attributes of the generated image finally generated. Optionally, the above-mentioned image attributes include at least one of resolution, size, color depth, color mode, contrast, brightness, saturation, hue, transparency, file format, image size (storage space occupied), and image quality.

[0232] That is, in the reasoning process through the diffusion model, the generated content is guided by the prompt word 905, and the pixel-level fusion control of the grayscale change map in the denoising process is combined to well achieve the controllable pixel-level generation intensity and content in the reasoning process.

[0233] Step 1020: input the image encoding representation into the forward diffusion network in the diffusion model to obtain a noise image feature representation corresponding to the noise image.

[0234] In the embodiment of the present application, the function of the forward diffusion network 921 in the diffusion model 900 is to convert a real image into Gaussian noise, that is, after the image encoding representation corresponding to the first image 903 is input into the U-Net network structure 920, it is noised through the forward diffusion network 921 in the U-Net network structure 920 to obtain a noise image feature representation corresponding to the noise image.

[0235] Step 1030 , input the noise image feature representation and the variation map encoding representation into the reverse diffusion network in the diffusion model to obtain the generated image feature representation.

[0236] In the embodiment of the present application, the noise image feature representation and the variation map encoding representation are input to the back diffusion network 922, and the back diffusion network 922 performs noise reduction to obtain the generated image feature representation.

[0237] Schematically, the back diffusion network 922 performs iterative denoising by modifying the first image as a denoising condition. During the iterative denoising process, in the i-th iterative denoising stage, denoising is performed from the noise image feature representation to obtain the i-th noise image feature representation corresponding to the i-th iterative denoising stage. Based on the variation map encoding representation, the i-th mask corresponding to the i-th iterative denoising stage is obtained. The i-th noise image feature representation and the i-1-th denoising stage feature representation obtained by the i-1-th round of iterative denoising are fused through the i-th mask to obtain the i-th fused image feature representation. The i-th fused image feature representation is denoised to obtain the i-th denoising stage feature representation, until the denoising is completed to obtain the generated image feature representation.

[0238] In some embodiments, when the input of the diffusion model 900 also includes a prompt word 905, the noise image feature representation, the change graph encoding representation and the text encoding representation corresponding to the prompt word 905 are input into the reverse diffusion network 922 in the diffusion model 900 to obtain a generated image feature representation.

[0239] Schematically, the back diffusion network 922 performs iterative denoising by modifying the first image as a denoising condition. During the iterative denoising process, in the i-th iterative denoising stage, denoising is performed from the noise image feature representation to obtain the i-th noise image feature representation corresponding to the i-th iterative denoising stage. Based on the variation graph encoding representation, the i-th mask corresponding to the i-th iterative denoising stage is obtained. The i-th noise image feature representation and the i-1-th denoising stage feature representation obtained by the i-1-th round of iterative denoising are fused through the i-th mask to obtain the i-th fused image feature representation. The text encoding representation is used as a guide for content generation, and the i-th fused image feature representation is denoised to obtain the i-th denoising stage feature representation, until the denoising is completed to obtain the generated image feature representation.

[0240] Specifically, during the iterative denoising process, for each time step t (from k-1 to 0), the following operations are performed:

[0241] S1, add noise to the encoded original image x according to the current time step t to obtain the noisy image z t ′.

[0242] S2, calculate a mask mask, which contains all the pixels in the grayscale change map that are lower than the threshold (kt) / k.

[0243] S3, use mask mask to transform the denoising stage image z of the previous time step t-1 With the noisy image z t Specifically, the area where the mask value is 1 indicates that the image of the denoising stage of the previous time step is retained, while the area where the mask value is 0 indicates that the noisy image is retained.

[0244] S4, denoise the mixed image to obtain the denoised image z at the current time step t .

[0245] In some embodiments, the text information guidance of the prompt word 905 is passed into the U-Net network structure 920 of the diffusion model 900 through the cross-attention layer, and the cross-attention layer is added between the ResNet blocks of the encoder 910 and the decoder 930 of the U-Net network structure 920.

[0246] Step 1040 , inputting the generated image feature representation into the decoder of the diffusion model to obtain the generated image.

[0247] In the embodiment of the present application, the feature representation of the generated image is input into the decoder 930, and the generated image is obtained by decoding by the decoder 930.

[0248] Specifically, after all time steps are completed, the denoising stage image z0 of the last time step is decoded to obtain the final generated image

[0249] In summary, when a new image is generated based on a first image, the grayscale change map is used to control the area in the first image that needs to be modified or retained, as well as the modification intensity during the modification process. That is, in the process of iteratively denoising the acquired noisy image by modifying the first image as a denoising condition, the second pixel value in the grayscale change map is used to control the fusion ratio between the original image content carried by the first image and the new image content generated from the noise, thereby achieving an image modification result with controllable generation intensity at the pixel level. Under the guidance of the grayscale change map, there are no longer only two generation options, namely, change and unchanged. The modification of the original image content can be more freely controllable. At the same time, there is a better transition effect at the junction of the change and the unchanged in the picture, thereby improving the generation effect of the image modification result.

[0250] In an embodiment of the present application, in each iterative denoising stage round, the pixel threshold corresponding to the current iterative denoising stage is used to screen the second pixel position in the grayscale change map, so as to obtain a mask of the fused noise image and the denoising stage image, and thus the fusion ratio between the noise image and the denoising stage image in each iterative denoising stage is controlled by the pixel threshold corresponding to each iterative denoising stage, so as to realize the gradual fusion of the original image content and the newly generated content in the iterative denoising process, improve the transition effect between the original image content and the newly generated content, and improve the image effect of the generated image.

[0251] In some optional embodiments, the grayscale change map includes multiple forms of expression. Fig.11 As shown, it shows a schematic diagram of the expression form of a grayscale change graph provided by an exemplary embodiment of the present application.

[0252] The first type is a candidate grayscale image 1110 that is preset and provided, including a gradient grayscale image or a pattern grayscale image ( Fig.11 The gradient type grayscale image is taken as an example). The gradient type of the gradient grayscale image includes at least one of linear gradient, ray gradient, rectangular gradient, and path gradient, and the gradient direction includes at least one of diagonal direction, up-down direction, left-right direction, and central radiation direction. The grayscale pattern in the pattern grayscale image includes at least one of rectangle, circle, star, triangle, and polygon.

[0253] The second type is a generated grayscale image 1120 generated from a picture. That is, based on the content or outline of the original picture 1121, an algorithm is used to identify and generate a corresponding black and white grayscale image. In addition to the gradient generation intensity information, this grayscale change map also retains part of the original image content information, which can achieve controllable changes to the background of the picture or a specified area.

[0254] The third type is to generate a refined grayscale image 1130 after the grayscale image is modified with custom details. That is, on the grayscale image obtained by recognition and generation based on the original image, the user can use tools such as a brush to select a specified area to modify the range of change. For example, the user selects the hair accessories and item decorations of the character separately and reduces the range of change, thereby obtaining the refined grayscale image 1130.

[0255] In some embodiments, the grayscale change map generated based on the original image may contain more information about the image in combination with the actual usage requirements of the business. At the same time, the business side also requires that the generated grayscale change map can be further conveniently modified manually to mark the areas that need to be changed or unchanged.

[0256] In one example, a custom editing method for a grayscale change map is provided, and the method is executed by a terminal. Fig.12 As shown, it shows a schematic diagram of the interface for custom editing of a grayscale change map provided by an exemplary embodiment of the present application. The terminal displays a plug-in interface 1200 provided by a grayscale change map generation plug-in. The plug-in page 1200 includes an image upload area 1210, an editing area 1220 and a grayscale change map display area 1230.

[0257] The user uploads the original image 1211 in the image upload area 1210, and the grayscale change map generation plug-in recognizes the content or outline of the original image 1211 through an algorithm, generates and displays the grayscale image 1221 in the editing area 1220. The generation process mainly converts the original image 1211 from a color mode to a grayscale mode, that is, each pixel in the original image 1211 is converted from an RGB (red, green, and blue) channel to a single brightness value. After conversion to the grayscale image 1221, the brightness value of each pixel will represent its intensity within the grayscale range, and the intensity control from 0 (black) to 255 (white) represents a change from completely changed to completely unchanged.

[0258] An editing control is provided in the editing area 1220 , and the user can select a custom adjustment area on the grayscale image 1221 through the editing control, and finally display the grayscale change map 1231 generated by the grayscale change map generation plug-in in the grayscale change map display area 1230 .

[0259] In some embodiments, after the grayscale change map 1231 is generated, the user can download the grayscale change map 1231 .

[0260] In other embodiments, the grayscale change map generation plug-in inputs the generated grayscale change map 1231 together with the original image 1211 and the user input Prompt into the pre-trained diffusion model to obtain a generated image, and displays the generated image.

[0261] The image generation method provided in the embodiment of the present application, when generating a new image based on a first image, controls the area in the first image that needs to be modified or retained, and controls the modification intensity in the modification process, through a grayscale change map, that is, in the process of iteratively denoising the acquired noisy image with the modification of the first image as the denoising condition, controls the fusion ratio between the original image content carried by the first image and the new image content generated from the noise through the second pixel value in the grayscale change map, thereby achieving an image modification result with controllable generation intensity at the pixel level. Under the guidance of the grayscale change map, there are no longer only two generation options, namely, change and unchanged. The modification of the original image content can be more freely controllable. At the same time, there is a better transition effect for the junction of the change and the unchanged in the picture, thereby improving the generation effect of the image modification result.

[0262] like Fig.13 As shown, it shows a schematic diagram of an image generation process provided by an exemplary embodiment of the present application, from left to right are the original image 1301, the initially generated grayscale image 1302, the edited grayscale change image 1303, and the final generated image 1304. In this process, the original 2D style and blurred original image 1301 is transformed into a generated image 1304 similar to a 3D style through controllable generation of this solution. In the generation process, the special hair accessories, ears, backpack and other details of the character in the original image 1301 are retained, and at the same time, the facial features, background and other areas that need to be changed of the character are well changed.

[0263] It should be noted that before collecting relevant data of users and during the process of collecting relevant data of users, this application can display a prompt interface, pop-up window or output voice prompt information, and the prompt interface, pop-up window or voice prompt information is used to prompt the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining relevant data of users after obtaining the confirmation operation issued by the user to the prompt interface or pop-up window, otherwise (that is, when the confirmation operation issued by the user to the prompt interface or pop-up window is not obtained), the relevant steps of obtaining relevant data of users are terminated, that is, the relevant data of users are not obtained. In other words, all user data collected by this application are collected with the consent and authorization of the user, and the collection, use and processing of relevant user data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0264] Please refer to Fig.14 , which shows a structural block diagram of an image generation device provided by an exemplary embodiment of the present application, the device includes the following modules:

[0265] A first acquisition module 1410 is used to acquire a first image and a grayscale change map, and to acquire a noise image; wherein a first pixel position of the first image corresponds to a first pixel value, and a second pixel position corresponding to the first pixel position in the grayscale change map corresponds to a second pixel value, and the second pixel value is used to indicate a change intensity of the first pixel position when generating a new image;

[0266] The second acquisition module 1420 is used to acquire an i-th noise image corresponding to the i-th round of iterative noise reduction stage during the process of iteratively denoising the noise image by modifying the first image as a noise reduction condition, where i is a positive integer, and when i=1, the noise image is an initial noise image, and when i>1, the noise intensity of the i-th noise image corresponds to the i-th round of iterative noise reduction stage;

[0267] A denoising module, used for denoising the first noise image when i=1, to obtain the first denoising stage image outputted by the first round of iterative denoising stage;

[0268] A fusion module 1430 is used for fusing the i-th noise image and the i-1th noise reduction stage image outputted from the i-1th iterative noise reduction stage based on the fusion ratio indicated by the second pixel value in the grayscale change map when i>1, to obtain an i-th fused image;

[0269] The denoising module 1440 is further used to denoise the i-th fused image to obtain the i-th denoising stage image outputted by the i-th round of iterative denoising stage, until it meets the iterative denoising requirements, and obtain a generated image as a modification result of the first image.

[0270] In some optional embodiments, such as Fig.15 As shown, the fusion module 1430 includes:

[0271] A screening unit 1431 is used to convert the grayscale change map into an i-th mask corresponding to the i-th round of iterative denoising stage, wherein the i-th mask is used to indicate the regions corresponding to the i-th noise image and the i-1-th denoising stage image in the i-th round of iterative denoising stage when participating in fusion;

[0272] The fusion unit 1432 is used to fuse the i-th noise image and the i-1-th noise reduction stage image based on the i-th mask to obtain the i-th fused image.

[0273] In some optional embodiments, the fusion module 1430 further includes:

[0274] A first acquisition unit 1433 is used to acquire an i-th pixel threshold corresponding to the i-th round of iterative noise reduction stage, wherein the i-th pixel threshold is used to select the second pixel position according to the second pixel value in the grayscale change map;

[0275] The screening unit 1431 is further configured to screen the second pixel value in the grayscale change map by using an i-th pixel threshold to obtain the i-th mask.

[0276] In some optional embodiments, the screening unit 1431 is further used to screen the second pixel positions in the grayscale change map where the second pixel value is greater than the i-th pixel threshold to obtain a first mask map;

[0277] The screening unit 1431 is further configured to screen the first pixels in the grayscale change map whose second pixel values ​​are less than or equal to the i-th pixel threshold to obtain a second mask map;

[0278] The fusion unit 1432 is further configured to superimpose the first mask image and the i-th noise image to obtain a first superimposed image;

[0279] The fusion unit 1432 is further configured to superimpose the second mask image and the image in the (i-1)th denoising stage to obtain a second superimposed image;

[0280] The fusion unit 1432 is further configured to fuse the first superimposed image and the second superimposed image to obtain the i-th fused image.

[0281] In some optional embodiments, the device further comprises:

[0282] The noise adding module 1450 is used to add noise to the first image to obtain the noise image.

[0283] In some optional embodiments, the noise adding module 1450 includes:

[0284] A first sampling unit 1451 is used to sample the standard Gaussian noise to obtain the sampled noise;

[0285] The noise adding unit 1452 is used to add noise to the first image according to the sampling noise and the diffusion time to obtain the noise image, and the diffusion time is used to indicate the number of steps required to generate the generated image from the noise image in the iterative denoising process.

[0286] In some optional embodiments, the second acquisition module 1420 further includes:

[0287] The second sampling unit 1421 is used to sample the diffusion duration to obtain the i-th sampling moment corresponding to the i-th round of iterative noise reduction stage;

[0288] A noise reduction unit 1422, configured to use the noise image as the first noise image corresponding to the first sampling moment when i=1;

[0289] The noise reduction unit 1422 is also used to, when i>1, perform noise reduction on the noise image to obtain the i-th noise image corresponding to the i-th sampling moment; or, when i>1, perform noise reduction on the i-1th noise image corresponding to the i-1th sampling moment to obtain the i-th noise image corresponding to the i-th sampling moment.

[0290] In some optional embodiments, the device further comprises:

[0291] An encoding module 1460, configured to input the first image into a first encoder of the diffusion model to obtain an image encoding representation;

[0292] The encoding module 1460 is further used to input the grayscale change map into the second encoder of the diffusion model to obtain a change map encoding representation;

[0293] The noise adding module 1450 is further used to input the image coding representation into the forward diffusion network in the diffusion model to obtain the noise image feature representation corresponding to the noise image;

[0294] The noise reduction module 1440 is further used to input the noise image feature representation and the change map encoding representation into the reverse diffusion network in the diffusion model to obtain the generated image feature representation;

[0295] The decoding module 1470 is used to input the generated image feature representation into the decoder of the diffusion model to obtain the generated image.

[0296] In some optional embodiments, the first acquisition module 1410 is further used to acquire a prompt word, where the prompt word is used to guide image changes when a new image is generated based on the first image through the diffusion model;

[0297] The encoding module 1460 is further used to input the prompt word into the third encoder of the diffusion model to obtain a text encoding representation;

[0298] The noise reduction module 1440 is further used to input the noise image feature representation, the change map encoding representation and the text encoding representation into the reverse diffusion network in the diffusion model to obtain the generated image feature representation.

[0299] In some optional embodiments, the prompt word includes at least one of the following contents:

[0300] Generate content prompt information, where the generated content prompt information is used to indicate the semantics of content that needs to be generated when generating a new image based on the first image;

[0301] generating style hint information, wherein the generated style hint information is used to indicate an image style of the new image generated based on the first image;

[0302] Image attribute prompt information, where the image attribute prompt information is used to indicate image attributes of the new image generated based on the first image.

[0303] In some optional embodiments, the first acquisition module 1410 includes:

[0304] The recognition unit 1411 is used to recognize the image content in the first image and determine multiple image subjects existing in the first image;

[0305] A segmentation unit 1412, configured to segment the first image into subject areas corresponding to the plurality of image subjects respectively;

[0306] The second acquisition unit 1413 is used to acquire subject modification requirement information corresponding to the multiple image subjects respectively;

[0307] The processing unit 1414 is used to perform grayscale processing on the main body area of ​​the first image based on the main body modification requirement information to obtain the grayscale change map.

[0308] In some optional embodiments, the processing unit 1414 is further configured to perform grayscale processing on the subject area of ​​the first image based on the subject modification requirement information to obtain a grayscale image corresponding to the first image;

[0309] The second acquisition unit 1413 is further configured to receive a region selection instruction for the grayscale image, wherein the region selection instruction is configured to indicate at least one target region, wherein the target region is a region for adjusting the first pixel value of the first image by a specified degree of change;

[0310] The processing unit 1414 is further configured to adjust the pixel values ​​in the at least one target area in the grayscale image to a target grayscale value corresponding to the specified degree of change based on the area selection instruction, so as to obtain the grayscale change map.

[0311] It should be noted that the image generation device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the image generation device provided in the above embodiment and the image generation method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0312] Fig.16 The schematic diagram of the structure of a server provided by an exemplary embodiment of the present application is shown, which specifically includes the following structure.

[0313] The server 1600 includes a central processing unit (CPU) 1601, a system memory 1604 including a random access memory (RAM) 1602 and a read only memory (ROM) 1603, and a system bus 1605 connecting the system memory 1604 and the central processing unit 1601. The server 1600 also includes a mass storage device 1606 for storing an operating system 1613, application programs 1614, and other program modules 1615.

[0314] The mass storage device 1606 is connected to the central processing unit 1601 through a mass storage controller (not shown) connected to the system bus 1605. The mass storage device 1606 and its associated computer readable media provide non-volatile storage for the server 1600. That is, the mass storage device 1606 may include a computer readable medium (not shown) such as a hard disk or a compact disc read only memory (CD-ROM) drive.

[0315] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other solid-state memory technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 1604 and mass storage device 1606 can be collectively referred to as memory.

[0316] According to various embodiments of the present application, the server 1600 can also be connected to a remote computer on the network through a network such as the Internet. That is, the server 1600 can be connected to the network 1612 through the network interface unit 1611 connected to the system bus 1605, or the network interface unit 1611 can be used to connect to other types of networks or remote computer systems (not shown).

[0317] The above-mentioned memory also includes one or more programs, and the one or more programs are stored in the memory and configured to be executed by the CPU.

[0318] The embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the image generation method provided by the above-mentioned method embodiments. Optionally, the computer device can be a terminal or a server.

[0319] An embodiment of the present application further provides a computer-readable storage medium, on which at least one computer program is stored, and the at least one computer program is loaded and executed by a processor to implement the image generation method provided by the above-mentioned method embodiments.

[0320] An embodiment of the present application further provides a computer program product, including a computer program, which is executed by a processor to implement the image generation method provided by the above-mentioned method embodiments.

[0321] Optionally, the computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0322] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0323] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating an image, characterized in that: The method comprises: Acquire a first image and a grayscale change map, and acquire a noise image; wherein a first pixel position of the first image corresponds to a first pixel value, a second pixel position corresponding to the first pixel position in the grayscale change map corresponds to a second pixel value, and the second pixel value is used to indicate the change intensity of the first pixel position when generating a new image; In the process of iteratively denoising the noise image by modifying the first image as a denoising condition, obtaining an i-th noise image corresponding to the i-th round of iterative denoising stage, where i is a positive integer, and when i=1, the noise image is an initial noise image, and when i>1, the noise intensity of the i-th noise image corresponds to the i-th round of iterative denoising stage; When i=1, the first noise image is denoised to obtain the first denoised stage image outputted by the first round of iterative denoising stage; When i>1, based on the fusion ratio indicated by the second pixel value in the grayscale change map, the i-th noise image and the i-1th noise reduction stage image outputted from the i-1th iterative noise reduction stage are fused to obtain the i-th fused image; Denoising is performed on the i-th fused image to obtain an i-th denoising stage image outputted in the i-th round of iterative denoising stage, until it meets the iterative denoising requirements, and a generated image is obtained as a modification result of the first image.

2. The method according to claim 1, characterized in that The step of fusing the i-th noise image and the i-1th noise reduction stage image obtained by the i-1th round of iterative noise reduction based on the fusion ratio indicated by the second pixel value in the grayscale change map to obtain the i-th fused image comprises: Converting the grayscale change map into an i-th mask corresponding to the i-th round of iterative denoising stage, wherein the i-th mask is used to indicate the regions corresponding to the i-th noise image and the i-1-th denoising stage image in the i-th round of iterative denoising stage when participating in fusion; The i-th noise image and the i-1-th denoising stage image are fused based on the i-th mask to obtain the i-th fused image.

3. The method according to claim 2, characterized in that The step of converting the grayscale change map into an i-th mask corresponding to the i-th round of iterative denoising process includes: Obtaining an i-th pixel threshold corresponding to the i-th round of iterative noise reduction stage, wherein the i-th pixel threshold is used to select the second pixel position according to the second pixel value in the grayscale change map; The second pixel value in the grayscale change map is screened by using an i-th pixel threshold to obtain the i-th mask.

4. The method according to claim 3, characterized in that The step of screening the second pixel value in the grayscale change map by using the i-th pixel threshold to obtain the i-th mask includes: Screening the second pixel positions in the grayscale change map where the second pixel value is greater than the i-th pixel threshold to obtain a first mask map; Filtering the first pixels whose second pixel values ​​in the grayscale change map are less than or equal to the i-th pixel threshold to obtain a second mask map; The step of fusing the i-th noise image and the i-1-th noise reduction stage image based on the i-th mask to obtain the i-th fused image comprises: Superimposing the first mask image and the i-th noise image to obtain a first superimposed image; Superimposing the second mask image and the image in the (i-1)th denoising stage to obtain a second superimposed image; The first superimposed image and the second superimposed image are fused to obtain the i-th fused image.

5. The method according to any one of claims 1 to 4, characterized in that: The step of acquiring a noise image comprises: Noise is added to the first image to obtain the noisy image.

6. The method according to claim 5, characterized in that The adding noise to the first image to obtain a noisy image includes: Sampling standard Gaussian noise to obtain the sampled noise; Noise is added to the first image according to the sampling noise and the diffusion time to obtain the noise image, wherein the diffusion time is used to indicate the number of steps required to generate the generated image from the noise image during an iterative denoising process.

7. The method according to claim 6, characterized in that The step of obtaining the i-th noise image corresponding to the i-th round of iterative noise reduction stage includes: Sampling the diffusion time to obtain the i-th sampling moment corresponding to the i-th round of iterative noise reduction stage; When i=1, the noise image is taken as the first noise image corresponding to the first sampling moment; When i>1, the noise image is denoised to obtain the i-th noise image corresponding to the i-th sampling moment; or, when i>1, the i-1th noise image corresponding to the i-1th sampling moment is denoised to obtain the i-th noise image corresponding to the i-th sampling moment.

8. The method according to any one of claims 1 to 4, characterized in that: The method is performed by a diffusion model, and the data processing process of the diffusion model includes: Inputting the first image into a first encoder of the diffusion model to obtain an image encoding representation; Inputting the grayscale change map into a second encoder of the diffusion model to obtain a change map encoding representation; Inputting the image encoding representation into the forward diffusion network in the diffusion model to obtain a noise image feature representation corresponding to the noise image; Inputting the noise image feature representation and the variation map encoding representation into a reverse diffusion network in the diffusion model to obtain a generated image feature representation; The generated image feature representation is input into the decoder of the diffusion model to obtain the generated image.

9. The method according to claim 8, characterized in that The step of inputting the noise image feature representation and the variation map encoding representation into a reverse diffusion network in the diffusion model to obtain a generated image feature representation comprises: Acquire a prompt word, where the prompt word is used to guide image changes when a new image is generated based on the first image through the diffusion model; Inputting the prompt word into the third encoder of the diffusion model to obtain a text encoding representation; The noise image feature representation, the variation map encoding representation and the text encoding representation are input into a reverse diffusion network in the diffusion model to obtain a generated image feature representation.

10. The method according to claim 9, characterized in that The prompt word includes at least one of the following contents: Generate content prompt information, where the generated content prompt information is used to indicate the semantics of content that needs to be generated when generating a new image based on the first image; generating style hint information, wherein the generated style hint information is used to indicate an image style of the new image generated based on the first image; Image attribute prompt information, where the image attribute prompt information is used to indicate image attributes of the new image generated based on the first image.

11. The method according to any one of claims 1 to 4, characterized in that: The step of obtaining a grayscale change map comprises: Identify image content in the first image and determine a plurality of image subjects present in the first image; dividing the first image into subject areas corresponding to the plurality of image subjects respectively; Acquire subject modification requirement information respectively corresponding to the plurality of image subjects; The subject area of ​​the first image is grayed based on the subject modification requirement information to obtain the grayscale change map.

12. The method according to claim 11, characterized in that The grayscale processing of the main body area of ​​the first image based on the main body modification requirement information to obtain the grayscale change map includes: grayscale the subject area of ​​the first image based on the subject modification requirement information to obtain a grayscale image corresponding to the first image; receiving a region selection instruction for the grayscale image, the region selection instruction being used to indicate at least one target region, the target region being a region for adjusting the first pixel value of the first image by a specified degree of change; Based on the area selection instruction, the pixel values ​​in the at least one target area in the grayscale image are adjusted to the target grayscale value corresponding to the specified change degree to obtain the grayscale change map.

13. An image generating device, characterized in that: The device comprises: A first acquisition module, used to acquire a first image and a grayscale change map, and to acquire a noise image; wherein a first pixel position of the first image corresponds to a first pixel value, a second pixel position corresponding to the first pixel position in the grayscale change map corresponds to a second pixel value, and the second pixel value is used to indicate a change intensity of the first pixel position when generating a new image; A second acquisition module is used to acquire an i-th noise image corresponding to an i-th round of iterative noise reduction stage during iterative noise reduction of the noise image by modifying the first image as a noise reduction condition, where i is a positive integer, and when i=1, the noise image is an initial noise image, and when i>1, the noise intensity of the i-th noise image corresponds to the i-th round of iterative noise reduction stage; A denoising module, used for denoising the first noise image when i=1, to obtain the first denoising stage image outputted by the first round of iterative denoising stage; a fusion module, configured to fuse the i-th noise image and the i-1th noise reduction stage image outputted from the i-1th iterative noise reduction stage based on a fusion ratio indicated by a second pixel value in the grayscale change map when i>1, to obtain an i-th fused image; The denoising module is further used to denoise the i-th fused image to obtain the i-th denoising stage image outputted by the i-th round of iterative denoising stage, until it meets the iterative denoising requirements, and obtain a generated image as a modification result of the first image.

14. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the image generation method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the computer program is loaded and executed by a processor to implement the image generation method according to any one of claims 1 to 12.

16. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method for generating an image as claimed in any one of claims 1 to 12.

Images