Image external drawing method and device, equipment and storage medium

Abstract

This application discloses an image outlining method, apparatus, device, and storage medium, relating to the field of image processing. The method includes: processing an original image using a pre-trained multimodal model to obtain image generation prompts; and then using a pre-trained image generation model to generate an outlined image based on the image generation prompts and the original image. Since the image generation prompts contain descriptive text for generating the outlined region of the original image, they provide outlined region information for subsequent outlined image generation, facilitating the generation of information-rich outlined regions. Furthermore, the image generation model is a stable diffusion model configured with an extended model, which is a pre-trained inline drawing control network ControlNet++. This network uses the original image to control the image generation process, ensuring that the region in the generated image from the stable diffusion model corresponds to the original image and closely matches the original image, ultimately achieving a vivid and natural image outlining task.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image drawing method, apparatus, device and storage medium. Background Technology

[0002] Image outpainting is an image editing task that involves drawing new image content outside the boundaries of a given original image to expand the image.

[0003] Currently, image outlining can be performed by copying or splicing similar texture content outside the boundaries of the original image based on the texture information in the original image. For example, when outlining a grassland image, grassland texture samples can be extracted from the grassland image and spliced ​​outside the image boundary directly or after reasonable deformation, thereby achieving image expansion; alternatively, similar image patches can be obtained from an image library and filled into the area to be outlined to achieve image expansion.

[0004] However, the texture and other information of the drawn area obtained based on the above scheme is relatively simple, and the existing image drawing schemes are difficult to achieve a vivid and natural extension of the original image. Summary of the Invention

[0005] In view of the above problems, this application provides an image drawing method, apparatus, device and storage medium to generate an information-rich drawing area and achieve a vivid and natural image drawing task.

[0006] The specific plan is as follows:

[0007] The first aspect of this application provides an image drawing method, including:

[0008] The pre-trained multimodal model is called to process the original image to obtain image generation prompts, which contain descriptive text for generating the drawn region of the original image;

[0009] A pre-trained image generation model is invoked to generate an externally drawn image based on the image generation prompt and the original image. The image generation model is a stable diffusion model configured with an extended model, which is a pre-trained in-line drawing control network ControlNet++. The in-line drawing control network ControlNet++ uses the original image to control the image generation process of the stable diffusion model so that the region in the generated image of the stable diffusion model that corresponds to the original image tends to be consistent with the original image.

[0010] A second aspect of this application provides an image drawing apparatus, comprising:

[0011] The prompt word generation unit is used to call a pre-trained multimodal model to process the original image and obtain image-generated prompt words. The image-generated prompt words contain descriptive text for generating the drawn region of the original image.

[0012] An image generation unit is used to call a pre-trained image generation model to generate an externally drawn image based on the image generation prompt and the original image; wherein, the image generation model is a stable diffusion model configured with an extended model, the extended model is a pre-trained in-line drawing control network ControlNet++, and the in-line drawing control network ControlNet++ uses the original image to control the image generation process of the stable diffusion model so that the region in the generated image of the stable diffusion model that corresponds to the original image tends to be consistent with the original image.

[0013] A third aspect of this application provides an image mapping device, comprising at least one processor and a memory connected to the processor, wherein:

[0014] The memory is used to store computer programs;

[0015] The processor is used to execute the computer program to implement the image drawing method described in the first aspect above.

[0016] A fourth aspect of this application provides a storage medium carrying one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the image drawing method described in the first aspect.

[0017] Using the above technical solution, this application first calls a pre-trained multimodal model to process the original image to obtain image generation prompts, and then calls a pre-trained image generation model to generate a drawn image based on the image generation prompts and the original image. Since the image generation prompts contain descriptive text for generating the drawn region of the original image, they provide drawn region information for the subsequent drawn image generation, which helps to generate an information-rich drawn region. On this basis, the image generation model is a stable diffusion model configured with an extension model, which is a pre-trained in-line drawing control network ControlNet++. Since the in-line drawing control network ControlNet++ uses the original image to control the image generation process of the stable diffusion model, it can make the region in the generated image of the stable diffusion model that corresponds to the original image consistent with the original image, thereby basically preserving the content of the original image. This allows the generated drawn image to serve as the target image for drawn extension of the original image, ultimately achieving a vivid and natural image drawing task. Attached Figure Description

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0019] Figure 1 A flowchart illustrating an image drawing method provided in this application;

[0020] Figure 2 A schematic diagram of the structure of an image generation model is shown;

[0021] Figure 3 A schematic diagram of the noise prediction process based on the joint guidance method is shown;

[0022] Figure 4 A schematic diagram of the external drawing process is shown;

[0023] Figure 5 This application provides a schematic diagram of the structure of an image drawing device.

[0024] Figure 6 This is a schematic diagram of the structure of an image drawing device provided in this application. Detailed Implementation

[0025] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is only for explaining specific embodiments and is not intended to limit the application. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0026] Figure 1 This is a schematic flowchart illustrating an image drawing method according to an embodiment of this application. (In conjunction with...) Figure 1 As shown, the method may include the following steps:

[0027] Step S101: Call the pre-trained multimodal model to process the original image and obtain image-generated prompts.

[0028] The image generation prompts include descriptive text for the drawn regions used to generate the original image. This text can be inferred by a multimodal model based on the content of the original image, providing an informational basis for subsequent image generation, particularly for the generation of images corresponding to the drawn regions. For example, the multimodal model mentioned above can be a multimodal large language model, such as GPT-4o, or other image understanding models, such as mini-CPM-v.

[0029] Step S102: Call the pre-trained image generation model to generate a drawn image based on the image prompt words and the original image.

[0030] The image generation model is a stable diffusion model configured with an extended model, namely a pre-trained In-Picture Control Network (ControlNet++). ControlNet++ uses the original image to control the image generation process of the stable diffusion model, ensuring that the regions in the generated image corresponding to the original image are consistent with the original image. ControlNet++ is obtained by optimizing ControlNet, and it exhibits better performance in maintaining consistency with the original image. By controlling the generation method with ControlNet++, it is unnecessary to introduce an additional consistency control module (such as an image cue adapter IP-Adapter), thereby reducing the computational resource requirements and task implementation cost to some extent. In one possible implementation, the stable diffusion model mentioned above can refer to the text-to-image generation model SDXL-Lighting. It's worth noting that this model performs well in terms of image quality, generating higher resolution and more detailed images while maintaining good diversity and image-text matching. Furthermore, the model offers flexible usage, generating images in 1, 2, 4, or 8 steps, with image quality improving as the number of inference steps increases. In addition, the model employs progressive adversarial distillation, generating high-quality and high-resolution images in 2 or 4 steps, reducing computational cost and time by a factor of ten. It can even generate images for time-sensitive applications in just one step, with a slight sacrifice in quality. Based on these findings, the SDXL-Lighting model can significantly reduce the number of sampling steps while maintaining a relatively consistent generation quality, thereby improving image generation efficiency.

[0031] This application first uses a pre-trained multimodal model to process the original image, obtaining image generation prompts. Then, it uses a pre-trained image generation model to generate a drawn image based on the image generation prompts and the original image. Since the image generation prompts contain descriptive text for the drawn region used to generate the original image, they provide drawn region information for the subsequent drawn image generation, helping to generate information-rich drawn regions. Furthermore, the image generation model is a stable diffusion model configured with an extension model, which is a pre-trained in-line drawing control network, ControlNet++. Because ControlNet++ uses the original image to control the image generation process of the stable diffusion model, it ensures that the regions in the generated image of the stable diffusion model that correspond to the original image are consistent with the original image, thus essentially preserving the content of the original image. This allows the generated drawn image to serve as the target image for drawn extension of the original image, ultimately achieving a vivid and natural image drawing task.

[0032] In one or more embodiments provided in this application, step S101, which involves calling a pre-trained multimodal model to process the original image and obtain image-generated prompts, may include:

[0033] The original image and pre-configured image processing prompts are input into the multimodal model so that the multimodal model can output the image generation prompts.

[0034] The image processing prompts are used to instruct the multimodal model to recognize and understand the input image, infer the outer drawing region of the input image based on the recognized image content, and generate and output prompts based on the recognized image content and the inferred outer drawing content.

[0035] The aforementioned step S101 can be referred to as the prompt word reverse inference stage. Using the above scheme, this application utilizes a multimodal model to achieve a deep understanding of the original image content and imagines the content of the drawn area, thereby obtaining rich supplementary information on the drawn area description, providing abundant reference information for applying the in-drawing model to solve the drawn area problem.

[0036] Compared to commonly used prompt word inference schemes that rely solely on visual language models to understand the original image, this scheme, which uses the original image description to guide the diffusion model generation, can supplement missing information about the drawn area. This can, to some extent, solve the problems of blurred content and poor image quality in the drawn area. Even when facing large-scale drawn area tasks, this scheme can provide certain information support for large drawn areas. Therefore, the image-generated prompt words obtained in the prompt word inference stage of this application can significantly improve drawn area quality and greatly solve the problem of blurred drawn areas at large drawn areas.

[0037] In one or more embodiments provided in this application, step S102, which involves calling a pre-trained image generation model to generate a drawn image based on the image prompt and the original image, may include the following steps:

[0038] Step S201: Fill the surrounding area of ​​the original image with a preset color to generate an extended image containing the original image and having the same size as the drawn image, as a control image.

[0039] For example, the preset color mentioned above can be white or black; the control chart can be centered on the original image. Specifically, if the outlay target specifies the position of the original image in the outlay image, then the position of the original image in the control chart is consistent with the position of the original image (or the image area corresponding to the original image) in the outlay image.

[0040] Step S202: Use the control graph as the conditional input of the in-line drawing control network ControlNet++, and use the image generation prompt as the prompt input of the in-line drawing control network ControlNet++ and the stable diffusion model. Call the configured stable diffusion model to generate the externally drawn image based on the input original image.

[0041] For example, Figure 2 This diagram illustrates the structure of an image generation model, showing the stable diffusion model and the structure of ControlNet++ for in-line drawing. Figure 2 As shown, the original image can be used as "input" to the stable diffusion model and the in-line drawing ControlNet++, the image generation prompt can be used as "prompt" to the stable diffusion model and the in-line drawing ControlNet++, and the control graph can be used as "condition N" to the in-line drawing ControlNet++.

[0042] In one or more embodiments provided in this application, the stable diffusion model adopts a sampling guidance method that combines perturbation attention guidance and classifier-free guidance.

[0043] It should be noted that Perturbed-Attention Guidance (PAG) aims to capture structural information by considering self-attention mechanisms, thereby progressively enhancing the structure of synthesized samples during denoising. Specifically, PAG generates structurally degraded intermediate samples by replacing selected self-attention maps in the diffusion U-Net with an identity matrix, and guides the denoising process away from these degraded samples. For example, by replacing the self-attention mask of a specified layer with the identity matrix, mutual attention between different regions in the image is completely deactivated. It should be noted that self-attention is mainly responsible for reconstructing the overall structure of the image, and deactivating self-attention can result in the loss of supervision over structural details. PAG can achieve sample quality improvement under both unconditional and conditional settings without further training or integration of external modules. Classifier-Free Guidance (CFG) amplifies the influence of conditional signals (such as textual conditions) by mixing the noisy prediction results of conditional branches and unconditional branches.

[0044] The joint guidance method combining PAG and CFG can be applied to the noise predicted by the U-Net of the diffusion model in each sampling step. For example, Figure 3 A schematic diagram of the noise prediction process based on the joint guidance method is shown, combined with Figure 3 As shown, the prediction noise at time t can be expressed as: σ t =σ φ +λ CFG ×(σ text -σ φ )+λ PAG ×(σ text -σ perturbed In the formula, σ φ This represents unconditional noise, which is the noisy prediction result obtained by injecting the text prompt word as an empty string into the model. In other words, it's the result generated by the model without any guidance or direction, and the output is usually very random. σ text This represents text conditional noise, which is the noise prediction result obtained by inputting text prompts into the model, and the result is expressed as σ. text During the process, all attention layers in U-Net are enabled normally, and the output usually follows the cue words; σ perturbed This represents perturbation noise, which is the noise prediction result obtained by inputting the text prompt words into the model, and is expressed as σ. text During the process, some attention layers in U-Net were deactivated; λ CFG This represents the weighting coefficient used to measure the intensity of CFG, via λ.CFG ×(σ text -σ φ This can cause the output image to be biased towards σ. text This adherence to cue words results in avoiding σ. φ This random result; λ PAG This represents the weighting coefficient for measuring the intensity of PAG, using λ PAG ×(σ text -σ perturbed This can cause the output image to be biased towards σ. text This well-structured image avoids σ perturbed This results in a cluttered and disorganized visual structure.

[0045] Based on the above, this application improves the quality of structure generation in the out-of-area region to a certain extent by introducing PAG in the sampling. By combining PAG and CFG, the advantages of CFG in better adherence to text prompts and the advantages of PAG in generating images with better structures are combined, which helps to improve the overall quality of the out-of-area image.

[0046] In one or more embodiments provided in this application, after generating the drawn image, the following steps may also be included:

[0047] Step S103: Post-process the drawn image to obtain the drawn result of the original image.

[0048] Post-processing of the drawn image may include:

[0049] The image region in the redrawn image corresponding to the original image is replaced with the original image, and image fusion processing is performed on the seam region between the redrawn image and the original image.

[0050] For example, the image fusion processing described above can refer to fusion of the stitching edges of the original image and the drawn image using the alpha blending / composition method. That is, the two original images and the drawn image are linearly weighted and fused at the stitching edges to eliminate the seams.

[0051] Specifically, a mask image can be generated first. The region corresponding to the original image in this mask image is white (alpha=1) to represent that region is entirely the original image, while the remaining regions are black (alpha=0) to represent that region is entirely the drawn image. At the seam, a portion of the image is taken inwards and another portion outwards to form a blending region (also called the seam region or transition region). In this region, the alpha gradually decreases from the inside out (from 1.0 to 0.0) to represent the gradual fusion of the original and drawn images. Assuming the original image is a rectangle with width W and height H, the transition region span in the width direction can be W / 8, consisting of 1 / 16 of the width taken inwards and 1 / 16 of the width taken outwards; correspondingly, the transition region span in the length direction can be H / 8. The gradual attenuation of alpha from the inside out in the transition region can be achieved by applying a filter to the mask image. The filter can be used to blur the white-black boundary of the mask image, ultimately forming the transition region. For example, the filter mentioned above can be a box filter, which updates the alpha value of the transition region by calculating the average alpha value of the surrounding pixels, thus achieving a smooth transition.

[0052] It should be noted that since the image generation model randomly samples from a distribution to generate images, it cannot guarantee that the regions in the drawn image corresponding to the original image are completely consistent with the original image. Based on this, the above scheme ensures the consistency of the original image by overlaying the original image onto the corresponding position in the drawn image, and eliminates the seams through image fusion, which can make the seam areas of the drawn image blend naturally, thereby improving the quality of image drawing.

[0053] In one or more embodiments provided in this application, post-processing of the drawn image may further include: performing the following steps before replacing the image region in the drawn image that corresponds to the original image:

[0054] Step A: Calculate the pixel values ​​of the image regions (which can be represented as I_original) in the original image and the outpainted image that correspond to the original image.

[0055] Step B: Calculate the pixel value distribution in the RGB color space of the image region corresponding to the original image in the original image and the drawn image respectively, based on the kernel density estimation method.

[0056] Step C: Based on the calculated pixel value distribution, establish a pixel value mapping relationship between the original image and the image region in the drawn image that corresponds to the original image.

[0057] Specifically, the distributions P_original and P_outpaint of I_original and I_outpaint in the RGB color space can be estimated using the kernel density estimation method. This includes estimating the distributions of the R, G, and B channels. Then, the cumulative distribution functions (CDFs) of I_original and I_outpaint in the RGB space are calculated using P_original and P_outpaint, respectively, and denoted as CDF_original and CDF_outpaint. Finally, the pixel value mapping relationship between I_original and I_outpaint is constructed using CDF_original and CDF_outpaint.

[0058] Step D: Correct the pixel values ​​of the drawn image according to the pixel value mapping relationship, and continue to execute the step of replacing the image region in the drawn image that corresponds to the original image with the original image based on the corrected drawn image.

[0059] The above scheme establishes a pixel value mapping relationship between the original image and the corresponding region in the external drawing result through kernel density estimation. Based on this, the external drawing image is corrected, eliminating the color difference between the external drawing image and the original image, and further ensuring the consistency between the external drawing image and the original image.

[0060] In one or more embodiments provided in this application, post-processing of the drawn image may further include:

[0061] Before replacing the image region in the redrawn image that corresponds to the original image, a pre-trained image inpainting model is invoked to inpaint the redrawn image, and the step of replacing the image region in the redrawn image that corresponds to the original image with the original image is continued based on the inpainted image.

[0062] The image inpainting model includes a face inpainting model and an image super-resolution model. The face inpainting model can refer to the GFPGAN (Generative Facial Prior - Guided Facial Attribute Editing Network) model, which utilizes a Generative Adversarial Network (GAN) architecture. Specifically, this model's generator network can perform inpainting and enhancement operations on input face images based on pre-trained face prior knowledge. During model training, a large amount of high-quality face image data can be used to allow the generator to learn key features such as facial structure and texture. Based on this, the discriminator network can distinguish between generated and real faces. Through adversarial training between the generator and discriminator, the generator's performance is optimized. Furthermore, GFPGAN also utilizes semantic information of facial components, such as the position and shape of facial features, to more accurately repair facial details, such as repairing facial blemishes, improving clarity, and reconstructing facial texture details, contributing to the generation of high-quality, natural, and realistic face images. Additionally, the image super-resolution model can be the basic image inpainting model Real-ESRGAN. For example, the GFPGAN model can be used to detect and repair faces to obtain clearer face region images, and the Real-ESRGAN model can be used to repair the entire image range to obtain clearer external drawing images. Finally, the corresponding regions in the repaired external drawing images can be replaced using the repaired face region images to obtain the repaired external drawing images.

[0063] Based on the above, this application uses a face restoration model to restore the faces in the drawn results, and integrates an image super-resolution model on this basis to improve the overall image clarity and ultimately improve the quality of image drawing.

[0064] Optionally, during post-processing, repair processing can be performed first, followed by color difference correction processing, and finally replacement and fusion processing.

[0065] Figure 4 The example illustrates the external drawing process using the scheme of this application, including the original image, the control chart C_image generated based on the original image, and the obtained external drawing result. Figure 4As shown, the original image is taken from the open-source anime dataset qkrwnstj / anime_dataset. The image-generated prompt C_text obtained by inputting the original image into the multimodal model can include: "A high-resolution image of an anime-style short-haired woman standing in a lush green park. The woman (i.e., the aforementioned short-haired woman) is wearing a light, flowing dress, raising one hand as if checking something. In the background, tall city buildings can be seen under a clear sky, and sunlight filters through the trees, creating a tranquil and peaceful atmosphere. The soft, natural lighting highlights the calm and serene environment." The original image and C_text are input into a stable diffusion model, and the original image, C_image, and C_text are input into the in-line drawing control network ControlNet++. Finally, post-processing yields the out-of-line drawing result of the original image.

[0066] The image drawing apparatus provided in the embodiments of this application is described below. The image drawing apparatus described below can be referred to in correspondence with the image drawing method described above.

[0067] Figure 5 This is a schematic diagram of the structure of an image drawing device disclosed in an embodiment of this application. Figure 5 As shown, the device may include:

[0068] The prompt word generation unit 11 is used to call a pre-trained multimodal model to process the original image and obtain image generation prompt words, wherein the image generation prompt words contain descriptive text for generating the drawn region of the original image;

[0069] Image generation unit 12 is used to call a pre-trained image generation model to generate an externally drawn image based on the image generation prompt and the original image; wherein, the image generation model is a stable diffusion model configured with an extended model, the extended model is a pre-trained internal drawing control network ControlNet++, and the internal drawing control network ControlNet++ uses the original image to control the image generation process of the stable diffusion model so that the region in the generated image of the stable diffusion model that corresponds to the original image tends to be consistent with the original image.

[0070] In one or more embodiments provided in this application, the process by which the prompt word generation unit 11 calls a pre-trained multimodal model to process the original image and obtain image-generated prompt words may include:

[0071] The original image and pre-configured image processing prompts are input into the multimodal model so that the multimodal model can output the image generation prompts. The image processing prompts are used to instruct the multimodal model to recognize and understand the input image, infer the outer drawing region of the input image based on the recognized image content, and generate and output prompts based on the recognized image content and the inferred outer drawing content.

[0072] In one or more embodiments provided in this application, the process by which the image generation unit 12 calls a pre-trained image generation model to generate a drawn image based on the image generation prompt and the original image may include:

[0073] The area surrounding the original image is filled with a preset color to generate an extended image containing the original image and having the same size as the drawn image, which serves as a control image.

[0074] The control graph is used as the conditional input to the in-line drawing control network ControlNet++, and the image generation prompt is used as the prompt input to the in-line drawing control network ControlNet++ and the stable diffusion model. The configured stable diffusion model is then invoked to generate the out-drawn image based on the input original image.

[0075] In one or more embodiments provided in this application, the stable diffusion model adopts a sampling guidance method that combines perturbation attention guidance and classifier-free guidance.

[0076] In one or more embodiments of this application, the apparatus may further include: a post-processing unit, which is configured to perform post-processing on the drawn image after generating the drawn image to obtain the drawn result of the original image.

[0077] Based on the above, the post-processing unit's post-processing of the drawn image may include:

[0078] The image region in the redrawn image corresponding to the original image is replaced with the original image, and image fusion processing is performed on the seam region between the redrawn image and the original image.

[0079] In one or more embodiments of this application, the post-processing unit may further include the following steps in post-processing the drawn image:

[0080] Before replacing the image region in the drawn image that corresponds to the original image, the following steps are performed:

[0081] Calculate the pixel values ​​of the image regions corresponding to the original image in both the original image and the drawn image;

[0082] The pixel value distribution in the RGB color space of the image region corresponding to the original image in the original image and the drawn image are calculated based on the kernel density estimation method.

[0083] Based on the calculated pixel value distribution, a pixel value mapping relationship is established between the original image and the image region in the drawn image that corresponds to the original image;

[0084] The pixel values ​​of the drawn image are corrected according to the pixel value mapping relationship, and the step of replacing the image region in the drawn image that corresponds to the original image with the original image is continued based on the corrected drawn image.

[0085] In one or more embodiments of this application, the post-processing unit may further include the following steps in post-processing the drawn image:

[0086] Before replacing the image region in the drawn image that corresponds to the original image, a pre-trained image inpainting model is called to inpaint the drawn image, and the step of replacing the image region in the drawn image that corresponds to the original image with the original image is continued based on the inpainted image.

[0087] The image restoration model includes a face restoration model and an image super-resolution model.

[0088] The image drawing device provided in this application embodiment can be applied to image drawing equipment, such as terminals with data processing capabilities: mobile phones, computers, etc. Optionally, Figure 6 The hardware structure block diagram of the image drawing device is shown below. Figure 6 The hardware structure of an image drawing device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0089] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0090] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0091] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0092] The memory is used to store computer programs, and the processor is used to execute the computer programs so that the image drawing device can implement any of the above-described image drawing methods.

[0093] This application also provides a storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the image drawing methods provided in this application.

[0094] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the image drawing methods provided in this application.

[0095] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0096] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0097] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image drawing method, characterized in that, include: The pre-trained multimodal model is called to process the original image to obtain image generation prompts, which contain descriptive text for generating the drawn region of the original image; A pre-trained image generation model is invoked to generate an externally drawn image based on the image generation prompt and the original image; wherein, the image generation model is a stable diffusion model configured with an extended model, the extended model is a pre-trained internal drawing control network ControlNet++, and the internal drawing control network ControlNet++ uses the original image to control the image generation process of the stable diffusion model so that the region in the generated image of the stable diffusion model that corresponds to the original image tends to be consistent with the original image; The step of calling the pre-trained image generation model to generate a drawn image based on the image generation prompt and the original image includes: The area surrounding the original image is filled with a preset color to generate an extended image containing the original image and having the same size as the drawn image, which serves as a control image. The control graph is used as the conditional input to the in-line drawing control network ControlNet++, and the image generation prompt is used as the prompt input to the in-line drawing control network ControlNet++ and the stable diffusion model. The original image is input to the stable diffusion model and the in-line drawing control network ControlNet++, and the configured stable diffusion model is invoked to generate the externally drawn image based on the input original image. If the position of the original image in the externally drawn image is specified, then the position of the original image in the control graph is consistent with the position of the original image in the externally drawn image.

2. The image drawing method according to claim 1, characterized in that, The process of calling a pre-trained multimodal model to process the original image and obtain image-generated prompts includes: The original image and pre-configured image processing prompts are input into the multimodal model so that the multimodal model can output the image generation prompts. The image processing prompts are used to instruct the multimodal model to recognize and understand the input image, infer the outer drawing region of the input image based on the recognized image content, and generate and output prompts based on the recognized image content and the inferred outer drawing content.

3. The image drawing method according to claim 1, characterized in that, The stable diffusion model employs a sampling guidance method that combines perturbation-attention guidance with classifier-free guidance.

4. The image drawing method according to claim 1 or 2, characterized in that, After generating the drawn image, the process also includes: Post-processing is performed on the drawn image to obtain the drawn result of the original image; wherein, the post-processing of the drawn image includes: The image region in the redrawn image corresponding to the original image is replaced with the original image, and image fusion processing is performed on the seam region between the redrawn image and the original image.

5. The image drawing method according to claim 4, characterized in that, Post-processing of the drawn image further includes: Before replacing the image region in the drawn image that corresponds to the original image, the following steps are performed: Calculate the pixel values ​​of the image regions corresponding to the original image in both the original image and the drawn image; The pixel value distribution in the RGB color space of the image region corresponding to the original image in the original image and the drawn image are calculated based on the kernel density estimation method. Based on the calculated pixel value distribution, a pixel value mapping relationship is established between the original image and the image region in the drawn image that corresponds to the original image; The pixel values ​​of the drawn image are corrected according to the pixel value mapping relationship, and the step of replacing the image region in the drawn image that corresponds to the original image with the original image is continued based on the corrected drawn image.

6. The image drawing method according to claim 4, characterized in that, Post-processing of the drawn image further includes: Before replacing the image region in the drawn image that corresponds to the original image, a pre-trained image inpainting model is called to inpaint the drawn image, and the step of replacing the image region in the drawn image that corresponds to the original image with the original image is continued based on the inpainted image. The image restoration model includes a face restoration model and an image super-resolution model.

7. An image drawing device, characterized in that, include: The prompt word generation unit is used to call a pre-trained multimodal model to process the original image and obtain image-generated prompt words. The image-generated prompt words contain descriptive text for generating the drawn region of the original image. An image generation unit is used to invoke a pre-trained image generation model to generate a drawn image based on the image generation prompt and the original image; wherein, the image generation model is a stable diffusion model configured with an extended model, the extended model being a pre-trained in-painting control network ControlNet++, the in-painting control network ControlNet++ using the original image to control the image generation process of the stable diffusion model, so that the region corresponding to the original image in the generated image of the stable diffusion model tends to be consistent with the original image; wherein, invoking the pre-trained image generation model to generate a drawn image based on the image generation prompt and the original image includes: The area surrounding the original image is filled with a preset color to generate an extended image containing the original image and having the same size as the drawn image, which serves as a control image. The control graph is used as the conditional input to the in-line drawing control network ControlNet++, and the image generation prompt is used as the prompt input to the in-line drawing control network ControlNet++ and the stable diffusion model. The original image is input to the stable diffusion model and the in-line drawing control network ControlNet++, and the configured stable diffusion model is invoked to generate the externally drawn image based on the input original image. If the position of the original image in the externally drawn image is specified, then the position of the original image in the control graph is consistent with the position of the original image in the externally drawn image.

8. An image rendering device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the image drawing device can implement the image drawing method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the image drawing method as described in any one of claims 1 to 6.

Images