Image generation method and system based on diffusion model, and storage medium

Abstract

The invention discloses an image generation method and system based on a diffusion model and a storage medium, and the method comprises the steps: obtaining an original image, and carrying out the preprocessing of the original image, obtaining a face image, an eye region mask, a face image with the eye region mask removed, and a reference image which only retains an eye region; encoding the face image, the face image without the eye area mask and the eye area mask, and inputting the face image, the face image without the eye area mask and the eye area mask into a Unet model; obtaining a text prompt word and performing coding processing to obtain a text feature vector; performing coding processing and feature mapping processing on the reference image to obtain an image feature vector; inputting the text feature vector and the image feature vector into a Unet model through a cross attention mechanism; obtaining an eye opening degree adjustment parameter, and adjusting the output of the Unet model through a pre-trained LoRA weight; decoding a latent variable output by the Unet model to obtain an eye opening image; and splicing and fusing corresponding areas of the eye opening image and the original image to obtain an image generation result.

Description

Technical Field

[0001] The present invention relates to the technical field of image generation, and in particular to an image generation method, system and storage medium based on a diffusion model. Background Art

[0002] In photo-taking scenarios, the problem of poor photo effects due to closing eyes or squinting eyes is common, especially in group photos, where it is more difficult to open eyes simultaneously.

[0003] To address this issue, existing technologies typically use convolutional neural network-based eye-opening methods, which directly output a generated image with a fixed eye-opening degree and cannot be personalized based on user needs. This limitation restricts the flexibility and applicability of eye-opening functions in practical applications, making it difficult for users to achieve the desired eye-opening effect using existing technologies in different scenarios and needs.

[0004] Furthermore, traditional methods have low fidelity and similarity to the original eyes in the generated eyes. This means that while the generated eyes can achieve the effect of open eyes, they lose the unique charm and iconic features of the person's eyes, making the generated images lack realism and recognition.

[0005] That is, the images generated by the existing technology are not good and cannot be personalized according to user needs. Summary of the Invention

[0006] The main purpose of the present invention is to provide an image generation method, system and storage medium based on a diffusion model, aiming to solve the technical problem that the existing image generation methods based on diffusion models have poor user experience.

[0007] To achieve the above object, the present invention provides an image generation method based on a diffusion model, which comprises the following steps:

[0008] Obtain the original image and perform preprocessing to obtain a face image, an eye region mask, a face image with the eye region mask removed, and a reference image retaining only the eye region;

[0009] The face image, the face image with the eye region mask removed, and the eye region mask are encoded and input into a Unet model;

[0010] Obtain text prompt words and perform encoding processing to obtain text feature vectors;

[0011] After encoding and feature mapping the reference image, an image feature vector is obtained;

[0012] Inputting the text feature vector and the image feature vector into the Unet model through a cross-attention mechanism to guide image generation;

[0013] Obtain eye opening adjustment parameters and adjust the output of the Unet model using pre-trained LoRA weights;

[0014] Decode the latent variables output by the Unet model to obtain the eye-opening image;

[0015] The open-eye image is spliced and fused with corresponding regions of the original image to obtain an image generation result.

[0016] Optionally, the preprocessing includes at least the following steps:

[0017] Performing face detection on the original image to obtain facial key points and face frames;

[0018] Correcting the angle of the face frame according to the facial key points;

[0019] Crop the corrected original image to obtain a face image;

[0020] Performing image segmentation processing on the face image to obtain an eye region mask;

[0021] The face image with the eye region mask removed and the reference image with only the eye region retained are generated respectively according to the eye region mask.

[0022] Optionally, the facial image, the facial image with the eye area mask removed, and the eye area mask are encoded using a VAE encoder.

[0023] Optionally, the text prompt word is encoded using a CLIP text encoder.

[0024] Optionally, after encoding and feature mapping the reference image, obtaining an image feature vector specifically includes the following steps:

[0025] Encoding the reference image using a CLIPVision image encoder to obtain a human eye feature vector;

[0026] After mapping the human eye feature vector to a low-dimensional space that matches the text feature vector, an image feature vector is obtained.

[0027] Optionally, the value range of the eye-opening degree adjustment parameter is [-1, 1], wherein a negative value indicates an enhanced eye-closing degree, and a positive value indicates an enhanced eye-opening degree.

[0028] Optionally, the stitched images are fused by gradient-based fusion or Poisson fusion.

[0029] Optionally, before fusing the stitched images, the following steps are further included:

[0030] Perform color correction and / or eye optimization on the stitched image.

[0031] Corresponding to the diffusion model-based image generation method, the present invention provides a diffusion model-based image generation system, which includes:

[0032] A preprocessing module is used to obtain the original image and perform preprocessing to obtain a face image, an eye area mask, a face image with the eye area mask removed, and a reference image with only the eye area retained;

[0033] A first encoding module is used to encode the face image, the face image with the eye region mask removed, and the eye region mask and then input them into a Unet model;

[0034] The second encoding module is used to obtain text prompt words and perform encoding processing to obtain text feature vectors;

[0035] A third encoding module is used to encode the reference image and input the processing result into the feature mapping module;

[0036] A feature mapping module is used to perform feature mapping processing on the processing result of the third encoding module to obtain an image feature vector;

[0037] A cross attention module is used to input the text feature vector and the image feature vector into the Unet model to guide image generation;

[0038] The LoRA module is used to obtain eye opening adjustment parameters and adjust the output of the Unet model through pre-trained LoRA weights;

[0039] A Unet model is configured to adjust parameters based on the output of the first encoding module, the output of the cross-attention module, and the degree of eye opening, and output a latent variable;

[0040] A decoding module, decoding the latent variables to obtain an eye-opening image;

[0041] The image generation module is used to splice and fuse the open-eye image with the corresponding area of the original image to obtain an image generation result.

[0042] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a diffusion model-based image generation program is stored. When the diffusion model-based image generation program is executed by a processor, the steps of the diffusion model-based image generation method described above are implemented.

[0043] The beneficial effects of the present invention are:

[0044] (1) Compared with the existing technology, the present invention obtains key information such as face image and eye area mask by acquiring the original image and performing preprocessing. Then, it uses encoding processing and Unet (U-shaped Network) model to perform noise prediction. It combines text features and image features to guide image generation through the cross-attention mechanism, and finally generates an open-eye image that meets user needs. It can realize intelligent open-eye processing of closed-eye images and generate high-quality, personalized image results.

[0045] (2) Compared with the existing technology, the present invention can accurately obtain facial images and eye area masks through preprocessing steps such as face detection, angle correction and cropping, providing accurate basic data for subsequent image generation; it helps to improve the accuracy and naturalness of image generation and ensure that the generated open-eye image is consistent with other parts of the original image;

[0046] (3) Compared with the prior art, the present invention uses a VAE (Encoder of Variational Autoencoder) encoder to encode the face image, the face image with the eye region mask removed, and the eye region mask, thereby converting the image data into a feature representation suitable for processing by the Unet model; this helps improve the model's ability to extract and understand image features, thereby generating an open-eye image that better meets user needs;

[0047] (4) Compared with the prior art, the present invention encodes text prompt words through the text encoder in CLIP (Contrastive Language-Image Pre-training), which can convert the user's text description into a text feature vector that matches the image features. This enables the model to understand the user's intention and make corresponding adjustments based on the text prompt during the image generation process, thereby improving the personalization of the generated image.

[0048] (5) Compared with the prior art, the present invention obtains an image feature vector that matches the text feature vector after encoding and feature mapping the reference image. This enables the model to combine the eye features in the reference image with the text prompt to generate an open-eye image that not only conforms to the user's description but also has the features of the reference image, thereby improving the realism and naturalness of the generated image.

[0049] (6) Compared with the prior art, the present invention sets the value range of the eye-opening adjustment parameter to [-1, 1], allowing users to flexibly control the eye-opening degree of the generated image; negative values indicate enhanced eye closure, while positive values indicate enhanced eye openness, which provides users with personalized adjustment options to meet the needs of different scenarios;

[0050] (7) Compared with the prior art, the present invention fuses the spliced images by gradient-based fusion or Poisson fusion, which can naturally fuse the generated open-eye image with the other parts of the original image, avoiding obvious splicing traces; thus, it helps to improve the visual effect of the final image, making it more natural and realistic;

[0051] (8) Compared with the prior art, the present invention performs color correction and / or eye optimization processing before fusing the spliced images, which can further improve the quality of the generated image; color correction can ensure that the color of the generated open-eye image is consistent with that of the original image, and eye optimization can improve the details of the eye area, making it more realistic and natural. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0053] Figure 1 This is a simplified flow chart of an embodiment of the image generation method based on the diffusion model of the present invention;

[0054] Figure 2 This is a framework diagram of an embodiment of an image generation system based on a diffusion model according to the present invention;

[0055] Figure 3 This is a structural diagram of the LoRA module of an embodiment of the image generation system based on the diffusion model of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0057] like Figure 1 As shown, the image generation method based on the diffusion model of the present invention includes the following steps:

[0058] Obtain the original image and perform preprocessing to obtain a face image, an eye region mask, a face image with the eye region mask removed, and a reference image retaining only the eye region;

[0059] The face image, the face image with the eye area mask removed, and the eye area mask are encoded and input into the Unet model;

[0060] Obtain text prompt words and perform encoding processing to obtain text feature vectors;

[0061] After encoding and feature mapping the reference image, an image feature vector is obtained;

[0062] The text feature vector and the image feature vector are input into the Unet model through the cross attention mechanism to guide image generation;

[0063] Obtain the eye opening adjustment parameters and adjust the output of the Unet model through the pre-trained LoRA weights;

[0064] Decode the latent variables output by the Unet model to obtain the eye-opening image;

[0065] The open-eye image is spliced and fused with the corresponding areas of the original image to obtain the image generation result.

[0066] It can be understood that the corresponding area of the original image is the corresponding eye area of the original image.

[0067] The present invention obtains key information such as face image and eye area mask by acquiring the original image and performing preprocessing. It then uses encoding processing and the Unet model to perform noise prediction, combines text features and image features to guide image generation through a cross-attention mechanism, and ultimately generates an open-eye image that meets user needs. It can realize intelligent open-eye processing of closed-eye images and generate high-quality, personalized image results.

[0068] In this embodiment, the preprocessing includes at least the following steps:

[0069] Perform face detection on the original image to obtain facial key points and face frames;

[0070] Correct the angle of the face frame based on the facial key points;

[0071] Crop the corrected original image to obtain a face image;

[0072] Perform image segmentation processing on the face image to obtain the eye area mask;

[0073] According to the eye region mask, a face image with the eye region mask removed and a reference image with only the eye region retained are generated respectively.

[0074] It should be noted that inputting the entire image not only slows down model processing but also introduces features unrelated to the human eye, hindering eye generation. Therefore, cropping the rectified original image to obtain a face image ensures that eye features are captured while providing contextual semantic information around the eyes, making the generated eyes blend more naturally with their surroundings.

[0075] Through preprocessing steps such as face detection, angle correction and cropping, the present invention can accurately obtain facial images and eye area masks, providing precise basic data for subsequent image generation; it helps to improve the accuracy and naturalness of image generation and ensure that the generated open-eye image is consistent with other parts of the original image.

[0076] In this embodiment, the face image, the face image with the eye region mask removed, and the eye region mask are encoded using a VAE encoder.

[0077] It should be noted that the VAE encoder converts the input face image into a low-dimensional feature vector that captures the key features of the face image, such as the overall shape of the face, the position and shape of the facial features, skin color, etc. This feature vector can be used for subsequent image generation tasks.

[0078] The VAE encoder converts the input face image, with the eye region mask removed, into a low-dimensional feature vector. Since the eye region is masked out, the encoder primarily captures features outside the eye region, such as the nose, mouth, and cheeks. This feature vector can be used in the generation process to ensure that the generated image remains consistent and natural outside the eye region.

[0079] The VAE encoder converts the input eye region mask into a low-dimensional feature vector. This feature vector represents the location and shape of the eye region, which is used to guide the processing of the eye region during the generation process. By encoding the eye region mask, the model can more accurately locate and generate the content of the eye region.

[0080] The present invention uses a VAE encoder to encode facial images, facial images with the eye area mask removed, and eye area masks, thereby converting image data into feature representations suitable for processing by the Unet model. This helps improve the model's ability to extract and understand image features, thereby generating open-eye images that better meet user needs.

[0081] In this embodiment, the text prompt word is encoded by the text encoder in CLIP (Contrastive Language-Image Pretraining). The text prompt word is used to describe the eye area. For example, the text prompt word entered by the user may be "make the eyes bright and lively." The text encoder in CLIP will convert it into a low-dimensional text feature vector that can capture the semantic information contained in the text, such as the description of the brightness of the eyes and the desired eye effect.

[0082] The present invention encodes text prompt words through the text encoder in CLIP, which can convert the user's text description into a text feature vector that matches the image features; this enables the model to understand the user's intention and make corresponding adjustments based on the text prompt during the image generation process, thereby improving the personalization of the generated image.

[0083] In this embodiment, after encoding and feature mapping the reference image, an image feature vector is obtained, which specifically includes the following steps:

[0084] The reference image is encoded using the CLIPVision (Vision Image Encoder of CLIP Model) image encoder to obtain the human eye feature vector;

[0085] After mapping the human eye feature vector to a low-dimensional space that matches the text feature vector, the image feature vector is obtained.

[0086] It should be noted that by encoding the reference image through the CLIPVision image encoder, high-dimensional features of the eyes (human eye feature vectors) can be extracted, which include at least the shape, color, texture and eye expression of the eyes; this can provide a reference for generating open-eye images, so that the generated eyes can be similar to the eye features in the reference image, thereby providing users with a personalized generation effect.

[0087] After encoding and feature mapping the reference image, the present invention obtains an image feature vector that matches the text feature vector; this enables the model to combine the eye features in the reference image with the text prompt to generate an open-eye image that conforms to the user's description and has the characteristics of the reference image, thereby improving the realism and naturalness of the generated image.

[0088] In this embodiment, the value range of the eye-opening degree adjustment parameter is [-1, 1], wherein a negative value indicates an increased eye-closing degree, and a positive value indicates an increased eye-opening degree.

[0089] The present invention sets the value range of the eye-opening degree adjustment parameter to [-1, 1], so that users can flexibly control the eye-opening degree of the generated image; negative values indicate enhanced eye closure, and positive values indicate enhanced eye opening, which provides users with personalized adjustment options to meet the needs of different scenarios.

[0090] In this embodiment, the spliced images are fused by gradient-based fusion or Poisson fusion.

[0091] The present invention fuses the spliced images through gradient-based fusion or Poisson fusion, which can naturally fuse the generated open-eye image with other parts of the original image to avoid obvious splicing marks; it helps to improve the visual effect of the final image, making it more natural and realistic.

[0092] In this embodiment, before fusing the spliced images, the following steps are further included:

[0093] Perform color correction and / or eye optimization on the stitched image.

[0094] The present invention fuses the spliced images through gradient-based fusion or Poisson fusion, which can naturally fuse the generated open-eye image with other parts of the original image to avoid obvious splicing marks; it helps to improve the visual effect of the final image, making it more natural and realistic.

[0095] like Figure 2 As shown, the present invention also provides an image generation system based on a diffusion model, which includes:

[0096] A preprocessing module 100 is used to obtain an original image and perform preprocessing to obtain a face image, an eye region mask, a face image with the eye region mask removed, and a reference image retaining only the eye region;

[0097] A first encoding module 210 is configured to encode the face image, the face image with the eye region mask removed, and the eye region mask, and then input the encoded data into the Unet model 600;

[0098] The second encoding module 220 is used to obtain text prompt words and perform encoding processing to obtain text feature vectors;

[0099] The third encoding module 230 is used to encode the reference image and input the processing result into the feature mapping module 300;

[0100] A feature mapping module 300 is configured to perform feature mapping on the processing result of the third encoding module 230 to obtain an image feature vector;

[0101] Cross-attention module 400, used to input text feature vector and image feature vector into Unet model 600 to guide image generation;

[0102] LoRA module (Low-Rank Adaptation of Large Language Models) 500, used to obtain eye opening adjustment parameters and adjust the output of the Unet model 600 through pre-trained LoRA weights;

[0103] The Unet model 600 is configured to adjust parameters based on the output of the first encoding module 210, the output of the cross-attention module 400, and the degree of eye opening, and output a latent variable;

[0104] The decoding module 700 decodes the latent variables to obtain an eye-opening image;

[0105] The image generation module 800 is used to splice and fuse the open-eye image with the corresponding area of the original image to obtain an image generation result.

[0106] Preferably, the first encoding module 210 specifically adopts a VAE encoder, the second encoding module 220 specifically adopts a CLIP text encoder, and the third encoding module 230 specifically adopts a CLIPVision image encoder.

[0107] In this embodiment, the input of the Unet model 600 mainly consists of three parts. The first part is the face image, the encoded output of the face image with the eye region mask removed, and the eye region mask downsampled to the same size as the encoded face image. The second part is two additional conditional inputs: the text feature vector and the image feature vector. The third part is the encoding T of a time series. The loss function L of the Unet model 600 is: d The calculation formula is as follows:

[0108] L d =MSE(∈ θ (x t ,c,t),∈);

[0109] Among them, ∈ is the noise in the diffusion model noise adding process, x t is the input of the first part, c is the input of the second part, t is the input of the third part, ∈ θ (·) is the noise prediction model, and MSE is the mean square error loss function.

[0110] In this embodiment, the structure of the LoRA module 500 can refer to Figure 3 As shown, it can be expressed as:

[0111] h1=Wh+αBAh;

[0112] Wherein, h represents the input, W represents the pre-trained weight parameter, which is frozen during training, A and B are trainable model parameters used for fine-tuning, A represents the first model parameter, B represents the second model parameter, α represents the eye opening degree adjustment parameter, and h1 represents the output of the pre-trained weight parameter (i.e., W) after fine-tuning.

[0113] Based on LoRA, the present invention designs two inputs, one input is an image with open eyes, which can be called a positive sample, and the other input is an image with closed eyes, which is called a negative sample. The positive sample loss function L during training is rp And the negative sample loss function L rn The calculation formula is as follows:

[0114]

[0115] in, and Represents the positive and negative samples at time t. is the face image corresponding to the positive sample, the encoded output of the face image with the eye area mask removed, and the eye area mask downsampled to the size of the face image after encoding, c + are the text feature vector and image feature vector corresponding to the positive sample, is the face image corresponding to the negative sample, the encoded output of the face image with the eye area mask removed, and the eye area mask downsampled to the size of the face image after encoding, c - are the text feature vector and image feature vector corresponding to the negative sample; this loss function optimizes the two linear directions of eye opening and closing in the generated image. Therefore, by adjusting the value of α in the LoRA formula, the degree of eye opening can be easily adjusted.

[0116] It is difficult to achieve a controllable degree of effect using a single data training in the training phase of the LoRA module 500. In order to train the LoRA module 500 with controllable degree of eye opening, the present invention cleverly uses the differences between data pairs for training. First, clear images of human faces with different degrees of eye opening are collected, and corresponding eye closing images are produced through data processing to combine them into paired data. Secondly, the corresponding prompt words are obtained for all images through the BLIP model. Two special text prompt words are added to the image prompt words of eyes open and eyes closed to distinguish the difference between the two. L is used in training r For supervision, the α value in the LoRA module 500 is 1 for positive samples and -1 for negative samples.

[0117] In this embodiment, the feature mapping module 300 specifically trains a small feature mapping network to keep its input dimension consistent with the text feature vector, which can effectively embed the acquired human eye feature vector into the diffusion model while avoiding a large number of parameters.

[0118] In this embodiment, the cross-attention mechanism guides the interaction of additional conditional features with the original image features by providing additional conditions, thereby generating results related to the guided features. This operation is a non-spatial guidance method that is independent of the position of the eyes in the reference image, making the selection of input data more arbitrary.

[0119] In the cross-attention module 400 designed by the present invention, the text feature vector and the image feature vector are independently cross-attentioned, which can effectively adjust the influence of the text and image conditions and avoid the entanglement of the influences exerted by the two. At the same time, a local cross-attention loss is designed in this embodiment to improve the attention score for the eye area, so that the generation of the eye part is more focused during the model generation process. The local cross-attention loss expression is as follows:

[0120]

[0121] Where q represents the feature extracted from the reference image by the cross attention module 400, i.e., the image feature vector, and k T Represents the conditional feature, that is, the text feature vector, d is the resolution size of q of the attention layer, S is the weight score of the cross attention calculated by the image feature vector and the text feature vector; M represents the eye area mask that is transformed into the same resolution size of the attention layer and flattened into one dimension, l represents the flattened length of the eye area mask, i represents each pixel of the flattened length of the eye area mask, P represents the calculated attention score of the eye area mask; n is the number of cross attention layers, j represents the number of feature layers, E represents the unit vector, L s It is a loss function for the attention score of the eye area. It is averaged after calculation in the cross-attention layer of the Unet model 600 to improve the model's attention to the eye area.

[0122] Preferably, the cross attention module 400 is trained using L=L d +βL s Supervision is performed, β is an adjustable weight factor, and its general value is 0.001. s If the resolution is too small, no calculation is performed (for example, if the resolution is less than 32 times the original image).

[0123] Furthermore, the image generation module 800 is further configured to perform color correction and / or eye optimization processing on the spliced images before fusing the spliced images.

[0124] The embodiment of the present invention further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement Figure 1 The image generation method based on the diffusion model is shown. The computer readable storage medium can be a read-only memory, a magnetic disk or an optical disk.

[0125] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between the various embodiments can be referred to in conjunction with each other. The system and storage medium embodiments are described briefly because they are generally similar to the method embodiments. For relevant parts, refer to the description of the method embodiments.

[0126] Furthermore, in this document, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0127] While the foregoing description shows and describes preferred embodiments of the present invention, it should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments, and can be modified within the scope of the present invention by the teachings herein or by techniques or knowledge in the relevant art. Modifications and variations made by those skilled in the art without departing from the spirit and scope of the present invention are intended to be within the scope of the appended claims.

Claims

1. An image generation method based on a diffusion model, characterized in that: The following steps are involved: Obtain the original image and perform preprocessing to obtain a face image, an eye region mask, a face image with the eye region mask removed, and a reference image retaining only the eye region; The face image, the face image with the eye region mask removed, and the eye region mask are encoded and input into a Unet model; Obtain text prompt words and perform encoding processing to obtain text feature vectors; After encoding and feature mapping the reference image, an image feature vector is obtained; Inputting the text feature vector and the image feature vector into the Unet model through a cross-attention mechanism to guide image generation; Obtain eye opening adjustment parameters and adjust the output of the Unet model using pre-trained LoRA weights; Decode the latent variables output by the Unet model to obtain the eye-opening image; The open-eye image is spliced and fused with corresponding regions of the original image to obtain an image generation result.

2. The image generation method based on the diffusion model according to claim 1, characterized in that: Preprocessing includes at least the following steps: Performing face detection on the original image to obtain facial key points and face frames; Correcting the angle of the face frame according to the facial key points; Crop the corrected original image to obtain a face image; Performing image segmentation processing on the face image to obtain an eye region mask; The face image with the eye region mask removed and the reference image with only the eye region retained are generated respectively according to the eye region mask.

3. The image generation method based on the diffusion model according to claim 1, characterized in that: The face image, the face image with the eye region mask removed, and the eye region mask are encoded using a VAE encoder.

4. The image generation method based on the diffusion model according to claim 1, characterized in that: The text prompt words are encoded using a CLIP text encoder.

5. The image generation method based on the diffusion model according to claim 1, characterized in that: After encoding and feature mapping the reference image, an image feature vector is obtained, which specifically includes the following steps: Encoding the reference image using a CLIPVision image encoder to obtain a human eye feature vector; After mapping the human eye feature vector to a low-dimensional space that matches the text feature vector, an image feature vector is obtained.

6. The image generation method based on the diffusion model according to claim 1, characterized in that: The value range of the eye-opening degree adjustment parameter is [-1, 1], wherein a negative value indicates an increased eye-closing degree, and a positive value indicates an increased eye-opening degree.

7. The image generation method based on the diffusion model according to claim 1, characterized in that: The stitched images are fused using gradient-based fusion or Poisson fusion.

8. The image generation method based on the diffusion model according to claim 1, characterized in that: Before fusing the stitched images, the following steps are also included: Perform color correction and / or eye optimization on the stitched image.

9. An image generation system based on a diffusion model, characterized in that: include: A preprocessing module is used to obtain the original image and perform preprocessing to obtain a face image, an eye area mask, a face image with the eye area mask removed, and a reference image with only the eye area retained; A first encoding module is used to encode the face image, the face image with the eye region mask removed, and the eye region mask and then input them into a Unet model; The second encoding module is used to obtain text prompt words and perform encoding processing to obtain text feature vectors; A third encoding module is used to encode the reference image and input the processing result into the feature mapping module; A feature mapping module is used to perform feature mapping processing on the processing result of the third encoding module to obtain an image feature vector; A cross attention module is used to input the text feature vector and the image feature vector into the Unet model to guide image generation; The LoRA module is used to obtain eye opening adjustment parameters and adjust the output of the Unet model through pre-trained LoRA weights; A Unet model is configured to adjust parameters based on the output of the first encoding module, the output of the cross-attention module, and the degree of eye opening, and output a latent variable; A decoding module, decoding the latent variables to obtain an eye-opening image; The image generation module is used to splice and fuse the open-eye image with the corresponding area of the original image to obtain an image generation result.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a diffusion model-based image generation program, which, when executed by a processor, implements the steps of the diffusion model-based image generation method according to any one of claims 1 to 8.

Images