Image generation method, device and storage medium

Abstract

The present application provides an image generation method, device and storage medium. The image generation method includes: inputting a prompt word to be processed into a pre-trained first stable diffusion model to obtain an apparent image corresponding to the prompt word to be processed, and inputting the prompt word to be processed into an edge map generation model to obtain a first edge image corresponding to the prompt word to be processed; and generating an image associated with the prompt word to be processed based on the apparent image and the first edge image. The image generation method, device and storage medium provided by the present application extract the apparent information and edge information of a small sample category image, and use them together to generate images of the corresponding category, thereby improving the training accuracy and generalization ability of the image generation model based on small sample images, thereby improving the accuracy of small sample image generation.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image generation method, device and storage medium. Background Art

[0002] Deep learning is widely used in the field of computer vision. In image generation technology, a large number of image samples are needed to train the image generation model. However, the sample distribution in the real world has a serious long tail phenomenon. A large number of rare images, such as rare works of art, can only provide a few scarce image samples as training samples when generating images of the corresponding category.

[0003] In practical applications, due to the scarcity of image samples of most categories, the image generation networks trained by existing image generation methods are prone to serious overfitting and exhibit poor generalization ability. The image generation network training accuracy is low, which leads to low image generation accuracy. Summary of the invention

[0004] The embodiments of the present application provide an image generation method, device and storage medium to solve the technical problem of low image generation accuracy in the prior art.

[0005] In a first aspect, an embodiment of the present application provides an image generation method, comprising:

[0006] Inputting the prompt word to be processed into a pre-trained first stable diffusion model to obtain an apparent image corresponding to the prompt word to be processed, and inputting the prompt word to be processed into an edge map generation model to obtain a first edge image corresponding to the prompt word to be processed;

[0007] An image associated with the prompt word to be processed is generated based on the appearance image and the first edge image.

[0008] In some embodiments, generating an image associated with the prompt word to be processed based on the surface image and the first edge image includes:

[0009] Determining a target embedding vector of the prompt word to be processed based on the appearance image and the first edge image;

[0010] An image associated with the prompt word to be processed is generated based on the target embedding vector of the prompt word to be processed and the first stable diffusion model.

[0011] In some embodiments, determining a target embedding vector of the prompt word to be processed based on the appearance image and the first edge image includes:

[0012] Using an encoder to convert and map the prompt word to be processed to obtain an initial embedding vector of the prompt word to be processed;

[0013] The initial embedding vector is adjusted based on the appearance image and the first edge image to obtain a target embedding vector of the prompt word to be processed.

[0014] In some embodiments, the edge map generation model is trained by the following steps:

[0015] Acquire a second edge image corresponding to the sample prompt word based on the sample prompt word and a pre-trained first stable diffusion model;

[0016] A second stable diffusion model is trained based on the second edge image to obtain an edge map generation model.

[0017] In some embodiments, the acquiring the second edge image corresponding to the sample prompt word based on the sample prompt word and the pre-trained first stable diffusion model includes:

[0018] Inputting a sample prompt word into a pre-trained first stable diffusion model to obtain a synthetic image corresponding to the sample prompt word;

[0019] An edge detector is used to extract edges from the composite image to obtain a second edge image.

[0020] In some embodiments, the training of the second stable diffusion model based on the second edge image to obtain an edge map generation model includes:

[0021] Based on the sample prompt word and the denoising network in the second stable diffusion model, a prediction tensor corresponding to the sample prompt word is obtained;

[0022] determining a loss function based on the second edge image and the prediction tensor;

[0023] The denoising network in the second stable diffusion model is trained with the goal of minimizing the loss function to obtain an edge map generation model.

[0024] In some embodiments, determining a loss function based on the second edge image and the prediction tensor comprises:

[0025] adding random noise to the second edge image;

[0026] The loss function is calculated based on the second edge image after random noise is added and the predicted tensor.

[0027] In a second aspect, an embodiment of the present application provides an image generating device, including:

[0028] A first acquisition module is used to input the prompt word to be processed into a pre-trained first stable diffusion model to obtain an apparent image corresponding to the prompt word to be processed, and input the prompt word to be processed into an edge map generation model to obtain a first edge image corresponding to the prompt word to be processed;

[0029] The second acquisition module is used to generate an image associated with the prompt word to be processed based on the appearance image and the first edge image.

[0030] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the image generation method as described in the first aspect above is implemented.

[0031] In a fourth aspect, an embodiment of the present application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image generation method as described in the first aspect above.

[0032] In a fifth aspect, an embodiment of the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the image generation method as described in the first aspect above.

[0033] The image generation method, device and storage medium provided in the embodiments of the present application respectively input the prompt word to be processed into the pre-trained first stable diffusion model and edge map generation model to obtain the apparent image and the first edge image of the prompt word to be processed, that is, obtain the image appearance information and edge structure information of the prompt word to be processed, and generate images of corresponding categories based on the apparent image and the first edge image of the prompt word to be processed, thereby improving the training accuracy and generalization ability of the image generation model based on small sample images, thereby improving the accuracy of small sample image generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 It is a flowchart of the image generation method provided in the embodiment of the present application;

[0036] Figure 2 It is a training framework diagram of an edge graph generation model for an example scenario provided in an embodiment of the present application;

[0037] Figure 3 It is an embedding vector optimization framework diagram of the prompt word to be processed in the example scenario provided in the embodiment of the present application;

[0038] Figure 4 is a structural schematic diagram of an image generating device provided in an embodiment of the present application;

[0039] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In the related art, three to five concept images are input into a pre-trained text-image diffusion model to obtain pseudo-words in the text latent space to describe the images of such concepts. These pseudo-words are then combined into natural language sentences to guide the generation of personalized images for the concept.

[0041] Existing AI-based text-based graph technologies all use deep networks as their main framework. Their powerful abstract understanding and distribution modeling capabilities for images rely on the drive of massive training data. When the supervision data is insufficient to effectively support model training, the existing image generation models are prone to serious overfitting and poor generalization capabilities. In addition, the existing small sample text inversion technology only uses the RGB pixels of the image as the final optimization target. However, the image contains both the apparent texture information of the target and the edge structure information, which are not considered in the existing technology. This makes the knowledge concepts extracted by the existing text-based graph model incomplete and inaccurate, and the generalization ability of the model is weak, which reduces the accuracy of small sample image generation.

[0042] How to better extract more complete knowledge concepts that contain both target appearance information and structural information through pre-training of the text graph model so that the text graph model has a more generalized small sample image generation capability is a key issue that needs to be solved urgently.

[0043] Based on the above technical problems, an embodiment of the present application proposes an image generation method, by respectively inputting the prompt word to be processed into a pre-trained first stable diffusion model and an edge map generation model, obtaining the apparent image and the first edge image of the prompt word to be processed, and performing image generation of corresponding categories based on the apparent image and the first edge image of the prompt word to be processed, extracting a more complete knowledge concept that contains both target appearance information and structural information, improving the training accuracy and generalization ability of the image generation model based on small sample images, and thereby improving the accuracy of small sample image generation.

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] Figure 1 is a flow chart of the image generation method provided in the embodiment of the present application, such as Figure 1 As shown, an embodiment of the present application provides an image generation method, the method comprising:

[0046] Step 101: input the prompt word to be processed into a pre-trained first stable diffusion model to obtain an apparent image corresponding to the prompt word to be processed, and input the prompt word to be processed into an edge map generation model to obtain a first edge image corresponding to the prompt word to be processed.

[0047] Specifically, the prompt words to be processed may refer to text prompt words of an image of unknown category to be acquired, especially text prompt words of a small sample image.

[0048] The first stable diffusion model is a pre-trained stable diffusion model (Stable Diffusion) used for text image, that is, the input of the first stable diffusion model is a text prompt word or a sentence based on the text prompt word, and the output is a corresponding RGB image.

[0049] The appearance image refers to the RGB image corresponding to the target described by the prompt word to be processed, and the image contains a variety of appearance information, such as color, texture and other information.

[0050] The edge map generation model is used to capture and understand the structural characteristics or information of the target described by the prompt word to be processed, and generate an edge map of the target, that is, a first edge image. In the embodiment of the present application, the edge map generation model is trained using the first stable diffusion model as a supervisor.

[0051] Step 102: Generate an image associated with the prompt word to be processed based on the appearance image and the first edge image.

[0052] Specifically, an image generation network with a dual-stream structure is used. After obtaining the apparent image and the first edge image corresponding to the prompt word to be processed, an image of the corresponding category of the prompt word to be processed is generated based on the apparent image and the first edge image to obtain a target image.

[0053] For example, after obtaining the apparent image and the first edge image corresponding to the prompt word to be processed, based on the textual inversion technology, the apparent image and the first edge image are used to reversely optimize the embedding vector of the prompt word to be processed to obtain the optimized embedding vector of the prompt word to be processed. Thus, a sentence is composed according to the optimized embedding vector of the prompt word to be processed and any context information, and is input into the pre-trained first stable diffusion model to generate an image corresponding to the sentence.

[0054] The image generation method provided in the embodiment of the present application can quickly learn the knowledge or concepts corresponding to the prompt words to be processed based on a small number of samples by extracting the appearance information and structural information corresponding to the prompt words to be processed, and obtain different information from multiple dimensions to train the model, thereby increasing the accuracy of model training, making it closer to the human cognitive mechanism and discrimination pattern of targets in the real world, enhancing the generalization ability of the model, and improving the accuracy of image generation.

[0055] In some embodiments, the edge map generation model is trained by the following steps:

[0056] Acquire a second edge image corresponding to the sample prompt word based on the sample prompt word and a pre-trained first stable diffusion model;

[0057] A second stable diffusion model is trained based on the second edge image to obtain an edge map generation model.

[0058] Specifically, a pre-trained first stable diffusion model is used as supervision, and a second stable diffusion model is trained to generate edge images, wherein the first stable diffusion model and the second stable diffusion model use a shared sample prompt word as input.

[0059] Firstly, based on the sample prompt word and the pre-trained first stable diffusion model, a second edge image corresponding to the sample prompt word is obtained, and then the second edge image is used as the regression target for training the second stable diffusion model, and the second stable diffusion model is trained to obtain an edge map generation model.

[0060] For example, the sample prompt word is "a picture of an airplane". The sample prompt word is input into the pre-trained first stable diffusion model, and the first stable diffusion model generates an airplane image corresponding to the sample prompt word, and then extracts the edge of the airplane image to obtain the edge image of the airplane, that is, the second edge image. Then, the edge image of the airplane is used as the regression target to train the second stable diffusion model, and obtain the edge map generation model after training.

[0061] The image generation method provided in the embodiment of the present application does not contain apparent information such as color or texture in the edge map. Therefore, the obtained edge map generation model can be more focused on capturing and understanding the structural characteristics required to generate the corresponding image compared to the traditional image generation model, which is beneficial to the subsequent extraction of structural information of the small sample target image and improves the accuracy of the edge map generation model in extracting structural information or generating edge images.

[0062] In some embodiments, the acquiring the second edge image corresponding to the sample prompt word based on the sample prompt word and the pre-trained first stable diffusion model includes:

[0063] Inputting a sample prompt word into a pre-trained first stable diffusion model to obtain a synthetic image corresponding to the sample prompt word;

[0064] An edge detector is used to extract edges from the composite image to obtain a second edge image.

[0065] Specifically, the sample prompt word is first input into the pre-trained first stable diffusion model to perform a text map operation to obtain a synthetic image corresponding to the sample prompt word. Then, an edge detector is used to extract the structural information or edge of the synthetic image corresponding to the sample prompt word to obtain a second edge image.

[0066] In an embodiment of the present application, the edge detector may be DeepEdge, N4-Fields or DeepContour, etc.

[0067] For example, Figure 2 is a training framework diagram of the edge graph generation model of the example scenario provided in the embodiment of the present application, such as Figure 2 As shown in FIG. 1 , the sample prompt word y is input into the pre-trained first stable diffusion model M for image synthesis, and the synthetic image I corresponding to the sample prompt word y is output. Then the edge detector E is used to extract the edge map of I to obtain the second edge image I containing structural information. edge .

[0068] The image generation method provided in the embodiment of the present application uses a pre-trained first stable diffusion model and an edge detector to obtain an edge image corresponding to a sample prompt word, which is used for supervised training of a second stable diffusion model. The important role of structural information in image generation is taken into account, so that a stable diffusion model with high accuracy can be trained as an edge map generation model.

[0069] In some embodiments, the training of the second stable diffusion model based on the second edge image to obtain an edge map generation model includes:

[0070] Based on the sample prompt word and the denoising network in the second stable diffusion model, a prediction tensor corresponding to the sample prompt word is obtained;

[0071] determining a loss function based on the second edge image and the prediction tensor;

[0072] The denoising network in the second stable diffusion model is trained with the goal of minimizing the loss function to obtain an edge map generation model.

[0073] Specifically, based on the sample cue words and the denoising network included in the second stable diffusion model, a prediction tensor corresponding to the sample cue words in the latent space (hidden space) is obtained, a loss function is determined based on the second edge image and the prediction tensor, and the denoising network in the second stable diffusion model is trained with the goal of minimizing the loss function to obtain an edge map generation model.

[0074] In an embodiment of the present application, the sample prompt words input into the second stable diffusion model are first encoded by an encoder in the second stable diffusion model, wherein the encoder includes a contrastive learning language image pre-training model CLIP (Contrastive Language-Image Pre-training) marker and a text converter, etc. The encoded sample prompt words are predicted by the noise prediction period in the second stable diffusion model, and then denoised by the denoising network in the second stable diffusion model, and finally a prediction tensor of the latent space is obtained.

[0075] For example, Figure 2 As shown, the sample prompt word y is input into the second stable diffusion model M edge In, through M edge The encoder c in θ (c θ The latent space noise is obtained by the noise predictor, which then passes through the denoising network ∈ θ Denoising is performed by subtracting the predicted latent space noise from the pre-generated random tensor in the latent space, and the difference is the predicted tensor. The predicted tensor can be passed through the decoder D to generate the edge image I e ′ dge The first stable diffusion model M uses the second edge image I corresponding to the sample prompt word y edge As the second stable diffusion model M edge The regression target is based on the prediction tensor to calculate the corresponding loss function and train M edge The denoising network ∈ θ , the trained M edge Generates models as edge graphs.

[0076] The image generation method provided in the embodiment of the present application optimizes the network parameters of the second stable diffusion model, such as optimizing the denoising network, to train and obtain an effective stable diffusion model for generating edge images containing structural information, thereby reducing the error in edge image generation and improving the effectiveness and accuracy of image generation based on edge images.

[0077] In some embodiments, determining a loss function based on the second edge image and the prediction tensor comprises:

[0078] adding random noise to the second edge image;

[0079] The loss function is calculated based on the second edge image after random noise is added and the predicted tensor.

[0080] Specifically, after obtaining the prediction tensor in the latent space corresponding to the second edge image and the sample prompt word, different random noises are added to the second edge image to obtain multiple expanded sample edge images, and the loss function is calculated based on the second edge image after adding random noise and the prediction tensor.

[0081] For example, Figure 2 As shown, when obtaining the second edge image I edge Then, the first stable diffusion model M uses the second edge image I edge As the second stable diffusion model M edge The regression target of I edge Add random noise, based on I after adding random noise edge The difference between the predicted and predicted tensors determines the loss function.

[0082] For example, for I edge Add random noise and train M edge The denoising network ∈ θ , the loss function L edge The calculation formula is as follows:

[0083]

[0084] Among them, L edge represents the loss function; E represents the edge extraction operation; z represents noise; y represents the sample prompt word; ∈ represents the image after random noise is added to the second edge image; ∈ θ represents the denoising network in the second stable diffusion model; t represents the time step; z t represents the noise at time t; c θ Represents the encoder in the second stable diffusion model.

[0085] The image generation method provided in the embodiment of the present application does not need to use real samples to train the second stable diffusion model in the process of training the second stable diffusion model to obtain the edge map generation model. Instead, a plurality of different training samples are obtained based on the second edge image for model training. Rich training samples can be obtained so that the edge map generation model can learn the target structure generation capability with strong generalization, so that the structural information of the generated target can be better maintained in the process of fine-tuning the Vincent map with small samples.

[0086] In some embodiments, generating an image associated with the prompt word to be processed based on the surface image and the first edge image includes:

[0087] Determining a target embedding vector of the prompt word to be processed based on the appearance image and the first edge image;

[0088] An image associated with the prompt word to be processed is generated based on the target embedding vector of the prompt word to be processed and the first stable diffusion model.

[0089] Specifically, after obtaining an edge map generation model with structural description and abstract generation capabilities, the apparent image corresponding to the prompt word to be processed generated by the first stable diffusion model and the first edge image corresponding to the prompt word to be processed generated by the edge map generation model are combined, and the text inversion method is used to reversely optimize the text embedding vector of the prompt word to be processed to obtain the target embedding vector of the prompt word to be processed. Then, based on the target embedding vector and the first stable diffusion model, an image associated with the prompt word to be processed is generated to obtain a target image.

[0090] For example, the target embedding vector v of the prompt word to be processed is determined based on the appearance image and the first edge image. * , and based on the target embedding vector of the prompt word to be processed and the first stable diffusion model, v * Then the user can use the optimized v * Combine any context information to form a complete sentence, such as "Please help me generate a v * The image of "is input into the first stable diffusion model to generate v * The corresponding small sample image.

[0091] The image generation method provided in the embodiment of the present application uses only a small number of small sample training images for multiple rounds of training, which makes the overall training time short and occupies less computing resources, and well extracts and condenses the abstract knowledge concepts such as the appearance and structure of the small sample category into the text embedding vector, thereby performing large-scale synthesis. The learning process is closer to the process of summarizing and inferring new knowledge for people to learn. The training obtains an image generation model and an edge map generation model with strong abstract expression ability and generalization ability, which improves the accuracy of image generation.

[0092] In some embodiments, determining a target embedding vector of the prompt word to be processed based on the appearance image and the first edge image includes:

[0093] Using an encoder to convert and map the prompt word to be processed to obtain an initial embedding vector of the prompt word to be processed;

[0094] The initial embedding vector is adjusted based on the appearance image and the first edge image to obtain a target embedding vector of the prompt word to be processed.

[0095] Specifically, after the prompt word to be processed is converted and mapped using an encoder, an initial embedding vector of the prompt word to be processed is obtained in the latent space. Then, the initial embedding vector is adjusted or optimized based on the obtained apparent image and the first edge image corresponding to the prompt word to be processed to obtain a target embedding vector of the prompt word to be processed.

[0096] For example, Figure 3 is an embedding vector optimization framework diagram of the prompt word to be processed in the example scenario provided in the embodiment of the present application, such as Figure 3 As shown, the prompt word to be processed is a text that the user can specify, using a pseudo word S * Instead. * Like other prompt words, after being converted and mapped by the encoder in the first stable diffusion model M, the corresponding pseudo-word embedding v is obtained in the latent space * (i.e., the initial embedding vector), v * The apparent image I corresponding to the prompt word to be processed is obtained through the first stable diffusion model M * , and then the edge detector is used to obtain the edge detection result I corresponding to the prompt word to be processed edge* ; v* obtains the first edge image I′ through the edge map generation model edge* According to the generation results of the dual stream (I edge* and I′ edge* ) for pseudo-word embedding v * Optimize and get the optimized v * , that is, the target embedding vector of the prompt word to be processed.

[0097] For another example, the prompt word S to be processed * After the encoder conversion and mapping in the first stable diffusion model M, the corresponding initial embedding vector is obtained in the latent space. The initial embedding vector is used to obtain the apparent image I corresponding to the prompt word to be processed through M * , and then the edge detector is used to obtain the edge detection result I corresponding to the prompt word to be processed edge* , the initial embedding vector is used to generate the first edge image I′ through the edge map generation model edge- , according to I edge* and I′ edge* The initial embedding vector is optimized, and the optimization formula is as follows:

[0098]

[0099] Among them, v * Represents the target embedding vector of the prompt word to be processed; argmin v To seek The embedding vector of the prompt word to be processed when the minimum value is reached; E represents the edge extraction operation; z represents noise; S * represents the prompt word to be processed; ∈ represents the image after random noise is added to the second edge image, and the second edge image is obtained by edge extraction of the apparent image corresponding to the prompt word to be processed; ∈ θ represents the denoising network in the second stable diffusion model; t represents the time step; z t represents the noise at time t; c θ Represents the encoder in the second stable diffusion model.

[0100] The image generation method provided in the embodiment of the present application uses the generation results of the dual streams, namely, the apparent image corresponding to the prompt word to be processed and the first edge image pair, to optimize the text embedding of the prompt word to be processed, thereby injecting the new concept of the unknown category of the prompt word to be processed into the generation process, and jointly extracts the text inversion optimization to obtain the text space embedding vector of the target corresponding to the prompt word to be processed.

[0101] The structural information of the generated target can be better maintained in the process of fine-tuning the text graph with small samples.

[0102] Figure 4 is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application, such as Figure 4 As shown, an embodiment of the present application provides an image generating device, including a first acquisition module 401 and a second acquisition module 402.

[0103] The first acquisition module 401 is used to input the prompt word to be processed into a pre-trained first stable diffusion model to obtain an apparent image corresponding to the prompt word to be processed, and input the prompt word to be processed into an edge map generation model to obtain a first edge image corresponding to the prompt word to be processed;

[0104] The second acquisition module 402 is used to generate an image associated with the prompt word to be processed based on the appearance image and the first edge image.

[0105] In some embodiments, the second acquisition module includes:

[0106] A determination unit, configured to determine a target embedding vector of the prompt word to be processed based on the appearance image and the first edge image;

[0107] A generating unit is used to generate an image associated with the prompt word to be processed based on the target embedding vector of the prompt word to be processed and the first stable diffusion model.

[0108] In some embodiments, the determining unit includes:

[0109] A first acquisition subunit is used to convert and map the prompt word to be processed by using an encoder to obtain an initial embedding vector of the prompt word to be processed;

[0110] The second acquisition subunit is used to adjust the initial embedding vector based on the appearance image and the first edge image to obtain a target embedding vector of the prompt word to be processed.

[0111] In some embodiments, the edge map generation model is trained by the following steps:

[0112] Acquire a second edge image corresponding to the sample prompt word based on the sample prompt word and a pre-trained first stable diffusion model;

[0113] A second stable diffusion model is trained based on the second edge image to obtain an edge map generation model.

[0114] In some embodiments, the acquiring the second edge image corresponding to the sample prompt word based on the sample prompt word and the pre-trained first stable diffusion model includes:

[0115] Inputting a sample prompt word into a pre-trained first stable diffusion model to obtain a synthetic image corresponding to the sample prompt word;

[0116] An edge detector is used to extract edges from the composite image to obtain a second edge map.

[0117] In some embodiments, the training of the second stable diffusion model based on the second edge image to obtain an edge map generation model includes:

[0118] Based on the sample prompt word and the denoising network in the second stable diffusion model, a prediction tensor corresponding to the sample prompt word is obtained;

[0119] determining a loss function based on the second edge image and the prediction tensor;

[0120] The denoising network in the second stable diffusion model is trained with the goal of minimizing the loss function to obtain an edge map generation model.

[0121] In some embodiments, determining a loss function based on the second edge image and the prediction tensor comprises:

[0122] adding random noise to the second edge image;

[0123] The loss function is calculated based on the second edge image after random noise is added and the predicted tensor.

[0124] Specifically, the above-mentioned image generating device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned image generating method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0125] It should be noted that the division of units / modules in the above embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, the functional units in the various embodiments of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated units may be implemented in the form of hardware or in the form of software functional units.

[0126] Figure 5 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 5 As shown, the electronic device may include: a processor 501, a communications interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communications interface 502 and the memory 503 communicate with each other via the communication bus 504. The processor 501 may call the logic instructions in the memory 503 to execute the image generation method, which includes:

[0127] Inputting the prompt word to be processed into a pre-trained first stable diffusion model to obtain an apparent image corresponding to the prompt word to be processed, and inputting the prompt word to be processed into an edge map generation model to obtain a first edge image corresponding to the prompt word to be processed;

[0128] An image associated with the prompt word to be processed is generated based on the appearance image and the first edge image.

[0129] Specifically, the processor 501 can be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a complex programmable logic device (CPLD), and the processor can also adopt a multi-core architecture.

[0130] The logic instructions in the memory 503 can be implemented in the form of software functional units and can be stored in a processor-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0131] In some embodiments, a computer program product is further provided, the computer program product comprising a computer program, the computer program may be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the image generation method provided by each of the above method embodiments, the method comprising:

[0132] Inputting the prompt word to be processed into a pre-trained first stable diffusion model to obtain an apparent image corresponding to the prompt word to be processed, and inputting the prompt word to be processed into an edge map generation model to obtain a first edge image corresponding to the prompt word to be processed;

[0133] An image associated with the prompt word to be processed is generated based on the appearance image and the first edge image.

[0134] Specifically, the above-mentioned computer program product provided in the embodiment of the present application can implement all the method steps implemented by the above-mentioned method embodiments, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0135] In some embodiments, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute the image generation method provided by the above-mentioned method embodiments.

[0136] Specifically, the above-mentioned computer-readable storage medium provided in the embodiment of the present application can implement all the method steps implemented by the above-mentioned method embodiments, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0137] It should be noted that the computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.

[0138] It should also be noted that the terms "first", "second", etc. in the embodiments of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0139] In the embodiments of the present application, the term "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0140] In the embodiments of the present application, the term "plurality" refers to two or more than two, and other quantifiers are similar.

[0141] In this application, "determine B based on A" means that the factor A should be considered when determining B. It is not limited to "B can be determined based on A alone", but should also include: "determine B based on A and C", "determine B based on A, C and E", "determine C based on A, and further determine B based on C", etc. In addition, it can also include taking A as a condition for determining B, for example, "when A meets the first condition, use the first method to determine B"; for example, "when A meets the second condition, determine B", etc.; for example, "when A meets the third condition, determine B based on the first parameter", etc. Of course, it can also be a condition that takes A as a factor for determining B, for example, "when A meets the first condition, use the first method to determine C, and further determine B based on C", etc.

[0142] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0143] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer executable instructions. These computer executable instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0144] These processor executable instructions may also be stored in a processor readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the processor readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0145] These processor-executable instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0146] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An image generation method, characterized in that: include: Inputting the prompt word to be processed into a pre-trained first stable diffusion model to obtain an apparent image corresponding to the prompt word to be processed, and inputting the prompt word to be processed into an edge map generation model to obtain a first edge image corresponding to the prompt word to be processed; the edge map generation model is a stable diffusion model trained with the first stable diffusion model as a supervisor; generating an image associated with the prompt word to be processed based on the surface image and the first edge image; The step of generating an image associated with the prompt word to be processed based on the apparent image and the first edge image comprises: Determining a target embedding vector of the prompt word to be processed based on the appearance image and the first edge image; An image associated with the prompt word to be processed is generated based on the target embedding vector of the prompt word to be processed and the first stable diffusion model.

2. The image generation method according to claim 1, characterized in that: The step of determining a target embedding vector of the prompt word to be processed based on the appearance image and the first edge image comprises: Using an encoder to convert and map the prompt word to be processed to obtain an initial embedding vector of the prompt word to be processed; The initial embedding vector is adjusted based on the appearance image and the first edge image to obtain a target embedding vector of the prompt word to be processed.

3. The image generation method according to claim 1, characterized in that: The edge map generation model is trained by the following steps: Acquire a second edge image corresponding to the sample prompt word based on the sample prompt word and a pre-trained first stable diffusion model; A second stable diffusion model is trained based on the second edge image to obtain an edge map generation model.

4. The image generation method according to claim 3, characterized in that: The method of obtaining a second edge image corresponding to the sample prompt word based on the sample prompt word and the pre-trained first stable diffusion model includes: Inputting a sample prompt word into a pre-trained first stable diffusion model to obtain a synthetic image corresponding to the sample prompt word; An edge detector is used to extract edges from the composite image to obtain a second edge image.

5. The image generation method according to claim 3, characterized in that: The step of training the second stable diffusion model based on the second edge image to obtain an edge map generation model includes: Based on the sample prompt word and the denoising network in the second stable diffusion model, a prediction tensor corresponding to the sample prompt word is obtained; determining a loss function based on the second edge map and the prediction tensor; The denoising network in the second stable diffusion model is trained with the goal of minimizing the loss function to obtain an edge map generation model.

6. The image generation method according to claim 5, characterized in that: The determining of the loss function based on the second edge image and the prediction tensor comprises: adding random noise to the second edge image; The loss function is calculated based on the second edge image after random noise is added and the predicted tensor.

7. An image generating device, characterized in that: include: A first acquisition module is used to input the prompt word to be processed into a pre-trained first stable diffusion model to obtain an apparent image corresponding to the prompt word to be processed, and input the prompt word to be processed into an edge map generation model to obtain a first edge image corresponding to the prompt word to be processed; the edge map generation model is a stable diffusion model trained with the first stable diffusion model as a supervisor; A second acquisition module, configured to generate an image associated with the prompt word to be processed based on the surface image and the first edge image; The step of generating an image associated with the prompt word to be processed based on the apparent image and the first edge image comprises: Determining a target embedding vector of the prompt word to be processed based on the appearance image and the first edge image; An image associated with the prompt word to be processed is generated based on the target embedding vector of the prompt word to be processed and the first stable diffusion model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the image generating method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image generating method according to any one of claims 1 to 6 is implemented.

Images