Image generation method, model training method, equipment, storage medium and product

Abstract

One or more embodiments of the invention provide an image generation method, a model training method, equipment, a storage medium and a product. The image generation method comprises the steps that an image generation instruction is received, the image generation instruction carries a reference image and a prompt text, and the prompt text is used for indicating generation of a target image; extracting depth-of-field information of a foreground target in the reference image, acquiring depth control information based on the depth-of-field information of the foreground target by using a trained first control network, extracting edge information of a background in the reference image, and acquiring edge control information based on the edge information of the background by using a trained second control network; and guiding the trained image generation model to generate a target image based on the reference image by using the depth control information, the edge control information and the prompt text. Through the edge control information and the depth control information, a target image with higher quality can be generated.

Description

Technical Field

[0001] One or more embodiments of the present application relate to the field of image processing technology, and in particular, to an image generation method, a model training method, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] AIGC (Artificial Intelligence Generated Content) refers to a technology based on artificial intelligence technical methods such as generative adversarial networks and large pre-trained models, which generates relevant content with appropriate generalization capabilities through learning and identifying existing data.

[0003] For example, in an AIGC-based image-to-image scheme, artificial intelligence technology can be used to generate an image with content that is highly consistent with the original image based on the original image. However, the AIGC-based image-to-image scheme in the related art has the problem of creating something out of nothing, such as generating a person in an unmanned background area, which is inconsistent with the original image content, resulting in poor generation effect. Summary of the invention

[0004] In view of this, one or more embodiments of the present application provide an image generation method, a model training method, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] To achieve the above objectives, one or more embodiments of the present application provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of the present application, a method for generating an image is provided, comprising:

[0007] receiving an image generation instruction, wherein the image generation instruction carries a reference image and a prompt text, wherein the prompt text is used to instruct generation of a target image;

[0008] Extracting depth information of a foreground object in the reference image, and using a trained first control network to obtain depth control information based on the depth information of the foreground object, and extracting edge information of a background in the reference image, and using a trained second control network to obtain edge control information based on the edge information of the background;

[0009] The depth control information, the edge control information and the prompt text are used to guide the trained image generation model to generate the target image based on the reference image.

[0010] In one implementation, the method further includes:

[0011] Segmenting the reference image into a reference foreground image and a reference background image;

[0012] The step of extracting depth information of a foreground object in the reference image comprises:

[0013] Performing depth extraction on the reference foreground image to obtain depth information of the foreground object;

[0014] The extracting edge information of the background in the reference image comprises:

[0015] Perform edge extraction on the reference background image to obtain edge information of the background.

[0016] In one implementation, the image generation model includes a first image generation model and a second image generation model; the first control network is connected to the first image generation model, and the second control network is connected to the second image generation model;

[0017] The using the depth control information, the edge control information, and the prompt text to guide the trained image generation model to generate the target image based on the reference image includes:

[0018] Using the depth control information and the prompt text, guiding the first image generation model to generate a target foreground image based on the reference foreground image; and

[0019] Using the edge control information and the prompt text, guiding the second image generation model to generate a target background image based on the reference background image;

[0020] The target foreground image and the target background image are fused to obtain the target image.

[0021] In one implementation, the first image generation model includes a first text encoder and a first image generation network;

[0022] The step of using the depth control information and the prompt text to guide the first image generation model to generate a target foreground image based on the reference foreground image includes:

[0023] Performing noise processing on the reference foreground image using a preset noise parameter to obtain a reference foreground image after noise processing;

[0024] The depth control information, the prompt text and the reference foreground image after noise processing are input into the first image generation model, so that the first text encoder extracts text features from the prompt text, and the first image generation network processes the depth control information, the text features and the reference foreground image after noise processing to generate the target foreground image.

[0025] In one implementation, the second image generation model includes a second text encoder and a second image generation network;

[0026] The step of using the edge control information and the prompt text to guide the second image generation model to generate a target background image based on the reference background image includes:

[0027] Performing noise processing on the reference background image using a preset noise parameter to obtain a reference background image after noise processing;

[0028] The edge control information, the prompt text and the reference background image after noise processing are input into the second image generation model, so that the second text encoder extracts text features from the prompt text, and the second image generation network processes the edge control information, the text features and the reference background image after noise processing to generate the target background image.

[0029] In one implementation, the image generation model includes a third image generation model; the third image generation model includes a third text encoder and a third image generation network;

[0030] The using the depth control information, the edge control information, and the prompt text to guide the trained image generation model to generate the target image based on the reference image includes:

[0031] Performing noise processing on the reference image using a preset noise parameter to obtain a reference image after noise processing;

[0032] The depth control information, the edge control information, the prompt text and the reference image after noise processing are input into the third image generation model, so that the third text encoder can extract text features from the prompt text, and the third image generation network can process the depth control information, the edge control information, the text features and the reference image after noise processing to generate the target image.

[0033] In one implementation, the first control network and the second control network each include a coding layer, an intermediate layer, a first zero-convolution layer, and a second zero-convolution layer connected in sequence;

[0034] The depth control information includes depth control information output by a first zero convolution layer in the first control network and depth control information output by a second zero convolution layer in the first control network;

[0035] The edge control information includes edge control information output by a first zero convolution layer in the second control network and edge control information output by a second zero convolution layer in the second control network.

[0036] In one implementation, the image generation model includes a first image generation model and a second image generation model;

[0037] The first image generation model includes a first text encoder and a first image generation network, and the second image generation model includes a second text encoder and a second image generation network;

[0038] The first image generation network and the second image generation network each include a coding layer, an intermediate layer, and a decoding layer connected in sequence;

[0039] The first text encoder is connected to the encoding layer, the intermediate layer and the decoding layer in the first image generation network; the first zero convolution layer in the first control network is connected to the intermediate layer in the first image generation network, and the second zero convolution layer in the first control network is connected to the decoding layer in the first image generation network in a one-to-one correspondence;

[0040] The second text encoder is connected to the encoding layer, the intermediate layer and the decoding layer in the second image generation network; and the first zero convolution layer in the second control network is connected to the intermediate layer in the second image generation network, and the second zero convolution layer in the second control network is connected to the decoding layer in the second image generation network in a one-to-one correspondence.

[0041] In one implementation, the image generation model includes a third image generation model; the third image generation model includes a third text encoder and a third image generation network, and the third image generation network includes an encoding layer, an intermediate layer, and a decoding layer connected in sequence;

[0042] Among them, the third text encoder is connected to the encoding layer, the intermediate layer and the decoding layer in the third image generation network; and the first zero convolution layer in the first control network and the first zero convolution layer in the second control network are both connected to the intermediate layer in the third image generation network; and the second zero convolution layer in the first control network and the second zero convolution layer in the second control network are both connected one-to-one with the decoding layer in the third image generation network.

[0043] In one implementation, the prompt text includes description information of the target image;

[0044] The description information at least includes target style information corresponding to the target image, and the prompt text is determined from a mapping relationship between preset different style information and prompt text templates based on the target style information.

[0045] In one implementation, the image generation model includes a diffusion model.

[0046] According to a second aspect of an embodiment of the present application, a model training method is provided, including:

[0047] Using a plurality of first training samples to perform supervised training on the image generation model to be trained, to obtain a trained image generation model; wherein the first training samples include a first prompt text, a first noise image generated based on a preset noise parameter, and a first image; the first image is used as a label in the supervised training process;

[0048] On the basis of locking the model parameters of the trained image generation model, a first control network to be trained and a second control network to be trained connected to the trained image generation model are supervisedly trained using a plurality of second training samples to obtain a trained first control network and a trained second control network; wherein the second training samples include a second prompt text, a second noise image generated based on preset noise parameters, a second image, depth of field information of the second image, and edge information; the second image is used as a label in the supervised training process;

[0049] Wherein, the trained first control network, the trained second control network and the trained image generation model are applied to the image generation method described in any one of the first aspects.

[0050] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including:

[0051] processor;

[0052] a memory for storing processor-executable instructions;

[0053] When the processor executes the executable instructions, it is used to implement the method described in the first aspect or the second aspect.

[0054] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any of the above methods are implemented.

[0055] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising a computer program, which implements the steps of any of the above methods when executed by a processor.

[0056] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0057] In an embodiment of the present application, edge information of the background in the reference image is used to emphasize the background, and the edge control information obtained by the second control network is used as a control condition of the image generation model to accurately control the background part of the target image, thereby effectively suppressing the problem of the image generation model generating a portrait in an unmanned background; and depth of field information of the foreground target in the reference image is used to restore the foreground target, and the depth control information obtained by the first control network is used as another control condition of the image generation model to accurately control the foreground part of the target image, thereby making the foreground target in the target image more three-dimensional; a higher quality target image can be generated, the foreground target is more distinct, and the coordination between the background and the foreground is better.

[0058] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a schematic diagram of generating a person in an unmanned background area in the related art.

[0060] Figure 2 It is a structural schematic diagram of an image generation system provided by an exemplary embodiment.

[0061] Figure 3A It is a schematic diagram of an application scenario provided by an exemplary embodiment.

[0062] Figure 3B It is a schematic diagram of another application scenario provided by an exemplary embodiment.

[0063] Figure 4 It is a flowchart of a model training method provided by an exemplary embodiment.

[0064] Figure 5 is a schematic diagram of a training image generation model provided by an exemplary embodiment.

[0065] Fig. 6A It is a schematic diagram of training a first control network provided by an exemplary embodiment.

[0066] Figure 6B It is a schematic diagram of training a first control network provided by an exemplary embodiment.

[0067] Figure 7 It is a schematic diagram of another method of training a first control network and a second control network provided by an exemplary embodiment.

[0068] Fig. 8A It is a flowchart of an image generating method provided by an exemplary embodiment.

[0069] Figure 8B is a schematic diagram of a reference image and a target image provided by an exemplary embodiment.

[0070] Fig. 9 It is a schematic diagram of obtaining a target image provided by an exemplary embodiment.

[0071] Fig.10 It is a structural schematic diagram of a first control network and a first image generation model provided by an exemplary embodiment.

[0072] Fig.11 It is another schematic diagram of obtaining a target image provided by an exemplary embodiment.

[0073] Fig.12 It is a schematic structural diagram of a first control network, a second control network and a third image generation model provided by an exemplary embodiment.

[0074] Fig.13 It is a schematic structural diagram of an electronic device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0075] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of the present application. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of the present application as detailed in the appended claims.

[0076] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this application. In some other embodiments, the steps included in the method may be more or less than those described in this application. In addition, a single step described in this application may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this application may also be combined into a single step for description in other embodiments.

[0077] In an AIGC-based image-to-image scheme, artificial intelligence technology can be used to generate an image with content that is highly consistent with the original image based on the original image. However, the AIGC-based image-to-image scheme in the related art has the problem of creating something out of nothing, for example, please refer to Figure 1 , generating characters in the unmanned background area does not match the original image content, resulting in poor generation effect. It is understandable that based on the need for personal privacy protection, Figure 1 Some of the photos of real people are blocked.

[0078] Based on this, an embodiment of the present application provides an image generation method, which can extract the depth information of the foreground target and the edge information of the background in the reference image in response to the image generation instruction, and then use the trained first control network to obtain the depth control information about the foreground target, and use the trained second control network to obtain the edge control information about the background. Finally, the depth control information, edge control information and prompt text can be used to guide the trained image generation model to generate the target image based on the reference image. This embodiment uses the depth information of the foreground target and the edge information of the background to achieve precise control of the generation of the foreground and background in the target image, which can effectively avoid the situation of creating something out of nothing, and the generated image content is more consistent with the reference image.

[0079] For example, see Figure 2 , provides an image generation system, which includes a server 100 and at least one client 200.

[0080] The server 100 may be a program installed in a background device to provide image generation services to users. Figure 2 As shown, the background device can be a server, which can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0081] The client 200 may be a program installed in a user terminal to provide image generation services to the user, and the client 200 includes but is not limited to an application APP, a web page, a small program, a plug-in or a component, etc. Figure 2 As shown, the user terminal includes but is not limited to smart phones, personal digital assistants, tablet computers, personal computers, laptops, virtual reality terminal devices, augmented reality terminal devices, etc.

[0082] An image generation method provided in an embodiment of the present application can be applied to a server or client in an image generation system. For example, when the computing resources of the user terminal where the client is located are sufficient, relevant models (such as the first control network, the second control network, and the image generation model mentioned above) can be deployed in the client to meet the user's image generation needs. For another example, relevant models (such as the first control network, the second control network, and the image generation model mentioned above) can be deployed in the server, so that the server can uniformly provide image generation services for each client.

[0083] In an exemplary application scenario, see Figure 3A The client 200 displays a picture upload interface, which includes an image upload control. The user can trigger the image upload control according to actual needs to upload the reference image. If the user confirms that it is correct, the client 200 can send the reference image to the server 100, so that after receiving the reference image, the server 100 obtains an image generation instruction based on the reference image, and executes the image generation method provided in the embodiment of the present application in response to the image generation instruction to generate a target image, and returns the target image to the client 200, so that the target image can be displayed in the display interface of the client 200. The content of the target image is similar to that of the reference image.

[0084] Further, such as Figure 3B As shown, the client 200 displays a picture upload interface, which includes not only an image upload control but also a style selection control. The user can trigger the image upload control to upload a reference image and trigger the style selection control to select the target style of the target image to be generated according to actual needs. If the user confirms that the target style information of the reference image and the target image to be generated is correct, the client 200 can send the target style information of the reference image and the target image to be generated to the server 100, so that the server 100 can generate an image generation instruction based on the reference image and the target style information of the target image to be generated. The image generation instruction carries the reference image and a prompt text, and the prompt text includes the target style information corresponding to the target image. Then, the server 11 executes the image generation method provided in the embodiment of the present application in response to the image generation instruction to generate a target image, and returns the target image to the client 200, so that the target image can be displayed in the display interface of the client 200.

[0085] In some embodiments, the joint training process of the first control network, the second control network and the image generation model required for the image generation process is first exemplarily described here:

[0086] See also Figure 4 , a flow chart of a model training method is provided, and the method can be performed by an electronic device. It is understandable that the electronic device used to perform the model training method can be a background device where the server is located, or other electronic devices, and this embodiment does not impose any restrictions on this.

[0087] The model training method includes:

[0088] In S401, supervised training is performed on the image generation model to be trained using a plurality of first training samples to obtain a trained image generation model; wherein the first training sample includes a first prompt text, a first noise image generated based on preset noise parameters, and a first image; and the first image is used as a label in the supervised training process. In this embodiment, through the first prompt text in the first training sample, the first noise image generated based on preset noise parameters, and the first image, the image generation model can learn how to generate an image according to semantic information, which ensures that the generated image can convey the specified semantic meaning.

[0089] The first prompt text includes description information of the image to be generated, and the first image, as a label in the supervised training process, has content related to the description information.

[0090] Exemplarily, the description information includes style information of the image to be generated, such as a specified image style. The first image has the specified image style, such as a comic style, but not limited thereto; so that the image generated by the image generation model visually conforms to the expected specified image style.

[0091] Exemplarily, the description information includes the specified content contained in the image to be generated, such as specifying that there is a cat in the image to be generated. The first image also shows a cat, so that the image generated by the image generation model visually contains the expected specified content.

[0092] In S402, on the basis of locking the model parameters of the trained image generation model, supervised training is performed on the first control network to be trained and the second control network to be trained connected to the trained image generation model using a number of second training samples to obtain the trained first control network and the trained second control network; wherein the second training samples include a second prompt text, a second noise image generated based on preset noise parameters, a second image, depth of field information of the second image, and edge information; the second image is used as a label in the supervised training process.

[0093] The trained first control network, the trained second control network and the trained image generation model can be applied to the image generation method mentioned below.

[0094] In this embodiment, the first control network and the second control network are trained by the depth information and edge information in the second training sample, and these control information can accurately adjust the depth effect and edge clarity of the generated image, so that the generated image has more details and realism. And on the basis of locking the model parameters of the trained image generation model, further training the first control network and the second control network can effectively utilize the knowledge and capabilities of the existing model, which not only improves the training efficiency, but also ensures the stability of the generated image quality and control accuracy.

[0095] Exemplarily, the image generation model can be a diffusion model. Diffusion models are a type of generation model based on a probability model. The core idea of ​​this type of model is to destroy the data distribution through a series of processes of gradually adding noise, and then learn how to reverse this process and restore the original data from the noise.

[0096] In the diffusion model, the above-mentioned preset noise parameters can be various types of parameters, including but not limited to noise intensity, noise distribution, time steps, and noise scheduling. Among them, noise intensity refers to the amount of noise added to the image in each diffusion process. Noise distribution includes but is not limited to Gaussian noise, uniform noise, Laplace noise, hyperbolic tangent noise or gamma noise. The number of time steps determines the total number of steps in the diffusion process. More time steps can refine the diffusion process, but also increase the computational complexity. Noise scheduling refers to the change law of noise intensity at different time steps. Common noise scheduling methods include linear scheduling and cosine scheduling.

[0097] In some embodiments, see Figure 5 In the process of supervised training of the image generation model to be trained, the electronic device can input the first prompt text and the first noise image in the first training sample into the image generation model to be trained, so that the image generation model to be trained generates a first predicted image based on the first prompt text and the first noise image, and trains the image generation model to be trained with minimizing the difference between the first predicted image and the first image as the optimization goal.

[0098] The image generation model includes a text encoder and an image generation network. Exemplarily, the image generation network may be a U-net neural network, but is not limited thereto, and may also be other deep neural networks. Exemplarily, the text encoder may generate text features of the prompt text, and then the image generation network processes the text features and the first noise image to generate a first predicted image.

[0099] After obtaining the first predicted image, the electronic device can calculate the difference between the first predicted image and the first image, for example, by using a loss function such as mean square error (MSE) or perceptual loss to calculate the difference between the first predicted image and the first image; and then taking minimizing the difference between the first predicted image and the first image as the optimization goal, minimize the loss value of the loss function through back propagation and gradient descent algorithms and update the model parameters of the image generation model.

[0100] In some embodiments, the first control network and the second control network may be connected to different image generation models, for example, a trained image generation model may be copied, the trained image generation model includes a trained first image generation model and a trained second image generation model; wherein, see Fig. 6A , the first control network to be trained is connected to the trained first image generation model, and, see Figure 6B , the second control network to be trained is connected to the trained second image generation model.

[0101] For example, see Fig. 6A In the process of supervised training of the first control network, on the basis of locking the model parameters of the trained first image generation model, the electronic device inputs the second prompt text and the second noise image in the second training sample into the trained first image generation model, and inputs the depth of field information of the second image into the first control network to be trained, so that the first control network to be trained generates and outputs depth control information according to the depth of field information, and the trained first image generation model generates a second predicted image based on the second prompt text, the second noise image and the depth control information output by the first control network to be trained, and trains the first control network to be trained with minimizing the difference between the second predicted image and the second image as the optimization goal.

[0102] When training the first control network, the model parameters of the trained first image generation model are first locked. This means that during the training process, the model parameters of the first image generation model will not change, thereby maintaining its generation capability and stability. After generating the second predicted image, the generated second predicted image is compared with the true second image to calculate the difference between them. For example, the loss function used may be mean square error (MSE), perceptual loss, etc. With the goal of minimizing the loss function, the model parameters of the first control network are updated using the back propagation algorithm. Since the model parameters of the first image generation model have been locked, only the model parameters of the first control network will be updated.

[0103] By inputting the depth of field information into the first control network, this embodiment can accurately control the depth of field effect of the generated image, making the generated image more realistic and three-dimensional visually. The introduction of depth control information can help the model generate images with complex scenes and details, making the generated results more delicate and realistic in detail expression.

[0104] For example, see Figure 6B In the process of supervised training of the second control network, on the basis of locking the model parameters of the trained second image generation model, the second prompt text and the second noise image in the second training sample are input into the trained second image generation model, and the edge information of the second image is input into the second control network to be trained, so that the second control network to be trained generates edge control information according to the edge information and outputs it, and the trained second image generation model generates a third predicted image based on the second prompt text, the second noise image and the edge control information output by the second control network to be trained, and the second control network to be trained is trained with minimizing the difference between the third predicted image and the second image as the optimization goal.

[0105] When training the second control network, the model parameters of the trained second image generation model are first locked, which means that during the training process, the model parameters of the second image generation model will not change, thereby maintaining its generation capability and stability. After generating the third predicted image, the generated third predicted image is compared with the true second image to calculate the difference between them. For example, the loss function used may be mean square error (MSE), perceptual loss, etc. With the goal of minimizing the loss function, the model parameters of the second control network are updated using the back propagation algorithm. Since the model parameters of the second image generation model have been locked, only the model parameters of the second control network will be updated.

[0106] In this embodiment, by inputting edge information, the second control network can obtain edge control information and transmit it to the second image generation model, so that the second image generation model can better capture and generate detailed parts in the image, avoid blurred or unclear edge areas, and can learn and generate more precise edge details, so that the contours and edges of the generated third predicted image are clearer; the generated third predicted image can better retain and display complex edge details, such as the contours and textures of objects, thereby improving the overall image quality.

[0107] In some embodiments, the first control network and the second control network can be connected to the same image generation model, for example, see Figure 7The trained image generation model includes a trained third image generation model; the first control network to be trained and the second control network to be trained are both connected to the trained third image generation model.

[0108] It should be noted that the model structures of the first image generation model, the second image generation model and the third image generation model mentioned in the embodiments of the present application are the same, and “first”, “second” and “third” are distinguished due to the differences in their connections with the control network.

[0109] In the process of supervised training of the first control network and the second control network, on the basis of locking the model parameters of the trained third image generation model, the electronic device inputs the second prompt text and the second noise image in the second training sample into the trained third image generation model, inputs the depth of field information of the second image into the first control network to be trained, and inputs the edge information of the second image into the second control network to be trained, so that the first control network to be trained generates and outputs depth control information according to the depth of field information, and the second control network to be trained generates and outputs edge control information according to the edge information, and the trained third image generation model generates a fourth predicted image based on the second prompt text, the second noise image, the depth control information output by the first control network to be trained, and the edge control information output by the second control network to be trained, and trains the first control network to be trained and the second control network to be trained with minimizing the difference between the fourth predicted image and the second image as the optimization goal.

[0110] When training the first control network and the second control network, the model parameters of the trained third image generation model are first locked, which means that during the training process, the model parameters of the third image generation model will not change, thereby maintaining its generation ability and stability. After generating the fourth predicted image, the generated fourth predicted image is compared with the real second image to calculate the difference between them. For example, the loss function used may be mean square error (MSE), perceptual loss, etc. With the goal of minimizing the loss function, the model parameters of the first control network and the second control network are updated using the back propagation algorithm.

[0111] In this embodiment, by combining depth of field information and edge information, the depth and edge details of the generated image are comprehensively controlled. The depth control information enables the image to have a realistic depth of field effect, while the edge control information ensures the clarity of the image contour and details. Through the fusion of these two types of control information, the generated image is visually richer and more refined, with higher quality and realism.

[0112] After the first control network, the second control network and the image generation model are trained, they can be applied to Figure 3Aor Figure 3B The image shown is generated in the scene.

[0113] In some embodiments, see Fig. 8A , a flow chart of an image generation method is provided, and an exemplary description is given by taking the image generation method applied to a server in an image generation system as an example. The method includes:

[0114] In S801, an image generation instruction is received, the image generation instruction carries a reference image and a prompt text, and the prompt text is used to instruct the generation of a target image.

[0115] Exemplarily, the reference image provides the basis of the content, and the prompt text can provide direction and guidance for the image generation model, telling the model descriptive information such as the style and content of the target image that the user expects to generate. For example, the prompt text can describe a specific artistic style or visual element, thereby helping the model understand and achieve target styles such as "comic style" and "oil painting style"; or, by specifying details such as color, light, texture, etc. in the prompt text, specific visual aspects of the generated image can be controlled; or, by specifying the content that needs to be displayed in the target image in the prompt text, such as an object or a building, etc., specific visual aspects of the generated image can be controlled.

[0116] Exemplarily, the prompt text includes description information of the target image; wherein the description information at least includes target style information corresponding to the target image, and the target style information defines image style features that are desired to be achieved in the generated target image.

[0117] As mentioned above Figure 3BAs shown, a variety of image styles can be provided, and users can choose according to actual needs. Image style refers to the unique characteristics and style of the image in visual expression, such as image styles including but not limited to: (1) Realistic style, imitating the appearance of the real world, striving for precision and realism in details. (2) Steampunk style, combining Victorian industrial steam technology and punk culture to create a unique retro-futuristic style. (3) Gothic style, derived from Gothic architecture and literature, characterized by darkness, mystery and high decorativeness. (4) Abstract style, does not directly imitate the real world, but expresses concepts or emotions through a combination of colors, shapes and lines. (5) Comic style, derived from the cartoon style in comics and animations, characterized by big eyes, exaggerated expressions and dynamics. (6) Minimalist style, expressed in the simplest form and the least elements, emphasizing space and blank space. (7) Watercolor style, imitating the transparent and flowing nature of watercolor paintings, usually with soft edges and gradient effects. (8) Oil painting style, imitating the texture and heaviness of traditional oil paintings, with rich color layers and brushstrokes. (9) Graffiti style, which originated from street art and is characterized by free-flowing lines and bright colors. (10) Futuristic style, which explores technology and future themes and is often used in science fiction themes.

[0118] For example, the mapping relationship between different style information and prompt text templates can be pre-stored in the server, and then after receiving the target style information, the server can determine the prompt text for this application from the preset mapping relationship between different style information and prompt text templates based on the target style information. This embodiment can quickly obtain the prompt text corresponding to the target style information by pre-storing the mapping relationship between different style information and prompt text templates, thereby improving efficiency.

[0119] For example, (1) the prompt text template corresponding to the oil painting style can be: Please transform the reference image into a classic oil painting, using rich color layers and delicate brushstrokes to capture the light and atmosphere of the scene. (2) The prompt text template corresponding to the comic style can be: Please present the reference image in anime style, giving the characters vivid expressions and features, and the scene should have rich details and dynamic perspectives. (3) The prompt text template corresponding to the Gothic style can be: Please reshape the reference image in a Gothic style, using deep colors and sharp lines to emphasize mystery and drama. (4) The prompt text template corresponding to the minimalist style can be: Please redesign the reference image in a minimalist style, removing unnecessary details, retaining only the most basic shapes and colors, and pursuing simplicity and purity. However, it is not limited to this, and the prompt text template can be specifically set according to the actual application scenario.

[0120] In S802, the depth information of the foreground target in the reference image is extracted, and the depth control information is obtained based on the depth information of the foreground target using the trained first control network, and the edge information of the background in the reference image is extracted, and the edge control information is obtained based on the edge information of the background using the trained second control network.

[0121] Exemplarily, extracting the depth information of the foreground object helps determine the relative distance and spatial relationship of the foreground object in the reference image. Extracting the edge information of the background in the reference image helps distinguish the foreground from the background and identify the features of the background.

[0122] Exemplarily, the electronic device may segment the reference image into a reference foreground image and a reference background image. It is understandable that the present embodiment does not impose any restrictions on the specific segmentation method, and specific settings may be made according to the actual application scenario. For example, by adopting a threshold segmentation method, the electronic device may compare the grayscale value of each pixel in the reference image with a predefined threshold value, so that the foreground and background in the reference image can be separated. For another example, by adopting an edge detection method, the electronic device uses an edge detection algorithm (such as Sobel, Canny, etc.) to identify edge information in the reference image, thereby separating the foreground target from the background. For another example, by adopting an image segmentation deep learning model, the electronic device uses a pre-trained foreground target segmentation network (such as FCN, U-Net, etc.) to perform semantic segmentation on the image to obtain a mask map for indicating the foreground target and the background; in the mask map, the pixel values ​​of the pixels used to indicate the foreground target are different from the pixel values ​​of the pixels used to indicate the background; and then the reference foreground image and the reference background image can be obtained using the mask map.

[0123] Exemplarily, the electronic device may perform depth extraction on the reference foreground image to obtain depth information of the foreground target. For example, a depth estimation algorithm (such as a deep neural network, stereo vision, etc.) may be used to extract the depth information of the foreground target from the reference foreground image.

[0124] Exemplarily, the electronic device may perform edge extraction on the reference background image to obtain edge information of the background, for example, using an edge detection algorithm (such as Canny edge detection, Sobel operator, etc.) to extract edge information of the background from the reference background image.

[0125] In S803, the depth control information, the edge control information and the prompt text are used to guide the trained image generation model to generate a target image based on the reference image.

[0126] Among them, the first control network, the second control network and the image generation model are obtained through joint training.

[0127] In this embodiment, the edge information of the background in the reference image is used to emphasize the background, and the edge control information obtained by the second control network is used as the control condition of the image generation model to achieve precise control of the background part of the target image, thereby effectively suppressing the problem of the image generation model generating a portrait in an unmanned background; and the depth of field information of the foreground target in the reference image is used to restore the foreground target, and the depth control information obtained by the first control network is used as another control condition of the image generation model to achieve precise control of the foreground part of the target image, thereby making the foreground target in the target image more three-dimensional; this embodiment uses the depth of field information of the foreground and the edge information of the background to achieve precise control of the generation of the foreground and background, and can generate higher quality images, with more distinct foreground targets and better coordination between the background and foreground.

[0128] For example, see Figure 8B , when using the graphics generation method provided in the embodiment of the present application Figure 8B After processing the reference image shown in the left figure, the generated target image ( Figure 8B (The right picture in the figure) There is no problem of the background being created out of nothing, and the generated foreground characters are more three-dimensional.

[0129] In some embodiments, see Fig. 9 , the image generation model includes a first image generation model and a second image generation model, the first control network is connected to the first image generation model, and the second control network is connected to the second image generation model. The electronic device can use the depth control information and the prompt text to guide the first image generation model to generate a target foreground image based on the reference foreground image, and the depth control information can guide the first image generation model to generate a more three-dimensional and deep image content inside the foreground area. And, using the edge control information and the prompt text, the second image generation model is guided to generate a target background image based on the reference background image, and the edge control information can guide the second image generation model to maintain a clear outline and structure inside the background area to avoid creating something out of nothing. Finally, the target foreground image and the target background image are fused to ensure that the two are naturally connected at the edges and color transitions, and finally the target image is obtained.

[0130] In this embodiment, by processing the foreground and background separately and using depth control information and edge control information in a targeted manner, the generated image can be closer to the real scene; the introduction of two control networks enables the two image generation models to accurately adjust the details of the foreground and background respectively, such as the performance in depth and edge structure, so that the target image is more in line with the user's expectations, retaining the three-dimensional sense of the foreground object and the clear outline of the background, while maintaining the overall consistency and naturalness visually, thereby improving the user experience and satisfaction.

[0131] The following is an exemplary description of the process of generating a target foreground image:

[0132] The first image generation model includes a first text encoder and a first image generation network. In the process of generating a target foreground image, the electronic device can use preset noise parameters to perform noise processing on the reference foreground image to obtain a reference foreground image after noise processing; wherein the first image generation model is a diffusion model, and the noise processing process is usually associated with the noise injection principle in the diffusion model. In the diffusion model, the process of generating an image is a process of gradually removing noise, and the noise processing can correspond to this denoising learning process. The preset noise parameters include but are not limited to noise intensity, noise distribution, time steps, and noise scheduling.

[0133] See also Fig. 9 , then the electronic device inputs the depth control information, the prompt text and the reference foreground image after the noise processing output by the first control network into the first image generation model, so that the first text encoder extracts text features from the prompt text, and the first image generation network processes the depth control information, text features and the reference foreground image after the noise processing to generate the target foreground image. In this embodiment, the prompt text can guide the first text encoder to extract text features related to the target style, and these text features can provide additional semantic information for the first image generation model, helping the first image generation model to better understand and capture the key features and context in the image generation task. The depth control information can make the generated target foreground image more visually three-dimensional and realistic by guiding the first image generation model to process depth perception. The reference foreground image after the noise processing introduces randomness and changes, which can help the first image generation model learn the ability to handle noise and complexity. In the process of generating the target foreground image, the first image generation model needs to adapt to and repair these noises, thereby generating a more diverse and realistic target foreground image.

[0134] See also Fig.10 , the first control network includes a coding layer, an intermediate layer, a first zero convolution layer, and a second zero convolution layer connected in sequence; the depth control information includes the depth control information output by the first zero convolution layer in the first control network and the depth control information output by the second zero convolution layer in the first control network. It can be understood that this embodiment does not impose any restrictions on the number of coding layers, intermediate layers, first zero convolution layers, and second zero convolution layers in the first control network. Fig.10 This is for illustrative purposes only and does not constitute a quantity limitation.

[0135] The first image generation model includes a first text encoder and a first image generation network. The first image generation network includes an encoding layer, an intermediate layer, and a decoding layer connected in sequence. These layers work together to effectively extract features from input data and generate a target image. It can be understood that this embodiment does not impose any restrictions on the number of encoding layers, intermediate layers, and decoding layers in the first image generation network. Fig.10 This is for illustrative purposes only and does not constitute a quantity limitation.

[0136] Among them, the first text encoder is connected to the encoding layer, the middle layer and the decoding layer in the first image generation network, which can pass the semantic information extracted from the prompt text to each layer in the first image generation network, and help guide the first image generation model to better capture and reconstruct the image features related to the prompt text.

[0137] The first zero convolution layer in the first control network is connected to the middle layer in the first image generation network, and the second zero convolution layer in the first control network is connected to the decoding layer in the first image generation network in a one-to-one correspondence, so that the depth control information can be passed to the first image generation network. This connection method ensures that the depth control information can affect the feature extraction and image reconstruction at different stages during the image generation process.

[0138] Through the above connection method, the depth control information of the first control network and the text features extracted by the first text encoder can effectively work with the encoding layer, intermediate layer and decoding layer of the first image generation network to achieve more accurate and targeted image generation tasks. This structured connection method helps to improve the expressiveness of the first image generation model and the quality of the generated images, while maintaining the efficiency and flexibility of the generation process.

[0139] The following is an exemplary description of the process of generating a background image:

[0140] The second image generation model includes a second text encoder and a second image generation network. In the process of generating the background image, the electronic device can use the preset noise parameters to perform noise processing on the reference background image to obtain the reference background image after noise processing. Similarly, the second image generation model is a diffusion model; in the diffusion model, the process of generating an image is a process of gradually removing noise, and the noise processing can correspond to this denoising learning process.

[0141] Then, see Fig. 9, the electronic device inputs the edge control information, the prompt text, and the reference background image after the noise processing into the second image generation model, so that the second text encoder extracts text features from the prompt text, and the second image generation network processes the edge control information, text features, and the reference background image after the noise processing to generate the target background image. In this embodiment, the prompt text can guide the second text encoder to extract text features related to the target style, and these text features can provide additional semantic information for the second image generation model, helping the second image generation model to better understand and capture the key features and context in the image generation task. The edge control information can keep the generated target background image clear in outline and structure by guiding the second image generation model to process edge perception. The reference background image after the noise processing introduces randomness and variation, which can help the second image generation model learn the ability to handle noise and complexity. In the process of generating the target background image, the second image generation model needs to adapt to and repair these noises, thereby generating a more diverse and realistic target background image.

[0142] and Fig.10 Similar to the structure shown, the second control network includes a coding layer, an intermediate layer, a first zero convolution layer and a second zero convolution layer connected in sequence. The edge control information includes the edge control information output by the first zero convolution layer in the second control network and the edge control information output by the second zero convolution layer in the second control network.

[0143] The second image generation model includes a second text encoder and a second image generation network. The second image generation network includes a coding layer, an intermediate layer, and a decoding layer connected in sequence, which work together to effectively extract features from the input data and generate a target image. The second text encoder is connected to the coding layer, the intermediate layer, and the decoding layer in the second image generation network, and can pass the semantic information extracted from the prompt text to each layer in the second image generation network, which helps to guide the second image generation model to better capture and reconstruct image features related to the prompt text.

[0144] The first zero convolution layer in the second control network is connected to the middle layer in the second image generation network, and the second zero convolution layer in the second control network is connected to the decoding layer in the second image generation network in a one-to-one correspondence, so that the edge control information can be passed to the second image generation network. This connection method ensures that the edge control information can affect the feature extraction and image reconstruction at different stages in the image generation process.

[0145] Through the above connection method, the edge control information of the second control network and the text features extracted by the second text encoder can effectively work with the encoding layer, intermediate layer and decoding layer of the second image generation network to achieve more accurate and targeted image generation tasks. This structured connection method helps to improve the expressive power of the second image generation model and the quality of the generated images, while maintaining the efficiency and flexibility of the generation process.

[0146] In some embodiments, see Fig.11 and Fig.12 , the image generation model includes a third image generation model; the first control network and the second control network are both connected to the third image generation model; the third image generation model includes a third text encoder and a third image generation network.

[0147] The electronic device can use preset noise parameters to perform noise processing on the reference image to obtain the reference image after noise processing. Among them, the third image generation model is a diffusion model; in the diffusion model, the process of generating an image is a process of gradually removing noise, and the noise processing can correspond to this denoising learning process. Then the electronic device inputs the depth control information, edge control information, prompt text and the reference image after noise processing into the third image generation model, so that the third text encoder extracts text features from the prompt text, and the third image generation network processes the depth control information, edge control information, text features and the reference image after noise processing to generate a target image.

[0148] In this embodiment, the first control network and the second control network are both connected to the third image generation model to achieve precise control of the generation process of the foreground and background parts of the target image, and directly output the target image that is highly consistent with the user's requirements in style and content, which is conducive to improving processing efficiency.

[0149] See also Fig.12 ( Fig.12 The number of layers in the description is for illustrative purposes only and is not intended to be limiting), the first control network and the second control network both include a coding layer, an intermediate layer, a first zero convolution layer, and a second zero convolution layer connected in sequence; the depth control information includes the depth control information output by the first zero convolution layer in the first control network and the depth control information output by the second zero convolution layer in the first control network; the edge control information includes the edge control information output by the first zero convolution layer in the second control network and the edge control information output by the second zero convolution layer in the second control network.

[0150] The image generation model includes a third image generation model; the third image generation model includes a third text encoder and a third image generation network; the third image generation network includes a coding layer, an intermediate layer, and a decoding layer connected in sequence, and these layers work together to effectively extract features from the input data and generate a target image.

[0151] Among them, the third text encoder is connected to the encoding layer, the middle layer and the decoding layer in the third image generation network, which can pass the semantic information extracted from the prompt text to each layer in the third image generation network, and help guide the third image generation model to better capture and reconstruct the image features related to the prompt text.

[0152] The first zero convolution layer in the first control network and the first zero convolution layer in the second control network are both connected to the middle layer in the third image generation network; and the second zero convolution layer in the first control network and the second zero convolution layer in the second control network are both connected one-to-one with the decoding layer in the third image generation network; the depth control information and the edge control information are passed to the third image generation network. This connection method ensures that the depth control information and the edge control information can affect the feature extraction and image reconstruction at different stages in the image generation process.

[0153] Through the above connection method, the depth control information of the first control network, the edge control information of the second control network, and the text features extracted by the third text encoder can effectively work with the encoding layer, intermediate layer, and decoding layer of the third image generation network to achieve more accurate and targeted image generation tasks. This structured connection method helps to improve the expressiveness of the third image generation model and the quality of the generated images, while maintaining the efficiency and flexibility of the generation process.

[0154] The various technical features in the above embodiments can be arbitrarily combined as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, any combination of the various technical features in the above embodiments also falls within the scope of this specification.

[0155] In some embodiments, an embodiment of the present application further provides an electronic device, comprising: a processor; a memory for storing processor executable instructions; wherein the processor implements any of the above methods by running the executable instructions.

[0156] For example, Fig.13 is a schematic structural diagram of a device provided by an exemplary embodiment. Fig.13At the hardware level, the device includes a processor 1302, an internal bus 1304, a network interface 1306, a memory 1308, and a non-volatile memory 1310, and may also include hardware required for other scenarios. One or more embodiments of the present application can be implemented based on software, such as the processor 1302 reading the corresponding computer program from the non-volatile memory 1310 into the memory 1308 and then running it. Of course, in addition to the software implementation, one or more embodiments of the present application do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0157] In some embodiments, the embodiments of the present application further provide a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by a processor, the steps of any of the methods described above are implemented.

[0158] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0159] In some embodiments, the embodiments of the present application further provide a computer program product, including a computer program, which implements the steps of any of the methods described above when executed by a processor.

[0160] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0161] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0162] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0163] The above description is only a preferred embodiment of one or more embodiments of the present application and is not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application shall be included in the scope of protection of one or more embodiments of the present application.

Claims

1. An image generation method, characterized in that: include: receiving an image generation instruction, wherein the image generation instruction carries a reference image and a prompt text, wherein the prompt text is used to instruct generation of a target image; Extracting depth information of a foreground object in the reference image, and using a trained first control network to obtain depth control information based on the depth information of the foreground object, and extracting edge information of a background in the reference image, and using a trained second control network to obtain edge control information based on the edge information of the background; The depth control information, the edge control information and the prompt text are used to guide the trained image generation model to generate the target image based on the reference image.

2. The method according to claim 1, characterized in that Also includes: Segmenting the reference image into a reference foreground image and a reference background image; The extracting depth information of the foreground object in the reference image includes: Performing depth extraction on the reference foreground image to obtain depth information of the foreground object; The extracting edge information of the background in the reference image comprises: Perform edge extraction on the reference background image to obtain edge information of the background.

3. The method according to claim 2, characterized in that The image generation model includes a first image generation model and a second image generation model; the first control network is connected to the first image generation model, and the second control network is connected to the second image generation model; The using the depth control information, the edge control information, and the prompt text to guide the trained image generation model to generate the target image based on the reference image includes: Using the depth control information and the prompt text, guiding the first image generation model to generate a target foreground image based on the reference foreground image; and Using the edge control information and the prompt text, guiding the second image generation model to generate a target background image based on the reference background image; The target foreground image and the target background image are fused to obtain the target image.

4. The method according to claim 3, characterized in that The first image generation model includes a first text encoder and a first image generation network; The step of using the depth control information and the prompt text to guide the first image generation model to generate a target foreground image based on the reference foreground image includes: Performing noise processing on the reference foreground image using a preset noise parameter to obtain a reference foreground image after noise processing; The depth control information, the prompt text and the reference foreground image after noise processing are input into the first image generation model, so that the first text encoder extracts text features from the prompt text, and the first image generation network processes the depth control information, the text features and the reference foreground image after noise processing to generate the target foreground image.

5. The method according to claim 3, characterized in that: The second image generation model includes a second text encoder and a second image generation network; The step of using the edge control information and the prompt text to guide the second image generation model to generate a target background image based on the reference background image includes: Performing noise processing on the reference background image using a preset noise parameter to obtain a reference background image after noise processing; The edge control information, the prompt text and the reference background image after noise processing are input into the second image generation model, so that the second text encoder extracts text features from the prompt text, and the second image generation network processes the edge control information, the text features and the reference background image after noise processing to generate the target background image.

6. The method according to claim 1, characterized in that The image generation model includes a third image generation model; the third image generation model includes a third text encoder and a third image generation network; The using the depth control information, the edge control information, and the prompt text to guide the trained image generation model to generate the target image based on the reference image includes: Performing noise processing on the reference image using a preset noise parameter to obtain a reference image after noise processing; The depth control information, the edge control information, the prompt text and the reference image after noise processing are input into the third image generation model, so that the third text encoder can extract text features from the prompt text, and the third image generation network can process the depth control information, the edge control information, the text features and the reference image after noise processing to generate the target image.

7. The method according to any one of claims 1 to 6, characterized in that: The first control network and the second control network both include a coding layer, an intermediate layer, a first zero-convolution layer and a second zero-convolution layer connected in sequence; The depth control information includes depth control information output by a first zero convolution layer in the first control network and depth control information output by a second zero convolution layer in the first control network; The edge control information includes edge control information output by a first zero convolution layer in the second control network and edge control information output by a second zero convolution layer in the second control network.

8. The method according to claim 7, characterized in that The image generation model includes a first image generation model and a second image generation model; The first image generation model includes a first text encoder and a first image generation network, and the second image generation model includes a second text encoder and a second image generation network; The first image generation network and the second image generation network each include a coding layer, an intermediate layer, and a decoding layer connected in sequence; The first text encoder is connected to the encoding layer, the intermediate layer and the decoding layer in the first image generation network; the first zero convolution layer in the first control network is connected to the intermediate layer in the first image generation network, and the second zero convolution layer in the first control network is connected to the decoding layer in the first image generation network in a one-to-one correspondence; The second text encoder is connected to the encoding layer, the intermediate layer and the decoding layer in the second image generation network; and the first zero convolution layer in the second control network is connected to the intermediate layer in the second image generation network, and the second zero convolution layer in the second control network is connected to the decoding layer in the second image generation network in a one-to-one correspondence.

9. The method according to claim 7, characterized in that: The image generation model includes a third image generation model; the third image generation model includes a third text encoder and a third image generation network, and the third image generation network includes a coding layer, an intermediate layer, and a decoding layer connected in sequence; Among them, the third text encoder is connected to the encoding layer, the intermediate layer and the decoding layer in the third image generation network; and the first zero convolution layer in the first control network and the first zero convolution layer in the second control network are both connected to the intermediate layer in the third image generation network; and the second zero convolution layer in the first control network and the second zero convolution layer in the second control network are both connected one-to-one with the decoding layer in the third image generation network.

10. The method according to claim 1, characterized in that The prompt text includes description information of the target image; The description information at least includes target style information corresponding to the target image, and the prompt text is determined from a mapping relationship between preset different style information and prompt text templates based on the target style information.

11. The method according to claim 1, characterized in that: The image generation model includes a diffusion model.

12. A model training method, characterized in that: include: Using a plurality of first training samples to perform supervised training on the image generation model to be trained, to obtain a trained image generation model; wherein the first training samples include a first prompt text, a first noise image generated based on a preset noise parameter, and a first image; the first image is used as a label in the supervised training process; On the basis of locking the model parameters of the trained image generation model, a first control network to be trained and a second control network to be trained connected to the trained image generation model are supervisedly trained using a plurality of second training samples to obtain a trained first control network and a trained second control network; wherein the second training samples include a second prompt text, a second noise image generated based on preset noise parameters, a second image, depth of field information of the second image, and edge information; the second image is used as a label in the supervised training process; Wherein, the trained first control network, the trained second control network and the trained image generation model are applied to the image generation method described in any one of claims 1 to 11.

13. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 12 by running the executable instructions.

14. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images