Image generation method and device, equipment and storage medium

Abstract

The invention discloses an image generation method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the steps that character information, prop information and a prompt text are acquired, the character information is used for indicating features needed by characters in an output image needing to be generated, and the prop information is used for indicating features needed by props in the output image; generating input information of a stable diffusion model and input information of a control network according to the character information, the prop information and the prompt text; generating output information of the control network according to the input information of the control network through the control network; and through the stable diffusion model, generating an output image according to the input information of the stable diffusion model and the output information of the control network. According to the method, the character information, the prop information and the prompt text are used as the input information of the image generation model, the posture, the position, the size and the texture of the prop are finely controlled, and the generation effect of the output image is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image generation method, device, equipment and storage medium. Background Art

[0002] The automated character design in the gaming field aims to use AI (Artificial Intelligence) to generate models and automatically generate character images based on given design requirements, including the joint generation of character subjects and props.

[0003] In the related technology, an auxiliary conditional control model (ControlNet model) based on the Stable Diffusion Model (SD model) is used. According to the input conditions of different modes such as posture graphs, segmentation graphs, depth graphs, etc., the feature information corresponding to the control conditions is extracted, and the feature information is fused with the text features of the input stable diffusion model to generate a conditionally controlled image.

[0004] However, the postures of the characters in the pictures generated by the above model can be precisely controlled, but various factors such as the type, size, and angle of the props held by the characters can only be coarsely controlled through text prompts, resulting in poor prop generation effects. Summary of the invention

[0005] The present application provides an image generation method, device, equipment and storage medium. The technical solution is as follows:

[0006] According to one aspect of an embodiment of the present application, there is provided an image generation method based on an image generation model, wherein the image generation model includes a stable diffusion model and a control network, and the method includes:

[0007] Acquire character information, prop information, and prompt text, wherein the character information is used to indicate the features that the character in the output image to be generated should have, the prop information is used to indicate the features that the prop in the output image should have, and the prompt text is used to indicate the interactive relationship between the character and the prop in the output image;

[0008] Generating input information of the stable diffusion model and input information of the control network according to the character information, the item information and the prompt text;

[0009] generating output information of the control network according to input information of the control network through the control network;

[0010] The output image is generated by the stable diffusion model according to input information of the stable diffusion model and output information of the control network.

[0011] According to one aspect of an embodiment of the present application, a training method for an image generation model is provided, wherein the image generation model includes a stable diffusion model and a control network, and the method includes:

[0012] Acquire a training sample set of the image generation model, wherein the training sample set includes at least one image-text pair, each of the image-text pairs includes a label text and a label image having a matching relationship, the label text is used to indicate an interactive relationship between a character and a prop in the label image, and the label image is an image containing the character and the prop having an interactive relationship with the character;

[0013] Acquire character information and prop information in the label image, wherein the character information is used to indicate the characteristics of the character, and the prop information is used to indicate the characteristics of the prop;

[0014] Generating input information of the stable diffusion model and input information of the control network according to the character information, the prop information and the label text;

[0015] generating output information of the control network according to input information of the control network through the control network;

[0016] Generate a prediction image through the stable diffusion model according to input information of the stable diffusion model and output information of the control network;

[0017] According to the predicted image and the label image, the parameters of the control network are adjusted to obtain a trained image generation model.

[0018] According to one aspect of an embodiment of the present application, there is provided an image generation device based on an image generation model, wherein the image generation model includes a stable diffusion model and a control network, and the device includes:

[0019] An information acquisition module, used to acquire character information, prop information and prompt text, wherein the character information is used to indicate the characteristics that the character in the output image to be generated should have, the prop information is used to indicate the characteristics that the prop in the output image should have, and the prompt text is used to indicate the interactive relationship between the character and the prop in the output image;

[0020] An input generation module, used to generate input information of the stable diffusion model and input information of the control network according to the character information, the prop information and the prompt text;

[0021] A control output module, used to generate output information of the control network according to input information of the control network through the control network;

[0022] The image output module is used to generate the output image through the stable diffusion model according to the input information of the stable diffusion model and the output information of the control network.

[0023] According to one aspect of an embodiment of the present application, a training device for an image generation model is provided, wherein the image generation model includes a stable diffusion model and a control network, and the device includes:

[0024] A sample set acquisition module, used to acquire a training sample set of the image generation model, wherein the training sample set includes at least one image-text pair, each of which includes a label text and a label image having a matching relationship, wherein the label text is used to indicate an interactive relationship between a character and a prop in the label image, and the label image is an image containing the character and the prop having an interactive relationship with the character;

[0025] An information acquisition module, used to acquire character information and prop information in the label image, wherein the character information is used to indicate the characteristics of the character, and the prop information is used to indicate the characteristics of the prop;

[0026] An input generation module, used to generate input information of the stable diffusion model and input information of the control network according to the character information, the prop information and the label text;

[0027] A control output module, used to generate output information of the control network according to input information of the control network through the control network;

[0028] An image output module, used to generate a predicted image through the stable diffusion model according to input information of the stable diffusion model and output information of the control network;

[0029] The model training module is used to adjust the parameters of the control network according to the predicted image and the label image to obtain a trained image generation model.

[0030] According to one aspect of an embodiment of the present application, a computer device is provided, which includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned image generation method based on the image generation model, or the training method of the image generation model.

[0031] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned image generation method based on the image generation model, or the training method of the image generation model.

[0032] According to one aspect of an embodiment of the present application, a computer program product is provided, which includes a computer program, and the computer program is loaded and executed by a processor to implement the above-mentioned image generation method based on the image generation model, or the training method of the image generation model.

[0033] The technical solution provided in the embodiments of the present application can bring the following beneficial effects:

[0034] By taking character information, prop information and prompt text as input information of the image generation model, and generating input information of the stable diffusion model and input information of the control network based on the character information, prop information and prompt text, the input information of the control network is combined with the input information of the stable diffusion model as a control condition, so that the text features in the prompt text and the character features in the character information can flexibly interact with the prop features in the prop information, so that an output image containing a specified character posture and a specified prop posture can be generated. Compared with the related art that generates an image based on a character posture image and a prompt text but cannot control the generation effect of the props therein, the technical solution provided by the present application can realize the joint control generation of multiple targets, and achieve refined control over the posture, position, size and texture of the props, thereby improving the generation effect and generation quality of the output image. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a structural diagram of the ControlNet model provided by the relevant technology;

[0036] Figure 2 It is a schematic diagram of the output image of the ControlNet model provided by the related art;

[0037] Figure 3 It is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;

[0038] Figure 4 is a schematic diagram of a model interaction process provided by an embodiment of the present application;

[0039] Figure 5 is a flowchart of an image generation method based on an image generation model provided by an embodiment of the present application;

[0040] Figure 6 It is a schematic diagram of the structure of the encoding block and the decoding block provided by an embodiment of the present application;

[0041] Figure 7 It is a flowchart of a method for generating input information of an image generation model provided by an embodiment of the present application;

[0042] Figure 8 It is a schematic diagram of a process of generating image features according to character information and prop information provided by an embodiment of the present application;

[0043] Fig. 9 It is a schematic diagram of an input situation of a stable diffusion model and a control network provided by an embodiment of the present application;

[0044] Fig.10 is a flowchart of another method for generating input information of an image generation model provided by an embodiment of the present application;

[0045] Fig.11 is a schematic diagram of another stable diffusion model and input situation of a control network provided by an embodiment of the present application;

[0046] Fig.12 is a flowchart of a method for training an image generation model provided by an embodiment of the present application;

[0047] Fig.13 is a schematic diagram of the training effect of the image generation model provided by one embodiment of the present application;

[0048] Fig.14 This is a schematic diagram comparing the effects of using one and multiple props to generate an image, provided by an embodiment of the present application;

[0049] Fig.15 is a block diagram of an image generation device based on an image generation model provided by an embodiment of the present application;

[0050] Fig.16 is a block diagram of a training device for an image generation model provided by an embodiment of the present application;

[0051] Fig.17 It is a structural block diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0053] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0054] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, pre-trained models are also called large models and basic models. After fine-tuning, they can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0055] Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning. The pre-trained model is the latest development of deep learning, which integrates the above technologies.

[0056] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, driverless cars, autonomous driving, drones, digital twins, virtual humans, robots, artificial intelligence generated content (Artificial Intelligence Generated Content, AIGC), conversational interaction, smart medical care, smart customer service, game AI, virtual reality (Virtual Reality, VR), augmented reality (Augmented Reality, AR), etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0057] The technical solution of this application mainly involves machine learning technology in artificial intelligence technology, mainly involving the training and use process of image generation models.

[0058] like Figure 1 As shown, the related art adopts an auxiliary conditional control model (ControlNet model) based on the stable diffusion model. The stable diffusion model only supports prompt text as input information, so generating a high-quality picture requires a lot of text design, and the content of the picture generation cannot be finely controlled, resulting in the inability to accurately generate pictures that meet user needs, which easily reduces the user experience.

[0059] Therefore, the ControlNet model introduces an additional control network (ControlNet, CN) based on the stable diffusion model. By inputting given conditional information, such as posture graphs, segmentation graphs, depth graphs, etc., into the control network, the control network extracts the feature information corresponding to the conditional information, and then the stable diffusion model interacts the conditional features with the text features input into the stable diffusion model to generate images controlled by the conditions. Taking the posture graph as an example of conditional information, the character posture image can be input into the control network as a control condition to generate different character images that match the character posture in the character posture image, and the character style can be controlled by the prompt text.

[0060] The process of stable diffusion model and control network generating pictures based on prompt text and character posture images can be referred to Figure 1 As shown, a text encoder is used to encode the prompt text to obtain the text features corresponding to the prompt text, and the text features and random noise are used as input information of the stable diffusion model, and the text features, random noise and character posture images are used as input information of the control network. Figure 1 The process of denoising the features after adding noise is shown in FIG. The input information of the control network is compressed by each coding block in the control network in turn to obtain the output information of the control network. The input information of the stable diffusion model is compressed by each coding block in the stable diffusion model in turn to obtain the compressed text features. Then, the features of the image information are interacted and reconstructed according to the compressed text features and the output information of the control network by each decoding block in the stable diffusion model to obtain the image information after feature interaction. Finally, the decoder in the VAE (Variational Auto-Encoder) decodes the image information after feature interaction to generate an output image.

[0061] However, although the posture of the characters in the images generated by the above ControlNet model can be accurately controlled, various factors such as the type, size, and angle of the props held by the characters can only be coarsely controlled through prompt text, resulting in poor prop generation effects.

[0062] For example, the input information of the ControlNet model cannot specify the number and location of prop generation, such as Figure 2 As shown in Figure (1), the prompt text is "an image of a pirate holding a sword on the beach". The prompt text requires that only one sword be generated, but two swords (props 201 and props 202) appear in the generated image, and the position of prop 202 is unreasonable, which does not meet the control requirements of the prompt text. Prop 202 is not held by the "pirate" in the image.

[0063] For another example, the props in the images generated by the ControlNet model cannot be customized, such as Figure 2 As shown in Figure (2), the prompt text is "an image of a person holding a sword, a shield, and armor." The prompt text requires the person to hold a sword and a shield, but the generated image only has a sword (prop 203), and the shield required by the prompt text does not appear, and prop 203 is not held by the person in the image. Therefore, the props in the image generated by the ControlNet model can only be controlled by the prompt text, and it is difficult to generate the prop image that the user wants.

[0064] Please refer to Figure 3 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment can be implemented as a training and use system for an image generation model. The solution implementation environment may include: a model training device 10 and a model use device 20.

[0065] The model training device 10 may be an electronic device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, a multimedia player, a vehicle terminal, a server, an intelligent robot, or other electronic devices with strong computing capabilities. The model training device 10 is used to train the image generation model.

[0066] In the embodiment of the present application, the image generation model is a machine learning model trained based on the training method of the image generation model, and is used to generate an image containing a person and a prop having an interactive relationship with the person according to the person information, the prop information and the prompt text. The model training device 10 can train the image generation model in a machine learning manner so that it has the ability to generate an image containing a person and a prop having an interactive relationship with the person. The specific model training method can refer to the following embodiment.

[0067] The image generation model includes a text encoder, a stable diffusion model and a control network. The input information of the image generation model includes character information, prop information, prompt text and random noise. Among them, the text encoder is used to encode the prompt text to obtain the text features corresponding to the prompt text. The input information of the stable diffusion model and the input information of the control network both include text features and random noise. For character information and prop information, the input information of the stable diffusion model and the input information of the control network can be generated according to the parameters of the stable diffusion model used and the corresponding input information construction method.

[0068] The stable diffusion model includes an encoding network, a first decoding network and a second decoding network, and the structure of the control network is the same as that of the encoding network. The encoding network is used to perform feature compression on the input information of the stable diffusion model, and the control network is used to perform feature compression on the input information of the control network; the first decoding network is used to perform feature interaction and feature reconstruction on the output information of the encoding network and the output information of the control network to obtain image information after feature interaction; the second decoding network is used to decode the output information of the first decoding network to obtain an output image, which is an image containing a character and a prop that has an interactive relationship with the character.

[0069] In the embodiment of the present application, character information, prop information, prompt text and random noise are input into the image generation model, and the text features corresponding to the prompt text are first generated by the text encoder, and then the input information of the stable diffusion model and the input information of the control network are generated according to the parameters of the stable diffusion model. After the encoding network and the control network perform feature compression on their respective input information, the first decoding network performs feature interaction on the output information of the encoding network and the output information of the control network, and finally the feature information is input into the second decoding network to generate an output image containing characters and props that have an interactive relationship with the characters.

[0070] The trained image generation model can be deployed in the model use device 20 for use. The model use device 20 can be a terminal device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, a multimedia playback device, a vehicle terminal, an intelligent robot, etc., or a server. When it is necessary to generate an output image that matches the input text based on the input text, the model use device 20 can achieve the above function through the trained image generation model.

[0071] The model training device 10 and the model using device 20 may be two independent devices or the same device. If the model training device 10 and the model using device 20 are the same device, the model training device 10 may be deployed in the model using device 20.

[0072] In the embodiments of the present application, the execution subject of each step may be a computer device, which refers to an electronic device with data calculation, processing and storage functions. The computer device may be a terminal device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, a multimedia playback device, a vehicle terminal, an intelligent robot, or a server. The server may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The computer device may be, for example, Figure 3 The model training device 10 may also be a model using device 20.

[0073] Figure 4 The schematic diagram of the model interaction process is shown. The user inputs character information, prop information and prompt text, and the image features such as the image ratio (width and height of the output image), image style, and image type of the output image are configured with parameters through random noise. After the input information is input into the image generation model, the stable diffusion model interacts with the character information, prop information and prompt text according to the parameter configuration requirements corresponding to the random noise, and generates an image that meets the user's requirements. Different random noises can be input to generate images with different configurations for users to choose.

[0074] Please refer to Figure 5 , which shows a flow chart of an image generation method based on an image generation model provided by an embodiment of the present application. The image generation model includes a stable diffusion model and a control network. The execution subject of each step of the method can be a computer device. The method can include at least one of the following steps 510 to 540:

[0075] Step 510, obtaining character information, prop information and prompt text, wherein the character information is used to indicate the features that the character in the output image to be generated must possess, the prop information is used to indicate the features that the prop in the output image must possess, and the prompt text is used to indicate the interactive relationship between the character and the prop in the output image.

[0076] A character may refer to a real person or a virtual person. If the character is a real person, it may be a well-known person, such as a well-known scientist, a well-known athlete, a well-known actor, etc., or it may be an unknown ordinary person, such as a classmate, a colleague, a neighbor, a stranger, etc. If the character is a virtual person, the character is not limited to human form, and may include animal form, any virtual character form created independently, etc. For example, it may be a film and television character in a film and television work, an animation character in an animation work, a game character in a game, etc.

[0077] Props can refer to any object other than characters. Similarly, props can be real objects in life or virtual objects. If props are real objects, they can be means of transportation, such as bicycles, cars, skateboards, etc., or decorative items, such as hats, watches, glasses, etc., or animals, such as cats, dogs, rabbits, etc., or any other objects. If props are virtual objects, props are not limited to the form of real objects, and can include any virtual object form created independently, for example, it can be a bicycle with wings, or a prop with offensive properties, or any animal form created independently, etc.

[0078] The interactive relationship between characters and props refers to the positional relationship between characters and props, which can refer to the positional relationship between characters and props that are in contact with each other, or the positional relationship between characters and props that are not in contact with each other, but there is a mutually mapped positional relationship. If there is a mutually contacting positional relationship between characters and props, it can be that the characters hold props, the characters step on props, the characters lean on props, etc. For example, it can be that the game characters hold attack props, a character is riding a bicycle, a character is holding a pet, etc. If there is no mutually contacting positional relationship between characters and props, the mutually mapped positional relationship can be that the props serve as the background or decoration of the characters, for example, the top of the game character's head can be decorated with a halo leaving the head, an attack prop can be erected beside the game character, a flag can be flying behind a character, etc.

[0079] Exemplarily, the character information may be at least one of the character information representing the character characteristics, such as a character mask, a character image, a character label, etc., and the prop information may refer to at least one of the prop information representing the prop characteristics, such as a prop mask, a prop image, a prop label, etc.

[0080] The character mask is a binary mask that uses different values ​​to distinguish the character area in the output image from other areas except the character area. The character mask is used to indicate the character area in the output image. For example, the character mask is a binary mask composed of 0 and 1, wherein 1 is used to indicate the character area in the output image, and 0 is used to indicate other areas except the character area in the output image. The prop mask is a binary mask that uses different values ​​to distinguish the prop area in the output image from other areas except the prop area. The prop mask is used to indicate the prop area in the output image. For example, the prop mask is a binary mask composed of 0 and 1, wherein 1 is used to indicate the prop area in the output image, and 0 is used to indicate other areas except the prop area in the output image.

[0081] A person image refers to an image containing only a person, and other regions in the person image except the person region can be represented by a uniform value, for example, other regions except the person region can be uniformly marked as 0. A prop image refers to an image containing only props, and other regions in the prop image except the prop region can be represented by a uniform value, for example, other regions except the prop region can be uniformly marked as 0.

[0082] A character label is text information used to indicate the characteristics of a character in an output image. For example, a character label can indicate the characteristics of a character such as the character's gender, body shape, and posture. The character characteristics mentioned here refer to the characteristics that a character in an output image needs to have. A prop label is text information used to indicate the characteristics of a prop in an output image. For example, a prop label can indicate the characteristics of a prop such as the prop type, prop size, and prop color. The prop characteristics mentioned here refer to the characteristics that a prop in an output image needs to have.

[0083] It should be noted that the character information of the input image generation model mentioned here can be one or more character information, but one or more character information are only used to indicate the character characteristics of one character, and the prop information of the input image generation model can be one or more prop characteristics, but one prop information is only used to indicate the prop characteristics of one prop, and multiple prop information can be used to indicate the prop characteristics of multiple different props, which means that the output image only contains one character, but can contain one or more props.

[0084] Step 520, generating input information of a stable diffusion model and input information of a control network according to the character information, the item information and the prompt text.

[0085] According to the parameters of the stable diffusion model, the input information of the stable diffusion model and the input information of the control network are determined. When the stable diffusion model adopts different parameters, the generation method of the input information of the stable diffusion model and the input information of the control network should be configured accordingly.

[0086] If the stable diffusion model is the first parameter, the input information of the stable diffusion model includes the prompt text, and the input information of the control network includes the character information, the prop information, and the prompt text. If the parameter of the stable diffusion model is the second parameter, the input information of the stable diffusion model includes the prompt text and the prop information, and the input information of the control network includes the character information and the prompt text. And the first parameter and the second parameter are different.

[0087] Step 530, generating output information of the control network according to input information of the control network through the control network.

[0088] The control network includes multiple encoding blocks and a middle-layer block. The encoding block is used to compress the features of the input information of the control network and further extract the features in the input information of the control network. The middle-layer block is used to buffer the feature information processed by the encoding block and does not have the ability to process information.

[0089] In some embodiments, random noise is added to input information of the control network to obtain first input information; and the control network is used to perform feature compression on the first input information to obtain output information of the control network.

[0090] Random noise is also one of the input information of the image generation model. Random noise is used to indicate the image features of the output image, such as image scale, image style, image type, image scene, etc. Random noise can refer to random numbers or random noise images. Usually, random noise images can be generated by random numbers. Different random numbers correspond to different random noise images, and different random noise images have different image features. The random noise added here can be random noise of any dimension. Adding random noise to the input information of the control network means adding the image features of the random noise image to the input information of the control network.

[0091] Random noise is added to the input information of the control network to obtain first input information. The first input information refers to the information after the noise is added to the input information of the control network, and is information obtained by stacking the input information of the control network and the random noise.

[0092] Exemplarily, if the dimension of the input information of the control network is N×512×H×W and the dimension of the random noise is N×4×H×W, then the dimension of the first input information is N×516×H×W, where N represents the number of images processed at a time (batch), H represents the height of the output image (Height), and W represents the width of the output image (Weight).

[0093] The structure of the control network is a downsampling network structure. Multiple encoding blocks in the control network are used to compress the features of the first input information in sequence. The structure of the encoding blocks in the control network can be referred to Figure 6 As shown in the figure. Each encoding block is composed of multiple stacked residual modules (ResBlock) and spatial transformers (Transfomer) cascaded. The residual network is used to solve the gradient vanishing problem in the neural network to learn more layers of features. The spatial transformer is used to perform spatial transformation on the features to increase the spatial invariance of the features. The output of the last spatial transformer in the encoding block will be used as the input of the next encoding block of the downsampling network.

[0094] The residual module includes a fully connected layer and a convolutional layer. The convolutional layer is used to extract the features of the image, and the fully connected layer acts as a classifier in the neural network. Each node is connected to all nodes in the previous layer, and is used to map the previously extracted features to the sample label space. Figure 6 As shown in the figure, after the information is input into the residual module, the time embedding information is spatially mapped through the fully connected layer, and the image features are extracted through the convolution layer, combined with the spatial features obtained through the fully connected layer, and combined with the input original information again in one step, and the image features are further extracted through the convolution layer, where the time embedding information is used to indicate the number of denoising times of the coding block.

[0095] The spatial transformer includes convolutional layer, fully connected layer, dot product layer (MatMul), scaling layer (Scaling Layer), and normalization layer (Softmax). The dot product layer obtains the feature interaction result by performing dot product between the representation vectors of different features. Dot product refers to the process of multiplying the elements at the same position of vectors of the same length and then adding them together. The scaling layer is used to scale the input information so that the input information has an appropriate range and can be easily calculated. The commonly used scaling methods of the scaling layer include maximum and minimum value scaling, mean and standard deviation scaling, and standard deviation scaling. The normalization layer is used to normalize the input information and convert the information into probability distribution.

[0096] like Figure 6 As shown in the figure, after the output information of the residual module is input into the spatial transformer, it first passes through the fully connected layer to map the input information to the sample tag space, and then the input information is subjected to three different linear transformations, namely query transformation, key transformation and value transformation, to obtain the query vector, key vector and value vector corresponding to the three linear transformations, namely the query feature Q, index feature K and content feature V corresponding to the input information. The query feature is used to represent the query feature to be queried by the input information, the index feature is used to represent the key features in the input information, and the content feature is used to represent the data features of the input information. Then, the query feature and the index feature are input together, and pass through the dot product layer, the scaling layer and the normalization layer in sequence. After the output information of the normalization layer is combined with the content feature, it is then passed through the dot product layer and the convolution layer in sequence to further extract the image features. The output of the last convolution layer in the spatial transformer will be used as the input of the next cascaded residual module.

[0097] By controlling multiple encoding blocks and an intermediate block in the network, feature extraction is performed on the first input information to obtain output information of the control network, which is used as a control condition of the stable diffusion model, so that the output image of the stable diffusion model is generated after being controlled by the control condition, which better meets the image generation needs of the user.

[0098] Step 540, generating an output image through a stable diffusion model according to input information of the stable diffusion model and output information of the control network.

[0099] The stable diffusion model includes an encoding network, a first decoding network, and a second decoding network. The encoding network includes multiple encoding blocks and a middle-layer block. The encoding network is used to perform feature compression on the input information of the encoding network and further extract features from the input information of the encoding network. In some embodiments, the structure of the control network is the same as that of the encoding network. By designing the structure of the control network to be the same as the encoding network, the design process of the control network is simplified, and the parameter setting of the control network can refer to the parameter setting of the encoding network, which is conducive to completing the design of the image generation model as soon as possible.

[0100] The first decoding network contains multiple decoding blocks. The structure of the decoding block is the same as that of the encoding block. Figure 6 The first decoding network is used to perform feature interaction on the output information of the encoding network and the output information of the control network, and to perform feature reconstruction on the feature information after interaction, so as to obtain image information after feature interaction.

[0101] The second decoding network includes a decoder, which may be a decoder in a variational autoencoder (VAE) for example. The second decoding network is used to decode the output information of the first decoding network to obtain an output image.

[0102] In some embodiments, random noise is added to the input information of the stable diffusion model to obtain second input information; the second input information is feature compressed using an encoding network to obtain third input information; image information after feature interaction is obtained through a first decoding network based on the third input information and output information of a control network; the image information after feature interaction is decoded using a second decoding network to obtain an output image.

[0103] The random noise mentioned here is the same random noise as the random noise added to the input information of the control network, and will not be described in detail here.

[0104] Random noise is added to the input information of the stable diffusion model to obtain second input information. The second input information refers to the information after the noise is added to the input information of the stable diffusion model, which is the information obtained by stacking the input information of the stable diffusion model and the random noise. The stacking process can be referred to above.

[0105] The second input information is the input information of the encoding network. After the encoding network performs feature compression on the second input information, the third input information is obtained. The third input information and the input information of the control network are the input information of the first decoding network. The structure of the first decoding network is an upsampling network structure. Through the first decoding network, the third input information and the output information of the control network are subjected to feature interaction, and the image features in the output information of the control network are combined with the image features of the third input information to obtain feature information after feature interaction. In the process of feature interaction, the feature information after feature interaction is gradually reconstructed to generate image information after feature interaction. Finally, the image information after feature interaction is decoded through the second decoding network to obtain an output image.

[0106] The image information after feature interaction refers to feature information obtained by performing feature interaction on the third input information and the output information of the control network. Feature interaction refers to associating and interacting the character features and prop features in the third input information and the output information of the control network according to the interactive relationship between the character and the prop contained in the prompt text, thereby generating interactive features between the character features and the prop features. The image information after feature interaction includes all image features required to generate the output image, that is, the features required for the characters in the output image, the features required for the props in the output image, the interactive features between the characters and props required to form the interactive relationship between the characters and props in the output image, and the image features of other areas in the output image except the character area and the prop area.

[0107] The output image is an image containing a character and a prop that has an interactive relationship with the character, and the character posture and prop posture in the output image meet the feature requirements of the character information, the prop information and the prompt text.

[0108] The first decoding network of the stable diffusion model performs feature interaction on the output information of the encoding network and the output information of the control network, so that the image features in the output information of the encoding network can be combined with the image features in the output information of the control network, so that the image information obtained after the feature interaction can include the features of the character information, props information and prompt text, thereby improving the image generation effect and making the output image more in line with the user's image generation needs.

[0109] The technical solution provided in the embodiment of the present application uses character information, prop information and prompt text as input information of the image generation model, and generates input information of the stable diffusion model and input information of the control network based on the character information, prop information and prompt text, thereby combining the input information of the control network with the input information of the stable diffusion model as a control condition, so that the text features in the prompt text and the character features in the character information can flexibly interact with the prop features in the prop information, thereby generating an output image containing a specified character posture and a specified prop posture. Compared with the related art that generates an image based on a character posture image and a prompt text but cannot control the generation effect of the props therein, the technical solution provided in the present application can realize the joint control generation of multiple targets, and achieve refined control over the posture, position, size and texture of the props, thereby improving the generation effect and generation quality of the output image.

[0110] Next, through Figure 7 and Fig.10 The two generation processes of the input information of the stable diffusion model and the input information of the control network in step 520 are specifically introduced.

[0111] Figure 7 In the method for generating input information of the image generation model shown, the input information of the stable diffusion model includes prompt text, and the input information of the control network includes character information, prop information and prompt text.

[0112] Fig.10 In the method for generating input information of the image generation model shown, the input information of the stable diffusion model includes prompt text and prop information, and the input information of the control network includes character information and prompt text.

[0113] Please refer to Figure 7 , which shows a flow chart of a method for generating input information of an image generation model provided by an embodiment of the present application. The image generation model includes a stable diffusion model and a control network, and also includes a text encoder and a feature extractor. The execution subject of each step of the method can be a computer device. The method can include at least one of the following steps 710 to 730:

[0114] Step 710, obtaining image features according to character information and prop information through a feature extractor.

[0115] The feature extractor includes two types of feature extractors: an image feature extractor and a text feature extractor. Exemplarily, the feature extractor may be a clip model. The image feature extractor is used to extract features from an image, and the text feature extractor is used to extract features from a text.

[0116] The character information and prop information are encoded by the feature extractor to obtain the character features corresponding to the character information and the prop features corresponding to the prop information. The character features and prop features are stacked to obtain the image features, which are used to indicate the character posture and prop posture in the output image. The image feature mentioned here is a spatial image feature.

[0117] In some embodiments, the feature extractor is an image feature extractor for extracting features from an image, the character information is a character image, and the prop information is a prop image. Then step 710 includes at least one sub-step of steps 711 to 712 (not shown in the figure).

[0118] Step 711, using an image feature extractor to encode the character image and the prop image respectively, to obtain character image features corresponding to the character image and prop image features corresponding to the prop image.

[0119] The character image is encoded by an image feature extractor to obtain the character image features corresponding to the character image, and the prop image is encoded by an image feature extractor to obtain the prop image features corresponding to the prop image. The character image features are used to indicate the character features in the character image, and the prop image features are used to indicate the prop features in the prop image. The character image features and prop image features obtained here are spatial image features.

[0120] Step 712, stacking the character image features and the prop image features to obtain image features.

[0121] For example, reference may be made to Figure 8 Figure (1) in Figure 1 shows the process of generating image features based on character images and prop images. If the dimension of the character image feature is N×C×1×1 and the dimension of the prop feature is N×C×1×1, then the dimension of the image feature is N×C×H×W.

[0122] By extracting corresponding character features and prop features from character images and prop images respectively and stacking them, image features containing character features and prop features are obtained, so that the generated image features can indicate the specified character posture and specified prop posture required in the character information and prop information.

[0123] In some embodiments, the feature extractor is an image feature extractor for extracting features from an image, the character information is a character mask, and the prop information is a prop mask. Then step 710 includes at least one sub-step of steps 713 to 715 (not shown in the figure).

[0124] Step 713, obtaining a character mask image corresponding to the character mask according to the character mask, and obtaining a prop mask image corresponding to the prop mask according to the prop mask.

[0125] A person mask image refers to a binary image that uses different colors to distinguish between the person area and other areas except the person area. For example, the person area can be marked as white, and other areas except the person area can be marked as black. A prop mask image refers to a binary image that uses different colors to distinguish between the prop area and other areas except the prop area in the output image. For example, the prop area can be marked as white, and other areas except the prop area can be marked as black.

[0126] You can refer to Figure 8 Figure (3) in FIG. 1 shows the process of generating image features based on the person mask and the prop mask. The person mask image and the prop mask image are shown in FIG. Figure 8 As shown in Figure (3), the character mask image includes a character area represented by white and other areas except the character area represented by black, and the prop mask image includes a prop area represented by white and other areas except the prop area represented by black.

[0127] Step 714, using an image feature extractor to encode the character mask image and the prop mask image respectively, to obtain character mask features corresponding to the character mask image and prop mask features corresponding to the prop mask image.

[0128] An image feature extractor is used to encode the character mask image to obtain character mask features corresponding to the character mask image, and an image feature extractor is used to encode the prop mask image to obtain prop mask features corresponding to the prop image.

[0129] The person mask feature is used to indicate the person region in the person mask image, and the prop mask feature is used to indicate the prop region in the prop mask image. The person mask feature and prop mask feature obtained here are spatial image features.

[0130] Step 715, stacking the character mask features and the prop mask features to obtain image features.

[0131] You can refer to Figure 8 In Figure (3), after stacking the person mask features and prop mask features, we get the image features, which include the person posture features and prop posture features. If the dimension of the person image feature is N×C×1×1, and the dimension of the prop feature is N×C×1×1, then the dimension of the image feature is N×C×H×W.

[0132] Through the character mask and the prop mask, a character mask image and a prop mask image are obtained, thereby obtaining the corresponding character mask features and prop mask features respectively. After stacking, image features containing character posture features and prop posture features are obtained, so that the generated image features can indicate the specified character posture and specified prop posture required in the character information and prop information.

[0133] In some embodiments, the feature extractor is a text feature extractor for extracting features from text, the character information is a character tag, and the prop information is a prop tag. Then step 710 includes at least one sub-step of steps 716 to 717 (not shown in the figure).

[0134] Step 716, using a text feature extractor to encode the character label and the prop label respectively, to obtain the character text features corresponding to the character label and the prop text features corresponding to the prop label.

[0135] The character label is encoded by a text feature extractor to obtain the character text feature corresponding to the character label, and the prop label is encoded by a text feature extractor to obtain the prop text feature corresponding to the prop label. The character text feature is used to indicate the character feature in the character label, and the prop text feature is used to indicate the prop feature in the prop label. The character text feature and prop text feature obtained here are one-dimensional text features.

[0136] Step 717, stacking the character text features and the prop text features to obtain image features.

[0137] For example, reference may be made to Figure 8 Figure (2) in the figure shows the process of generating image features based on character labels and prop labels. Stacking character text features and prop text features is to stack one-dimensional text features into spatial image features. If the dimension of the character image feature is N×C×1×1, and the dimension of the prop feature is N×C×1×1, then the dimension of the image feature is N×C×H×W.

[0138] By extracting the corresponding character features and prop features from the character labels and prop labels respectively and stacking them, image features containing the character features and prop features are obtained, so that the generated image features can indicate the specified character posture and specified prop posture required in the character information and prop information.

[0139] Optionally, the character information and the prop information may also be different types of information, for example, the character information may be a character image, and the prop information may be a prop mask or a prop label, which is not limited in this application. If the character information is a character image, and the prop information is a prop label, then after obtaining the character image features corresponding to the character image, and obtaining the prop text features corresponding to the prop label, it is necessary to convert the prop text features into prop image features through a feature conversion network, so that the converted feature form is the same as the feature form of the character image feature.

[0140] Step 720: Encode the prompt text using a text encoder to obtain text features corresponding to the prompt text.

[0141] The text encoder can be any encoder with text encoding capability, which is used to encode the prompt text to obtain text features corresponding to the prompt text, and the text features are used to indicate the interactive relationship between the characters and props in the output image.

[0142] Step 730: Using the text features as input information of the stable diffusion model, and using the image features and the text features as input information of the control network.

[0143] Fig. 9 A schematic diagram showing the input situation of a stable diffusion model and control network is shown in Fig. 9 As shown, the input information of the stable diffusion model includes text features, and the input information of the control network includes image features and text features, that is, the input information of the control network includes text features, character features, and prop features. In addition, random noise will also be input into the stable diffusion model and the control network as input information. The stable diffusion model performs feature compression on the second input information after noise addition through the encoding block in the encoding network, and the control network performs feature compression on the first input information after noise addition through the encoding block. Then, the stable diffusion model performs feature interaction on the output information of the encoding network and the output information of the control network through the decoding block in the first decoding network, and gradually reconstructs the feature information after feature interaction during the feature interaction process to generate image information after feature interaction. Finally, the image information after feature interaction is decoded through the second decoding network to obtain the output image.

[0144] Figure 7 The illustrated embodiment provides a method for generating input information of a stable diffusion model and input information of a control network, so that text features are ultimately used as input information of the stable diffusion model, and image features and text features are used as input information of the control network.

[0145] Please refer to Fig.10, which shows a flowchart of a method for generating input information of another image generation model provided by an embodiment of the present application. The image generation model includes a stable diffusion model and a control network, and also includes a text encoder and a posture extraction network. The character information is a character image, and the prop information includes a prop image and a prop mask. The execution subject of each step of the method can be a computer device. The method can include at least one of the following steps 1010 to 1040:

[0146] Step 1010, obtaining a prop reverse mask according to the prop mask; the prop reverse mask is a binary mask after exchanging the binary numerical values ​​in the prop mask, and the prop reverse mask is used to indicate other areas in the output image except the prop area, and the image generation model is used to generate characters in the character image in other areas.

[0147] Exemplarily, if the prop mask is a binary mask composed of 0 and 1, wherein 1 is used to indicate the prop area in the output image, and 0 is used to indicate other areas in the output image except the prop area, then the prop anti-mask is also a binary mask composed of 0 and 1, wherein 0 is used to indicate the prop area in the output image, and 1 is used to indicate other areas in the output image except the prop area.

[0148] Specifically, the prop mask image corresponding to the prop mask is a binary image that represents the prop area with white and other areas except the prop area with black, and the prop reverse mask image corresponding to the prop reverse mask is a binary image that represents the prop area with black and other areas except the prop area with white.

[0149] The input information of the stable diffusion model includes a prop image, a prop mask and a prompt text, wherein the prop inverse mask obtained from the prop mask is used to indicate other areas in the output image except the prop area, so that the image generation model can generate a character image in the character image according to the prompt text and character information in other areas, and the prop image is used to generate a prop image in the prop area of ​​the output image.

[0150] Prop anti-mask and prop image can be referenced Fig.11 , wherein the prop anti-mask image corresponding to the prop anti-mask is a binary image in which black is used to represent the prop area, and white is used to represent other areas except the prop area.

[0151] Step 1020: extract the person's posture in the person image using a posture extraction network to obtain a person's posture image.

[0152] The posture extraction network is a neural network with the ability to extract character postures. Character postures are used to indicate character movements in character images, including hand movements, leg movements, body postures, and other body postures. The character postures in character images are extracted through the posture extraction network to obtain character posture images. Character posture images are used to indicate character postures in character images. For details, please refer to Fig.11 Character pose images in .

[0153] Step 1030: Encode the prompt text using a text encoder to obtain text features corresponding to the prompt text.

[0154] Step 1040, using the prop image, prop inverse mask and text features as input information of the stable diffusion model, and using the character posture image and text features as input information of the control network.

[0155] Fig.11 A schematic diagram showing another stable diffusion model and the input situation of the control network is shown in Fig.11 As shown, the input information of the stable diffusion model includes text features, props reverse masks and prop images, and the input information of the control network includes text features and character posture images, wherein the props reverse masks can be obtained through the prop masks, and the character posture images can be obtained by extracting the character images through the posture extraction network. In addition, random noise will also be input into the stable diffusion model and the control network as input information. The stable diffusion model performs feature compression on the second input information after noise addition through the encoding block in the encoding network, and the control network performs feature compression on the first input information after noise addition through the encoding block. Then, the stable diffusion model performs feature interaction on the output information of the encoding network and the output information of the control network through the decoding block in the first decoding network, and gradually reconstructs the feature information after feature interaction in the process of feature interaction to generate image information after feature interaction, and finally decodes the image information after feature interaction through the second decoding network to obtain the output image.

[0156] Fig.10 The illustrated embodiment provides another method for generating input information of a stable diffusion model and input information of a control network, so that ultimately the prop image, prop reverse mask and text features are used as input information of the stable diffusion model, and the character posture image and text features are used as input information of the control network.

[0157] Please refer to Fig.12 , which shows a flow chart of a training method for an image generation model provided by an embodiment of the present application. The image generation model includes a stable diffusion model and a control network. The execution subject of each step of the method can be a computer device. The method can include at least one of the following steps 1210 to 1260:

[0158] Step 1210, obtaining a training sample set of the image generation model, the training sample set including at least one image-text pair, each image-text pair including a label text and a label image having a matching relationship, the label text is used to indicate the interactive relationship between the character and the prop in the label image, and the label image is an image containing the character and the prop having an interactive relationship with the character.

[0159] The label text and the label image have a matching relationship, which means that the label text can indicate the interactive relationship between the character and the prop in the label image, and the label image is an image containing a character and at least one prop, wherein there is an interactive relationship between the prop and the character.

[0160] Step 1220, obtaining character information and prop information in the label image, wherein the character information is used to indicate the characteristics of the character, and the prop information is used to indicate the characteristics of the prop.

[0161] Before training the model, it is necessary to obtain the character information and prop information in the label image, that is, obtain the character image, prop image, character mask and prop mask.

[0162] In some embodiments, at least one of steps 1270 to 1280 (not shown in the figure) is included before step 1220 .

[0163] Step 1270, use an image segmentation network to perform region recognition on the person area and the prop area in the label image respectively, and obtain a person mask and a prop mask. The person mask is a binary mask that uses different values ​​to distinguish the person area in the label image from other areas except the person area, and the prop mask is a binary mask that uses different values ​​to distinguish the prop area in the label image from other areas except the prop area.

[0164] An image segmentation network is used to perform region recognition on the person region in the label image, and the person region and other regions except the person region in the label image are obtained, thereby generating a person mask based on the person region and other regions except the person region. An image segmentation network is used to perform region recognition on the prop region in the label image, and the prop region and other regions except the prop region in the label image are obtained, thereby generating a prop mask based on the prop region and other regions except the prop region.

[0165] Step 1280, obtaining a character image according to the character mask and the label image, and obtaining a prop image according to the prop mask and the label image.

[0166] According to the person mask, the person area and other areas except the person area in the label image can be obtained. According to the person area and other areas except the person area, other areas except the person area in the label image are uniformly marked as a value, for example, they can be uniformly marked as 0 to obtain a person image.

[0167] According to the prop mask, the prop area and other areas except the prop area in the label image can be obtained. According to the prop area and other areas except the prop area, other areas except the prop area in the label image are uniformly marked as a value, for example, they can be uniformly marked as 0 to obtain the prop image.

[0168] By segmenting the label image, we can obtain the character image, prop image, character mask and prop mask, so as to obtain the input information of the image generation model.

[0169] Step 1230, generating input information of a stable diffusion model and input information of a control network according to the character information, the prop information and the label text.

[0170] In some embodiments, the image generation model also includes a text encoder and a feature extractor; the generation process of the input information of the stable diffusion model and the input information of the control network is: through the feature extractor, the image features are obtained according to the character information and props information; the text encoder is used to encode the label text to obtain the text features corresponding to the label text; the text features are used as the input information of the stable diffusion model, and the image features and text features are used as the input information of the control network.

[0171] In some embodiments, the feature extractor is an image feature extractor for extracting features from an image, the character information is a character image, and the prop information is a prop image; the image feature extractor is used to encode the character image and the prop image respectively to obtain character image features corresponding to the character image and prop image features corresponding to the prop image; the character image features and the prop image features are stacked to obtain image features.

[0172] In some embodiments, the feature extractor is an image feature extractor for extracting features from an image, the character information is a character mask, and the prop information is a prop mask; based on the character mask, a character mask image corresponding to the character mask is obtained, and based on the prop mask, a prop mask image corresponding to the prop mask is obtained; the image feature extractor is used to encode the character mask image and the prop mask image respectively to obtain character mask features corresponding to the character mask image and prop mask features corresponding to the prop mask image; the character mask features and the prop mask features are stacked to obtain image features.

[0173] In some embodiments, the feature extractor is a text feature extractor for extracting features from text, the character information is a character label, and the prop information is a prop label. The character label is text information for indicating the character features in the label image, and the prop label is text information for indicating the prop features in the label image; the text feature extractor is used to encode the character label and the prop label respectively to obtain the character text features corresponding to the character label and the prop text features corresponding to the prop label; the character text features and the prop text features are stacked to obtain image features.

[0174] The training effect of training the model by using the feature extractor to generate image features can be referred to Fig.13 As shown in Figure (1), the training effect of the image encoding feature driven by using an image feature extractor to generate image features is different from the training effect of the text encoding feature driven by using a text feature extractor to generate image features. The prop poses and character poses of the generated images of both meet the input character conditions and prop conditions, but there are differences in details. Therefore, users can choose the corresponding encoding feature driving method according to actual image generation requirements.

[0175] In some embodiments, the image generation model also includes a text encoder and a posture extraction network; the character information is a character image, and the prop information includes a prop image and a prop mask; then the generation process of the input information of the stable diffusion model and the input information of the control network is: according to the prop mask, a prop reverse mask is obtained; the prop reverse mask is a binary mask after the binary numerical value in the prop mask is exchanged, and the prop reverse mask is used to indicate other areas in the predicted image except the prop area, and the image generation model is used to generate characters in the character image in other areas; the posture extraction network is used to extract the character posture in the character image to obtain a character posture image; the text encoder is used to encode the label text to obtain the text features corresponding to the label text; the prop image, prop reverse mask and text features are used as input information of the stable diffusion model, and the character posture image and text features are used as input information of the control network.

[0176] The training effect of using human posture images as input information for the control network can be referenced Fig.13 As shown in Figure (2), the generated image before adjustment refers to the output effect of the image generated by only using the stable diffusion model, and the generated image after adjustment refers to the generated effect of the image generated by using the stable diffusion model and the control network. It can be seen that the images generated by using the stable diffusion model and the control network have certain improvements in the details of the characters and props, and the posture, position, size and texture of the props have been finely controlled, which improves the image generation effect.

[0177] Fig.14A schematic diagram comparing the effects of generating images using one and multiple prop conditions is shown. It can be seen that the image generation model provided in the present application can realize the joint control generation of multiple objectives, improve the richness of the generated images, and thus enhance the image generation effect.

[0178] The process of generating the input information of the stable diffusion model and the output information of the control network in step 1230 can be referred to the above embodiment and will not be described in detail here.

[0179] Step 1240, generating output information of the control network according to input information of the control network through the control network.

[0180] In some embodiments, random noise is added to the label image to obtain a label noisy image; first input information is obtained based on the input information of the control network and the label noisy image; and the control network is used to perform feature compression on the first input information to obtain output information of the control network.

[0181] The labeled noisy image is used to indicate image features of the predicted image, such as image scale, image style, image type, and image scene, so that the generated predicted image can meet the image features of the labeled noisy image.

[0182] Step 1250, generating a predicted image through a stable diffusion model according to input information of the stable diffusion model and output information of the control network.

[0183] In some embodiments, the stable diffusion model includes an encoding network, a first decoding network, and a second decoding network; random noise is added to the label image to obtain a label noisy image; second input information is obtained based on the input information of the stable diffusion model and the label noisy image; the encoding network is used to perform feature compression on the second input information to obtain third input information; through the first decoding network, image information after feature interaction is obtained based on the third input information and the output information of the control network; the second decoding network is used to decode the image information after feature interaction to obtain a predicted image.

[0184] In some embodiments, the structure of the control network is the same as the structure of the encoding network.

[0185] Step 1260, adjusting the parameters of the control network according to the predicted image and the label image to obtain a trained image generation model.

[0186] In some embodiments, according to the denoising process of the encoding network and the first decoding network, the predicted noise corresponding to the predicted image is obtained; according to the difference between the random noise and the predicted noise, the loss function value is calculated; according to the loss function value, the parameters of the control network are adjusted to obtain the trained image generation model.

[0187] The encoding network and the first decoding network are used to implement the denoising process of the stable diffusion model. According to the denoising process of the encoding network and the first decoding network, the noise of each denoising in the encoding network and the first decoding network can be obtained, so that the noise removed in the denoising process can be obtained, that is, the prediction noise that needs to be removed to generate a predicted image.

[0188] According to the difference between the random noise and the predicted noise, the loss function value is calculated. For example, the difference between the random noise and the predicted noise can be calculated using MSE (Mean Squared Error) loss, and the loss function value can be expressed as the following formula.

[0189]

[0190] Among them, ∈ represents random noise, represents the prediction noise.

[0191] According to the loss function value The parameters of the control network are adjusted. Optionally, the training condition of the loss function value may be that the loss function value is less than a set threshold, or that the loss function value is within a set threshold range, etc., which is not limited in the present application.

[0192] The technical solution provided in the embodiment of the present application adopts an image segmentation network to obtain the person area and other areas except the person area, as well as the prop area and other areas except the prop area in the label image, to obtain the person mask and the prop mask, thereby obtaining the person image and the prop image, and based on the person image, the prop image, the person mask and the prop mask, generates the input information of the stable diffusion model and the input information of the control network, so that the training data of the image generation model are all derived from the label image, which can ensure the accuracy and effectiveness of the input information, thereby improving the image generation effect.

[0193] The image generation method based on the image generation model and the training method of the image generation model provided in the embodiment of the present application are the model training process and the use process corresponding to each other. For details not described in detail on one side, please refer to the introduction on the other side.

[0194] The image generation method based on the image generation model provided in the embodiment of the present application can be used to generate promotional images for game characters, can also be used to generate promotional images for animation characters, can also be used to generate images of any virtual characters that have an interactive relationship with props, and can also be used in other scenarios without limitation and other expandable descriptions.

[0195] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0196] Please refer to Fig.15 , which shows a block diagram of an image generation device based on an image generation model provided by an embodiment of the present application. The device has the function of implementing the above-mentioned image generation method based on the image generation model, and the function can be implemented by hardware, or by hardware executing corresponding software. The device can be the computer device introduced above, or it can be set in a computer device. Fig.15 As shown, the device 1500 may include: an information acquisition module 1510 , an input generation module 1520 , a control output module 1530 and an image output module 1540 .

[0197] The information acquisition module 1510 is used to obtain character information, prop information and prompt text, wherein the character information is used to indicate the characteristics that the character in the output image to be generated must have, the prop information is used to indicate the characteristics that the prop in the output image must have, and the prompt text is used to indicate the interactive relationship between the character and the prop in the output image.

[0198] The input generation module 1520 is used to generate input information of the stable diffusion model and input information of the control network according to the character information, the prop information and the prompt text.

[0199] The control output module 1530 is used to generate output information of the control network according to input information of the control network through the control network.

[0200] The image output module 1540 is used to generate the output image through the stable diffusion model according to the input information of the stable diffusion model and the output information of the control network.

[0201] In some embodiments, the image generation model further includes a text encoder and a feature extractor; the input generation module 1520 includes:

[0202] The image feature generating unit is used to obtain image features according to the character information and the prop information through the feature extractor.

[0203] The text feature generating unit is used to encode the prompt text by using the text encoder to obtain the text feature corresponding to the prompt text.

[0204] An input information generating unit is used to use the text feature as input information of the stable diffusion model, and use the image feature and the text feature as input information of the control network.

[0205] In some embodiments, the feature extractor is an image feature extractor for extracting features from an image, the character information is a character image, and the prop information is a prop image; the image feature generating unit is used to:

[0206] The image feature extractor is used to encode the character image and the prop image respectively to obtain character image features corresponding to the character image and prop image features corresponding to the prop image;

[0207] The character image feature and the prop image feature are stacked to obtain the image feature.

[0208] In some embodiments, the feature extractor is an image feature extractor for extracting features from an image, the character information is a character mask, the prop information is a prop mask, the character mask is a binary mask for distinguishing a character area in the output image from other areas except the character area by using different values, and the prop mask is a binary mask for distinguishing a prop area in the output image from other areas except the prop area by using different values; the image feature generating unit is used to:

[0209] According to the character mask, a character mask image corresponding to the character mask is obtained, and according to the prop mask, a prop mask image corresponding to the prop mask is obtained;

[0210] The image feature extractor is used to encode the character mask image and the prop mask image respectively to obtain character mask features corresponding to the character mask image and prop mask features corresponding to the prop mask image;

[0211] The character mask feature and the prop mask feature are stacked to obtain the image feature.

[0212] In some embodiments, the feature extractor is a text feature extractor for extracting features from text, the character information is a character label, the prop information is a prop label, the character label is text information for indicating a character feature in the output image, and the prop label is text information for indicating a prop feature in the output image; the image feature generation unit is used to:

[0213] The text feature extractor is used to encode the character label and the prop label respectively to obtain the character text feature corresponding to the character label and the prop text feature corresponding to the prop label;

[0214] The character text feature and the prop text feature are stacked to obtain the image feature.

[0215] In some embodiments, the image generation model further includes a text encoder and a posture extraction network; the character information is a character image, and the prop information includes a prop image and a prop mask; the input generation module 1520 is used to:

[0216] According to the prop mask, a prop inverse mask is obtained; the prop inverse mask is a binary mask after the binary numerical values ​​in the prop mask are exchanged and represented, the prop inverse mask is used to indicate other areas in the output image except the prop area, and the image generation model is used to generate the character in the character image in the other areas;

[0217] Using the posture extraction network to extract the posture of the person in the person image to obtain a person posture image;

[0218] Encoding the prompt text using the text encoder to obtain text features corresponding to the prompt text;

[0219] The prop image, the prop inverse mask and the text feature are used as input information of the stable diffusion model, and the character posture image and the text feature are used as input information of the control network.

[0220] In some embodiments, the control output module 1530 is used to:

[0221] Adding random noise to the input information of the control network to obtain first input information;

[0222] The control network is used to perform feature compression on the first input information to obtain output information of the control network.

[0223] In some embodiments, the stable diffusion model includes an encoding network, a first decoding network, and a second decoding network; the image output module 1540 is used to:

[0224] Adding random noise to the input information of the stable diffusion model to obtain second input information;

[0225] Using the encoding network to perform feature compression on the second input information to obtain third input information;

[0226] Obtaining image information after feature interaction according to the third input information and the output information of the control network through the first decoding network;

[0227] The second decoding network is used to decode the image information after the feature interaction to obtain the output image.

[0228] In some embodiments, the structure of the control network is the same as the structure of the encoding network.

[0229] The technical solution provided in the embodiment of the present application uses character information, prop information and prompt text as input information of the image generation model, and generates input information of the stable diffusion model and input information of the control network based on the character information, prop information and prompt text, thereby combining the input information of the control network with the input information of the stable diffusion model as a control condition, so that the text features in the prompt text and the character features in the character information can flexibly interact with the prop features in the prop information, thereby generating an output image containing a specified character posture and a specified prop posture. Compared with the related art that generates an image based on a character posture image and a prompt text but cannot control the generation effect of the props therein, the technical solution provided in the present application can realize the joint control generation of multiple targets, and achieve refined control over the posture, position, size and texture of the props, thereby improving the generation effect and generation quality of the output image.

[0230] Please refer to Fig.16 , which shows a block diagram of a training device for an image generation model provided by an embodiment of the present application. The device has the function of implementing the training method for the above-mentioned image generation model, and the function can be implemented by hardware, or by hardware executing corresponding software. The device can be the computer device described above, or it can be set in a computer device. Fig.16 As shown, the device 1600 may include: a sample set acquisition module 1610, an information acquisition module 1620, an input generation module 1630, a control output module 1640, an image output module 1650 and a model training module 1660.

[0231] The sample set acquisition module 1610 is used to acquire a training sample set for the image generation model, wherein the training sample set includes at least one image-text pair, each of which includes a label text and a label image with a matching relationship, wherein the label text is used to indicate the interactive relationship between the character and the prop in the label image, and the label image is an image containing the character and the prop that has an interactive relationship with the character.

[0232] The information acquisition module 1620 is used to acquire character information and prop information in the label image, wherein the character information is used to indicate the characteristics of the character, and the prop information is used to indicate the characteristics of the prop.

[0233] The input generation module 1630 is used to generate input information of the stable diffusion model and input information of the control network according to the character information, the prop information and the label text.

[0234] The control output module 1640 is used to generate output information of the control network according to input information of the control network through the control network.

[0235] The image output module 1650 is used to generate a predicted image through the stable diffusion model according to the input information of the stable diffusion model and the output information of the control network.

[0236] The model training module 1660 is used to adjust the parameters of the control network according to the predicted image and the label image to obtain a trained image generation model.

[0237] In some embodiments, the apparatus 1600 further includes an image segmentation module, wherein the image segmentation module is configured to:

[0238] An image segmentation network is used to perform region recognition on a person region and a prop region in the label image, respectively, to obtain a person mask and a prop mask, wherein the person mask is a binary mask that uses different values ​​to distinguish between a person region and other regions except the person region in the label image, and the prop mask is a binary mask that uses different values ​​to distinguish between a prop region and other regions except the prop region in the label image;

[0239] The character image is obtained according to the character mask and the label image, and the prop image is obtained according to the prop mask and the label image.

[0240] In some embodiments, the control output module 1640 is used to:

[0241] Obtaining first input information according to the input information of the control network and the labeled noisy image;

[0242] The control network is used to perform feature compression on the first input information to obtain output information of the control network.

[0243] In some embodiments, the stable diffusion model includes an encoding network, a first decoding network, and a second decoding network; the image output module 1650 is used to:

[0244] Adding random noise to the label image to obtain a label noisy image;

[0245] Obtaining second input information according to the input information of the stable diffusion model and the labeled noisy image;

[0246] Using the encoding network to perform feature compression on the second input information to obtain third input information;

[0247] Obtaining image information after feature interaction according to the third input information and the output information of the control network through the first decoding network;

[0248] The second decoding network is used to decode the image information after the feature interaction to obtain the output image.

[0249] In some embodiments, the model training module 1660 is used to:

[0250] According to the denoising process of the encoding network and the first decoding network, the prediction noise of the predicted image for Jiuyin is obtained;

[0251] Calculating a loss function value according to the random noise and the predicted noise;

[0252] The parameters of the control network are adjusted according to the loss function value to obtain the trained image generation model.

[0253] The technical solution provided in the embodiment of the present application adopts an image segmentation network to obtain the person area and other areas except the person area, as well as the prop area and other areas except the prop area in the label image, to obtain the person mask and the prop mask, thereby obtaining the person image and the prop image, and based on the person image, the prop image, the person mask and the prop mask, generates the input information of the stable diffusion model and the input information of the control network, so that the training data of the image generation model are all derived from the label image, which can ensure the accuracy and effectiveness of the input information, thereby improving the image generation effect.

[0254] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0255] Please refer to Fig.17 , which shows a block diagram of a computer device 1700 provided in one embodiment of the present application. The computer device 1700 may be any electronic device having data calculation, processing and storage functions. The computer device 1700 may be used to implement the image generation method based on the image generation model or the training method of the image generation model provided in the above embodiments.

[0256] Typically, the computer device 1700 includes a processor 1701 and a memory 1702 .

[0257] The processor 1701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1701 may also include an AI processor for processing computing operations related to machine learning.

[0258] The memory 1702 may include one or more computer-readable storage media, which may be non-transitory. The memory 1702 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1702 is used to store a computer program, which is configured to be executed by one or more processors to implement the above-mentioned image generation method based on the image generation model or the training method of the image generation model.

[0259] Those skilled in the art will understand that Fig.17 The structure shown in the figure does not constitute a limitation on the computer device 1700, and the computer device 1700 may include more or less components than shown in the figure, or combine some components, or adopt a different arrangement of components.

[0260] In an illustrative embodiment, a computer-readable storage medium is also provided, in which a computer program is stored, and when the computer program is executed by a processor of a computer device, the image generation method based on the image generation model or the training method of the image generation model is implemented. Optionally, the computer-readable storage medium can be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0261] In an exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned image generation method based on the image generation model or the training method of the image generation model.

[0262] It should be noted that this application can display a prompt interface, pop-up window or output voice prompt information before collecting relevant data of users and during the process of collecting relevant data of users. The prompt interface, pop-up window or voice prompt information is used to prompt the user that relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining relevant data of users after obtaining the confirmation operation issued by the user to the prompt interface or pop-up window, otherwise (that is, when the confirmation operation issued by the user to the prompt interface or pop-up window is not obtained), the relevant steps of obtaining relevant data of users are terminated, that is, the relevant data of users are not obtained. In other words, all user data collected by this application are processed strictly in accordance with the requirements of relevant national laws and regulations, and the informed consent or separate consent of the subject of personal information is obtained with the consent and authorization of the user. The subsequent data use and processing behavior is carried out within the scope of authorization of laws and regulations and the subject of personal information, and the collection, use and processing of relevant user data shall comply with the relevant laws, regulations and standards of relevant countries and regions.

[0263] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0264] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image generation method based on an image generation model, characterized in that: The image generation model includes a stable diffusion model and a control network, and the method includes: Acquire character information, prop information, and prompt text, wherein the character information is used to indicate the features that the character in the output image to be generated should have, the prop information is used to indicate the features that the prop in the output image should have, and the prompt text is used to indicate the interactive relationship between the character and the prop in the output image; Generating input information of the stable diffusion model and input information of the control network according to the character information, the item information and the prompt text; generating output information of the control network according to input information of the control network through the control network; The output image is generated by the stable diffusion model according to input information of the stable diffusion model and output information of the control network.

2. The method according to claim 1, characterized in that The image generation model also includes a text encoder and a feature extractor; The step of generating input information of the stable diffusion model and input information of the control network according to the character information, the item information and the prompt text includes: Obtaining image features according to the character information and the prop information through the feature extractor; Encoding the prompt text using the text encoder to obtain text features corresponding to the prompt text; The text features are used as input information of the stable diffusion model, and the image features and the text features are used as input information of the control network.

3. The method according to claim 2, characterized in that The feature extractor is an image feature extractor for extracting features from an image, the character information is a character image, and the prop information is a prop image; The step of obtaining image features according to the character information and the prop information by the feature extractor includes: The image feature extractor is used to encode the character image and the prop image respectively to obtain character image features corresponding to the character image and prop image features corresponding to the prop image; The character image feature and the prop image feature are stacked to obtain the image feature.

4. The method according to claim 2, characterized in that: The feature extractor is an image feature extractor for extracting features from an image, the character information is a character mask, the prop information is a prop mask, the character mask is a binary mask for distinguishing a character region in the output image from other regions except the character region by using different values, and the prop mask is a binary mask for distinguishing a prop region in the output image from other regions except the prop region by using different values; The step of obtaining image features according to the character information and the prop information by the feature extractor includes: According to the character mask, a character mask image corresponding to the character mask is obtained, and according to the prop mask, a prop mask image corresponding to the prop mask is obtained; The image feature extractor is used to encode the character mask image and the prop mask image respectively to obtain character mask features corresponding to the character mask image and prop mask features corresponding to the prop mask image; The character mask feature and the prop mask feature are stacked to obtain the image feature.

5. The method according to claim 2, characterized in that: The feature extractor is a text feature extractor for extracting features from text, the character information is a character tag, the prop information is a prop tag, the character tag is text information for indicating a character feature in the output image, and the prop tag is text information for indicating a prop feature in the output image; The step of obtaining image features according to the character information and the prop information by the feature extractor includes: The text feature extractor is used to encode the character label and the prop label respectively to obtain the character text feature corresponding to the character label and the prop text feature corresponding to the prop label; The character text feature and the prop text feature are stacked to obtain the image feature.

6. The method according to claim 1, characterized in that The image generation model further includes a text encoder and a posture extraction network; the character information is a character image, and the prop information includes a prop image and a prop mask; The step of generating input information of the stable diffusion model and input information of the control network according to the character information, the item information and the prompt text includes: According to the prop mask, a prop inverse mask is obtained; the prop inverse mask is a binary mask after the binary numerical values ​​in the prop mask are exchanged and represented, the prop inverse mask is used to indicate other areas in the output image except the prop area, and the image generation model is used to generate the character in the character image in the other areas; Using the posture extraction network to extract the posture of the person in the person image to obtain a person posture image; Encoding the prompt text using the text encoder to obtain text features corresponding to the prompt text; The prop image, the prop inverse mask and the text feature are used as input information of the stable diffusion model, and the character posture image and the text feature are used as input information of the control network.

7. The method according to claim 1, characterized in that The step of generating output information of the control network according to input information of the control network through the control network includes: Adding random noise to the input information of the control network to obtain first input information; The control network is used to perform feature compression on the first input information to obtain output information of the control network.

8. The method according to claim 1, characterized in that The stable diffusion model includes an encoding network, a first decoding network and a second decoding network; The step of generating the output image by using the stable diffusion model according to input information of the stable diffusion model and output information of the control network includes: Adding random noise to the input information of the stable diffusion model to obtain second input information; Using the encoding network to perform feature compression on the second input information to obtain third input information; Obtaining image information after feature interaction according to the third input information and the output information of the control network through the first decoding network; The second decoding network is used to decode the image information after the feature interaction to obtain the output image.

9. The method according to claim 8, characterized in that The structure of the control network is the same as that of the encoding network.

10. A training method for an image generation model, characterized in that: The image generation model includes a stable diffusion model and a control network, and the method includes: Acquire a training sample set of the image generation model, wherein the training sample set includes at least one image-text pair, each of the image-text pairs includes a label text and a label image having a matching relationship, the label text is used to indicate an interactive relationship between a character and a prop in the label image, and the label image is an image containing the character and the prop having an interactive relationship with the character; Acquire character information and prop information in the label image, wherein the character information is used to indicate the characteristics of the character, and the prop information is used to indicate the characteristics of the prop; Generating input information of the stable diffusion model and input information of the control network according to the character information, the prop information and the label text; generating output information of the control network according to input information of the control network through the control network; Generate a prediction image through the stable diffusion model according to input information of the stable diffusion model and output information of the control network; According to the predicted image and the label image, the parameters of the control network are adjusted to obtain a trained image generation model.

11. The method according to claim 10, characterized in that Before obtaining the character information and prop information in the label image, the method further includes: An image segmentation network is used to perform region recognition on a person region and a prop region in the label image, respectively, to obtain a person mask and a prop mask, wherein the person mask is a binary mask that uses different values ​​to distinguish between a person region and other regions except the person region in the label image, and the prop mask is a binary mask that uses different values ​​to distinguish between a prop region and other regions except the prop region in the label image; The character image is obtained according to the character mask and the label image, and the prop image is obtained according to the prop mask and the label image.

12. The method according to claim 10, characterized in that The step of generating output information of the control network according to input information of the control network through the control network includes: Adding random noise to the label image to obtain a label noisy image; Obtaining first input information according to the input information of the control network and the labeled noisy image; The control network is used to perform feature compression on the first input information to obtain output information of the control network.

13. The method according to claim 10, characterized in that The stable diffusion model includes an encoding network, a first decoding network and a second decoding network; The step of generating a predicted image by using the stable diffusion model according to input information of the stable diffusion model and output information of the control network includes: Adding random noise to the label image to obtain a label noisy image; Obtaining second input information according to the input information of the stable diffusion model and the labeled noisy image; Using the encoding network to perform feature compression on the second input information to obtain third input information; Obtaining image information after feature interaction according to the third input information and the output information of the control network through the first decoding network; The second decoding network is used to decode the image information after the feature interaction to obtain the output image.

14. The method according to any one of claims 10 to 13, characterized in that The step of adjusting the parameters of the control network according to the predicted image and the label image to obtain a trained image generation model comprises: Obtaining prediction noise corresponding to the predicted image according to the denoising process of the encoding network and the first decoding network; Calculating a loss function value according to a difference between the random noise and the predicted noise; The parameters of the control network are adjusted according to the loss function value to obtain the trained image generation model.

15. An image generation device based on an image generation model, characterized in that: The image generation model includes a stable diffusion model and a control network, and the device includes: An information acquisition module, used to acquire character information, prop information and prompt text, wherein the character information is used to indicate the characteristics that the character in the output image to be generated should have, the prop information is used to indicate the characteristics that the prop in the output image should have, and the prompt text is used to indicate the interactive relationship between the character and the prop in the output image; An input generation module, used to generate input information of the stable diffusion model and input information of the control network according to the character information, the prop information and the prompt text; A control output module, used to generate output information of the control network according to input information of the control network through the control network; The image output module is used to generate the output image through the stable diffusion model according to the input information of the stable diffusion model and the output information of the control network.

16. A training device for an image generation model, characterized in that: The image generation model includes a stable diffusion model and a control network, and the device includes: A sample set acquisition module, used to acquire a training sample set of the image generation model, wherein the training sample set includes at least one image-text pair, each of which includes a label text and a label image having a matching relationship, wherein the label text is used to indicate an interactive relationship between a character and a prop in the label image, and the label image is an image containing the character and the prop having an interactive relationship with the character; An information acquisition module, used to acquire character information and prop information in the label image, wherein the character information is used to indicate the characteristics of the character, and the prop information is used to indicate the characteristics of the prop; An input generation module, used to generate input information of the stable diffusion model and input information of the control network according to the character information, the prop information and the label text; A control output module, used to generate output information of the control network according to input information of the control network through the control network; An image output module, used to generate a predicted image through the stable diffusion model according to input information of the stable diffusion model and output information of the control network; The model training module is used to adjust the parameters of the control network according to the predicted image and the label image to obtain a trained image generation model.

17. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the image generation method based on the image generation model as described in any one of claims 1 to 9, or to implement the training method of the image generation model as described in any one of claims 10 to 14.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the image generation method based on the image generation model as described in any one of claims 1 to 9, or to implement the training method of the image generation model as described in any one of claims 10 to 14.

19. A computer program product, characterized in that The computer program product includes a computer program, which is loaded and executed by a processor to implement the image generation method based on the image generation model as described in any one of claims 1 to 9, or to implement the training method of the image generation model as described in any one of claims 10 to 14.

Images