Image synthesis model construction method, image generation method and device

Abstract

The invention provides an image synthesis model construction method and an image generation method and device, relates to the technical field of artificial intelligence such as computer vision, deep learning and large models, and can be applied to scenes such as AIGC content generation based on artificial intelligence. According to the specific implementation scheme, the method comprises the steps of obtaining a sample image comprising a plurality of figures, and obtaining a sample prompt word corresponding to the sample image; extracting figure related features of each figure from the sample image through a first processing network, wherein the figure related features comprise an appearance feature map, a facial feature map and a description text vector; converting the sample cue word into a cue text vector through a second processing network; and training a to-be-trained multi-modal diffusion network based on the sample image, the character related features and the prompt text vector to obtain a target multi-modal diffusion network, and constructing an image synthesis model based on the target multi-modal diffusion network.

Description

Technical Field

[0001] The present disclosure relates to artificial intelligence technology fields such as computer vision, deep learning, and large models, and can be applied to scenarios such as AIGC content generation based on artificial intelligence. Specifically, it relates to a method for constructing an image synthesis model, an image generation method, and a device thereof. Background Art

[0002] Image synthesis technology is an important direction in the field of computer vision and image processing, and is widely used in many industries such as entertainment and advertising. By combining images from different sources or with different contents, synthesis technology can create new visual effects. With the development of computer image processing technology, image synthesis technology has become an indispensable technical tool in many fields. Summary of the invention

[0003] The present disclosure provides a method for constructing an image synthesis model, an image generation method, an apparatus, a device, and a storage medium.

[0004] According to one aspect of the present disclosure, a method for constructing an image synthesis model is provided, comprising: obtaining a sample image including a plurality of characters, and obtaining a sample prompt word corresponding to the sample image; extracting character-related features of each character from the sample image through a first processing network, the character-related features including an appearance feature map, a facial feature map and a description text vector; converting the sample prompt word into a prompt text vector through a second processing network; training a multimodal diffusion network to be trained based on the sample image, the character-related features and the prompt text vector to obtain a target multimodal diffusion network, and constructing an image synthesis model based on the target multimodal diffusion network.

[0005] The present application trains a multimodal diffusion network in combination with an appearance feature map, a facial feature map, a description text vector, and a prompt text vector, so that the trained target multimodal diffusion network can understand the structure and semantic information of a person at a higher level, thereby enabling an image synthesis model constructed based on the target multimodal diffusion network to generate a more natural and realistic synthesis effect when processing image synthesis tasks, thereby improving user satisfaction.

[0006] According to another aspect of the present disclosure, there is provided an image generation method, comprising: obtaining a plurality of input images input by a user, wherein the input images include a target object; obtaining a target prompt word input by the user; inputting the plurality of input images and the target prompt word into an image synthesis model to obtain a target image output by the image synthesis model; wherein the image synthesis model is constructed and generated using the image synthesis model construction method introduced above.

[0007] According to another aspect of the present disclosure, a device for constructing an image synthesis model is provided, comprising: an acquisition module, used to acquire sample images including multiple characters, and to acquire sample prompt words corresponding to the sample images; an extraction module, used to extract character-related features of each character from the sample images through a first processing network, the character-related features including an appearance feature map, a facial feature map and a description text vector; a conversion module, used to convert the sample prompt words into prompt text vectors through a second processing network; and a training module, used to train a multimodal diffusion network to be trained based on the sample images, the character-related features and the prompt text vectors to obtain a target multimodal diffusion network, and to construct an image synthesis model based on the target multimodal diffusion network.

[0008] According to another aspect of the present disclosure, there is provided an image generating device, comprising: a first acquiring module, used to acquire multiple input images input by a user, wherein the input images include a target object; a second acquiring module, used to acquire a target prompt word input by the user; and a generating module, used to input the multiple input images and the target prompt word into an image synthesis model to obtain a target image output by the image synthesis model; wherein the image synthesis model is constructed and generated using the image synthesis model construction device introduced above.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned image synthesis model construction method or image generation method.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned image synthesis model construction method or image generation method.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the above-mentioned image synthesis model construction method or image generation method when executed by a processor.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1It is a schematic diagram of an exemplary implementation of a method for constructing an image synthesis model according to an exemplary embodiment of the present disclosure.

[0015] Figure 2 It is a schematic diagram of an exemplary implementation of a method for constructing an image synthesis model according to an exemplary embodiment of the present disclosure.

[0016] Figure 3 It is a schematic diagram of an exemplary implementation of a method for constructing an image synthesis model according to an exemplary embodiment of the present disclosure.

[0017] Figure 4 It is a simplified schematic diagram of a certain layer in a multimodal diffusion network according to an exemplary embodiment of the present disclosure.

[0018] Figure 5 It is a structural diagram of an image synthesis model according to an exemplary embodiment of the present disclosure.

[0019] Figure 6 is a schematic diagram of an exemplary implementation of an image generating method according to an exemplary embodiment of the present disclosure.

[0020] Figure 7 is a schematic diagram of a device for constructing an image synthesis model according to an exemplary embodiment of the present disclosure.

[0021] Figure 8 is an exemplary schematic diagram of an image generating device according to an exemplary embodiment of the present disclosure.

[0022] Fig. 9 is a schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] Image processing is a technology that uses computers to analyze images to achieve the desired results. It is also called image processing. Image processing generally refers to digital image processing. A digital image is a large two-dimensional array obtained by shooting with industrial cameras, video cameras, scanners and other equipment. The elements of the array are called pixels, and their values ​​are called grayscale values. Image processing technology generally includes three parts: image compression, enhancement and restoration, matching, description and recognition.

[0025] Deep learning (DL) is a new research direction in the field of machine learning (ML). It is introduced into machine learning to make it closer to its original goal - artificial intelligence. Deep learning is the internal laws and representation levels of learning sample data. The information obtained in the learning process is of great help in the interpretation of data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition that far exceed previous related technologies.

[0026] Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform image processing so that the computer processes the images into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain "information" from images or multidimensional data. The information here refers to information defined by Shannon that can be used to help make a "decision". Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data.

[0027] Artificial Intelligence (AI) is a discipline that studies how computers can simulate certain thought processes and intelligent behaviors of human beings (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level technologies and software-level technologies. AI hardware technologies generally include computer vision technology, speech recognition technology, natural language processing technology, as well as learning / deep learning, big data processing technology, knowledge graph technology, and other aspects.

[0028] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0029] Figure 1 is a schematic diagram of an exemplary implementation of a method for constructing an image synthesis model shown in the present application, such as Figure 1 As shown, the method for constructing the image synthesis model includes the following steps:

[0030] S101, obtaining sample images including multiple persons, and obtaining sample prompt words corresponding to the sample images.

[0031] Each sample image includes multiple characters. For example, a sample image may include three characters.

[0032] In some embodiments, the sample prompt word corresponding to each sample image can be a natural language description of the sample image as a whole, such as "3 women in red dresses, standing on the beach, smiling", or a shorter prompt word, such as "3 women, smiling, beach".

[0033] In some embodiments, the sample prompt word corresponding to each sample image can be generated based on the background information of the sample image, for example, "daytime, beach, sun, gravel, stone".

[0034] S102, extracting character-related features of each character from the sample image through a first processing network, where the character-related features include an appearance feature map, a facial feature map, and a description text vector.

[0035] It is not difficult to understand that since the above sample image includes multiple characters, in order to facilitate the subsequent multimodal diffusion network to be trained to accurately perform multimodal constraints based on the relevant features of each character, it is necessary to first obtain the character-related features corresponding to each character in the sample image.

[0036] Among them, each person in the sample image corresponds to a group of person-related features, and each group of person-related features includes an appearance feature map, a facial feature map and a description text vector corresponding to the person.

[0037] Among them, the appearance feature map is used to represent the appearance information of the character, such as body posture, movement and other information.

[0038] Among them, the facial feature map is used to represent the features of a person's face, such as facial expressions, eyes, mouth, nose, etc.

[0039] Among them, the description text vector is used to represent the overall description information of the character, such as clothing style, color, emotion, etc.

[0040] S103, converting the sample prompt word into a prompt text vector through a second processing network.

[0041] The second processing network is a text encoder, which is used to convert sample prompt words into prompt text vectors.

[0042] Optionally, the text encoder in the second processing network can use a pre-trained language model, such as a bidirectional encoder representation model (Bidirectional Encoder Representations from Transformers, BERT), a generative pretrained transformer (Generative Pretrained Transformer, GPT), a contrastive language-image pretraining (Contrastive Language-Image Pretraining, CLIP), etc. to process and encode the sample prompt words and output a high-dimensional prompt text vector. The vector can capture the semantic information in the sample prompt words, thereby guiding the subsequent training process.

[0043] S104, training the multimodal diffusion network to be trained based on the sample image, the person-related features and the prompt text vector to obtain a target multimodal diffusion network, and constructing an image synthesis model based on the target multimodal diffusion network.

[0044] Based on the scalability and excellent performance of the Transformer structure, in some embodiments, the multimodal diffusion network can adopt a Transformer-based Multimodal Diffusion Transformer (MMDiT) architecture.

[0045] The embodiment of the present application proposes a method for constructing an image synthesis model, including: obtaining a sample image including multiple characters, and obtaining a sample prompt word corresponding to the sample image; extracting character-related features of each character from the sample image through a first processing network, the character-related features including an appearance feature map, a facial feature map, and a description text vector; converting the sample prompt word into a prompt text vector through a second processing network; training a multimodal diffusion network to be trained based on the sample image, the character-related features, and the prompt text vector to obtain a trained target multimodal diffusion network, and constructing an image synthesis model based on the target multimodal diffusion network. The present application trains a multimodal diffusion network in combination with an appearance feature map, a facial feature map, a description text vector, and a prompt text vector, so that the trained target multimodal diffusion network can understand the structure and semantic information of the character at a higher level, so that the image synthesis model constructed based on the target multimodal diffusion network can generate a more natural and realistic synthesis effect when processing image synthesis tasks, thereby improving user satisfaction.

[0046] Figure 2 is a schematic diagram of an exemplary implementation of a method for constructing an image synthesis model shown in the present application, such as Figure 2 As shown, the method for constructing the image synthesis model includes the following steps:

[0047] S201, obtaining sample images including multiple persons, and obtaining sample prompt words corresponding to the sample images.

[0048] Each sample image includes multiple characters. For example, a sample image may include three characters.

[0049] In some embodiments, when obtaining sample prompt words corresponding to sample images, background information of the sample images may be obtained first, and then sample prompt words corresponding to the sample images may be generated based on the background information. For example, the sample prompt words are "daytime, beach, sun, gravel, stone".

[0050] The sample prompt words generated based on background information can provide more detailed background generation guidance for the training process, helping the network understand the environment or other non-person elements of the image, so that when users use it online, the image synthesis model constructed based on the trained target multimodal diffusion network can generate more accurate and coordinated person backgrounds based on the prompt words input by the user.

[0051] S202, extracting character-related features of each character from the sample image through a first processing network, where the character-related features include an appearance feature map, a facial feature map, and a description text vector.

[0052] S203, converting the sample prompt word into a prompt text vector through a second processing network.

[0053] Regarding the specific implementation of steps S202 to S203, reference may be made to the specific introduction of the relevant parts in the above embodiments, which will not be elaborated here.

[0054] S204, encode the sample image to obtain a first latent space code, and perform noise processing on the first latent space code to obtain a second latent space code.

[0055] In some embodiments, a variational autoencoder (VAE) encoder may be used to encode the sample image to obtain a first latent space encoding.

[0056] S205: Determine the face area of ​​the sample image.

[0057] It is not difficult to understand that since each sample image contains multiple persons, each sample image contains multiple face areas.

[0058] S206: Based on the face region, generate a mask image corresponding to each face region in the sample image.

[0059] In some embodiments, a separate mask image is generated for each person in the sample image. For example, if there are three people in a sample image (each person has a face region), three mask images are usually generated, each mask image corresponding to a face region.

[0060] In some embodiments, it is also possible that a sample image corresponds to a single mask image, and the single mask image marks all face regions in the sample image.

[0061] S207, weighting the loss function of the multimodal diffusion network to be trained with face weights based on the face region mask image to obtain a weighted target loss function.

[0062] This step is mainly to weight the loss function of the multimodal diffusion network to be trained with face weights to obtain the weighted target loss function, which can make the multimodal diffusion network pay more attention to the details and consistency of facial features during training, thereby improving the quality of images generated by the image synthesis model constructed based on the target multimodal diffusion network when used online, and ensuring that the facial features of the characters are natural and realistic.

[0063] Among them, the target loss function is applied in the subsequent training process of the multimodal diffusion network to be trained.

[0064] S208, inputting the character-related features and the prompt text vector as control conditions into the multimodal diffusion network to be trained, and combining the input second latent space encoding to obtain the noise distribution of the multimodal diffusion network output.

[0065] S209, obtaining a loss value of the multimodal diffusion network based on the noise distribution, iteratively training the multimodal diffusion network to be trained based on the loss value to obtain a target multimodal diffusion network, and constructing an image synthesis model based on the target multimodal diffusion network.

[0066] Among them, the goal of the multimodal diffusion network is to gradually denoise the second latent space encoding after noisy based on the person-related features and the prompt text vector as control conditions, so as to deduce the image generation process.

[0067] In the present application, the multimodal diffusion network to be trained calculates a loss value through the noise distribution of each output, and iteratively trains the network based on the loss value until a trained target multimodal diffusion network is obtained.

[0068] After training, the target multimodal diffusion network has the ability to gradually denoise according to control conditions to recover feature representations from noise. Ultimately, an image synthesis model can be constructed based on the target multimodal diffusion network to generate high-quality synthetic images.

[0069] In an embodiment of the present application, by weighting the loss function of the multimodal diffusion network to be trained with face weights to obtain a weighted target loss function, the multimodal diffusion network can pay more attention to the details and consistency of facial features during training, thereby improving the quality of images generated by the image synthesis model constructed based on the target multimodal diffusion network when used online, ensuring that the facial features of the characters are natural and realistic.

[0070] Figure 3 is a schematic diagram of an exemplary implementation of a method for constructing an image synthesis model shown in the present application, such as Figure 3 As shown, the method for constructing the image synthesis model includes the following steps:

[0071] First, obtain a sample image including multiple people ( Figure 3 In the example, the sample image includes N people), and the sample prompt words corresponding to the sample image are obtained.

[0072] On the one hand, the character-related features of each character are extracted from the sample image through the first processing network 301, and the character-related features include an appearance feature map, a facial feature map and a description text vector.

[0073] Specifically, Figure 3 As shown, the first processing network 301 includes an image segmentation unit 3011, an appearance feature extraction unit 3012, a facial feature extraction unit 3013, an image annotation unit 3014 and a first text encoder 3015. The character-related features of each character are extracted from the sample image through the first processing network, specifically including the following steps:

[0074] 1. Based on the image segmentation unit 3011, the person in the sample image is segmented to obtain multiple person images, such as Figure 3 As shown, after segmentation, character image 1, character image 2, ..., character image N are obtained.

[0075] Optionally, the image segmentation unit 3011 may adopt a closed set segmentation model.

[0076] 2. Based on the appearance feature extraction unit 3012, an appearance feature map is extracted from each character image, and based on the facial feature extraction unit 3013, a facial feature map is extracted from each character image, and based on the pre-trained image annotation unit 3014, each character image is annotated to obtain a description text, and the description text is converted into a description text vector based on the first text encoder 3015.

[0077] 3. Construct the character-related features of each character based on the appearance feature map, facial feature map and description text vector.

[0078] Optionally, the character-related features may be arranged in order from left to right of each character on the sample image, namely, character 1-related features, character 2-related features, ..., character N-related features.

[0079] Among them, the character-related features of each character can be regarded as a set of features, which includes 1 appearance feature map, 1 facial feature map and 1 description text vector.

[0080] Since visual information (appearance features and facial features) can help the network better understand the structure of the image, the movements of the characters, and sensory details such as facial expressions, while text information provides a higher level of contextual interpretation for visual information, helping the network understand the semantics behind the image. Therefore, when constructing character-related features, for each character-related feature, the appearance feature map and the facial feature map are placed before the descriptive text vector. In this way, for each character-related feature, when the character-related feature is subsequently input into the multimodal diffusion network to be trained, the input of the appearance feature map and the facial feature map is also placed before the descriptive text vector. In this way, the network's understanding of the text can be constrained and guided by visual information, ensuring that when the network generates images, it is first established on an accurate visual basis, and then the semantics are supplemented by text information, making the generated results more consistent and reasonable.

[0081] On the other hand, the sample prompt word is converted into a prompt text vector by the second processing network 302. Figure 3 As shown, the second processing network 302 includes a second text encoder 3021. The sample prompt word is input into the second text encoder 3021 to obtain a prompt text vector.

[0082] The first text encoder 3015 and the second text encoder 3021 may be independent or the same text encoder.

[0083] On the other hand, Figure 3 As shown, the sample image is encoded based on the image encoder 3031 in the third processing network 303 to obtain a first latent space code, and the first latent space code is denoised by the denoising unit 3032 in the third processing network 303 to obtain a second latent space code.

[0084] Finally, the character-related features and the prompt text vector are used as control conditions ( Figure 3 The multimodal diffusion network 304 to be trained is input and combined with the second latent space encoding ( Figure 3 The noise distribution of the multimodal diffusion network output is obtained by using the hidden layer flow shown above.

[0085] For ease of understanding, Figure 4is a simplified schematic diagram of a layer in a multimodal diffusion network shown in this application, such as Figure 4 As shown in the figure, in a certain layer of the multimodal diffusion network (the multimodal diffusion network includes multiple layers of such a structure), after the control flow and the hidden layer flow are input into the multimodal diffusion network, the information interaction between the hidden layer flow and the conditional flow is realized through the self-attention module, and then the next information interaction between the control flow and the hidden layer flow is carried out based on the control flow and the hidden layer flow output by this layer.

[0086] Furthermore, after obtaining the noise distribution of the multimodal diffusion network output as described above, the loss value of the multimodal diffusion network is obtained based on the noise distribution, and the multimodal diffusion network to be trained is iteratively trained based on the loss value, and multiple iterations are performed until the loss function converges to obtain a trained target multimodal diffusion network.

[0087] The above process introduces in detail the detailed structure of the first processing network and the second processing network, and introduces in detail the training process of the multimodal diffusion network. The multimodal diffusion network is combined with the appearance feature map, the facial feature map, the description text vector, the prompt text vector, and the second latent space encoding to gradually perform denoising training for the multimodal diffusion network, so that the trained target multimodal diffusion network can understand the structure and semantic information of the character at a higher level. The trained target multimodal diffusion network has the ability to guide the denoising process according to the control conditions and gradually recover the feature representation from the noise.

[0088] After the above training, the target multimodal diffusion network has the ability to gradually remove noise according to the control conditions and gradually recover the feature representation from the noise. Finally, an image synthesis model can be built based on the network to generate high-quality synthetic images. Figure 5 A structural schematic diagram of an image synthesis model shown in this application is described in detail. Figure 5 As shown, an image synthesis model is constructed based on a first processing network 501, a second processing network 502, a target multimodal diffusion network 504, a preset decoder 505 and a noise introduction unit 503.

[0089] Among them, the first processing network 501 includes an image segmentation unit 5011, an appearance feature extraction unit 5012, a facial feature extraction unit 5013, an image annotation unit 5014 and a first text encoder 5015, whose structures and respective functions have been introduced in detail in the above training process and will not be repeated here.

[0090] Among them, the second processing network 502 includes a second text encoder 5021, whose function has been introduced in detail in the above training process and will not be repeated here.

[0091] The noise introduction unit 503 is used to introduce noise into the target multimodal diffusion network 504 to initialize the diffusion process. Optionally, Gaussian noise can be used.

[0092] The target multimodal diffusion network 504 is used to gradually recover feature representations from the noise introduced by the noise introduction unit 503 through a denoising process of multiple time steps based on the outputs of the first processing network 501 and the second processing network 502 as control conditions.

[0093] The decoder 505 is used to convert the feature representation obtained by the target multimodal diffusion network 504 after denoising at multiple time steps into an output image.

[0094] The image synthesis model constructed above can introduce multimodal control information into the target multimodal diffusion network through the first processing network and the second processing network, so that the target multimodal diffusion network can gradually denoise the introduced noise and gradually recover the feature representation from the noise, so that the decoder is used to convert the feature representation obtained by the target multimodal diffusion network after denoising at multiple time steps into an output image, thereby ensuring the quality of the image generated by the image synthesis model when used online and ensuring that the facial features of the characters are natural and realistic.

[0095] The following mainly introduces the online use process of the image synthesis model built based on the above solution. Figure 6 is a schematic diagram of an exemplary implementation of an image generation method shown in the present application, such as Figure 6 As shown, the image generation method comprises the following steps:

[0096] S601, obtaining a plurality of input images input by a user, wherein the input images include a target object.

[0097] The input images refer to the multiple input images input by the user when submitting the image synthesis task online.

[0098] It is not difficult to understand that the image synthesis model constructed in the present application is mainly used to realize the synthesis of various characters in multiple input images into one image, so the input image needs to include a target object. Among them, the target object can be a person or a digital person, etc. S602, obtaining a target prompt word input by a user.

[0099] Among them, the target prompt words input by the user are mainly used to guide the generation process of the image synthesis model. For example, it can guide the generation of background information in the synthesized image, such as the target prompt words are "night, moon, France, street".

[0100] In addition to guiding the generation of background information, it can also guide information such as each character's clothing, appearance, emotions, facial expressions, etc. For example, target prompt words can also include "laughing, holding hands, red skirt" and so on.

[0101] S603, inputting a plurality of input images and a target prompt word into an image synthesis model to obtain a target image output by the image synthesis model.

[0102] The following will be combined Figure 5 Introduce the image generation method, such as Figure 5 As shown, the input images include input image 1, input image 2, ..., input image M, and it is assumed that each image includes one person. The M input images input by the user are input into the first processing network, and are processed by the image segmentation unit 5011, the appearance feature extraction unit 5012, the facial feature extraction unit 5013, the image annotation unit 5014, and the first text encoder 5015 in the first processing network 501 to obtain the person-related features of each of the M persons, and each person-related feature includes an appearance feature map, a facial feature map, and a description text vector.

[0103] The target prompt word input by the user is input into the second processing network 502 to obtain a prompt text vector corresponding to the target prompt word.

[0104] The character-related features and the prompt text vector are used as control conditions of the target multimodal diffusion network 504 in the image synthesis model, guiding the target multimodal diffusion network to gradually recover the feature representation from the introduced noise, and converting the obtained feature representation into an output image based on the decoder 505.

[0105] It is not difficult to understand that, taking the input images including input image 1, input image 2... input image M, assuming that each image includes a woman, and the target prompt words are "night, moon, France, street, laughing, arm in arm, red skirt" as an example, the target image output by the image synthesis model will be: M women in red skirts are arm in arm and laughing on the streets of France at night, and there is a moon in the night sky, and the characters in this target image retain the original characteristics of the characters (such as facial features).

[0106] The image synthesis model is constructed and generated using the image synthesis model construction method introduced in the above embodiment.

[0107] The embodiment of the present application proposes an image generation method, which can guide the model to generate a synthetic image based on the character features of each character in the input image and the target prompt word. The output target image character has a more natural appearance and can achieve personalized generation.

[0108] Figure 7 is a schematic diagram of a device for constructing an image synthesis model shown in the present application, such as Figure 7 As shown, the image synthesis model construction device 700 includes an acquisition module 701, an extraction module 702, a conversion module 703 and a training module 704, wherein:

[0109] The acquisition module 701 is used to acquire sample images including multiple characters, and acquire sample prompt words corresponding to the sample images.

[0110] The extraction module 702 is used to extract the character-related features of each character from the sample image through the first processing network, and the character-related features include an appearance feature map, a facial feature map and a description text vector.

[0111] The conversion module 703 is used to convert the sample prompt word into a prompt text vector through a second processing network.

[0112] The training module 704 is used to train the multimodal diffusion network to be trained based on the sample image, the person-related features and the prompt text vector to obtain the target multimodal diffusion network, and to construct an image synthesis model based on the target multimodal diffusion network.

[0113] The device is trained by combining a multimodal diffusion network with an appearance feature map, a facial feature map, a description text vector, and a prompt text vector, so that the trained target multimodal diffusion network can understand the structure and semantic information of the character at a higher level, so that the image synthesis model constructed based on the target multimodal diffusion network can generate a more natural and realistic synthesis effect when processing image synthesis tasks, thereby improving user satisfaction.

[0114] Furthermore, the first processing network includes an image segmentation unit, an appearance feature extraction unit, a facial feature extraction unit, an image annotation unit and a first text encoder, and the extraction module 702 is also used to: segment the characters in the sample image based on the image segmentation unit to obtain multiple character images; extract an appearance feature map from each character image based on the appearance feature extraction unit; extract a facial feature map from each character image based on the facial feature extraction unit; annotate each character image based on the image annotation unit to obtain a description text, and convert the description text into a description text vector based on the first text encoder; construct character-related features for each character based on the appearance feature map, the facial feature map and the description text vector.

[0115] Furthermore, the second processing network includes a second text encoder, and the conversion module 7603 is also used to: input the sample prompt word into the second text encoder to obtain a prompt text vector.

[0116] Furthermore, the training module 704 is also used to: encode the sample image to obtain a first latent space code, and perform noise processing on the first latent space code to obtain a second latent space code; input the person-related features and the prompt text vector as control conditions into the multimodal diffusion network to be trained, and combine the input second latent space code to obtain the noise distribution of the multimodal diffusion network output; obtain the loss value of the multimodal diffusion network based on the noise distribution, and iteratively train the multimodal diffusion network to be trained based on the loss value to obtain the target multimodal diffusion network.

[0117] Furthermore, the training module 704 is also used to: construct an image synthesis model based on the first processing network, the second processing network, the target multimodal diffusion network, a preset decoder and a noise introduction unit; the noise introduction unit is used to introduce noise into the target multimodal diffusion network to initialize the diffusion process; the target multimodal diffusion network is used to gradually recover the feature representation from the noise through a denoising process of multiple time steps; the decoder is used to convert the feature representation obtained by the target multimodal diffusion network after denoising at multiple time steps into an output image.

[0118] Furthermore, the acquisition module is further used to: acquire background information of the sample image; and generate a sample prompt word corresponding to the sample image based on the background information.

[0119] Furthermore, the image synthesis model construction device 700 also includes an adjustment module, which is used to: weight the loss function of the multimodal diffusion network to be trained with face weights to obtain a weighted target loss function; and train the multimodal diffusion network to be trained based on the target loss function.

[0120] Furthermore, the adjustment module is also used to: determine the face area of ​​the sample image; based on the face area, generate a mask map corresponding to each face area in the sample image; based on the face area mask map, weight the loss function of the multimodal diffusion network to be trained with face weights to obtain a weighted target loss function.

[0121] Further, in the training module 704, for each person-related feature, when the person-related feature is input into the multimodal diffusion network to be trained, the appearance feature map and the facial feature map are located before the description text vector.

[0122] Figure 8 is an exemplary schematic diagram of an image generating device shown in the present application, such as Figure 8 As shown, the image generating device 800 comprises:

[0123] A first acquisition module 801 is used to acquire multiple input images input by a user, wherein the input images include a target object;

[0124] The second acquisition module 802 is used to acquire the target prompt word input by the user;

[0125] A generation module 803 is used to input multiple input images and target prompt words into the image synthesis model to obtain a target image output by the image synthesis model;

[0126] Among them, the image synthesis model is constructed and generated using the image synthesis model construction device introduced above.

[0127] The device can guide the model to generate a synthetic image based on the character features of each character in the input image and the target prompt word. The output target image character has a more natural appearance and can be personalized.

[0128] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0129] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0130] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0131] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0132] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the construction method of the image synthesis model or the image generation method. For example, in some embodiments, the construction method of the image synthesis model or the image generation method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the construction method of the image synthesis model or the image generation method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the image synthesis model construction method or the image generation method in any other appropriate manner (for example, by means of firmware).

[0133] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0134] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0135] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0136] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0137] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0138] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0139] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0140] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for constructing an image synthesis model, comprising: Acquire sample images including a plurality of characters, and acquire sample prompt words corresponding to the sample images; Extracting character-related features of each character from the sample image by a first processing network, wherein the character-related features include an appearance feature map, a facial feature map, and a description text vector; Converting the sample prompt word into a prompt text vector through a second processing network; The multimodal diffusion network to be trained is trained based on the sample image, the character-related features and the prompt text vector to obtain a target multimodal diffusion network, and an image synthesis model is constructed based on the target multimodal diffusion network.

2. The method according to claim 1, wherein: The first processing network includes an image segmentation unit, an appearance feature extraction unit, a facial feature extraction unit, an image annotation unit, and a first text encoder. The extracting of the character-related features of each character from the sample image through the first processing network includes: Segmenting the person in the sample image based on the image segmentation unit to obtain a plurality of person images; Extracting an appearance feature map from each of the character images based on the appearance feature extraction unit; Extracting a facial feature map from each of the character images based on the facial feature extraction unit; Based on the image annotation unit, annotate each of the character images to obtain a description text, and based on the first text encoder, convert the description text into a description text vector; Based on the appearance feature map, the facial feature map and the description text vector, character-related features of each character are constructed.

3. The method according to claim 2, wherein: The second processing network includes a second text encoder, and the converting the sample prompt word into a prompt text vector through the second processing network includes: The sample prompt word is input into the second text encoder to obtain the prompt text vector.

4. The method according to claim 3, wherein: The method of training the multimodal diffusion network to be trained based on the sample image, the character-related features and the prompt text vector to obtain a target multimodal diffusion network includes: Encoding the sample image to obtain a first latent space code, and performing noise processing on the first latent space code to obtain a second latent space code; Inputting the character-related features and the prompt text vector as control conditions into the multimodal diffusion network to be trained, and combining the second latent space encoding input to obtain a noise distribution output by the multimodal diffusion network; A loss value of the multimodal diffusion network is obtained based on the noise distribution, and the multimodal diffusion network to be trained is iteratively trained based on the loss value to obtain a target multimodal diffusion network.

5. The method according to claim 4, wherein: The constructing of an image synthesis model based on the target multimodal diffusion network includes: Building an image synthesis model based on the first processing network, the second processing network, the target multimodal diffusion network, a preset decoder and a noise introduction unit; The noise introduction unit is used to introduce noise into the target multimodal diffusion network to initialize the diffusion process; The target multimodal diffusion network is used to gradually recover feature representations from noise through a denoising process of multiple time steps; The decoder is used to convert the feature representation obtained by the target multimodal diffusion network after denoising at multiple time steps into an output image.

6. The method according to any one of claims 1 to 5, wherein: The obtaining of the sample prompt word corresponding to the sample image includes: Acquiring background information of the sample image; A sample prompt word corresponding to the sample image is generated based on the background information.

7. The method according to claim 6, wherein: The method further comprises: Weighting the loss function of the multimodal diffusion network to be trained with face weights to obtain a weighted target loss function; The multimodal diffusion network to be trained is trained based on the target loss function.

8. The method according to claim 7, wherein: The step of weighting the loss function of the multimodal diffusion network to be trained by face weights to obtain a weighted target loss function includes: Determine the face area of ​​the sample image; Based on the face region, generating a mask image corresponding to each face region in the sample image; The loss function of the multimodal diffusion network to be trained is weighted by face weights based on the face region mask image to obtain a weighted target loss function.

9. The method according to claim 8, wherein: For each of the character-related features, when the character-related features are input into the multimodal diffusion network to be trained, the appearance feature map and the facial feature map are located before the description text vector.

10. A method for generating an image, comprising: Acquire multiple input images input by a user, wherein the input images include a target object; Get the target prompt word input by the user; Inputting the plurality of input images and the target prompt word into an image synthesis model to obtain a target image output by the image synthesis model; Wherein, the image synthesis model is constructed and generated using the method described in any one of claims 1 to 9.

11. A device for constructing an image synthesis model, comprising: An acquisition module, used to acquire sample images including a plurality of characters, and to acquire sample prompt words corresponding to the sample images; An extraction module, configured to extract character-related features of each character from the sample image through a first processing network, wherein the character-related features include an appearance feature map, a facial feature map, and a description text vector; A conversion module, used for converting the sample prompt word into a prompt text vector through a second processing network; A training module is used to train the multimodal diffusion network to be trained based on the sample image, the character-related features and the prompt text vector to obtain a target multimodal diffusion network, and to construct an image synthesis model based on the target multimodal diffusion network.

12. The device according to claim 11, wherein The first processing network includes an image segmentation unit, an appearance feature extraction unit, a facial feature extraction unit, an image annotation unit and a first text encoder, and the extraction module is further used to: Segmenting the person in the sample image based on the image segmentation unit to obtain a plurality of person images; Based on the appearance feature extraction unit, extracting an appearance feature map from each of the character images; Based on the facial feature extraction unit, extract a facial feature map from each of the character images; Based on the image annotation unit, annotate each of the character images to obtain a description text, and based on the first text encoder, convert the description text into a description text vector; Based on the appearance feature map, the facial feature map and the description text vector, character-related features of each character are constructed.

13. The device according to claim 12, wherein: The second processing network includes a second text encoder, and the conversion module is further used for: The sample prompt word is input into the second text encoder to obtain the prompt text vector.

14. The device according to claim 13, wherein: The training module is also used for: Encoding the sample image to obtain a first latent space code, and performing noise processing on the first latent space code to obtain a second latent space code; Inputting the character-related features and the prompt text vector as control conditions into the multimodal diffusion network to be trained, and combining the second latent space encoding input to obtain a noise distribution output by the multimodal diffusion network; A loss value of the multimodal diffusion network is obtained based on the noise distribution, and the multimodal diffusion network to be trained is iteratively trained based on the loss value to obtain a target multimodal diffusion network.

15. The device according to claim 14, wherein: The training module is also used for: Building an image synthesis model based on the first processing network, the second processing network, the target multimodal diffusion network, a preset decoder and a noise introduction unit; The noise introduction unit is used to introduce noise into the target multimodal diffusion network to initialize the diffusion process; The target multimodal diffusion network is used to gradually recover feature representations from noise through a denoising process of multiple time steps; The decoder is used to convert the feature representation obtained by the target multimodal diffusion network after denoising at multiple time steps into an output image.

16. The device according to any one of claims 11 to 15, wherein: The acquisition module is further used for: Acquiring background information of the sample image; A sample prompt word corresponding to the sample image is generated based on the background information.

17. The device according to claim 16, wherein: The device also includes an adjustment module, which is used to: Weighting the loss function of the multimodal diffusion network to be trained with face weights to obtain a weighted target loss function; The multimodal diffusion network to be trained is trained based on the target loss function.

18. The device according to claim 17, wherein: The adjustment module is further used for: Determine the face area of ​​the sample image; Based on the face region, generating a mask image corresponding to each face region in the sample image; The loss function of the multimodal diffusion network to be trained is weighted by face weights based on the face region mask image to obtain a weighted target loss function.

19. The device according to claim 18, wherein: For each of the character-related features, when the character-related features are input into the multimodal diffusion network to be trained, the appearance feature map and the facial feature map are located before the description text vector.

20. An image generating device, comprising: A first acquisition module, used to acquire a plurality of input images input by a user, wherein the input images include a target object; The second acquisition module is used to acquire the target prompt word input by the user; A generation module, used for inputting the plurality of input images and the target prompt word into an image synthesis model to obtain a target image output by the image synthesis model; Wherein, the image synthesis model is constructed and generated using the device described in any one of claims 11 to 19.

21. An electronic device comprising: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 9 or claim 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9 or claim 10.

23. A computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9 or claim 10.

Images