Image generation model training method and image generation method

Abstract

The invention relates to an image generation model training method and an image generation method. The training method comprises the steps that a control model, a pre-trained image generator and a pre-trained text encoder are acquired; a sample text and a sample reference image are obtained, the sample text is used for describing text content of a portrait in the to-be-generated image, and the sample reference image indicates the form of the whole body portrait in the to-be-generated image; inputting the sample text into a text encoder for text encoding to obtain sample text features; performing feature processing on the sample reference image based on the control model to obtain a sample image guide feature; the sample text features and the sample image guide features are input into an image generator for image generation, a prediction image is obtained, and the whole body portrait of the prediction image comprises a human face; adjusting control model parameters based on the predicted image; and combining the parameter-adjusted control model, the text encoder and the image generator to obtain an image generation model. The present disclosure provides the ability to generate a whole body portrait with a face.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image generation model training method and an image generation method. Background Art

[0002] Full-body portrait generation technology is one of the important applications of artificial intelligence in the field of design and creation. With the development of digitalization and virtual reality technology, full-body portraits play an increasingly important role in many industries. They can not only reduce labor costs, but also promote the creation of highly personalized and customized content. However, the generation method of full-body portraits in related technologies has the problem of low quality and unstable quality of the generated images.

[0003] Related technologies can use the advantages of diffusion models in the field of image generation to generate full-body portraits, but currently they cannot directly generate full-body portraits based on diffusion models. Most of them can only generate half-body portraits based on diffusion models, and then obtain full-body portraits by expanding the image. However, this method has many disadvantages, such as complex process, lack of details, unstable quality, and uncontrollable posture. Summary of the invention

[0004] The present disclosure provides an image generation model training method and an image generation method to at least solve at least one problem in the related art. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image generation model training method, comprising:

[0006] Get the control model, as well as the pre-trained image generator and text encoder;

[0007] Acquire sample text and a sample reference image, wherein the sample text is text content used to describe a portrait in an image to be generated, and the sample reference image indicates the form of a full-body portrait in the image to be generated;

[0008] Inputting the sample text into the text encoder for text encoding to obtain sample text features;

[0009] Performing feature processing on the sample reference image based on the control model to obtain a sample image guiding feature;

[0010] Inputting the sample text feature and the sample image guide feature into the image generator to generate an image, thereby obtaining a generated predicted image, wherein the full-body portrait in the predicted image includes a face;

[0011] Based on the predicted image, the parameters of the control model are adjusted; the adjusted control model, the text encoder and the image generator are combined to obtain an image generation model.

[0012] In an exemplary embodiment, the sample reference image is a full-body posture diagram of a person in the image to be generated; and the control model is a posture control model.

[0013] In an exemplary embodiment, the sample reference image is a schematic diagram of the whole body contour including the height and weight of the person in the image to be generated, and the sample reference image indicates the correspondence between the height, the weight and the whole body contour; the control model is a contour control model.

[0014] In an exemplary embodiment, the acquiring of sample text and sample reference image includes:

[0015] Performing key point detection on a two-dimensional image including a preset portrait to obtain key point data of the preset portrait;

[0016] Obtaining the height and weight of the preset portrait;

[0017] Constructing a three-dimensional human body model based on the key point data and the height and weight of the preset portrait;

[0018] The three-dimensional human body model is projected to obtain the sample reference image.

[0019] In an exemplary embodiment, the control model includes a feature extractor and a fusion control model, and the feature processing of the sample reference image based on the control model to obtain the sample image guiding feature includes:

[0020] Inputting the sample reference image into the feature extractor to extract features and obtain sample image features;

[0021] The sample image features are input into the fusion control model for feature fusion to obtain the sample image guiding features.

[0022] In an exemplary embodiment, the feature extractor includes a pre-trained image encoder and a mapping component, wherein the mapping component is used to map information in an image feature space to a text feature space; the step of inputting the sample reference image into the feature extractor for feature extraction to obtain sample image features includes:

[0023] Inputting the sample reference image into the image encoder for image encoding to obtain a sample image code;

[0024] Input the sample image code into the mapping component for information mapping to obtain the sample image features;

[0025] The adjusting the parameters of the control model based on the predicted image includes:

[0026] While freezing the parameters of the image encoder, the parameters of the mapping component and the fusion control model are adjusted based on the predicted image.

[0027] In an exemplary embodiment, the step of inputting the sample image features into the fusion control model for feature fusion to obtain the sample image guidance features includes:

[0028] Inputting the sample image features into at least one network layer of the fusion control model for feature fusion to obtain a single-layer guided feature output by each network layer, wherein the single-layer guided feature belongs to the sample image guided feature;

[0029] The step of inputting the sample text feature and the sample image guide feature into the image generator to generate an image and obtain a generated predicted image includes:

[0030] Each of the single-layer guiding features is input into a corresponding network layer in the image generator, and the sample text features are input into at least one network layer in the image generator, so that the image generator generates the predicted image by fusing the sample text features and the sample image guiding features.

[0031] In an exemplary embodiment, the image generator is a diffusion model, and the predicted image generated by the image generator includes an image prediction result corresponding to each diffusion step in at least one diffusion step; and adjusting the parameters of the control model based on the predicted image includes:

[0032] Determine, based on a base image, a reference image corresponding to each diffusion step in the at least one diffusion step, wherein the base image includes a portrait with a face, the portrait with a face conforms to the description of the sample text, and the morphology of the portrait with a face is consistent with the sample reference image;

[0033] For any diffusion step of the at least one diffusion step, the parameters of the control model are adjusted based on a difference between a reference image corresponding to the diffusion step and a corresponding image prediction result.

[0034] In an exemplary embodiment, the step of inputting the sample text feature and the sample image guide feature into the image generator to generate an image to obtain a generated predicted image includes:

[0035] Initialize the current diffusion steps;

[0036] Based on the preset noise and the current number of diffusion steps, denoising the preset noise is performed by fusing the sample text feature and the sample image guide feature to obtain an image prediction result corresponding to the current number of diffusion steps;

[0037] Update the preset noise based on the difference between the preset noise and the image prediction result corresponding to the current diffusion step number;

[0038] Update the current diffusion step number, and when the updated current diffusion step number is less than the diffusion step number threshold, repeat the step of denoising the preset noise based on the preset noise and the current diffusion step number by fusing the sample text feature and the sample image guide feature to obtain the image prediction result corresponding to the current diffusion step number.

[0039] In an exemplary embodiment, the fusion control model and the image generator have the same structure, and initial parameters of the fusion control model are determined based on parameters of the image generator.

[0040] According to a second aspect of an embodiment of the present disclosure, there is provided an image generating method, the method comprising:

[0041] Acquire a target text and a target reference image, wherein the target text is a text content used to describe a portrait in an image to be generated, and the target reference image indicates the form of a full-body portrait in the image to be generated;

[0042] Inputting the target text and the target reference image into an image generation model to obtain a target image, wherein the full-body portrait in the target image includes a face;

[0043] Wherein, the image generation model is trained by the image generation model training method described in any one of the first aspects.

[0044] According to a third aspect of an embodiment of the present disclosure, there is provided an image generation model training device, comprising:

[0045] A component acquisition module, configured to execute an acquisition control model, and a pre-trained image generator and text encoder;

[0046] A sample acquisition module is configured to acquire sample text and sample reference images, wherein the sample text is text content used to describe the portrait in the image to be generated, and the sample reference image indicates the form of the full-body portrait in the image to be generated;

[0047] The training module is configured to execute:

[0048] Inputting the sample text into the text encoder for text encoding to obtain sample text features;

[0049] Performing feature processing on the sample reference image based on the control model to obtain a sample image guiding feature;

[0050] Inputting the sample text feature and the sample image guide feature into the image generator to generate an image, thereby obtaining a generated predicted image, wherein the full-body portrait in the predicted image includes a face;

[0051] Based on the predicted image, the parameters of the control model are adjusted; the adjusted control model, the text encoder and the image generator are combined to obtain an image generation model.

[0052] In an exemplary embodiment, the sample reference image is a full-body posture diagram of a person in the image to be generated; and the control model is a posture control model.

[0053] In an exemplary embodiment, the sample reference image is a schematic diagram of the whole body contour including the height and weight of the person in the image to be generated, and the sample reference image indicates the correspondence between the height, the weight and the whole body contour; the control model is a contour control model.

[0054] In an exemplary embodiment, the sample acquisition module is configured to execute:

[0055] Performing key point detection on a two-dimensional image including a preset portrait to obtain key point data of the preset portrait;

[0056] Obtaining the height and weight of the preset portrait;

[0057] Constructing a three-dimensional human body model based on the key point data and the height and weight of the preset portrait;

[0058] The three-dimensional human body model is projected to obtain the sample reference image.

[0059] In an exemplary embodiment, the control model includes a feature extractor and a fusion control model, and the training module is configured to perform:

[0060] Inputting the sample reference image into the feature extractor to extract features and obtain sample image features;

[0061] The sample image features are input into the fusion control model for feature fusion to obtain the sample image guiding features.

[0062] In an exemplary embodiment, the feature extractor includes a pre-trained image encoder and a mapping component, wherein the mapping component is used to map information in the image feature space to the text feature space; the training module is configured to perform:

[0063] Inputting the sample reference image into the image encoder for image encoding to obtain a sample image code;

[0064] Input the sample image code into the mapping component for information mapping to obtain the sample image features;

[0065] The adjusting the parameters of the control model based on the predicted image includes:

[0066] While freezing the parameters of the image encoder, the parameters of the mapping component and the fusion control model are adjusted based on the predicted image.

[0067] In an exemplary embodiment, the training module is configured to execute:

[0068] Inputting the sample image features into at least one network layer of the fusion control model for feature fusion to obtain a single-layer guided feature output by each network layer, wherein the single-layer guided feature belongs to the sample image guided feature;

[0069] The step of inputting the sample text feature and the sample image guide feature into the image generator to generate an image and obtain a generated predicted image includes:

[0070] Each of the single-layer guiding features is input into a corresponding network layer in the image generator, and the sample text features are input into at least one network layer in the image generator, so that the image generator generates the predicted image by fusing the sample text features and the sample image guiding features.

[0071] In an exemplary embodiment, the image generator is a diffusion model, and the predicted image generated by the image generator includes an image prediction result corresponding to each diffusion step in at least one diffusion step; the training module is configured to execute:

[0072] Determine, based on a base image, a reference image corresponding to each diffusion step in the at least one diffusion step, wherein the base image includes a portrait with a face, the portrait with a face conforms to the description of the sample text, and the morphology of the portrait with a face is consistent with the sample reference image;

[0073] For any diffusion step of the at least one diffusion step, the parameters of the control model are adjusted based on a difference between a reference image corresponding to the diffusion step and a corresponding image prediction result.

[0074] In an exemplary embodiment, the training module is configured to execute:

[0075] Initialize the current diffusion steps;

[0076] Based on the preset noise and the current number of diffusion steps, denoising the preset noise is performed by fusing the sample text feature and the sample image guide feature to obtain an image prediction result corresponding to the current number of diffusion steps;

[0077] Update the preset noise based on the difference between the preset noise and the image prediction result corresponding to the current diffusion step number;

[0078] Update the current diffusion step number, and when the updated current diffusion step number is less than the diffusion step number threshold, repeat the step of denoising the preset noise based on the preset noise and the current diffusion step number by fusing the sample text feature and the sample image guide feature to obtain the image prediction result corresponding to the current diffusion step number.

[0079] In an exemplary embodiment, the fusion control model and the image generator have the same structure, and initial parameters of the fusion control model are determined based on parameters of the image generator.

[0080] According to a fourth aspect of an embodiment of the present disclosure, there is provided an image generating device, including:

[0081] A target data acquisition module is configured to acquire a target text and a target reference image, wherein the target text is a text content used to describe a portrait in an image to be generated, and the target reference image indicates the form of a full-body portrait in an image to be generated;

[0082] An image generation module is configured to input the target text and the target reference image into an image generation model to obtain a target image, wherein the full-body portrait in the target image includes a face;

[0083] Wherein, the image generation model is trained by the image generation model training method described in any one of the first aspects.

[0084] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0085] processor;

[0086] a memory for storing instructions executable by the processor;

[0087] The processor is configured to execute the instructions to implement the image generation model training method or image generation method as described in any of the above embodiments.

[0088] According to a fourth aspect of an embodiment of the present disclosure, a computer storage medium is provided. When instructions in the computer storage medium are executed by a processor of an electronic device, the electronic device executes the image generation model training method or the image generation method described in any of the above embodiments.

[0089] According to a sixth aspect of an embodiment of the present disclosure, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, it implements the image generation model training method or the image generation method described in any of the above-mentioned embodiments.

[0090] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0091] The disclosed embodiments may use a pre-trained image generator and a text encoder to form the basic skeleton of an image generation model. By additionally training a control model in the image generation model, the trained image generation model may have the ability to directly generate an image of a full-body portrait with a face based on the text content used to describe the portrait in the image to be generated and the image used to describe the shape of the full-body portrait in the image to be generated.

[0092] In the training stage, a full-body portrait morphology control signal is directly introduced using a sample reference image indicating the morphology of the full-body portrait in the image to be generated, so that the control model can guide the image generator to directly generate an image including a full-body portrait, without the need to generate only half-body portraits and then rely on image expansion to form a full-body portrait as in the related art, thereby improving the generation efficiency and quality of full-body portraits. The use of sample text and sample reference images can introduce full-body portrait morphology control signals and text signals for describing portraits to the image generator, so that the portraits with faces generated by the image generator are more realistic, with more coherent parts, and have the significant advantages of high quality and controllable morphology, meeting the needs of users to generate arbitrary portraits and arbitrary settings of portrait morphology, and the generated full-body portraits with faces are realistic and vivid.

[0093] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] The drawings herein are incorporated into the specification and constitute a part of the present disclosure, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0095] Figure 1 The figure is a flowchart of a method for training an image generation model according to an exemplary embodiment.

[0096] Figure 2The figure is a flowchart of a method for obtaining a sample reference image according to an exemplary embodiment.

[0097] Figure 3 is a flow chart of a method for acquiring sample image features according to an exemplary embodiment.

[0098] Figure 4 It is a framework diagram of an image generator training method according to an exemplary embodiment.

[0099] Figure 5 The figure is a flowchart of a method for generating a predicted image according to an exemplary embodiment.

[0100] Figure 6 The figure is a flowchart of an image generating method according to an exemplary embodiment.

[0101] Figure 7 The figure is a block diagram of an image generation model training device according to an exemplary embodiment.

[0102] Figure 8 is a block diagram of an image generating device according to an exemplary embodiment.

[0103] Fig. 9 is a structural frame of a computer device according to an exemplary embodiment Figure 1 ;

[0104] Fig.10 is a structural frame of a computer device according to an exemplary embodiment Figure 2 . DETAILED DESCRIPTION

[0105] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0106] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0107] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0108] Figure 1 It is a flowchart of an image generation model training method according to an exemplary embodiment. The image generation model training method can be applied to an electronic device, which can be implemented by a server or a terminal alone, or can be implemented by a terminal and a server in collaboration. Among them, the terminal can be but is not limited to physical devices such as smart phones, tablet computers, laptops, desktop computers, smart speakers, smart wearable devices, digital assistants, augmented reality devices, virtual reality devices, etc., and can also include software such as applications running in physical devices. The server can be but is not limited to an independent server, or a server cluster or distributed system composed of multiple physical servers, or it can be one or more cloud servers that provide cloud services, cloud databases, cloud computing, cloud, cloud storage, network services, cloud communications, intermediate services, domain name services, security services, and basic cloud computing services such as big data and artificial intelligence platforms. Reference Figure 1 As shown, the method includes the following steps.

[0109] In step S101, a control model, and a pre-trained image generator and text encoder are obtained.

[0110] The present disclosure does not limit the pre-trained image generator and text encoder, which can be any model component with image generation capability and text encoding capability obtained through pre-training in the related art. The function of the image generator is to generate an image with a face, and the function of the text encoder is to encode the text content into text features, and use the text features as text signals to guide the image generator to generate an image with a face based on the text features. For example, if the text content is "a person with feature A", then the image generator can generate a portrait of a "person with feature A" with a face.

[0111] In some exemplary embodiments, the pre-trained image generator and text encoder may come from a pre-trained Wensheng graph model. The present disclosure does not limit the pre-trained Wensheng graph model, which may be any artificial intelligence model with Wensheng graph capabilities, which at least includes the aforementioned text encoder and the aforementioned image generator. In one embodiment, the pre-trained Wensheng graph model can generate images from text, and can also modify images based on text. The pre-trained Wensheng graph model includes several components. The text encoder and image generator extracted in step S101 are components in the branch of the pre-trained Wensheng graph model that implements the Wensheng graph function.

[0112] The present disclosure does not limit the specific structure of the control model, which may include a feature extractor and a fusion control model. The feature extractor is used to extract features, and the fusion control model is used to guide the image generator to generate images based on the extracted features.

[0113] In step S102, sample text and sample reference images are obtained, wherein the sample text is text content used to describe the portrait in the image to be generated, and the sample reference image indicates the form of the full-body portrait in the image to be generated.

[0114] The sample text may describe the person in the image to be generated, such as his / her clothing, clothes, facial features, gender, body shape, posture, outline, etc. The sample reference image does not include the face of the person, but only describes the form of the full-body portrait in the image to be generated.

[0115] In an exemplary embodiment, the sample reference image is a full-body posture diagram of the portrait in the image to be generated; and the control model is a posture control model. The full-body posture diagram is a full-body posture diagram of a stick figure, i.e., a Stick Figure. This type of full-body posture diagram does not include a face, and usually refers to a very simple human figure composed of several lines to represent the basic shapes of a person's head, torso, limbs, etc., thereby focusing on conveying posture information. Using a full-body posture diagram to train an image generation model can explicitly introduce a clearer full-body posture signal for the image generation model, so that the image generation model can generate a full-body image with a face based on text signals and full-body posture signals, thereby meeting the requirements of accurate and controllable posture of the generated full-body portrait, improving the ability to generate high-quality and highly controllable full-body portraits, meeting the requirements of different scenarios and applications, and increasing the diversity and flexibility of generated images. In addition, the ability to generate details is gradually enhanced during the training process, so that the final generated full-body portrait has high-quality and high-detail image performance.

[0116] Neither the text description nor the stick figure posture diagram can make the model understand the control effect of height and weight on the portrait contour, and the mechanism of their influence on the portrait proportion display effect. In order to increase the model's understanding ability in this regard, so that it can have the ability to generate an adaptive full-body portrait based on height and weight, in an exemplary embodiment, the sample reference image is a full-body contour diagram including the height and weight of the portrait in the image to be generated, and the sample reference image indicates the corresponding relationship between the height, the weight and the full-body contour; the control model is a contour control model. The full-body contour diagram only includes the contour of the portrait, which can roughly show the height, weight and muscle direction of the portrait, and the direction of the contour also reflects the posture, but does not include the face. In fact, the full-body contour diagram indicates the relationship between the height, weight and full-body contour of the portrait, and explicitly introduces clearer height and weight signals, full-body posture signals and full-body contour signals to the image generation model, so that the image generation model can generate a full-body image with a face based on text signals, height and weight. The contour of the full-body portrait meets the requirements of the corresponding height and the corresponding weight, and the posture is reasonable, which can ensure the high quality and rich details of the generated full-body portrait.

[0117] Of course, the present disclosure does not limit the method for obtaining the sample reference image. Figure 2 The flowchart of a method for obtaining a sample reference image according to an exemplary embodiment is shown. The method of obtaining a sample text and a sample reference image includes:

[0118] In step S201, key point detection is performed on a two-dimensional image including a preset portrait to obtain key point data of the preset portrait; and the height and weight of the preset portrait are obtained.

[0119] The present disclosure does not limit the preset portraits, because no human face is required, so the source of the preset portraits is not limited. It only needs to be a two-dimensional image taken by a real person or virtual person with a certain height and weight. The present disclosure can obtain corresponding two-dimensional images including preset portraits for various combinations of heights and weights, so as to obtain sample reference images under various heights and combinations. Of course, the present disclosure does not limit the method of detecting key points. For example, an artificial intelligence model can be used to realize automatic extraction of key points, or some regression algorithms can be used, and the present disclosure does not limit this.

[0120] In step S202, a three-dimensional human body model is constructed based on the key point data and the height and weight of the preset portrait.

[0121] The present disclosure does not limit the method for constructing a three-dimensional human body model. Constructing a three-dimensional human body model is a complex process, which usually involves computer graphics, human anatomy and three-dimensional modeling technology. For example, the coordinate points of various parts of the human body, such as joints and muscle attachment points, can be determined based on key point data, and the proportional relationship of various parts of the three-dimensional human body model can be determined based on height and weight, and then the three-dimensional human body model can be constructed using modeling software.

[0122] In step S203, the three-dimensional human body model is projected to obtain the sample reference image.

[0123] The present disclosure does not limit the projection method. For example, a forward projection may be performed to obtain a sample reference image.

[0124] The present invention obtains sample reference images through key point detection, three-dimensional human body model fitting and projection, thereby achieving the acquisition of a two-dimensional full-body contour schematic diagram, and making this two-dimensional full-body contour schematic diagram indicate the relationship between height, weight and full-body contour posture, providing rich sample reference images.

[0125] The present invention integrates three-dimensional height and weight information into a two-dimensional sample reference image. In a full-body portrait generation scenario, the user can generate a virtual model that better meets actual needs and application scenarios based on the height and weight of the specified model. The three-dimensional information is successfully introduced into the field of two-dimensional image generation, and precise control of the height and weight of the virtual model is achieved, providing a more realistic simulation effect.

[0126] In step S103, the sample text is input into the text encoder for text encoding to obtain sample text features.

[0127] The text encoder is a pre-trained component that is directly extracted for use. Inputting a sample text into the text encoder can trigger the text encoder to encode the sample text and obtain the sample text features in the text feature space.

[0128] In step S104, feature processing is performed on the sample reference image based on the control model to obtain sample image guiding features.

[0129] In an exemplary embodiment, the sample reference image is input into the feature extractor of the control model for feature extraction to obtain sample image features; the sample image features are input into the fusion control model of the control model for feature fusion to obtain the sample image guiding features. In an exemplary implementation, the inputting the sample image features into the fusion control model for feature fusion to obtain the sample image guiding features includes: inputting the sample image features into at least one network layer of the fusion control model for feature fusion to obtain a single-layer guiding feature output by each network layer, and the single-layer guiding feature belongs to the sample image guiding feature.

[0130] The present disclosure does not limit the feature extractor, as long as it can achieve the technical purpose of extracting features from the sample reference image, for example, convolutional neural networks and various image encoders in related technologies can be used as feature extractors. The present disclosure implements image feature extraction through a feature extractor, guides sample image features to perform feature fusion through a fusion control model, and obtains sample image guide features that can be used to guide an image generator to generate a predicted image, thereby improving the utilization rate of information in the sample reference image by the image generator and improving the image generation quality.

[0131] In an exemplary embodiment, the feature extractor includes a pre-trained image encoder and a mapping component, wherein the mapping component is used to map information in the image feature space to the text feature space. Figure 3 , which is a flow chart of a sample image feature acquisition method according to an exemplary embodiment. The step of inputting the sample reference image into the feature extractor for feature extraction to obtain the sample image features includes:

[0132] In step S301, the sample reference image is input into the image encoder for image encoding to obtain a sample image code.

[0133] The present disclosure does not limit the image encoder, which may be any pre-trained image encoder.

[0134] In step S302, the sample image code is input into the mapping component for information mapping to obtain the sample image features.

[0135] Since the image generator and text encoder used in the present disclosure are both pre-trained components, the image generator itself is trained to have the ability to generate images based on the features in the text feature space, that is, the image generator is good at processing the features in the text feature space. However, the sample image encoding is located in the image feature space. In order to enable the image generator to function, the present disclosure uses a mapping component to map the sample image encoding located in the image feature space to the text feature space to obtain the sample image features. Of course, the present disclosure does not limit the structure of the mapping component. For example, it can be formed by a convolutional network. The present disclosure uses a high-performance image encoder and a mapping component to map the image information to the text feature space, so that when the image generator generates an image, it not only considers the text features but also the mapping results of the sample image encoding in the text feature space. This makes the generated image not only have the information corresponding to the text features, but also have the morphological information in the sample image encoding, thereby fully meeting the user's expectations.

[0136] In step S105, the sample text features and the sample image guide features are input into the image generator to generate an image, thereby obtaining a generated predicted image, wherein the full-body portrait in the predicted image includes a face.

[0137] The present disclosure does not limit the method for fusing the sample text features and the sample image guiding features in the image generator, and the specific method for the image generator to generate a predicted image based on the sample text features and the sample image guiding features. In an exemplary embodiment, the sample text features and the sample image guiding features are input into the image generator for image generation to obtain a generated predicted image, including: inputting each of the single-layer guiding features into the corresponding network layer in the image generator, and inputting the sample text features into at least one network layer in the image generator, so that the image generator generates the predicted image by fusing the sample text features and the sample image guiding features. By inputting the sample image guiding features and the sample text features into at least one network layer of the image generator, the image generator is guided to gradually fuse these features during the image generation process, and the fusion effect of multiple information is improved through multi-level fusion, thereby improving the quality of the generated image.

[0138] In step S106, the parameters of the control model are adjusted based on the predicted image; the adjusted control model, the text encoder and the image generator are combined to obtain an image generation model.

[0139] The present disclosure does not limit the parameter adjustment method, for example, the gradient descent method can be used. When the number of parameter adjustments reaches a preset number threshold, or when the difference between the predicted image and the expected prediction result is less than the preset difference, the parameter adjustment can be stopped to obtain an image generation model. The present disclosure does not set a preset number threshold and a preset difference, which can be freely selected according to actual conditions and does not constitute an implementation obstacle. The present disclosure may not adjust the parameters of the pre-trained components, but only adjust the parameters of the control model. In the case where the control model includes a feature extractor and a fusion control model, and the feature extractor includes a pre-trained image encoder and a mapping component, the adjustment of the parameters of the control model based on the predicted image includes: in the case of freezing the parameters of the image encoder, adjusting the parameters of the mapping component and the fusion control model based on the predicted image. Under this implementation, there is no need to adjust the parameters of each pre-trained component, and the parameter adjustment effect is applied to the mapping component and the fusion control model, so as to quickly train the mapping component and the fusion control model and improve the training speed.

[0140] The disclosed embodiment can use a pre-trained image generator and text encoder to form the basic skeleton of the image generation model. By additionally training the control model in the image generation model, the image generation model's ability to accurately control the full-body morphology of the portrait is significantly improved, so that the trained image generation model has the ability to directly generate an image of a full-body portrait with a face based on the text content used to describe the portrait in the image to be generated and the image used to describe the morphology of the full-body portrait in the image to be generated. In the training stage, a sample reference image indicating the morphology of the full-body portrait in the image to be generated is directly used to introduce a full-body portrait morphology control signal, so that the control model can guide the image generator to directly generate an image including a full-body portrait, without the need to generate only a half-body portrait and then rely on image expansion to form a full-body portrait as in the related art, thereby improving the generation efficiency and generation quality of the full-body portrait. The use of sample text and sample reference images can introduce full-body portrait morphology control signals and text signals for describing portraits into the image generator, so that the portraits with faces generated by the image generator are more realistic, with more coherent parts, and have the significant advantages of high quality and controllable morphology. It can meet the needs of users to generate arbitrary portraits and arbitrarily set portrait morphologies, and the generated full-body portraits with faces are realistic and vivid.

[0141] In an exemplary embodiment, the image generator is a diffusion model, and the predicted image generated by the image generator includes an image prediction result corresponding to each diffusion step in at least one diffusion step; adjusting the parameters of the control model based on the predicted image includes: determining a reference image corresponding to each diffusion step in the at least one diffusion step based on a base image, the base image includes a portrait with a face, the portrait with a face conforms to the description of the sample text, and the morphology of the portrait with a face is consistent with the sample reference image; for any diffusion step in the at least one diffusion step, adjusting the parameters of the control model based on the difference between the reference image corresponding to the diffusion step and the corresponding image prediction result.

[0142] The present disclosure does not limit the source of the basic image, as long as it has a face, and the portrait with the face conforms to the description of the sample text, and the morphology of the portrait with the face is consistent with the sample reference image. The present disclosure does not limit the structure of the diffusion model, which can refer to the relevant technology and does not constitute an implementation obstacle. For any diffusion step, the corresponding reference image under the diffusion step can be calculated based on the basic image, and the reference image can be understood as the "true value" feature used to determine the loss of the image prediction result generated under the diffusion step. The present disclosure does not limit and explain the specific algorithm for determining the reference image based on the diffusion step, because the specific algorithm can refer to the relevant technology and does not constitute an implementation obstacle to the present disclosure. The present disclosure can adjust parameters based on any diffusion step to improve the training speed. Of course, the present disclosure does not limit the quantization method of the difference between the reference image corresponding to the diffusion step and the corresponding image prediction result, and does not constitute an implementation obstacle. The difference between the reference image corresponding to the diffusion step and the corresponding image prediction result can be understood as the loss of the image prediction result generated under the diffusion step.

[0143] The present disclosure uses a diffusion model for image generation. The diffusion model gradually reduces noise, restores image details, and finally generates a realistic image through a series of noise addition operations and reverse diffusion operations. The core of the diffusion model lies in its denoising process, which reconstructs high-quality image details by gradually reducing noise. In this process, the text signal derived from the sample text and the full-body morphology signal derived from the sample reference image can gradually guide the diffusion model to generate clear, high-quality full-body portraits. The text signal can provide a description of the appearance, details, and style of the full-body portrait, and the morphology signal can specify the posture, action, or outline corresponding to the height and weight of the full-body portrait, thereby achieving highly customized image generation.

[0144] The present disclosure does not elaborate on the denoising and denoising process of the diffusion model, and does not constitute an implementation obstacle. The present disclosure does not limit the structure of the diffusion model, and the text signal and the whole body morphology signal of the present disclosure can be introduced into the attention layer of the base of the diffusion model. The sample text features in the present disclosure can be introduced into at least one network layer or attention layer of the base of the diffusion model as a text signal. In one embodiment, the fusion control model has the same structure as the image generator, and the fusion control model introduces the sample image guide feature as a morphology signal into at least one network layer or attention layer of the base of the diffusion model. And the initial parameters of the fusion control model are determined based on the parameters of the image generator, that is, the initial parameters of the fusion control model can be partially integrated or completely inherited from the parameters of the image generator. The diffusion model is a mature architecture, and a large number of experiments have proved that it has excellent performance. The fusion control model has the same structure as the diffusion model and the initial parameters are inherited, so that the fusion control model can inherit the excellent performance advantages and cooperate better with the diffusion model, so that the sample image guide features output by it are better utilized to generate images.

[0145] Please refer to Figure 4 , which is a framework diagram of an image generator training method according to an exemplary embodiment. The image generator is a convolutional neural network architecture, which has a symmetrical "U"-shaped structure, including a contraction (encoding) path and a symmetrical expansion (decoding) path, which are connected by skip connections.

[0146] The text encoder can encode the sample text and input the obtained sample text features into each network layer of the image generator.

[0147] The control model includes a feature extractor and a fusion control model. The feature extractor extracts sample image features from a sample reference image, and inputs the sample image features into the fusion control model for feature fusion to obtain sample image guiding features. The fusion control model has the same structure as the image generator, and each single-layer guiding feature in the sample image guiding feature is input into the network layer corresponding to the image generator. The sample text feature includes a text signal, and the sample image guiding feature includes a whole body morphology signal, specifically, a whole body posture signal or a whole body contour signal. In one embodiment, each single-layer guiding feature of the sample image guiding feature is added to the upsampling corresponding network layer in the denoising network of the diffusion model in a vector addition manner.

[0148] The present disclosure does not limit the process of generating a prediction image using a diffusion model. Figure 5The method of generating a predicted image according to an exemplary embodiment is shown in the flowchart. The method of inputting the sample text feature and the sample image guide feature into the image generator to generate an image to obtain a generated predicted image includes:

[0149] In step S501, the current diffusion step number is initialized.

[0150] Exemplarily, when step S501 is executed for the first time, the current diffusion step number may be set to 0, that is, the current diffusion step number step=0.

[0151] In step S502, based on the preset noise and the current number of diffusion steps, the preset noise is denoised by fusing the sample text feature and the sample image guide feature to obtain an image prediction result corresponding to the current number of diffusion steps.

[0152] The present disclosure does not limit the preset noise obtained for the first time. Exemplarily, when the preset noise is obtained for the first time, the preset noise can be set to random white noise. When step S502 is executed for the first time, the random white noise, step=0, the sample text feature and the sample image guide feature can be input into the diffusion model together to perform denoising on the preset noise, and obtain the image prediction result corresponding to the current diffusion step number. Of course, the present disclosure does not limit the structure of the diffusion model.

[0153] In step S503, the preset noise is updated based on the difference between the preset noise and the image prediction result corresponding to the current diffusion step number.

[0154] When step=0, the image prediction result corresponding to step=0 is predicted, then the difference between the preset noise input at step=0 and the output image prediction result can be calculated, and the difference can be used as the new preset noise to participate in the next step, that is, as the preset noise when step=1.

[0155] In step S504, the current diffusion step is updated, and when the updated current diffusion step is less than the diffusion step threshold, the step of denoising the preset noise based on the preset noise and the current diffusion step is repeated by fusing the sample text feature and the sample image guide feature to obtain the image prediction result corresponding to the current diffusion step.

[0156] For example, the current diffusion step number can be incremented by 1. If step=0, then step can be updated to 1, thereby starting the denoising process of the next step, until each diffusion step number less than the diffusion step number threshold has a corresponding image prediction result. The present disclosure does not limit the diffusion step number threshold, for example, it can be limited to 100.

[0157] The image prediction results corresponding to each diffusion step are gradually generated through the diffusion model, thereby gradually optimizing the quality of the image prediction results until the image prediction result generated under the last diffusion step is obtained, which can be used as the predicted image finally output by the image generator. This process is a gradual noise reduction process, and the predicted image finally generated has good image quality.

[0158] In one exemplary embodiment, a method of generating an image is disclosed. Figure 6 FIG. 1 is a flow chart of an image generation method according to an exemplary embodiment. The image generation method comprises:

[0159] In step S601, a target text and a target reference image are obtained, wherein the target text is a text content used to describe a portrait in an image to be generated, and the target reference image indicates the form of a full-body portrait in an image to be generated.

[0160] The meanings of the target text and the target reference image are based on the same inventive concept as the meanings of the sample text and the sample reference image in the training phase, and will not be elaborated here.

[0161] In step S602, the target text and the target reference image are input into an image generation model to obtain a target image, wherein the full-body portrait in the target image includes a face; wherein the image generation model is trained by the aforementioned image generation model training method.

[0162] The image generation model trained by the present disclosure can accurately generate a full-body portrait based on any target text and any target reference image. The target text is used to accurately control the full-body portrait through text signals, and the target reference image can accurately control the morphology of the full-body portrait, for example, the posture and contour of the full-body portrait can be controlled, thereby achieving direct generation of the full-body portrait with rich details and high image quality.

[0163] Figure 7 is a block diagram of an image generation model training device according to an exemplary embodiment. Figure 7 , the device comprises:

[0164] A component acquisition module 701 is configured to execute acquisition control model, and pre-trained image generator and text encoder;

[0165] The sample acquisition module 702 is configured to acquire sample text and sample reference images, wherein the sample text is text content used to describe the portrait in the image to be generated, and the sample reference image indicates the form of the full-body portrait in the image to be generated;

[0166] The training module 703 is configured to perform:

[0167] Inputting the sample text into the text encoder for text encoding to obtain sample text features;

[0168] Performing feature processing on the sample reference image based on the control model to obtain a sample image guiding feature;

[0169] Inputting the sample text feature and the sample image guide feature into the image generator to generate an image, thereby obtaining a generated predicted image, wherein the full-body portrait in the predicted image includes a face;

[0170] Based on the predicted image, the parameters of the control model are adjusted; the adjusted control model, the text encoder and the image generator are combined to obtain an image generation model.

[0171] In an exemplary embodiment, the sample reference image is a full-body posture diagram of a person in the image to be generated; and the control model is a posture control model.

[0172] In an exemplary embodiment, the sample reference image is a schematic diagram of the whole body contour including the height and weight of the person in the image to be generated, and the sample reference image indicates the correspondence between the height, the weight and the whole body contour; the control model is a contour control model.

[0173] In an exemplary implementation, the sample acquisition module 702 is configured to execute:

[0174] Performing key point detection on a two-dimensional image including a preset portrait to obtain key point data of the preset portrait;

[0175] Obtaining the height and weight of the preset portrait;

[0176] Constructing a three-dimensional human body model based on the key point data and the height and weight of the preset portrait;

[0177] The three-dimensional human body model is projected to obtain the sample reference image.

[0178] In an exemplary embodiment, the control model includes a feature extractor and a fusion control model, and the training module 703 is configured to perform:

[0179] Inputting the sample reference image into the feature extractor to extract features and obtain sample image features;

[0180] The sample image features are input into the fusion control model for feature fusion to obtain the sample image guiding features.

[0181] In an exemplary embodiment, the feature extractor includes a pre-trained image encoder and a mapping component, wherein the mapping component is used to map information in the image feature space to the text feature space; the training module 703 is configured to perform:

[0182] Inputting the sample reference image into the image encoder for image encoding to obtain a sample image code;

[0183] Input the sample image code into the mapping component for information mapping to obtain the sample image features;

[0184] The adjusting the parameters of the control model based on the predicted image includes:

[0185] While freezing the parameters of the image encoder, the parameters of the mapping component and the fusion control model are adjusted based on the predicted image.

[0186] In an exemplary embodiment, the training module 703 is configured to execute:

[0187] Inputting the sample image features into at least one network layer of the fusion control model for feature fusion to obtain a single-layer guided feature output by each network layer, wherein the single-layer guided feature belongs to the sample image guided feature;

[0188] The step of inputting the sample text feature and the sample image guide feature into the image generator to generate an image and obtain a generated predicted image includes:

[0189] Each of the single-layer guiding features is input into a corresponding network layer in the image generator, and the sample text features are input into at least one network layer in the image generator, so that the image generator generates the predicted image by fusing the sample text features and the sample image guiding features.

[0190] In an exemplary embodiment, the image generator is a diffusion model, and the predicted image generated by the image generator includes an image prediction result corresponding to each diffusion step in at least one diffusion step; the training module 703 is configured to execute:

[0191] Determine, based on a base image, a reference image corresponding to each diffusion step in the at least one diffusion step, wherein the base image includes a portrait with a face, the portrait with a face conforms to the description of the sample text, and the morphology of the portrait with a face is consistent with the sample reference image;

[0192] For any diffusion step of the at least one diffusion step, the parameters of the control model are adjusted based on a difference between a reference image corresponding to the diffusion step and a corresponding image prediction result.

[0193] In an exemplary embodiment, the training module 703 is configured to execute:

[0194] Initialize the current diffusion steps;

[0195] Based on the preset noise and the current number of diffusion steps, denoising the preset noise is performed by fusing the sample text feature and the sample image guide feature to obtain an image prediction result corresponding to the current number of diffusion steps;

[0196] Update the preset noise based on the difference between the preset noise and the image prediction result corresponding to the current diffusion step number;

[0197] Update the current diffusion step number, and when the updated current diffusion step number is less than the diffusion step number threshold, repeat the step of denoising the preset noise based on the preset noise and the current diffusion step number by fusing the sample text feature and the sample image guide feature to obtain the image prediction result corresponding to the current diffusion step number.

[0198] In an exemplary embodiment, the fusion control model and the image generator have the same structure, and initial parameters of the fusion control model are determined based on parameters of the image generator.

[0199] Regarding the device in the above embodiment, the specific manner of each step has been described in detail in the embodiment of the aforementioned method, and will not be elaborated here.

[0200] Figure 8 is a block diagram of an image generating device according to an exemplary embodiment. Figure 8 , the device comprises:

[0201] The target data acquisition module 801 is configured to acquire a target text and a target reference image, wherein the target text is a text content used to describe a portrait in an image to be generated, and the target reference image indicates the form of a full-body portrait in an image to be generated;

[0202] An image generation module 802 is configured to input the target text and the target reference image into an image generation model to obtain a target image, wherein the full-body portrait in the target image includes a face;

[0203] Wherein, the image generation model is trained by the image generation model training method described in any one of the first aspects.

[0204] Regarding the device in the above embodiment, the specific manner of each step has been described in detail in the embodiment of the aforementioned method, and will not be elaborated here.

[0205] Please refer to Fig. 9 , which shows a structural frame of a computer device provided by an exemplary embodiment of the present disclosure Figure 1 The computer device may be a terminal. The computer device is used to implement the image generation model training method or image generation method provided in the above embodiment. Specifically:

[0206] Typically, the computer device 900 includes a processor 901 and a memory 902 .

[0207] The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In an exemplary embodiment, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In an exemplary embodiment, the processor 901 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0208] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In an exemplary embodiment, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one instruction, at least one program, code set or instruction set, and the above at least one instruction, at least one program, code set or instruction set is configured to be executed by one or more processors to implement the above-mentioned image generation model training method or image generation method.

[0209] In an exemplary embodiment, the computer device 900 may further optionally include: a peripheral device interface 903 and at least one peripheral device. The processor 901, the memory 902 and the peripheral device interface 903 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 903 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 904, a touch display screen 905, a camera assembly 906, an audio circuit 907, a positioning assembly 908 and a power supply 909.

[0210] Those skilled in the art will understand that Fig. 9 The structure shown in the figure does not constitute a limitation on the computer device 900, and the computer device 900 may include more or less components than those shown in the figure, or combine some components, or adopt a different arrangement of components.

[0211] Please refer to Fig.10 It shows a structural frame of a computer device provided by another exemplary embodiment of the present disclosure. Figure 2 The computer device may be a server for executing the above-mentioned image generation model training method or image generation method. Specifically:

[0212] The computer device 1000 includes a central processing unit (CPU) 1001, a system memory 1004 including a random access memory (RAM) 1002 and a read only memory (ROM) 1003, and a system bus 1005 connecting the system memory 1004 and the central processing unit 1001. The computer device 1000 also includes a basic input / output system (I / O (Input / Output) system) 1006 for facilitating information transmission between various components in the computer, and a large-capacity storage device 1007 for storing an operating system 1013, application programs 1014, and other program modules 1011.

[0213] The basic input / output system 1006 includes a display 1008 for displaying information and an input device 1009 such as a mouse and a keyboard for user inputting information. The display 1008 and the input device 1009 are connected to the central processing unit 1001 through an input / output controller 1100 connected to the system bus 1005. The basic input / output system 1006 may also include an input / output controller 1100 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1100 also provides output to a display screen, a printer, or other types of output devices.

[0214] The mass storage device 1007 is connected to the central processing unit 1001 through a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1007 and its associated computer readable medium provide non-volatile storage for the computer device 1000. That is, the mass storage device 1007 may include a computer readable medium (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0215] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 1004 and mass storage device 1007 can be collectively referred to as memory.

[0216] According to various embodiments of the present disclosure, the computer device 1000 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 1000 can be connected to the network 1012 through the network interface unit 1011 connected to the system bus 1005, or the network interface unit 1011 can be used to connect to other types of networks or remote computer systems (not shown).

[0217] The above-mentioned memory also includes a computer program, which is stored in the memory and configured to be executed by one or more processors to implement the above-mentioned image generation model training method or image generation method.

[0218] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. When the at least one instruction, the at least one program, the code set or the instruction set is executed by a processor, the image generation model training method or the image generation method is implemented.

[0219] Optionally, the computer readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives) or optical disks, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0220] In an exemplary embodiment, a computer-readable storage medium including a program code is also provided, such as a memory including the program code, and the program code can be executed by a processor to complete the above-mentioned image generation model training method or image generation method. Optionally, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact-disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0221] In an exemplary embodiment, a computer program product is also provided, including a computer program, which implements the above-mentioned image generation model training method or image generation method when executed by a processor.

[0222] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0223] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training an image generation model, characterized in that: include: Get the control model, as well as the pre-trained image generator and text encoder; Acquire sample text and a sample reference image, wherein the sample text is text content used to describe a portrait in an image to be generated, and the sample reference image indicates the form of a full-body portrait in the image to be generated; Inputting the sample text into the text encoder for text encoding to obtain sample text features; Performing feature processing on the sample reference image based on the control model to obtain a sample image guiding feature; Inputting the sample text feature and the sample image guide feature into the image generator to generate an image, thereby obtaining a generated predicted image, wherein the full-body portrait in the predicted image includes a face; Based on the predicted image, the parameters of the control model are adjusted; the adjusted control model, the text encoder and the image generator are combined to obtain an image generation model.

2. The method according to claim 1, characterized in that The sample reference image is a full-body posture schematic diagram of a person in the image to be generated; and the control model is a posture control model.

3. The method according to claim 1, characterized in that The sample reference image is a schematic diagram of the whole body contour including the height and weight of the person in the image to be generated, and the sample reference image indicates the corresponding relationship between the height, the weight and the whole body contour; the control model is a contour control model.

4. The method according to claim 3, characterized in that The obtaining of sample text and sample reference image comprises: Performing key point detection on a two-dimensional image including a preset portrait to obtain key point data of the preset portrait; Obtaining the height and weight of the preset portrait; Constructing a three-dimensional human body model based on the key point data and the height and weight of the preset portrait; The three-dimensional human body model is projected to obtain the sample reference image.

5. The method according to any one of claims 1 to 4, characterized in that: The control model includes a feature extractor and a fusion control model, and the feature processing of the sample reference image based on the control model to obtain the sample image guiding feature includes: Inputting the sample reference image into the feature extractor to extract features and obtain sample image features; The sample image features are input into the fusion control model for feature fusion to obtain the sample image guiding features.

6. The method according to claim 5, characterized in that The feature extractor includes a pre-trained image encoder and a mapping component, wherein the mapping component is used to map information in the image feature space to the text feature space; The step of inputting the sample reference image into the feature extractor for feature extraction to obtain sample image features comprises: Inputting the sample reference image into the image encoder for image encoding to obtain a sample image code; Input the sample image code into the mapping component for information mapping to obtain the sample image features; The adjusting the parameters of the control model based on the predicted image includes: While freezing the parameters of the image encoder, the parameters of the mapping component and the fusion control model are adjusted based on the predicted image.

7. The method according to claim 5, characterized in that The step of inputting the sample image features into the fusion control model for feature fusion to obtain the sample image guiding features includes: Inputting the sample image features into at least one network layer of the fusion control model for feature fusion to obtain a single-layer guided feature output by each network layer, wherein the single-layer guided feature belongs to the sample image guided feature; The step of inputting the sample text feature and the sample image guide feature into the image generator to generate an image and obtain a generated predicted image includes: Each of the single-layer guiding features is input into a corresponding network layer in the image generator, and the sample text features are input into at least one network layer in the image generator, so that the image generator generates the predicted image by fusing the sample text features and the sample image guiding features.

8. The method according to claim 1, characterized in that The image generator is a diffusion model, and the predicted image generated by the image generator includes an image prediction result corresponding to each diffusion step in at least one diffusion step; and adjusting the parameters of the control model based on the predicted image includes: Determine, based on a base image, a reference image corresponding to each diffusion step in the at least one diffusion step, wherein the base image includes a portrait with a face, the portrait with a face conforms to the description of the sample text, and the morphology of the portrait with a face is consistent with the sample reference image; For any diffusion step of the at least one diffusion step, the parameters of the control model are adjusted based on a difference between a reference image corresponding to the diffusion step and a corresponding image prediction result.

9. The method according to claim 8, characterized in that The step of inputting the sample text feature and the sample image guide feature into the image generator to generate an image and obtain a generated predicted image includes: Initialize the current diffusion steps; Based on the preset noise and the current number of diffusion steps, denoising the preset noise is performed by fusing the sample text feature and the sample image guide feature to obtain an image prediction result corresponding to the current number of diffusion steps; Update the preset noise based on the difference between the preset noise and the image prediction result corresponding to the current diffusion step number; Update the current diffusion step number, and when the updated current diffusion step number is less than the diffusion step number threshold, repeat the step of denoising the preset noise based on the preset noise and the current diffusion step number by fusing the sample text feature and the sample image guide feature to obtain the image prediction result corresponding to the current diffusion step number.

10. The method according to claim 5, characterized in that The fusion control model and the image generator have the same structure, and initial parameters of the fusion control model are determined based on parameters of the image generator.

11. An image generation method, characterized in that: include: Acquire a target text and a target reference image, wherein the target text is a text content used to describe a portrait in an image to be generated, and the target reference image indicates the form of a full-body portrait in the image to be generated; Inputting the target text and the target reference image into an image generation model to obtain a target image, wherein the full-body portrait in the target image includes a face; Wherein, the image generation model is trained by the image generation model training method described in any one of claims 1 to 10.

12. An image generation model training device, characterized in that: include: A component acquisition module, configured to execute an acquisition control model, and a pre-trained image generator and text encoder; A sample acquisition module is configured to acquire sample text and sample reference images, wherein the sample text is text content used to describe the portrait in the image to be generated, and the sample reference image indicates the form of the full-body portrait in the image to be generated; The training module is configured to execute: Inputting the sample text into the text encoder for text encoding to obtain sample text features; Performing feature processing on the sample reference image based on the control model to obtain a sample image guiding feature; Inputting the sample text feature and the sample image guide feature into the image generator to generate an image, thereby obtaining a generated predicted image, wherein the full-body portrait in the predicted image includes a face; Based on the predicted image, the parameters of the control model are adjusted; the adjusted control model, the text encoder and the image generator are combined to obtain an image generation model.

13. An image generating device, characterized in that: include: A target data acquisition module is configured to acquire a target text and a target reference image, wherein the target text is a text content used to describe a portrait in an image to be generated, and the target reference image indicates the form of a full-body portrait in an image to be generated; An image generation module is configured to input the target text and the target reference image into an image generation model to obtain a target image, wherein the full-body portrait in the target image includes a face; Wherein, the image generation model is trained by the image generation model training method described in any one of claims 1 to 10.

14. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image generation model training method as described in any one of claims 1 to 10, or the image generation method as described in claim 11.

15. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device executes the image generation model training method as described in any one of claims 1 to 10, or the image generation method as described in claim 11.

16. A computer program product, characterized in that The computer program product includes a computer program, which, when executed by a processor, implements the image generation model training method as described in any one of claims 1 to 10, or the image generation method as described in claim 11.

Images