Image generation model training method and device, electronic equipment and storage medium

Abstract

The invention provides an image generation model training method and device, electronic equipment and a storage medium, and belongs to the technical field of computers. In the method, on the premise that a sample model image corresponding to a sample clothing image is input into an image generation module for training, control data is selectively input, namely, an empty set is selectively input, so that the sample model image corresponding to the sample clothing image is obtained. Or at least one of the sample clothing feature corresponding to the sample clothing object and the sample text feature corresponding to the sample model image, so that the image generation model learns the sample clothing feature and the sample text feature in a more balanced manner. Compared with an image generation model obtained in a traditional training mode, the image generation model in the mode can realize balance control on the result image after deformation of the garment object is directly fused with the model image, so that the visual quality and the aesthetic degree of the result image are improved; and the generalization performance and the control performance of the image generation model are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a training method, device, electronic device and storage medium for an image generation model. Background Art

[0002] With the continuous development of computer technology, image generation technology is also improving. Whether in virtual try-on scenes or in item display scenes, image generation technology for clothing objects plays an important role. Through this technology, the wearing effect image of the clothing object can be quickly obtained based on the tiled image of the clothing object, which greatly improves the convenience. Therefore, how to train the image generation model used to obtain the wearing effect image is a problem that needs to be solved.

[0003] At present, a method of inputting sample clothing images and sample model images into an image generation model for training is generally adopted. The image generation model obtained by the above training method can deform the clothing objects in the clothing images, and then merge the deformed clothing objects with the model objects in the model images, so that the deformed clothing objects replace the original clothing objects in the model images, thereby obtaining the result image.

[0004] However, in the result images generated by the above image generation model, the clothing objects worn by the model objects are prone to unnatural texture distortion, which not only affects the visual quality and aesthetics of the result images, but also reduces the credibility and usability of the result images. Summary of the invention

[0005] The present disclosure provides a training method, device, electronic device and storage medium for an image generation model, which can realize balanced control of the result image, not only improving the visual quality and aesthetics of the result image, but also improving the generalization performance and control performance of the image generation model. The technical solution of the present disclosure is as follows:

[0006] According to one aspect of an embodiment of the present disclosure, a method for training an image generation model is provided, the method comprising:

[0007] For the i-th round of training, a plurality of sample clothing images are input into the feature extraction module of the image generation model of the i-1th round for feature extraction, and sample clothing features of the sample clothing objects in each sample clothing image are obtained, wherein the sample clothing features include at least one of the shape features, color features and texture features of the sample clothing objects;

[0008] For any sample clothing image, a sample model image and control data corresponding to the sample clothing image are input into the image generation module of the image generation model of the i-1th round for denoising to obtain a predicted image, wherein the control data is empty or includes at least one of a sample clothing feature and a sample text feature, a sample model object in the sample model image wears the sample clothing object in the sample clothing image, and the sample text feature is used to represent the feature of the sample model image;

[0009] Based on the differences between the multiple predicted images obtained in the i-th round of training and the corresponding sample model images, the parameters of the feature extraction module of the image generation model of the i-1th round are updated through the loss function to obtain the feature extraction module of the image generation model of the i-th round.

[0010] According to another aspect of an embodiment of the present disclosure, a training device for an image generation model is provided, the device comprising:

[0011] The feature extraction unit is configured to, for the i-th round of training process, input a plurality of sample clothing images into a feature extraction module of the image generation model of the i-1th round to perform feature extraction, and obtain sample clothing features of the sample clothing object in each sample clothing image, wherein the sample clothing features include at least one of the shape features, color features, and texture features of the sample clothing object;

[0012] The prediction unit is configured to, for any sample clothing image, input a sample model image and control data corresponding to the sample clothing image into the image generation module of the image generation model of the i-1th round for denoising to obtain a predicted image, wherein the control data is empty or the control data includes at least one of a sample clothing feature and a sample text feature, the sample model object in the sample model image wears the sample clothing object in the sample clothing image, and the sample text feature is used to represent the feature of the sample model image;

[0013] The updating unit is configured to update the parameters of the feature extraction module of the image generation model of the i-1th round through a loss function based on the differences between the multiple predicted images obtained in the i-th round training process and the corresponding sample model images, so as to obtain the feature extraction module of the image generation model of the i-th round.

[0014] In some embodiments, the feature extraction unit is configured to input multiple sample clothing images into the input layer of the feature extraction module of the i-1th round for the i-th round of training process; perform feature extraction based on each layer of the feature extraction module of the i-1th round to obtain sample clothing features corresponding to the sample clothing objects in each sample clothing image at each layer, and each layer of the feature extraction module includes a residual block, a self-attention mechanism block, and a cross-attention mechanism block.

[0015] In some embodiments, the prediction unit is configured to, for any sample clothing image, input the sample model image corresponding to the sample clothing image into the input layer of the image generation module, and input the control data as conditional data into each layer of the image generation module, each layer of the image generation module including a residual block, a self-attention mechanism block, and a cross-attention mechanism block; perform feature fusion and denoising based on each layer of the image generation module to obtain the predicted image.

[0016] In some embodiments, each layer of the feature extraction module corresponds to each layer of the image generation module one by one;

[0017] The prediction unit is also configured to input the image features corresponding to the previous layer, the sample clothing features corresponding to the layer, and the sample text features corresponding to the layer into the self-attention mechanism block for splicing for any layer of the image generation module to obtain the image features corresponding to the layer, and the sample clothing features corresponding to the layer are the sample clothing features obtained by feature extraction of the layer of the feature extraction module.

[0018] In some embodiments, the apparatus further comprises:

[0019] A text description unit is configured to, for any sample model image, input the sample model image into an image processing module for text description to obtain sample text prompt words, wherein the sample text prompt words are used to describe the sample model image, and the image processing module is used to perform text description on the input image to generate the text prompt words of the input image;

[0020] The encoding unit is configured to input the sample text prompt word into a text encoder for encoding, and use the obtained embedding vector of the sample text prompt word as the sample text feature corresponding to the sample model image.

[0021] In some embodiments, the apparatus further comprises:

[0022] A first generating unit is configured to input a clothing image into the feature extraction module to perform feature extraction, and obtain clothing features of a clothing object in the clothing image, wherein the clothing features include at least one of a shape feature, a color feature, and a texture feature of the clothing object;

[0023] The second generating unit is configured to input the clothing features and the text features corresponding to the text prompt words into the image generating module for denoising to obtain a result image, wherein the text prompt words are used to describe the features of the image to be generated, the result image conforms to the description of the text prompt words and the model object in the result image wears the clothing object.

[0024] In some embodiments, the multiple time steps corresponding to the image generation module are used to represent the step-by-step denoising process of the image generation module, the noise prediction values ​​corresponding to different time steps of the image generation module are different, and the noise prediction value corresponding to any time step is used to denoise the intermediate image obtained in the previous time step;

[0025] The second generating unit is configured to determine, based on the clothing feature and the text feature, a reference noise component, a first noise component and a second noise component through the image generating module, wherein the reference noise component is used to represent the noise value corresponding to the time step when the reference condition data is not input, the reference condition data includes the clothing feature and the text feature, the first noise component is used to represent the noise change amount when the clothing feature is input, and the second noise component is used to represent the noise change amount when the text feature is input; the reference noise component, the first noise component and the second noise component are summed to obtain the noise prediction value corresponding to the time step.

[0026] In some embodiments, the second generating unit is further configured to subtract the noise value when the reference condition data is not input from the noise value when the clothing feature is input to obtain a first difference value, and the first difference value is used to represent the degree of adjustment of the clothing feature to the noise prediction value; and multiply the first weight value and the first difference value to obtain the first noise component, and the first weight value is used to adjust the degree of adjustment of the clothing feature to the noise prediction value.

[0027] In some embodiments, the second generating unit is further configured to subtract the noise value when the reference condition data is not input from the noise value when the text feature is input to obtain a second difference value, and the second difference value is used to represent the degree of adjustment of the text feature on the noise prediction value; and multiply the second weight value and the second difference value to obtain the second noise component, and the second weight value is used to adjust the degree of adjustment of the text feature on the noise prediction value.

[0028] In some embodiments, the second generating unit is further configured to subtract the noise value when the clothing feature is input from the noise value when the reference condition data is input to obtain a third difference value, wherein the third difference value is used to represent the degree to which the text feature adjusts the noise prediction value when the clothing feature adjusts the noise prediction value; and multiply a third weight value and the third difference value to obtain the second noise component, wherein the third weight value is used to adjust the degree to which the text feature adjusts the noise prediction value when the clothing feature adjusts the noise prediction value.

[0029] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0030] one or more processors;

[0031] A memory for storing program codes executable by the processor;

[0032] Among them, the processor is configured to execute the program code to implement the training method of the above-mentioned image generation model.

[0033] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When a program code in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device is enabled to perform the above-mentioned training method of the image generation model.

[0034] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the above-mentioned training method of the image generation model when executed by a processor.

[0035] The disclosed embodiment provides a training method for an image generation model, in which, under the premise of inputting a sample model image corresponding to a sample clothing image into an image generation module for training, by selectively inputting control data, that is, selectively inputting an empty set, or at least one of the sample clothing features corresponding to the sample clothing object and the sample text features corresponding to the sample model image, the image generation model learns the sample clothing features and the sample text features in a more balanced manner. Compared with the image generation model obtained by the traditional training method, which directly deforms the clothing object and fuses it with the model image, the image generation model in this method can achieve balanced control of the result image, which not only improves the visual quality and aesthetics of the result image, but also improves the generalization performance and control performance of the image generation model.

[0036] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0038] Figure 1 It is a schematic diagram of an implementation environment of a training method for an image generation model according to an exemplary embodiment.

[0039] Figure 2 The figure is a flowchart of a method for training an image generation model according to an exemplary embodiment.

[0040] Figure 3 The figure is a flowchart of another method for training an image generation model according to an exemplary embodiment.

[0041] Figure 4 is a schematic diagram of an image generation model according to an exemplary embodiment.

[0042] Figure 5 It is a block diagram of a training device for an image generation model according to an exemplary embodiment.

[0043] Figure 6 It is a block diagram of another training device for an image generation model according to an exemplary embodiment.

[0044] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0045] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0046] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0047] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the sample clothing images, sample model images, clothing images, text prompt words, etc. involved in this disclosure are all obtained with full authorization.

[0048] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a training method for an image generation model according to an exemplary embodiment. Figure 1 The implementation environment specifically includes: a terminal 101 and a server 102. The terminal 101 can be connected to the server 102 via a wireless network or a wired network.

[0049] The terminal 101 may be at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), and a laptop computer. An application may be installed and run on the terminal 101, and the application may generate a result image based on a model image and a text prompt word. The application is associated with the server 102, and the server 102 provides a background service to the terminal 101.

[0050] The terminal 101 may generally refer to one of a plurality of terminals. This embodiment is described by taking the terminal 101 as an example. Those skilled in the art may know that the number of the above terminals may be more or less. For example, the above terminals may be several, or the above terminals may be dozens or hundreds, or a greater number. The embodiment of the present disclosure does not limit the number and device type of the terminals.

[0051] The server 102 is at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Optionally, the number of the above servers may be more or less, which is not limited in the embodiments of the present disclosure. Of course, the server 102 may also include other functional servers to provide more comprehensive and diversified services. In some embodiments, the server 102 undertakes the main computing work, and the terminal 101 undertakes the secondary computing work; or, the server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or, the server 102 and the terminal 101 use a distributed computing architecture for collaborative computing. The server 102 can be connected to the terminal 101 and other terminals via a wireless network or a wired network. Optionally, the number of the above servers may be more or less, which is not limited in the embodiments of the present disclosure.

[0052] Figure 2 is a flowchart of a method for training an image generation model according to an exemplary embodiment. Figure 2 As shown, the method is executed by the terminal and includes the following steps:

[0053] In step S201, for the i-th round of training process, the terminal inputs multiple sample clothing images into the feature extraction module of the image generation model of the i-1th round for feature extraction, and obtains sample clothing features of the sample clothing objects in each sample clothing image, and the sample clothing features include at least one of the shape features, color features and texture features of the sample clothing objects.

[0054] In the disclosed embodiment, the sample clothing image is a front image of the sample clothing object. There is a single sample clothing object in each sample clothing image. The sample clothing object shown in the sample clothing image is not worn by the model. Optionally, the sample clothing image is a front image of the sample clothing object when it is flattened or hung, that is, the sample clothing object is presented in an unobstructed, folded, and relatively unfolded form.

[0055] The image generation model at least includes a feature extraction module and an image generation module. The feature extraction module and the image generation module have the same structure, that is, the layers of the feature extraction module and the image generation module correspond one to one. Before training the image generation model, the feature extraction module and the image generation module have the same structure, and the terminal initializes the parameters of the feature extraction module to the parameters of the image generation module.

[0056] In step S202, for any sample clothing image, the terminal inputs the sample model image and control data corresponding to the sample clothing image into the image generation module for denoising to obtain a predicted image. The control data is empty, or the control data includes at least one of the sample clothing feature and the sample text feature. The sample model object in the sample model image wears the sample clothing object in the sample clothing image, and the sample text feature is used to represent the feature of the sample model image.

[0057] In the disclosed embodiment, the sample model image is a frontal image wearing a sample clothing object. There is only a single sample model object in each sample model image, and the sample clothing object worn by the sample model object is complete. The sample model image and the sample clothing image correspond one to one. For ease of description, the corresponding sample model image and sample clothing image are referred to as a sample image pair. Accordingly, the sample model image and the sample clothing image in each sample image pair correspond to the same sample clothing object.

[0058] The control data is used as conditional data to be input into the image generation module. Optionally, the terminal inputs the sample clothing features and sample text features as conditional data into the image generation module; or, the terminal discards the sample clothing features corresponding to a certain sample image pair, and inputs the sample text features corresponding to the sample image pair as conditional data into the image generation module; or, the terminal discards the sample text features corresponding to a certain sample image pair, and inputs the sample clothing features corresponding to the sample image pair as conditional data into the image generation module; or, the terminal discards both the sample text features and the sample clothing features corresponding to a certain sample image pair, and the conditional data is empty at this time.

[0059] In step S203, the terminal updates the parameters of the feature extraction module of the image generation model of the i-1th round through a loss function based on the differences between the multiple predicted images obtained in the i-th round of training and the corresponding sample model images, thereby obtaining the feature extraction module of the image generation model of the i-th round.

[0060] In the disclosed embodiment, during the training of the image generation model, the parameters of the feature extraction module are updated, and the parameters of the image generation module do not need to be updated. Since the structure and parameters of the image generation module are not adjusted, other functional plug-in modules adapted by the image generation module before training can still be used.

[0061] Optionally, the loss function uses a mean square error loss function to measure the difference between the predicted image and the sample model image. During the iterative training process, the prediction of the image generation model is made more accurate by continuously minimizing the loss value of the loss function.

[0062] The disclosed embodiment provides a training method for an image generation model, in which, under the premise of inputting a sample model image corresponding to a sample clothing image into an image generation module for training, by selectively inputting control data, that is, selectively inputting an empty set, or at least one of the sample clothing features corresponding to the sample clothing object and the sample text features corresponding to the sample model image, the image generation model learns the sample clothing features and the sample text features in a more balanced manner. Compared with the image generation model obtained by the traditional training method, which directly deforms the clothing object and fuses it with the model image, the image generation model in this method can achieve balanced control of the result image, which not only improves the visual quality and aesthetics of the result image, but also improves the generalization performance and control performance of the image generation model.

[0063] In some embodiments, for the i-th round of training, multiple sample clothing images are input into the feature extraction module of the image generation model of the i-1th round for feature extraction, and sample clothing features of the sample clothing objects in each sample clothing image are obtained, including:

[0064] For the i-th round of training, multiple sample clothing images are input into the input layer of the feature extraction module of the i-1th round;

[0065] Feature extraction is performed based on each layer of the feature extraction module of the i-1th round to obtain sample clothing features corresponding to the sample clothing objects in each sample clothing image at each layer. Each layer of the feature extraction module includes a residual block, a self-attention mechanism block, and a cross-attention mechanism block.

[0066] In the disclosed embodiment, through the above multi-level and multi-structure feature extraction process, the sample clothing features obtained by the feature extraction module are more accurate and comprehensive, which improves the accuracy and effectiveness of the feature extraction process. At the same time, each layer of the feature extraction module and the corresponding layer of the image generation module have the same structure, which makes it easier to input the sample clothing features obtained by each layer in the feature extraction module into the corresponding layer of the image generation module, thereby improving the model processing efficiency.

[0067] In some embodiments, for any sample clothing image, a sample model image and control data corresponding to the sample clothing image are input into an image generation module for denoising to obtain a predicted image, including:

[0068] For any sample clothing image, the sample model image corresponding to the sample clothing image is input into the input layer of the image generation module, and the control data is input into each layer of the image generation module as conditional data. Each layer of the image generation module includes a residual block, a self-attention mechanism block, and a cross-attention mechanism block.

[0069] Based on the layers of the image generation module, feature fusion and denoising are performed to obtain the predicted image.

[0070] In the disclosed embodiment, the control data is input as conditional data into the image generation module. Since the control data may be an empty set or include at least one of the sample clothing feature and the sample text feature, the image generation model can learn to generate images when a variety of conditional data are input, which not only improves the quality of the generated images of the image generation model, but also improves the generalization performance of the image generation model.

[0071] In some embodiments, each layer of the feature extraction module corresponds to each layer of the image generation module one by one;

[0072] The steps of feature fusion in the image generation module include:

[0073] For any layer of the image generation module, the image features corresponding to the previous layer, the sample clothing features corresponding to the layer, and the sample text features corresponding to the layer are input into the self-attention mechanism block for splicing to obtain the image features corresponding to the layer. The sample clothing features corresponding to the layer are the sample clothing features obtained by feature extraction of the layer of the feature extraction module.

[0074] In the disclosed embodiment, since the architecture of the feature extraction module and the image generation module are the same, that is, each layer of the feature extraction module corresponds to each layer of the image generation module one by one, the sample clothing features obtained in each layer of the feature extraction module are input into the corresponding layer of the image generation module, which can quickly fuse multi-modal and multi-level features, thereby improving the quality of generated images of subsequent image generation models and improving the generalization performance of the image generation model.

[0075] In some embodiments, the method further comprises:

[0076] For any sample model image, the sample model image is input into the image processing module for text description to obtain sample text prompt words, the sample text prompt words are used to describe the sample model image, and the image processing module is used to perform text description on the input image to generate text prompt words for the input image;

[0077] The sample text prompt words are input into the text encoder for encoding, and the obtained embedding vector of the sample text prompt words is used as the sample text feature corresponding to the sample model image.

[0078] In the disclosed embodiment, the terminal obtains sample text prompt words of the sample model image, and then obtains an embedding vector as a sample text feature, which facilitates the subsequent training of the image generation model through the sample text features, which not only improves the image generation model's processing ability for text prompt words, but also improves the quality of the generated images of the image generation model, thereby improving the overall performance of the image generation model.

[0079] In some embodiments, the method further comprises:

[0080] Inputting the clothing image into the feature extraction module for feature extraction, obtaining clothing features of the clothing object in the clothing image, wherein the clothing features include at least one of the shape features, color features, and texture features of the clothing object;

[0081] The clothing features and the text features corresponding to the text prompt words are input into the image generation module for denoising to obtain a result image. The text prompt words are used to describe the features of the image to be generated. The result image conforms to the description of the text prompt words and the model object in the result image wears the clothing object.

[0082] In the disclosed embodiment, by inputting clothing features and text features to the image generation module, that is, using clothing images and text prompt words as input data, rich information is provided, so that the image generation model can simultaneously learn clothing images and text prompt words, thereby generating images that better meet the needs, and improving the quality of the generated images of the image generation model. At the same time, since both the text prompt words and clothing images can be adjusted by themselves, the user can make the image generation module generate images of different styles and features by inputting different text prompt words and clothing images, thereby improving the flexibility of image generation, meeting diverse needs, and improving the level of personalization.

[0083] In some embodiments, multiple time steps corresponding to the image generation module are used to represent a step-by-step denoising process of the image generation module, the noise prediction values ​​corresponding to different time steps of the image generation module are different, and the noise prediction value corresponding to any time step is used to denoise the intermediate image obtained at the previous time step;

[0084] The steps for determining the noise prediction value corresponding to any time step include:

[0085] Based on clothing features and text features, a reference noise component, a first noise component and a second noise component are determined by an image generation module, wherein the reference noise component is used to represent a noise value corresponding to a time step when reference condition data is not input, the reference condition data includes clothing features and text features, the first noise component is used to represent a noise change amount when clothing features are input, and the second noise component is used to represent a noise change amount when text features are input;

[0086] The reference noise component, the first noise component and the second noise component are summed to obtain a noise prediction value corresponding to the time step.

[0087] In the disclosed embodiment, since the influence of reference condition data such as clothing features and text features is simultaneously considered in the calculation process of the noise prediction value, the image generation model can determine the adjustment noise value according to the clothing features and text features, and then determine the noise prediction value. In this way, the image generation model can better process the input of various reference condition data, enhance the generalization performance of the image generation model, and improve the quality of the generated image of the image generation model.

[0088] In some embodiments, the method further comprises:

[0089] The noise value when the clothing feature is input is subtracted from the noise value when the reference condition data is not input, to obtain a first difference value, wherein the first difference value is used to indicate the degree of adjustment of the clothing feature to the noise prediction value;

[0090] The first weight value and the first difference value are multiplied to obtain a first noise component, and the first weight value is used to adjust the degree to which the clothing feature adjusts the noise prediction value.

[0091] In the embodiment of the present disclosure, by introducing the first weight value in the calculation process of the first noise component, the terminal can adjust the influence of clothing features on the noise prediction value by adjusting the size of the first weight value, thereby balancing clothing retention and text following, thereby improving the flexibility and overall performance of the image generation model.

[0092] In some embodiments, the method further comprises:

[0093] The noise value when the text feature is input is subtracted from the noise value when the reference condition data is not input, to obtain a second difference value, the second difference value is used to indicate the degree of adjustment of the text feature to the noise prediction value;

[0094] The second weight value and the second difference value are multiplied to obtain a second noise component, and the second weight value is used to adjust the degree to which the text feature adjusts the noise prediction value.

[0095] In the embodiment of the present disclosure, by introducing the second weight value in the calculation process of the second noise component, the terminal can adjust the influence of text features on the noise prediction value by regulating the size of the second weight value, thereby balancing clothing retention and text following, thereby improving the flexibility and overall performance of the image generation model.

[0096] In some embodiments, the method further comprises:

[0097] The noise value when the reference condition data is input is subtracted from the noise value when the clothing feature is input to obtain a third difference value, wherein the third difference value is used to indicate the degree to which the text feature adjusts the noise prediction value when the clothing feature adjusts the noise prediction value;

[0098] The third weight value and the third difference value are multiplied to obtain a second noise component, and the third weight value is used to adjust the degree to which the text feature adjusts the noise prediction value when the clothing feature adjusts the noise prediction value.

[0099] In the disclosed embodiment, the above method not only achieves the decoupling of clothing features and text features by adding coupling-related data into the second noise component, but also can adjust the text tracking and clothing retention, thereby improving the flexibility and overall performance of the image generation model.

[0100] Above Figure 2 Shown is a flow chart of a training method for an image generation model of the present disclosure. The training scheme for the image generation model provided by the present disclosure is further elaborated below. Figure 3 is a flowchart of another method for training an image generation model according to an exemplary embodiment. Figure 3 The method is executed by the terminal. The method comprises the following steps:

[0101] In step S301, for the i-th round of training, the terminal inputs a plurality of sample clothing images into the input layer of the feature extraction module of the image generation model of the i-1th round.

[0102] In the disclosed embodiment, the terminal performs multiple rounds of iterative training on the image generation model. The i-th round of training process is one of the exemplary iterative training processes. Wherein, i is an integer greater than 1. In the i-th round of training process, the terminal updates the parameters of the feature extraction module of the i-1-th round. Accordingly, in the case where the i-th round of training process is the first round of training process, the terminal updates the parameters of the initialized feature extraction module, wherein the parameters of the initialized feature extraction module are the same as the parameters of the image generation module in the image generation model; in the case where the i-th round of training process is an intermediate round of training process, the terminal updates the parameters of the feature extraction module obtained in the previous round of training process.

[0103] The image generation model at least includes a feature extraction module and an image generation module. The feature extraction module and the image generation module have the same structure, that is, the layers of the feature extraction module and the image generation module correspond one to one. Before the image generation model is trained, the feature extraction module and the image generation module have the same structure, and the terminal initializes the parameters of the feature extraction module to the parameters of the image generation module. After the image generation model is trained, the structures of the feature extraction module and the image generation module are still the same, but the parameters of the feature extraction module and the image generation module are not necessarily the same.

[0104] Among them, the sample clothing image is a front image of the sample clothing object. Optionally, the sample clothing object displayed in each sample clothing image can be a single piece of clothing; or it can be a complete set of clothing. For example, the sample clothing object is a pair of jeans; or, the sample clothing object is a sports suit, which includes a sports short-sleeved shirt and sports shorts. The sample clothing object displayed in the sample clothing image is not worn by the model. Optionally, the sample clothing image is a front image of the sample clothing object when it is flattened or hung, that is, the sample clothing object appears in an unobstructed, folded, and relatively unfolded form.

[0105] For the convenience of describing the image generation model, see Figure 4 As shown, Figure 4 401 is a schematic diagram of an image generation model according to an exemplary embodiment. 402 is a feature extraction module of the image generation model. Accordingly, the terminal encodes the sample clothing image 401 to obtain a latent vector of the sample clothing image, and inputs the latent vector into the feature extraction module 402.

[0106] In step S302, the terminal performs feature extraction based on each layer of the feature extraction module of the i-1th round to obtain sample clothing features corresponding to the sample clothing objects in each sample clothing image at each layer.

[0107] In the disclosed embodiment, the sample clothing feature includes at least one of the shape feature, color feature and texture feature of the sample clothing object. Wherein, each layer of the feature extraction module includes a residual block, a self-attention mechanism block and a cross-attention mechanism block. The residual block is a unit constituting the residual network (Res Net), and the residual block is used to solve the gradient disappearance or gradient explosion problem of the neural network model during the training process through a jump connection mechanism. The self-attention mechanism block is used to make the neural network model pay attention to the information in the input data, so as to capture richer context information. In the embodiment of the present application, the self-attention mechanism block in the feature extraction module enables the feature extraction module to pay attention to the information in the sample clothing image, so as to generate richer and more accurate sample clothing features. The cross-attention mechanism block is used to make the neural network model pay attention to the information in the input data when generating output data. In the embodiment of the present application, the cross-attention mechanism block in the feature extraction module enables the feature extraction module to pay attention to the information in the sample clothing image when generating sample clothing features.

[0108] During the i-th round of training, the terminal performs feature extraction on the sample clothing image input into the feature extraction model through the residual block, self-attention mechanism block and cross-attention mechanism block of each layer of the feature extraction model to obtain the sample clothing features corresponding to each layer.

[0109] It should be noted that, through the above multi-level and multi-structure feature extraction process, the sample clothing features obtained by the feature extraction module are more accurate and comprehensive, which improves the accuracy and effectiveness of the feature extraction process. At the same time, each layer of the feature extraction module and the corresponding layer of the image generation module have the same structure, which makes it easier to input the sample clothing features obtained in each layer of the feature extraction module into the corresponding layer of the image generation module, thereby improving the model processing efficiency.

[0110] In step S303, for any sample model image, the terminal inputs the sample model image into the image processing module for text description to obtain sample text prompt words, and the sample text prompt words are used to describe the sample model image. The image processing module is used to perform text description on the input image to generate text prompt words for the input image.

[0111] In the disclosed embodiment, the sample model image is a frontal image of the sample model object wearing the sample clothing object. There is only a single sample model object in each sample model image, and the sample model object wears a complete sample clothing object. Usually, a small part of the sample clothing object in the sample model image may be blocked. For example, when the sample clothing object is a shirt, the sleeve part of the shirt may have some wrinkles. The main features of the sample clothing object in the sample model image are usually not blocked. For example, when the sample clothing object is a printed short-sleeved shirt, the printed pattern in the printed short-sleeved shirt is not blocked.

[0112] It should be noted that in the training data set, the sample model images and the sample clothing images correspond one to one. For ease of description, the corresponding sample model images and sample clothing images are referred to as sample image pairs. Accordingly, the sample model images and sample clothing images in each sample image pair correspond to the same sample clothing object. In other words, for each sample image pair, the sample clothing object worn by the sample model object in the sample model image is the same as the sample clothing object in the sample clothing image. For example, if the sample clothing image in a sample image pair is wearing shirt A, then the sample model image in the sample image pair is wearing shirt A. It should be noted that there is no restriction on the bottoms worn by the sample model image.

[0113] In some embodiments, the image processing module is a BLIP model (Bootstrapping Language-Image Pre-training). When a sample model image is input, the image processing module outputs text prompt words of the sample model image. For example, if the sample model image is an image of a girl wearing a dress standing on the grass, the text prompt words of the sample model image at least include girl, dress, grass, etc., and may also include words describing the girl's actions, the color of the dress, the length of the dress, etc., which will not be repeated here.

[0114] To facilitate the description of the process of obtaining sample text prompt words, see Figure 4 As shown. 403 is a sample model image, 404 is an image processing module, and 405 is a text prompt word of the sample model image. Accordingly, the terminal inputs the sample model image 403 into the image processing module 404 to obtain the text prompt word 405 of the sample model image.

[0115] In step S304, the terminal inputs the sample text prompt word into the text encoder for encoding, and uses the obtained embedding vector of the sample text prompt word as the sample text feature corresponding to the sample model image.

[0116] In the disclosed embodiment, the embedding vector is usually a vector in a high-dimensional space, and different dimensions are used to represent the information of the text in different semantic spaces. For ease of description, the sample text prompt word of each sample model image can be called a descriptive tag of the sample model image.

[0117] Through the above method, the terminal obtains the sample text prompt words of the sample model image, and then obtains the embedding vector as the sample text feature, which is convenient for subsequent training of the image generation model through the sample text feature, which not only improves the image generation model's processing ability for text prompt words, but also improves the quality of the generated images of the image generation model, and improves the overall performance of the image generation model.

[0118] In some embodiments, the terminal can obtain the sample text prompt words and corresponding sample text features of each sample clothing image as optional training data for the image generation model. Similarly, the sample text prompt words of each sample clothing image can also be called the descriptive label of the sample clothing image. Accordingly, for any sample clothing image, the terminal inputs the sample clothing image into the image processing module for text description to obtain the sample text prompt words of the sample clothing image; the terminal inputs the sample text prompt words of the sample clothing image into the text encoder for encoding, and uses the obtained embedding vector as the sample text feature corresponding to the sample clothing image.

[0119] It should be noted that whether the terminal uses the sample text prompt words of the sample clothing image to train the image generation model can be set according to actual needs. That is, the terminal can use the sample text prompt words of the sample model image as training data, or can use the sample text prompt words of the sample model image and the sample text prompt words of the sample clothing image as training data at the same time.

[0120] Optionally, before training the image generation model, each sample model image and the sample text prompt word of the sample model image are stored in a one-to-one correspondence; each sample clothing image and the sample text prompt word of the sample clothing image are stored in a one-to-one correspondence. In the above manner, it is convenient to improve the performance of the image generation model in the image recognition task in the subsequent training process of the image generation model. Alternatively, in the process of training the image generation model, an image processing module is used as a training auxiliary module, and a text description is performed in real time for the image input into the image generation model in each iteration process to obtain sample text features, so as to improve the performance of the image generation model.

[0121] It should be noted that the embodiment of the present application is described by taking the image processing module as the training auxiliary module as an example. The above steps S301 to S302 are steps for determining sample clothing features, and the above steps S303 to S304 are steps for determining sample text features. The order of the steps for determining sample clothing features and the steps for determining sample text features is not limited here.

[0122] In step S305, for any sample clothing image, the terminal inputs the sample model image corresponding to the sample clothing image into the input layer of the image generation module, and inputs the control data as conditional data into each layer of the image generation module, where the control data is empty, or the control data includes at least one of the sample clothing feature and the sample text feature.

[0123] In the disclosed embodiment, the sample model object in the sample model image wears the sample clothing object in the corresponding sample clothing image, and the sample text feature is used to represent the feature of the sample model image. Each layer of the image generation module includes a residual block, a self-attention mechanism block, and a cross-attention mechanism block.

[0124] Optionally, the terminal randomly discards sample text features or sample clothing features during the training process. In other words, under the premise of inputting the sample model image into the image generation module, the terminal inputs the sample clothing features and sample text features into the image generation module as conditional data; or, the terminal discards the sample clothing features corresponding to a certain sample image pair, and inputs the sample text features corresponding to the sample image pair into the image generation module as conditional data; or, the terminal discards the sample text features corresponding to a certain sample image pair, and inputs the sample clothing features corresponding to the sample image pair into the image generation module as conditional data; or, the terminal discards both the sample text features and the sample clothing features corresponding to a certain sample image pair, and the conditional data is empty at this time.

[0125] In some embodiments, the feature discard probability is used to indicate the probability of randomly discarding sample text features and sample clothing features during the training process. For example, the feature discard probability is set to 5%. At this time, for every 100 sample image pairs used as training data, the terminal randomly discards the sample text features and sample clothing features of 5 of the sample image pairs. For another example, the feature discard probability is set to 10%. At this time, for every 100 sample image pairs used as training data, the terminal randomly discards the sample text features and sample clothing features of 10 of the sample image pairs.

[0126] It should be noted that the control data is input into the image generation module as conditional data. Since the control data may be an empty set or include at least one of the sample clothing feature and the sample text feature, the image generation model can learn to generate images when a variety of conditional data are input, which not only improves the quality of the generated images of the image generation model, but also improves the generalization performance of the image generation model.

[0127] In some embodiments, before the terminal inputs the sample clothing image into the feature extraction module of the image generation model and inputs the sample model image into the image generation module of the image generation model, the terminal uses a variational autoencoder (VAE) to encode the sample clothing image and the sample model image respectively, and obtains the latent vector corresponding to the sample clothing image and the latent vector corresponding to the sample model image, so as to facilitate the subsequent feature extraction module and the image generation module to process. It should be noted that before the sample model image is input into the image generation module of the image generation model, noise is usually randomly added to the sample model image, and then the sample model image with the random noise added is encoded to obtain the latent vector of the sample model image with the random noise added, so that the image generation module can better learn the denoising process, which will not be repeated here.

[0128] The variational autoencoder is used to encode the input data into data in the latent space, and can also be used to map the data in the latent space back to the original space. For ease of description, the sample clothing image is referred to as I G , the sample model image is called I C The latent vector corresponding to the sample clothing image obtained by the variational autoencoder is I G ∈R 3×H×W , the latent vector I corresponding to the sample model image obtained by the variational autoencoder C ∈R 3×H×W .

[0129] To facilitate the description of the input process of training data, see Figure 4 As shown. 403 is a sample model image, 406 is a sample model image after random noise is added, and 407 is an image generation module in the image generation model. Accordingly, the terminal adds random noise to the sample model image 403 to obtain the sample model image 406 after random noise is added, and the terminal encodes the sample model image 406 after random noise is added to obtain a latent vector of the sample model image 406 after random noise is added, and inputs the latent vector into the image generation module 407. The terminal determines the control data based on the sample image features obtained by the feature extraction module 402 and the text prompt word 405 of the sample model image obtained by the image processing module 404, and inputs the control data into each layer of the image generation module 407 as conditional data.

[0130] In step S306, the terminal performs feature fusion and denoising based on each layer of the image generation module to obtain a predicted image.

[0131] In the disclosed embodiment, since the image generation module is a diffusion model, and the terminal inputs sample model images and control data into the image generation module, the terminal performs feature fusion at each layer of the diffusion model while performing the denoising process of the diffusion model to obtain a predicted image that meets the requirements.

[0132] In some embodiments, the terminal uses a splicing method to perform feature fusion. Accordingly, for any layer of the image generation module, the terminal inputs the image features corresponding to the previous layer, the sample clothing features corresponding to the layer, and the sample text features corresponding to the layer into the self-attention mechanism block for splicing to obtain the image features corresponding to the layer. Among them, each layer of the feature extraction module corresponds to each layer of the image generation module one by one, and the sample clothing features corresponding to any layer are the sample clothing features obtained by feature extraction of the layer of the feature extraction module.

[0133] It should be noted that since the architecture of the feature extraction module and the image generation module is the same, inputting the sample clothing features obtained in each layer of the feature extraction module into the corresponding layer of the image generation module can quickly realize multimodal and multi-level feature fusion, thereby improving the quality of generated images of subsequent image generation models and improving the generalization performance of the image generation model.

[0134] To facilitate the description of the structure of the image generation module, see Figure 4 As shown. Each layer of the feature extraction module 402 corresponds to each layer of the image generation module 407 one by one, and one layer is used for exemplary description. 4021 is a residual block of a layer in the feature extraction module, 4022 is a self-attention mechanism block of a layer in the feature extraction module, and 4023 is a cross-attention mechanism block of a layer in the feature extraction module. Correspondingly, 4071 is a residual block of the corresponding layer in the image generation module, 4072 is a self-attention mechanism block of the corresponding layer in the image generation module, and 4073 is a cross-attention mechanism block of the corresponding layer in the image generation module. The terminal inputs the sample clothing features obtained in this layer in the feature extraction module 402 into the corresponding layer of the image generation module 407 for feature splicing.

[0135] In step S307, the terminal updates the parameters of the feature extraction module of the image generation model of the i-1th round through a loss function based on the differences between the multiple predicted images obtained in the i-th round of training and the corresponding sample model images, thereby obtaining the feature extraction module of the image generation model of the i-th round.

[0136] In the disclosed embodiment, during the training of the image generation model, the parameters of the feature extraction module are updated, and there is no need to update the parameters of the image generation module, the image processing module, etc. Since the structure and parameters of the image generation module are not adjusted, other functional plug-in modules adapted to the image generation module before training can still be used. For example, plug-ins include ControlNet and IP-Adapter, etc. Among them, ControlNet is used to enhance the user's control over the model-generated images. IP-Adapter is used to enhance the processing capabilities of the diffusion model in the text-to-image process. It should be noted that since the feature extraction module is a plug-in module that can be applied to various base diffusion models, in the embodiment of the present application, the base diffusion model is an image generation module and can be used in combination with other functional plug-in modules, thereby improving the control ability of image generation and enhancing the diversity and controllability of image generation. For ease of description, see Figure 4 As shown, 408 is used to indicate other functional plug-in modules.

[0137] Optionally, the loss function is shown in the following formula.

[0138]

[0139] Among them, L is used to represent the loss value. ∈ refers to the sample model image, ∈ θ is the predicted image, and θ is used to indicate the parameters of the image generation model. It is used to represent the latent variable of the sample model image after adding noise at time step t. β is the sample clothing feature obtained by the feature extraction module. t refers to the time step in the diffusion process. T y is the sample text feature, that is, the embedding vector of the sample prompt word, ∈~N(0,1) is used to represent random noise.

[0140] The above loss function uses the mean square error loss function to measure the difference between the predicted image and the sample model image. During the iterative training process, by continuously minimizing the loss value of the loss function, the prediction of the image generation model is made more accurate, thereby generating high-quality images during the use phase of the image generation model.

[0141] It should be noted that, for the image generation model obtained by the above training method, since the feature extraction module in the image generation model can better obtain detailed clothing features, and can fuse the clothing features into the pre-trained base diffusion model, that is, the image generation module, through the self-attention fusion mechanism, it is ensured that the detailed features of the clothing objects in the generated image are consistent with the input clothing image, thereby improving the quality of the generated image. At the same time, since sample text features or sample clothing features are randomly discarded for training during the training process, the image generation model can better balance the degree of learning of clothing features and text features, thereby improving the control ability of the generated image.

[0142] In some embodiments, the terminal uses a trained image generation model to obtain a result image. Accordingly, the terminal inputs the clothing image into a feature extraction module for feature extraction to obtain clothing features of the clothing object in the clothing image, and the clothing features include at least one of the shape features, color features, and texture features of the clothing object; the terminal inputs the clothing features and the text features corresponding to the text prompt words into the image generation module for denoising to obtain a result image, and the text prompt words are used to describe the features of the image to be generated, and the result image conforms to the description of the text prompt words and the model object in the result image wears the clothing object.

[0143] At this time, the terminal does not need to input the model image for the image generation model. By inputting the clothing image and text prompt words, a customized model object wearing the clothing object can be generated. Among them, by using the clothing image and feature extraction module, detailed clothing features are obtained to ensure the consistency of the detailed features of the clothing object; by using a variety of text prompt words and image generation modules, the controllability and details of the generated image are guaranteed. At the same time, since the text prompt words and clothing images can be adjusted by themselves, the user can make the image generation module generate images of different styles and features by inputting different text prompt words and clothing images, which improves the flexibility of image generation, meets diverse needs, and improves the level of personalization.

[0144] In some embodiments, multiple time steps corresponding to the image generation module are used to represent the step-by-step denoising process of the image generation module. The noise prediction values ​​corresponding to different time steps of the image generation module are different. The noise prediction value corresponding to any time step is used to denoise the intermediate image obtained at the previous time step.

[0145] Correspondingly, the step of determining the noise prediction value corresponding to any time step includes: the terminal determines the reference noise component, the first noise component and the second noise component through the image generation module based on the clothing features and the text features, the reference noise component is used to represent the noise value corresponding to the time step when the reference condition data is not input, the reference condition data includes the clothing features and the text features, the first noise component is used to represent the noise change when the clothing features are input, and the second noise component is used to represent the noise change when the text features are input; the terminal sums the reference noise component, the first noise component and the second noise component to obtain the noise prediction value corresponding to the time step.

[0146] It should be noted that, since the influence of reference condition data such as clothing features and text features is taken into account in the calculation process of the noise prediction value, the image generation model can determine the adjustment noise value according to the clothing features and text features, and then determine the noise prediction value. In this way, the image generation model can better handle the input of various reference condition data, enhance the generalization performance of the image generation model, and improve the quality of the generated image of the image generation model.

[0147] In some embodiments, a joint classifier is used to achieve balanced control of clothing features and text features, as shown in the following formula.

[0148]

[0149] Among them, t is the time step and c is the reference condition data. Used to indicate the noise prediction value when inputting reference condition data, Used to represent the noise value when no reference condition data is input, ∈θ (z t ,c) is used to represent the noise value when inputting reference condition data, It is used to indicate the noise change when the reference condition data is input. s is the control intensity parameter, which is used to adjust the degree of adjustment of the reference condition data to the noise prediction value.

[0150] In some embodiments, the terminal determines the first noise component by a first weight value and a first difference value. Accordingly, the terminal subtracts the noise value when the clothing feature is input from the noise value when the reference condition data is not input to obtain a first difference value, and the first difference value is used to indicate the degree of adjustment of the clothing feature to the noise prediction value; the terminal multiplies the first weight value and the first difference value to obtain the first noise component, and the first weight value is used to adjust the degree of adjustment of the clothing feature to the noise prediction value.

[0151] In some embodiments, the terminal determines the second noise component by using two weight values ​​and a second difference value. Accordingly, the terminal subtracts the noise value when the reference condition data is not input from the noise value when the text feature is input to obtain a second difference value, and the second difference value is used to indicate the degree of adjustment of the text feature to the noise prediction value; the terminal multiplies the second weight value and the second difference value to obtain the second noise component, and the second weight value is used to adjust the degree of adjustment of the text feature to the noise prediction value.

[0152] It should be noted that the value ranges of the first weight value and the second weight value are both between 0 and 1. By introducing the first weight value in the calculation process of the first noise component and the second weight value in the calculation process of the second noise component, the terminal can adjust the influence of clothing features and text features on the noise prediction value by adjusting the size of the first weight value and the second weight value, thereby balancing clothing retention and text followability, and improving the flexibility and overall performance of the image generation model.

[0153] To facilitate description of the determination method of the first noise component and the second noise component, refer to the following formula.

[0154]

[0155] Where t is the time step, c T is the text feature, c G For clothing features, s G is the first weight value, s T is the second weight value. is the noise prediction value when the reference condition data is input, is the reference noise component, is the first noise component, is the second noise component.

[0156] In some embodiments, the terminal determines the second noise component by three weight values ​​and a third difference value. Accordingly, the terminal subtracts the noise value when the clothing feature is input from the noise value when the reference condition data is input, and obtains a third difference value, and the third difference value is used to indicate the degree of adjustment of the text feature to the noise prediction value when the clothing feature adjusts the noise prediction value; the terminal multiplies the third weight value and the third difference value to obtain the second noise component, and the third weight value is used to adjust the degree of adjustment of the text feature to the noise prediction value when the clothing feature adjusts the noise prediction value. The third weight value is the same as the second weight value.

[0157] To facilitate description of the method for determining the noise prediction value when the third difference value is used to determine the second noise component, refer to the following formula.

[0158]

[0159] Among them, s T is the third weight value, is the third difference value, is the second noise component.

[0160] It should be noted that the above method not only achieves the decoupling of clothing features and text features by adding coupling-related data to the second noise component, but also can adjust the text followability and clothing retention, thereby improving the flexibility and overall performance of the image generation model.

[0161] The disclosed embodiment provides a training method for an image generation model, in which, under the premise of inputting a sample model image corresponding to a sample clothing image into an image generation module for training, by selectively inputting control data, that is, selectively inputting an empty set, or at least one of the sample clothing features corresponding to the sample clothing object and the sample text features corresponding to the sample model image, the image generation model learns the sample clothing features and the sample text features in a more balanced manner. Compared with the image generation model obtained by the traditional training method, which directly deforms the clothing object and fuses it with the model image, the image generation model in this method can achieve balanced control of the result image, which not only improves the visual quality and aesthetics of the result image, but also improves the generalization performance and control performance of the image generation model.

[0162] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0163] Figure 5 FIG. 1 is a block diagram of a training device for an image generation model according to an exemplary embodiment. Figure 5As shown, the device includes: a feature extraction unit 501, a prediction unit 502 and an update unit 503.

[0164] The feature extraction unit 501 is configured to, for the i-th round of training process, input a plurality of sample clothing images into the feature extraction module of the image generation model of the i-1th round to perform feature extraction, and obtain sample clothing features of the sample clothing objects in each sample clothing image, wherein the sample clothing features include at least one of the shape features, color features, and texture features of the sample clothing objects;

[0165] The prediction unit 502 is configured to, for any sample clothing image, input the sample model image and control data corresponding to the sample clothing image into the image generation module for denoising to obtain a predicted image, wherein the control data is empty or the control data includes at least one of a sample clothing feature and a sample text feature, the sample model object in the sample model image wears the sample clothing object in the sample clothing image, and the sample text feature is used to represent the feature of the sample model image;

[0166] The updating unit 503 is configured to update the parameters of the feature extraction module of the image generation model of the i-1th round through a loss function based on the differences between the multiple predicted images obtained in the i-th round of training and the corresponding sample model images, so as to obtain the feature extraction module of the image generation model of the i-th round.

[0167] In some embodiments, the feature extraction unit 501 is configured to input multiple sample clothing images into the input layer of the feature extraction module of the i-1th round for the i-th round of training process; perform feature extraction based on each layer of the feature extraction module of the i-1th round to obtain sample clothing features corresponding to the sample clothing objects in each sample clothing image in each layer, and each layer of the feature extraction module includes a residual block, a self-attention mechanism block, and a cross-attention mechanism block.

[0168] In some embodiments, the prediction unit 502 is configured to, for any sample clothing image, input the sample model image corresponding to the sample clothing image into the input layer of the image generation module, and input the control data as conditional data into each layer of the image generation module, each layer of the image generation module includes a residual block, a self-attention mechanism block, and a cross-attention mechanism block; perform feature fusion and denoising based on each layer of the image generation module to obtain a predicted image.

[0169] In some embodiments, each layer of the feature extraction module corresponds to each layer of the image generation module one by one;

[0170] The prediction unit 502 is also configured to input the image features corresponding to the previous layer, the sample clothing features corresponding to the layer, and the sample text features corresponding to the layer into the self-attention mechanism block for splicing for any layer of the image generation module, so as to obtain the image features corresponding to the layer, and the sample clothing features corresponding to the layer are the sample clothing features obtained by feature extraction of the layer of the feature extraction module.

[0171] In some embodiments, Figure 6 is a block diagram of another image generation model training device according to an exemplary embodiment. Figure 6 As shown, the device also includes:

[0172] The text description unit 601 is configured to input any sample model image into the image processing module for text description to obtain sample text prompt words, where the sample text prompt words are used to describe the sample model image, and the image processing module is used to perform text description on the input image to generate text prompt words for the input image;

[0173] The encoding unit 602 is configured to input the sample text prompt word into the text encoder for encoding, and use the obtained embedding vector of the sample text prompt word as the sample text feature corresponding to the sample model image.

[0174] In some embodiments, the apparatus further comprises:

[0175] The first generating unit 603 is configured to input the clothing image into the feature extraction module to extract features, and obtain clothing features of the clothing objects in the clothing image, where the clothing features include at least one of the shape features, color features, and texture features of the clothing objects;

[0176] The second generation unit 604 is configured to input the clothing features and the text features corresponding to the text prompt words into the image generation module for denoising to obtain a result image, the text prompt words are used to describe the features of the image to be generated, the result image conforms to the description of the text prompt words and the model object in the result image wears the clothing object.

[0177] In some embodiments, multiple time steps corresponding to the image generation module are used to represent a step-by-step denoising process of the image generation module, the noise prediction values ​​corresponding to different time steps of the image generation module are different, and the noise prediction value corresponding to any time step is used to denoise the intermediate image obtained at the previous time step;

[0178] The second generation unit 604 is configured to determine a reference noise component, a first noise component and a second noise component through an image generation module based on clothing features and text features, wherein the reference noise component is used to represent the noise value corresponding to a time step when reference condition data is not input, the reference condition data includes clothing features and text features, the first noise component is used to represent the noise change when clothing features are input, and the second noise component is used to represent the noise change when text features are input; the reference noise component, the first noise component and the second noise component are summed to obtain a noise prediction value corresponding to the time step.

[0179] In some embodiments, the second generation unit 604 is further configured to subtract the noise value when the reference condition data is not input from the noise value when the clothing feature is input to obtain a first difference value, and the first difference value is used to represent the degree of adjustment of the clothing feature to the noise prediction value; multiply the first weight value and the first difference value to obtain a first noise component, and the first weight value is used to adjust the degree of adjustment of the clothing feature to the noise prediction value.

[0180] In some embodiments, the second generation unit 604 is further configured to subtract the noise value when the text feature is input from the noise value when the reference condition data is not input to obtain a second difference value, and the second difference value is used to represent the degree of adjustment of the text feature to the noise prediction value; multiply the second weight value and the second difference value to obtain a second noise component, and the second weight value is used to adjust the degree of adjustment of the text feature to the noise prediction value.

[0181] In some embodiments, the second generation unit 604 is further configured to subtract the noise value when the clothing feature is input from the noise value when the reference condition data is input to obtain a third difference value, and the third difference value is used to represent the degree of adjustment of the noise prediction value by the text feature when the clothing feature adjusts the noise prediction value; and multiply the third weight value and the third difference value to obtain a second noise component, and the third weight value is used to adjust the degree of adjustment of the noise prediction value by the text feature when the clothing feature adjusts the noise prediction value.

[0182] The disclosed embodiment provides a training device for an image generation model. On the premise that the terminal inputs the sample model image corresponding to the sample clothing image into the image generation module for training, the terminal selectively inputs control data, that is, selectively inputs an empty set, or at least one of the sample clothing features corresponding to the sample clothing object and the sample text features corresponding to the sample model image, so that the image generation model can learn the sample clothing features and the sample text features in a more balanced manner. Compared with the image generation model obtained by the traditional training method, which directly deforms the clothing object and merges it with the model image, the image generation model in this method can achieve balanced control of the result image, which not only improves the visual quality and aesthetics of the result image, but also improves the generalization performance and control performance of the image generation model.

[0183] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

[0184] It should be noted that the training device for the image generation model provided in the above embodiment is only illustrated by the division of the above functional units. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the training device for the image generation model provided in the above embodiment and the training method embodiment of the image generation model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0185] Regarding the training device for the image generation model in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0186] Figure 7 7 is a block diagram of an electronic device according to an exemplary embodiment. Generally, the electronic device 700 includes: a processor 701 and a memory 702 .

[0187] The processor 701 may include one or more processing cores, such as a 4-core processor, a 7-core processor, etc. The processor 701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 701 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0188] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 702 is used to store at least one program code, which is used to be executed by the processor 701 to implement the training method of the image generation model provided by the method embodiment of the present disclosure.

[0189] In some embodiments, the electronic device 700 may further optionally include: a peripheral device interface 703 and at least one peripheral device. The processor 701, the memory 702 and the peripheral device interface 703 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 703 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 704, a display screen 705, a camera assembly 706, an audio circuit 707 and a power supply 708.

[0190] The peripheral device interface 703 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 701 and the memory 702. In some embodiments, the processor 701, the memory 702, and the peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 701, the memory 702, and the peripheral device interface 703 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0191] The radio frequency circuit 704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 704 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 704 can communicate with other electronic devices through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 704 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0192] The display screen 705 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 705 is a touch display screen, the display screen 705 also has the ability to collect touch signals on the surface or above the surface of the display screen 705. The touch signal can be input to the processor 701 as a control signal for processing. At this time, the display screen 705 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 705 can be one, and the front panel of the electronic device 700 is set; in other embodiments, the display screen 705 can be at least two, which are respectively set on different surfaces of the electronic device 700 or are folded; in some other embodiments, the display screen 705 can be a flexible display screen, which is set on the curved surface or folded surface of the electronic device 700. Even, the display screen 705 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 705 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0193] The camera assembly 706 is used to capture images or videos. Optionally, the camera assembly 706 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the electronic device, and the rear camera is arranged on the back of the electronic device. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 706 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0194] The audio circuit 707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 701 for processing, or input them into the radio frequency circuit 704 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the electronic device 700. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 707 may also include a headphone jack.

[0195] The power supply 708 is used to power various components in the electronic device 700. The power supply 708 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 708 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0196] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the electronic device 700, and may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0197] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 702 including instructions, and the instructions can be executed by a processor 701 of an electronic device 700 to complete the training method of the image generation model. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0198] A computer program product includes a computer program, which implements the above-mentioned training method of the image generation model when executed by a processor.

[0199] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0200] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for an image generation model, characterized in that: The method comprises: For the i-th round of training, a plurality of sample clothing images are input into the feature extraction module of the image generation model of the i-1th round for feature extraction, and sample clothing features of the sample clothing objects in each sample clothing image are obtained, wherein the sample clothing features include at least one of the shape features, color features and texture features of the sample clothing objects; For any sample clothing image, a sample model image and control data corresponding to the sample clothing image are input into the image generation module of the image generation model of the i-1th round for denoising to obtain a predicted image, wherein the control data is empty or includes at least one of a sample clothing feature and a sample text feature, a sample model object in the sample model image wears the sample clothing object in the sample clothing image, and the sample text feature is used to represent the feature of the sample model image; Based on the differences between the multiple predicted images obtained in the i-th round of training and the corresponding sample model images, the parameters of the feature extraction module of the image generation model of the i-1th round are updated through the loss function to obtain the feature extraction module of the image generation model of the i-th round.

2. The training method of the image generation model according to claim 1, characterized in that: For the i-th round of training, multiple sample clothing images are input into the feature extraction module of the image generation model of the i-1th round for feature extraction, and sample clothing features of the sample clothing objects in each sample clothing image are obtained, including: For the i-th round of training, a plurality of sample clothing images are input into the input layer of the feature extraction module of the i-1th round; Feature extraction is performed based on each layer of the feature extraction module of the i-1th round to obtain sample clothing features corresponding to the sample clothing objects in each sample clothing image at each layer, wherein each layer of the feature extraction module includes a residual block, a self-attention mechanism block, and a cross-attention mechanism block.

3. The training method of the image generation model according to claim 1, characterized in that: For any sample clothing image, the sample model image and control data corresponding to the sample clothing image are input into the image generation module of the image generation model of the i-1th round for denoising to obtain a predicted image, including: For any sample clothing image, a sample model image corresponding to the sample clothing image is input into an input layer of the image generation module, and the control data is input into each layer of the image generation module as conditional data, wherein each layer of the image generation module includes a residual block, a self-attention mechanism block, and a cross-attention mechanism block; Feature fusion and denoising are performed based on each layer of the image generation module to obtain the predicted image.

4. The training method of the image generation model according to claim 2 or 3, characterized in that: Each layer of the feature extraction module corresponds to each layer of the image generation module one by one; The step of performing feature fusion in the image generation module includes: For any layer of the image generation module, the image features corresponding to the previous layer, the sample clothing features corresponding to the layer, and the sample text features corresponding to the layer are input into the self-attention mechanism block for splicing to obtain the image features corresponding to the layer, and the sample clothing features corresponding to the layer are the sample clothing features obtained by feature extraction of the layer of the feature extraction module.

5. The training method of the image generation model according to claim 1, characterized in that: The method further comprises: For any sample model image, the sample model image is input into the image processing module for text description to obtain sample text prompt words, the sample text prompt words are used to describe the sample model image, and the image processing module is used to perform text description on the input image to generate the text prompt words of the input image; The sample text prompt word is input into a text encoder for encoding, and the obtained embedding vector of the sample text prompt word is used as the sample text feature corresponding to the sample model image.

6. The training method of the image generation model according to claim 1, characterized in that: The method further comprises: Inputting a clothing image into the feature extraction module to perform feature extraction, and obtaining clothing features of the clothing object in the clothing image, wherein the clothing features include at least one of a shape feature, a color feature, and a texture feature of the clothing object; The clothing features and text features corresponding to the text prompt words are input into the image generation module for denoising to obtain a result image, wherein the text prompt words are used to describe the features of the image to be generated, the result image conforms to the description of the text prompt words and the model object in the result image wears the clothing object.

7. The method for training an image generation model according to claim 6, characterized in that: The multiple time steps corresponding to the image generation module are used to represent the step-by-step denoising process of the image generation module, the noise prediction values ​​corresponding to different time steps of the image generation module are different, and the noise prediction value corresponding to any time step is used to denoise the intermediate image obtained in the previous time step; The steps for determining the noise prediction value corresponding to any time step include: Based on the clothing feature and the text feature, determining a reference noise component, a first noise component and a second noise component through the image generation module, the reference noise component is used to represent the noise value corresponding to the time step when reference condition data is not input, the reference condition data includes the clothing feature and the text feature, the first noise component is used to represent the noise change amount when the clothing feature is input, and the second noise component is used to represent the noise change amount when the text feature is input; The reference noise component, the first noise component, and the second noise component are summed to obtain a noise prediction value corresponding to the time step.

8. The method for training an image generation model according to claim 7, characterized in that: The method further comprises: Subtracting the noise value when the reference condition data is not input from the noise value when the clothing feature is input, to obtain a first difference value, wherein the first difference value is used to indicate the degree of adjustment of the clothing feature on the noise prediction value; The first noise component is obtained by multiplying a first weight value and the first difference value, wherein the first weight value is used to adjust the degree to which the clothing feature adjusts the noise prediction value.

9. The method for training an image generation model according to claim 7, characterized in that: The method further comprises: Subtracting the noise value when the reference condition data is not input from the noise value when the text feature is input, to obtain a second difference value, wherein the second difference value is used to indicate the degree of adjustment of the text feature on the noise prediction value; The second noise component is obtained by multiplying a second weight value and the second difference value, wherein the second weight value is used to adjust the degree to which the text feature adjusts the noise prediction value.

10. The method for training an image generation model according to claim 7, characterized in that: The method further comprises: Subtracting the noise value when the clothing feature is input from the noise value when the reference condition data is input, to obtain a third difference value, wherein the third difference value is used to indicate the degree to which the text feature adjusts the noise prediction value when the clothing feature adjusts the noise prediction value; The second noise component is obtained by multiplying a third weight value and the third difference value, wherein the third weight value is used to adjust the degree to which the text feature adjusts the noise prediction value when the clothing feature adjusts the noise prediction value.

11. A training device for an image generation model, characterized in that: The device comprises: The feature extraction unit is configured to, for the i-th round of training process, input a plurality of sample clothing images into a feature extraction module of the image generation model of the i-1th round to perform feature extraction, and obtain sample clothing features of the sample clothing object in each sample clothing image, wherein the sample clothing features include at least one of the shape features, color features, and texture features of the sample clothing object; The prediction unit is configured to, for any sample clothing image, input a sample model image and control data corresponding to the sample clothing image into the image generation module of the image generation model of the i-1th round for denoising to obtain a predicted image, wherein the control data is empty or the control data includes at least one of a sample clothing feature and a sample text feature, the sample model object in the sample model image wears the sample clothing object in the sample clothing image, and the sample text feature is used to represent the feature of the sample model image; The updating unit is configured to update the parameters of the feature extraction module of the image generation model of the i-1th round through a loss function based on the differences between the multiple predicted images obtained in the i-th round training process and the corresponding sample model images, so as to obtain the feature extraction module of the image generation model of the i-th round.

12. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; Wherein, the processor is configured to execute the program code to implement the training method of the image generation model as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method for the image generation model as described in any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the training method for the image generation model as described in any one of claims 1 to 10.

Images