Image generation model training method and device, and image generation method

Abstract

The application is suitable for the technical field of data processing, and provides an image generation model training method and device and an image generation method. The image generation model training method comprises the following steps: obtaining a training set, wherein the training set comprises a plurality of sample pairs, and each sample pair comprises a text prompt word and an instance image corresponding to the text prompt word; for each sample pair in the training set, extracting a feature of the text prompt word in the sample pair to obtain a text feature, and extracting an appearance feature of the instance image in the sample pair to obtain an appearance feature; and training a to-be-trained image generation model according to the text feature and the appearance feature corresponding to the text feature, to obtain a trained image generation model. Through the above method, the accuracy of the generated image can be improved.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to image generation model training methods, apparatus, image generation methods, electronic devices, computer-readable storage media, and computer program products. Background Technology

[0002] Text-to-image generation is a cutting-edge technology in the field of artificial intelligence that allows users to generate corresponding images or paintings based on simple text descriptions. For example, when a user inputs a text prompt into an image generation model, the model extracts features from the prompt, obtains the corresponding text features, and then generates an image based on those features.

[0003] However, in existing text-to-image generation technologies, the images generated by the image generation models may have a low degree of matching with the text prompts, meaning that the accuracy of the generated images is low.

[0004] Therefore, a new image generation method is needed to solve the above-mentioned technical problems. Summary of the Invention

[0005] This application provides an image generation model training method, apparatus, image generation method, electronic device, computer-readable storage medium, and computer program product, which can solve the problem of low appearance matching between generated images and text prompts.

[0006] In a first aspect, embodiments of this application provide an image generation model training method, including:

[0007] Obtain a training set, which includes multiple sample pairs, each sample pair including a text prompt word and an instance image corresponding to the text prompt word;

[0008] For the sample pairs in the training set, extract the features of the text prompt words in the sample pairs to obtain text features, and extract the appearance features of the instance images in the sample pairs to obtain appearance features;

[0009] Based on the text features and the corresponding appearance features, the image generation model to be trained is trained to obtain the trained image generation model.

[0010] The beneficial effects of the embodiments in this application compared with the prior art are:

[0011] In this embodiment, after obtaining the training set, for each sample pair in the training set, features of the text prompt words in the sample pair are extracted to obtain text features. Furthermore, appearance features of the instance images in the sample pair are extracted to obtain appearance features. Then, based on the text features and their corresponding appearance features, the image generation model to be trained is trained to obtain the trained image generation model. Since the image generation model is trained based on the text features and their corresponding appearance features, the trained image generation model considers not only the text features of the text prompt words but also their appearance features when generating images. This improves the matching degree between the generated image and the text prompt words in terms of appearance, thus improving the accuracy of the generated image.

[0012] Secondly, embodiments of this application provide an image generation method, including:

[0013] Receive text messages;

[0014] Extract the features of the text information;

[0015] An image corresponding to the extracted features is generated using the trained image generation model as described in the first aspect.

[0016] Thirdly, embodiments of this application provide an image generation model training apparatus, comprising:

[0017] A training set acquisition module is used to acquire a training set, which includes multiple sample pairs, and each sample pair includes a text prompt word and an instance image corresponding to the text prompt word.

[0018] The feature extraction module is used to extract the features of the text prompt words in the sample pairs in the training set to obtain text features, and to extract the appearance features of the instance images in the sample pairs to obtain appearance features.

[0019] The model training module is used to train the image generation model to be trained based on the text features and the appearance features corresponding to the text features, so as to obtain the trained image generation model.

[0020] Fourthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in the first or second aspect.

[0021] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the method described in the first or second aspect.

[0022] Sixthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to perform the methods described in the first or second aspect above.

[0023] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0025] Figure 1 This is a schematic flowchart of an image generation model training method provided in an embodiment of this application;

[0026] Figure 2 This is a schematic flowchart of another image generation model training method provided in an embodiment of this application;

[0027] Figure 3 This is a schematic flowchart of an image generation method provided in an embodiment of this application;

[0028] Figure 4 This is a schematic diagram of the structure of an image generation model training device provided in one embodiment of this application;

[0029] Figure 5 This is a schematic diagram of the structure of an image generation apparatus provided in another embodiment of this application;

[0030] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0032] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0033] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0034] Furthermore, in the description of this application and the appended claims, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0035] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0036] With the development of artificial intelligence technology, image generation models can generate corresponding images based on text prompts input by users.

[0037] However, existing image generation models that generate images from text often fail to reflect the visual information of the text prompts. For example, assuming the text prompt is "Generate a person wearing red clothes," an existing image generation model might generate an image that doesn't actually depict a person wearing red clothes. This is because existing image generation models only extract the textual features of the prompts and train the model based on these features. Since training relies on textual features and neglects visual features, the resulting image generation model, when applied in practice, easily loses the visual characteristics of the text prompts.

[0038] In order to improve the matching degree between the images generated by the image generation model and the text prompts in appearance, that is, to improve the accuracy of the generated images, this application provides an image generation model training method.

[0039] In this training method, since the image generation model to be trained is trained based on the text features and appearance features of the text prompt words, it is beneficial to improve the appearance matching degree between the image generated by the trained image generation model and the text prompt words.

[0040] The image generation model training method provided in the embodiments of this application is described below with reference to the accompanying drawings.

[0041] Figure 1 The diagram illustrates a flowchart of an image generation model training method provided in an embodiment of this application. This training method can be applied to electronic devices, such as servers, and is described in detail below:

[0042] S11, Obtain the training set, which includes multiple sample pairs, each of which includes a text prompt word and an instance image corresponding to the text prompt word.

[0043] In this embodiment, the training set typically includes positive and negative sample pairs. In positive sample pairs, the instance image matches the text prompt, while in negative sample pairs, the instance image does not match the text prompt. The text prompt may contain specific scene, style, details, and emotional information to help the subsequent image generation model better understand the user's intent and thus generate high-quality images.

[0044] For example, if the text prompt 1 in a sample pair is "Generate a person wearing red clothes", the instance in instance image 1 is "a child wearing red clothes", the instance in instance image 2 is "a child wearing red clothes on the beach", and the instance in instance image 3 is "a child wearing red clothes in a shopping mall", then text prompt 1 can form a positive sample pair with instance image 1, text prompt 1 can also form a positive sample pair with instance image 2, and text prompt 1 can also form a positive sample pair with instance image 3.

[0045] For example, if the text prompt word 2 in a sample pair is "generate a girl wearing red clothes" and the instance in instance image 4 is "a girl wearing yellow clothes", then the text prompt word 2 can form a negative sample pair with instance image 4.

[0046] In this embodiment of the application, in a positive sample pair, the number of instances included in the instance image corresponding to the text prompt is the same as the number of instances included in the text prompt. For example, when the text prompt includes two (or more) instances, the number of instances included in the instance image corresponding to the text prompt is also two (or more).

[0047] In this embodiment, to improve the generalization ability of the trained image generation model, multiple text prompts can be generated. For each text prompt, multiple instance images corresponding to that text prompt are generated. Based on these text prompts and their corresponding instance images, multiple sample pairs are constructed. That is, in the multiple sample pairs constructed in the training set, the same text prompt corresponds to different instance images. By constructing these sample pairs, the probability that the trained image generation model generates different images when the user inputs the same text prompt is increased, thus improving the generalization ability of the trained image generation model.

[0048] S12, for the sample pairs in the training set, extract the features of the text prompt words in the sample pairs to obtain text features, and extract the appearance features of the instance images in the sample pairs to obtain appearance features.

[0049] In this embodiment, text features refer to information extracted from text prompt words that can represent text content and semantics in numerical or vector form. These text features include one or more of the following: word frequency, topic information, and dependency relationships between words. For a given pair of text prompt words, text features are extracted using text feature extraction methods. These methods include statistical methods (such as bag-of-words model, Term Frequency-Inverse Document Frequency (TF-IDF)), word embedding methods (such as Word2Vec, Bidirectional Encoder Representations from Transformers (BERT)), and deep learning methods (such as Long Short-Term Memory (LSTM)). In practice, other methods can also be selected to extract text features from text prompt words, which will not be listed here.

[0050] In this embodiment, the appearance features of an instance image refer to the attributes of the instance image that can be directly observed visually. These appearance features include one or more of the following: color features, texture features, shape features, and spatial relationship features. For an instance image in a sample pair, the appearance features of the instance image are extracted using an appearance feature extraction method. This method includes: representing the color features of the instance image using a color histogram, representing the texture features of the instance image using an oriented gradient histogram, and identifying the contours or boundaries of the instance image using an edge detection method. In practice, other methods can also be selected to extract the appearance features of the instance image, which will not be listed here.

[0051] S13, Based on the above text features and the above appearance features corresponding to the above text features, train the image generation model to be trained to obtain the trained image generation model.

[0052] The image generation model can be a U-Net: a convolutional network for biomedical image segmentation. It can also be a generative adversarial network (GAN) or other models, which are not limited here.

[0053] In this embodiment, the image generation model to be trained includes an initial image generation model that has not been trained, and an intermediate image generation model that has been trained but still needs further training. After extracting image features from text features, the image generation model to be trained generates the corresponding image based on the extracted image features.

[0054] In this embodiment, training the image generation model actually involves adjusting its parameters. For example, when the image generation model is a U-Net model, if it includes a cross-attention layer, adjusting the parameters of the image generation model includes adjusting the parameters of that cross-attention layer. This cross-attention layer enhances the interaction between image features by calculating the similarity between different image features extracted by the image generation model and dynamically adjusting the weights of these features. When the image generation model includes a cross-attention layer, it helps improve the relevance between the images generated by the model and the text prompts.

[0055] In this embodiment, after obtaining the training set, for each sample pair in the training set, features of the text prompt words in the sample pair are extracted to obtain text features. Furthermore, appearance features of the instance images in the sample pair are extracted to obtain appearance features. Then, based on the text features and their corresponding appearance features, the image generation model to be trained is trained to obtain the trained image generation model. Since the image generation model is trained based on the text features and their corresponding appearance features, the trained image generation model considers not only the text features of the text prompt words but also their appearance features when generating images. This improves the matching degree between the generated image and the text prompt words in terms of appearance, thus improving the accuracy of the generated image.

[0056] In this embodiment of the application, when training the image generation model to be trained, text features, appearance features, or a combination of the text features and appearance features can be used. For example, text features and appearance features can be used as inputs to the image generation model to be trained, and the image generation model can be trained based on the processing results of the image generation model to be trained. As another example, appearance features can be used as inputs to the image generation model to be trained, as well as text features and appearance features as inputs to the image generation model to be trained, and the parameters of the image generation model can be adjusted based on the processing results of both features.

[0057] In some embodiments, when text features and appearance features are used as input to the image generation model to be trained, the parameters of the image generation model to be trained can be adjusted by comparing the consistency between the appearance of the image output by the image generation model to be trained and the appearance of the instance image. At this time, S13 above, training the image generation model to be trained based on the text features and the appearance features corresponding to the text features, includes:

[0058] Extract the appearance features of the instance images from the above sample pairs to obtain the first appearance feature to be evaluated;

[0059] Based on the image generation model to be trained, the text features, and the appearance features corresponding to the text features, an image to be evaluated is generated.

[0060] Extract the appearance features of the image to be evaluated to obtain the second appearance feature to be evaluated;

[0061] The appearance consistency loss is calculated based on the first appearance feature to be evaluated and the second appearance feature to be evaluated. The image generation model to be trained is then trained based on the calculated appearance consistency loss result.

[0062] Among them, the above sample pairs are positive sample pairs, that is, the instance image in the sample pair matches the text prompt word, that is, the instance image can accurately reflect the appearance features of the text prompt word.

[0063] In this embodiment, considering that the instance image can accurately reflect the appearance features of the text prompt, the appearance of the image generated by the image generation model to be trained can be judged to meet the requirements by comparing the appearance features of the instance image with the appearance features of the image generated by the image generation model to be trained. Specifically, the appearance features of the instance image are extracted to obtain the first appearance feature to be evaluated. The text features and the corresponding appearance features are used as input to the image generation model to be trained to obtain the image to be evaluated output by the image generation model. The appearance features of the image to be evaluated are extracted to obtain the second appearance feature to be evaluated. Finally, the difference between the first appearance feature to be evaluated and the second appearance feature to be evaluated is calculated to calculate the appearance consistency loss. The difference can be represented by the distance between the first appearance feature to be evaluated and the second appearance feature to be evaluated. For example, it can be represented by the cosine distance between the first appearance feature to be evaluated and the second appearance feature to be evaluated. Of course, it can also be represented by the Euclidean distance between the first appearance feature to be evaluated and the second appearance feature to be evaluated; this is not limited here.

[0064] Optionally, when the difference (or appearance consistency loss) between the first appearance feature to be evaluated and the second appearance feature to be evaluated is represented by the cosine distance between them, the difference can be calculated using the following formula:

[0065] L apper =1-Cos(φ(I) instance )-φ(G ta ))).

[0066] Among them, L apper Cos(,) represents the appearance consistency loss between the first appearance feature to be evaluated and the second appearance feature to be evaluated, Cos(,) represents the cosine similarity, φ(·) is the encoder for extracting appearance features, and I instance It is an instance image, G ta It refers to images generated using appearance and text features as control conditions for the image generation model to be trained. That is, it uses an encoder with the same structure to extract I... instance Appearance features and extraction of G ta The appearance features are extracted, and then the cosine similarity between the two extracted appearance features is calculated. Finally, the appearance consistency loss is represented by the obtained cosine distance.

[0067] When the loss of appearance consistency is significant, it indicates that I instance Appearance features and G taWhen the similarity of appearance features is low, it is necessary to adjust the parameters of the image generation model to be trained. Backpropagation (BP) can be used for parameter adjustment. The core idea of ​​BP is to calculate the gradient of the appearance consistency loss function with respect to the network parameters using the chain rule, thereby updating the weights and biases of the network in the image generation model to be trained. Of course, in practice, other algorithms (such as genetic algorithms) can also be used to adjust the parameters of the image generation model to be trained; this is not a limitation here.

[0068] In some embodiments, considering the complex relationship between appearance features and text features, i.e., the need for the model to automatically learn feature interactions, the appearance features and text features can be concatenated before training the pre-trained image generation model, and then the concatenated features can be used to train the pre-trained image generation model. Before concatenation, the dimensions of the text features need to be the same as the dimensions of the appearance features. That is, in S12 above, the appearance features of the instance images in the above sample pairs are extracted to obtain the appearance features, including:

[0069] The appearance features of the instance images in the above sample pairs are extracted, and the extracted appearance features are dimensionally processed according to the dimensions of the above text features to obtain the appearance features with the same dimensions as the above text features.

[0070] Specifically, based on the dimension of the text features, if the dimension of the extracted instance image's appearance features is equal to the dimension of the text features, then no dimensionality processing is required for the appearance features; if the dimension of the extracted instance image's appearance features is greater than the dimension of the text features, then dimensionality reduction processing is required for the appearance features, such as through Principal Component Analysis (PCA). Of course, other methods can also be used for dimensionality reduction, such as through Linear Discriminant Analysis (LDA), which is not limited here; if the dimension of the extracted instance image's appearance features is less than the dimension of the text features, then dimensionality increase processing is required for the appearance features (i.e., mapping the appearance features from the original feature space to a higher-dimensional feature space through some mathematical transformation), such as through feature crossing.

[0071] Once appearance features with the same dimensions as text features are obtained, they can be concatenated with the text features. The resulting concatenated features are then used to train the image generation model. Because the concatenated appearance and text features are input as a whole into the image generation model, they help the model better learn the interaction between the appearance and text features, thereby improving the accuracy of the images generated by the trained model.

[0072] In some embodiments, when extracting appearance features from an instance image, one encoder may be used for extraction, or more than one encoder may be used for extraction followed by concatenation. When at least two different types of encoders are used for extraction, the above-mentioned extraction of appearance features from the instance image in the above-mentioned sample pair, and the dimensionality processing of the extracted appearance features according to the dimension of the above-mentioned text features to obtain the appearance features with the same dimension as the above-mentioned text features, includes:

[0073] The appearance features of the instance images in the above sample pairs are extracted using at least two encoders of different types.

[0074] The extracted appearance features are concatenated according to the dimensions of the text features to obtain the appearance features with the same dimensions as the text features.

[0075] Specifically, different types of encoders refer to encoders with different functions. When two different types of encoders are used to extract appearance features from the same instance image, the obtained appearance features usually have some differences.

[0076] Optionally, considering that the extracted appearance features will be used to train the image generation model to be trained along with the text features, and the core idea of ​​the Contrastive Language-Image Pre-training (CLIP) encoder is to use a large amount of paired image and text data for pre-training to learn the alignment relationship between the image and the text, that is, the appearance features extracted by the CLIP encoder have a certain relationship with the text, therefore, in this embodiment of the application, the encoder used to extract appearance features includes the CLIP encoder.

[0077] Optionally, to improve the accuracy of the extracted appearance features, an encoder (let's call it the appearance encoder) can be pre-trained, and the trained appearance encoder can be used to extract appearance features from the instance image. The training process can be as follows:

[0078] Collect multiple images of a large number of different instances, such as images of instances from different angles, under different lighting conditions, and / or with different degrees of occlusion. Multiple images of the same instance constitute positive samples, while images of different instances constitute negative samples. Use these positive and negative samples to train the appearance encoder, resulting in the trained appearance encoder.

[0079] Optionally, the encoder in this embodiment includes a CLIP encoder and a trained appearance encoder. These two encoders are used to extract appearance features from the instance image, and then a Multilayer Perceptron (MLP) is used to concatenate the two extracted appearance features to obtain a concatenated appearance feature with the same dimension as the text feature corresponding to the instance image. The main function of the MLP is to learn the mapping relationship between the two input appearance features and the target output (the output dimension is the same as the text feature dimension), thereby achieving approximation and classification or regression tasks for complex functions.

[0080] In some embodiments, S13 above, training the image generation model to be trained based on the text features and the appearance features corresponding to the text features, includes:

[0081] Based on the image generation model to be trained described above, image features are extracted from the text features to obtain the first image features;

[0082] Based on the image generation model to be trained, image features are extracted from the text features and the appearance features corresponding to the text features to obtain the second image features;

[0083] The semantic alignment loss is calculated based on the first image features and the second image features mentioned above, and the semantic alignment loss calculation result is obtained.

[0084] The image generation model to be trained is trained based on the semantic alignment loss calculation results.

[0085] Specifically, in the process of the image generation model to be trained generating images based on text features, it extracts corresponding image features from the text features and adds noise (such as Gaussian noise) to the extracted image features to make the noise-added image features approximate pure noise. Then, it recovers the image features from the pure noise to obtain the image corresponding to the image features. The noise-adding operation can be performed once or multiple times. When noise is added multiple times, the aforementioned first image feature (or second image feature) can be the image feature obtained after any one noise addition, but the first image feature and the second image feature are different image features obtained after the same noise addition. For example, the first image feature is the image feature extracted from the text features and obtained after the first noise addition, and the second image feature is the image feature extracted from both text features and appearance features and obtained after the first noise addition.

[0086] The semantic alignment loss result is obtained by calculating the semantic alignment loss between the first image feature extracted from text features by the image generation model to be trained and the second image feature extracted from the text features and corresponding appearance features. In other words, the semantic alignment loss result reflects the difference between the image features extracted by the image generation model to be trained when the input only has text features and when it has both text features and appearance features. Therefore, training the image generation model to be trained based on the semantic alignment loss calculation result is beneficial to improving the ability of the trained image generation model to generate image features with more appearance features.

[0087] In some embodiments, considering that the dimension of image features may be greater than 1, and that different dimensions result in different image features, the semantic alignment loss calculation can be performed in conjunction with the dimension of the image features to improve the accuracy of the obtained semantic alignment loss calculation result. In this case, the semantic alignment loss calculation based on the first image feature and the second image feature, to obtain the semantic alignment loss calculation result, includes:

[0088] Based on the aforementioned text features, the aforementioned first image features, the aforementioned second image features, and the dimensions of the aforementioned first image features, semantic alignment loss is calculated to obtain the semantic alignment loss calculation result.

[0089] In this case, the dimensions of the first image feature and the second image feature are the same. Calculating the semantic alignment loss by combining the dimensions of the first image feature is equivalent to calculating the semantic alignment loss by combining the dimensions of the second image feature.

[0090] Specifically, the semantic alignment loss can be calculated using the following formula:

[0091]

[0092] Among them, L align This represents the result of semantic alignment loss calculation. The Softmax function, representing the square of the 2-norm, is a function that transforms an input vector into a probability distribution. K represents the text features (or image features extracted from text features without added noise), d is the dimension of the first or second image features, and Q represents the second. t Q represents the first image feature. ta Indicates the second image feature, Q represents t transpose, Q represents ta The transpose of .

[0093] To more clearly describe the image generation model training method provided in the embodiments of this application, the following is combined with... Figure 2 Describe it.

[0094] exist Figure 2 In this model, the topmost branch (let's call it the first branch) is equivalent to a diffusion model. This diffusion model includes a text encoder and a trainable image generation model, and the trainable image generation model is located at... Figure 2 The position in the middle resembles a book shape.

[0095] exist Figure 2 In the first branch, the text encoder extracts the corresponding text features from the text prompts. The image generation model to be trained then extracts image features based on these text features, adds noise to these image features, and obtains the first image feature (i.e., Q). t The image generated based on the second image feature is G. t .

[0096] In the second branch (i.e., the branch following the first branch), text features and appearance features are concatenated to obtain concatenated features. The image generation model to be trained extracts image features based on these concatenated features, and then adds noise to these image features to obtain the second image features (i.e., Q). ta The image generated based on the first image feature is G. ta .

[0097] In generating G t and G ta Previously, regarding Q t and Q ta The semantic alignment loss is calculated to obtain the corresponding semantic alignment loss calculation result. The parameters of the image generation model to be trained are then adjusted based on the semantic alignment loss calculation result.

[0098] In addition to training the image generation model using the semantic alignment loss calculation results, the appearance consistency loss can also be used to train the image generation model.

[0099] Specifically, a large number of images of different instances are collected, and an encoder is trained based on these images to obtain the appearance encoder.

[0100] Obtain the instance image corresponding to the text prompt word. Extract the appearance features of the instance in the instance image using the appearance encoder and the CLIP encoder. Concatenate the two appearance features extracted by the appearance encoder and the CLIP encoder using MLP to obtain the concatenated appearance features. The concatenated appearance features will be further concatenated with the text features extracted by the text encoder. For details of the concatenation process, please refer to the description of the second branch, which will not be repeated here.

[0101] To ensure that the appearance of the images generated by the image generation model under training is as consistent as possible with the appearance of the instance images, the appearance features extracted by the appearance encoder from the instance images (let's call them the first appearance features to be evaluated) are compared with the appearance features extracted by the appearance encoder from the instance images. ta The extracted appearance features (let's call them the second appearance features to be evaluated) are used to calculate the appearance consistency loss, and the image generation model to be trained is trained based on the appearance consistency loss result obtained from the calculation.

[0102] It should be pointed out that, in Figure 2 In this paper, there are only two encoders for extracting appearance features, namely the CLIP encoder and the appearance encoder. In practice, the encoders for extracting appearance features can also be other encoders or other numbers of encoders, which is not limited here.

[0103] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0104] After training the image generation model, images corresponding to text prompts can be generated based on the trained image generation model.

[0105] Figure 3 A schematic flowchart of an image generation method provided in an embodiment of this application is shown, and is described in detail below:

[0106] S31, Receive text information.

[0107] The text information here is also called text prompts. These text prompts include descriptive information about the image, guiding the image generation model to generate an image that matches the description. Text prompts can contain specific scene, style, details, emotion, and other information.

[0108] S32, Extract the features of the text information.

[0109] In screenshots, text features can be extracted using a text encoder to obtain text features.

[0110] S33 uses a trained image generation model to generate images corresponding to the extracted features.

[0111] Here, the trained image generation model is the model trained according to the image generation model training method provided in the embodiments of this application.

[0112] In this embodiment of the application, since the trained image generation model is a model trained according to the image generation model training method provided in this embodiment of the application, and the trained image generation model incorporates appearance features during the training process, when generating an image corresponding to the extracted features according to the trained image generation model, it is beneficial to improve the matching degree between the appearance of the generated image and the appearance of the text information, that is, it is beneficial to improve the accuracy of the generated image.

[0113] Corresponding to the image generation model training method described in the above embodiments, Figure 4 The diagram shows a structural block diagram of the image generation model training device provided in the embodiments of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0114] Reference Figure 4 The image generation model training device 4 includes: a training set acquisition module 41, a feature extraction module 42, and a model training module 43. Wherein:

[0115] Training set acquisition module 41 is used to acquire a training set, which includes multiple sample pairs, and each sample pair includes a text prompt word and an instance image corresponding to the text prompt word.

[0116] The feature extraction module 42 is used to extract the features of the text prompt words in the sample pairs in the training set to obtain text features, and to extract the appearance features of the instance images in the sample pairs to obtain appearance features.

[0117] The model training module 43 is used to train the image generation model to be trained based on the above-mentioned text features and the appearance features corresponding to the above-mentioned text features, so as to obtain the trained image generation model.

[0118] In this embodiment, after obtaining the training set, for each sample pair in the training set, features of the text prompt words in the sample pair are extracted to obtain text features. Furthermore, appearance features of the instance images in the sample pair are extracted to obtain appearance features. Then, based on the text features and their corresponding appearance features, the image generation model to be trained is trained to obtain the trained image generation model. Since the image generation model is trained based on the text features and their corresponding appearance features, the trained image generation model considers not only the text features of the text prompt words but also their appearance features when generating images. This improves the matching degree between the generated image and the text prompt words in terms of appearance, thus improving the accuracy of the generated image.

[0119] In some embodiments, when the feature extraction module 42 extracts the appearance features of the instance images in the sample pair to obtain the appearance features, it is specifically used for:

[0120] The appearance features of the instance images in the above sample pairs are extracted, and the extracted appearance features are dimensionally processed according to the dimensions of the above text features to obtain the appearance features with the same dimensions as the above text features.

[0121] In some embodiments, the above-mentioned extraction of appearance features from instance images in the sample pair, and dimensional processing of the extracted appearance features according to the dimensions of the text features to obtain appearance features with the same dimensions as the text features, includes:

[0122] The appearance features of the instance images in the above sample pairs are extracted using at least two encoders of different types.

[0123] The extracted appearance features are concatenated according to the dimensions of the text features to obtain the appearance features with the same dimensions as the text features.

[0124] In some embodiments, when the model training module 43 trains the image generation model to be trained based on the text features and the appearance features corresponding to the text features, it is specifically used for:

[0125] Extract the appearance features of the instance images from the above sample pairs to obtain the first appearance feature to be evaluated;

[0126] Based on the image generation model to be trained, the text features, and the appearance features corresponding to the text features, an image to be evaluated is generated.

[0127] Extract the appearance features of the image to be evaluated to obtain the second appearance feature to be evaluated;

[0128] The appearance consistency loss is calculated based on the first appearance feature to be evaluated and the second appearance feature to be evaluated. The image generation model to be trained is then trained based on the calculated appearance consistency loss result.

[0129] In some embodiments, when the model training module 43 trains the image generation model to be trained based on the text features and the appearance features corresponding to the text features, it is specifically used for:

[0130] Based on the image generation model to be trained described above, image features are extracted from the text features to obtain the first image features;

[0131] Based on the image generation model to be trained, image features are extracted from the text features and the appearance features corresponding to the text features to obtain the second image features;

[0132] The semantic alignment loss is calculated based on the first image features and the second image features mentioned above, and the first loss calculation result is obtained.

[0133] The image generation model to be trained is trained based on the first loss calculation results.

[0134] In some embodiments, the calculation of semantic alignment loss based on the first image features and the second image features to obtain a first loss calculation result includes:

[0135] Based on the aforementioned text features, the aforementioned first image features, the aforementioned second image features, and the dimensions of the aforementioned first image features, semantic alignment loss is calculated to obtain the first loss calculation result.

[0136] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0137] Corresponding to the image generation method described in the above embodiments, Figure 5 A structural block diagram of an image generation apparatus provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0138] Reference Figure 5 The image generation device 5 includes: a text information receiving module 51, a feature extraction module 52, and an image generation module 53. Wherein:

[0139] The text information receiving module 51 is used to receive text information.

[0140] The feature extraction module 52 is used to extract the features of the text information.

[0141] Image generation module 53 is used to generate an image corresponding to the extracted features using a trained image generation model.

[0142] In this embodiment of the application, since the trained image generation model is a model trained according to the image generation model training method provided in this embodiment of the application, and the trained image generation model incorporates appearance features during the training process, when generating an image corresponding to the extracted features according to the trained image generation model, it is beneficial to improve the matching degree between the appearance of the generated image and the appearance of the text information, that is, it is beneficial to improve the accuracy of the generated image.

[0143] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 6 of this embodiment includes: at least one processor 60 ( Figure 6 The diagram shows only one processor, a memory 61, and a computer program 62 stored in the memory 61 and executable on the at least one processor 60, which, when executing the computer program 62, implements the steps in any of the above method embodiments.

[0144] The electronic device 6 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. This electronic device may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will understand that... Figure 6 This is merely an example of electronic device 6 and does not constitute a limitation on electronic device 6. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0145] The processor 60 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0146] In some embodiments, the memory 61 may be an internal storage unit of the electronic device 6, such as a hard disk or memory of the electronic device 6. In other embodiments, the memory 61 may be an external storage device of the electronic device 6, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 6. Furthermore, the memory 61 may include both internal and external storage units of the electronic device 6. The memory 61 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0147] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0148] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0149] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above-described method embodiments.

[0150] This application provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps described in the various method embodiments above.

[0151] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0152] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0153] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0154] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0155] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for training an image generation model, characterized in that, include: Obtain a training set, which includes multiple sample pairs, each sample pair including a text prompt word and an instance image corresponding to the text prompt word; For the sample pairs in the training set, extract the features of the text prompt words in the sample pairs to obtain text features, and extract the appearance features of the instance images in the sample pairs to obtain appearance features; Based on the text features and the appearance features corresponding to the text features, the image generation model to be trained is trained to obtain the trained image generation model; The step of training the image generation model to be trained based on the text features and the appearance features corresponding to the text features includes: Based on the image generation model to be trained, image features are extracted from the text features to obtain the first image features; Based on the image generation model to be trained, image features are extracted from the text features and the appearance features corresponding to the text features to obtain the second image features; The semantic alignment loss is calculated based on the first image features and the second image features to obtain the semantic alignment loss calculation result; The image generation model to be trained is trained based on the semantic alignment loss calculation results.

2. The image generation model training method as described in claim 1, characterized in that, The step of extracting appearance features from instance images in the sample pair to obtain appearance features includes: Extract the appearance features of the instance images in the sample pair, and perform dimensional processing on the extracted appearance features according to the dimension of the text features to obtain the appearance features with the same dimension as the text features.

3. The image generation model training method as described in claim 2, characterized in that, The step of extracting appearance features from instance images in the sample pair and performing dimensionality processing on the extracted appearance features according to the dimensions of the text features to obtain appearance features with the same dimensions as the text features includes: The appearance features of the instance images in the sample pair are extracted by at least two encoders of different types. The extracted appearance features are concatenated according to the dimension of the text features to obtain the appearance features with the same dimension as the text features.

4. The image generation model training method as described in claim 1, characterized in that, The step of training the image generation model to be trained based on the text features and the appearance features corresponding to the text features includes: Extract the appearance features of the instance images in the sample pair to obtain the first appearance feature to be evaluated; Based on the image generation model to be trained, the text features, and the appearance features corresponding to the text features, an image to be evaluated is generated; Extract the appearance features of the image to be evaluated to obtain the second appearance feature to be evaluated; The appearance consistency loss is calculated based on the first appearance feature to be evaluated and the second appearance feature to be evaluated, and the image generation model to be trained is trained based on the calculated appearance consistency loss result.

5. The image generation model training method as described in claim 1, characterized in that, The step of calculating the semantic alignment loss based on the first image features and the second image features to obtain the semantic alignment loss calculation result includes: Based on the text features, the first image features, the second image features, and the dimension of the first image features, the semantic alignment loss is calculated to obtain the semantic alignment loss calculation result.

6. An image generation method, characterized in that, include: Receive text messages; Extract the features of the text information; An image corresponding to the extracted features is generated using the trained image generation model as described in any one of claims 1 to 5.

7. An image generation model training device, characterized in that, include: A training set acquisition module is used to acquire a training set, which includes multiple sample pairs, and each sample pair includes a text prompt word and an instance image corresponding to the text prompt word. The feature extraction module is used to extract the features of the text prompt words in the sample pairs in the training set to obtain text features, and to extract the appearance features of the instance images in the sample pairs to obtain appearance features. The model training module is used to train the image generation model to be trained based on the text features and the appearance features corresponding to the text features, so as to obtain the trained image generation model. When the model training module trains the image generation model to be trained based on the text features and the appearance features corresponding to the text features, it is used to: Based on the image generation model to be trained, image features are extracted from the text features to obtain the first image features; Based on the image generation model to be trained, image features are extracted from the text features and the appearance features corresponding to the text features to obtain the second image features; The semantic alignment loss is calculated based on the first image features and the second image features to obtain the semantic alignment loss calculation result; The image generation model to be trained is trained based on the semantic alignment loss calculation results.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program, which, when run, causes the method as described in any one of claims 1 to 6 to be performed.

Images