Image generation method and method for training image generation model

Abstract

According to the image generation method and the method for training the image generation model, a feature extraction model is utilized to determine template features corresponding to a template image, the template features comprise a plurality of feature vectors, and in a pre-constructed semantic dictionary, a plurality of semantic base vectors matched with the plurality of feature vectors are determined to serve as target feature vectors corresponding to the template image; the semantic dictionary is constructed based on an image sample set and comprises a plurality of semantic base vectors, each semantic base vector represents the feature of one semantic category, a target image is generated by using an image generation model according to each target feature vector, the semantic base vectors are introduced as limiting conditions and guide information in the image generation process, and the target image is obtained. The problem of unreasonable composition or content distortion caused by overlarge search space in the image generation process can be solved. And on the other hand, the process of determining the target feature vector is simple, convenient and efficient, and the image generation efficiency is not affected while the image generation effect is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification belong to the technical field of data processing, and particularly relate to an image generation method and a method for training an image generation model. BACKGROUND

[0002] With the development of deep learning, image generation technology has also been widely applied in various fields. For example, generative adversarial networks and diffusion models are often used to generate images that meet the text description or specific style.

[0003] However, the existing image generation method still faces some technical problems, such as lack of dynamic adaptability of the model to diversified input, lack of semantic consistency between the generated results and the input, etc. Therefore, the diversity and authenticity of the generated content obtained by the existing image generation technology are insufficient.

[0004] Therefore, it is expected to have an improved image generation scheme to overcome at least part of the above technical problems. SUMMARY

[0005] The purpose of the present application is to provide an image generation method and a method for training an image generation model, comprising:

[0006] The first aspect of the present specification provides an image generation method, comprising:

[0007] determining the template features corresponding to the template image using a feature extraction model, wherein the template features include a plurality of feature vectors;

[0008] determining a plurality of semantic basis vectors matching the plurality of feature vectors in a pre-constructed semantic dictionary as a plurality of target feature vectors corresponding to the template image; the semantic dictionary is constructed based on an image sample set and includes a plurality of semantic basis vectors, each of which represents the features of a semantic category;

[0009] generating a target image according to the target feature vectors using an image generation model.

[0010] In some implementations, wherein in the pre-constructed semantic dictionary, determining a plurality of semantic basis vectors matching the plurality of feature vectors, specifically comprising:

[0011] For any feature vector, determine the semantic basis vector with the smallest difference from the feature vector in the semantic dictionary as the matching semantic basis vector.

[0012] In some implementations, wherein the semantic dictionary is constructed according to the following steps:

[0013] initialize a plurality of semantic basis vectors;

[0014] Determine, by using the feature extraction model, a plurality of first sample features respectively corresponding to a plurality of first sample images in the image sample set, each first sample feature including a plurality of first feature vectors;

[0015] Update the plurality of semantic basis vectors according to distances between the first feature vectors and the plurality of semantic basis vectors.

[0016] In some implementations, the updating of the plurality of semantic basis vectors according to distances between the first feature vectors and the plurality of semantic basis vectors includes:

[0017] Cluster the first feature vectors through a plurality of iterations, taking the initialized plurality of semantic basis vectors as initial cluster centers, wherein each iteration includes updating the cluster centers according to distances between the first feature vectors and the plurality of semantic basis vectors.

[0018] Determine the plurality of semantic basis vectors in the semantic dictionary as cluster centers determined after the plurality of iterations.

[0019] In some implementations, the image generation model is a diffusion model; and the generating of the target image according to the target feature vectors includes:

[0020] Performing a plurality of rounds of denoising operations corresponding to a plurality of time steps, and taking an output image of a last round as the target image, wherein a denoising operation of any round includes:

[0021] Predicting a target noise of the current round according to the target feature vectors and a current time step;

[0022] Performing denoising processing on an input image of the current round and the target noise to obtain a denoised image of the current round; wherein a denoised image output of a non-final round is taken as an input image of a next round; and an input image of a first round is a random noise image.

[0023] In some implementations, the generating of the target image according to the target feature vectors includes:

[0024] Generating the target image according to the target feature vectors and a text encoding representation corresponding to a description text of the template image.

[0025] In some implementations, the image generation model is a diffusion model; and the generating of the target image according to the target feature vectors includes:

[0026] Performing a plurality of rounds of denoising operations corresponding to a plurality of time steps, and taking an output image of a last round as the target image, wherein a denoising operation of any round includes:

[0027] predict target noise of the current round according to the target feature vectors, the text encoding representation, and a current time step;

[0028] perform denoising processing according to the input image of the current round and the target noise to obtain a denoised image of the current round; wherein the denoised image of a non-final round is output as an input image of a next round; and the input image of the first round is a random noise image.

[0029] The second aspect of the present specification provides a method for training an image generation model, comprising:

[0030] determining sample features corresponding to a sample image by using a feature extraction model, wherein the sample features include a plurality of feature vectors;

[0031] determining a plurality of semantic basis vectors matched with the plurality of feature vectors in a pre-constructed semantic dictionary as a plurality of sample feature vectors corresponding to the sample image; the semantic dictionary is constructed based on an image sample set and includes a plurality of semantic basis vectors, and each semantic basis vector represents features of a semantic category;

[0032] generating a predicted image according to the sample feature vectors by using an image generation model;

[0033] updating parameters of the image generation model according to sample image information corresponding to the sample image and predicted image information corresponding to the predicted image.

[0034] In some implementations, the image generation model is a diffusion model; and the method further comprises:

[0035] adding random noise to the sample image at a plurality of time steps to obtain a sample noise image;

[0036] The generating a predicted image according to the sample feature vectors by using an image generation model specifically comprises:

[0037] performing a plurality of rounds of denoising operations corresponding to a plurality of time steps, and taking an output image of a last round as the predicted image, wherein the denoising operation of any round includes:

[0038] predicting target noise of the current round according to the sample feature vectors and a current time step;

[0039] performing denoising processing according to the input image of the current round and the target noise to obtain a denoised image of the current round; wherein the denoised image of a non-final round is output as an input image of a next round; and the input image of the first round is the sample noise image.

[0040] In some implementations, the updating of the parameters of the image generation model according to the sample image information corresponding to the sample image and the prediction image information corresponding to the prediction image comprises:

[0041] obtaining random noise added at each time step as the sample image information, and obtaining target noise predicted at each time step as the prediction image information;

[0042] determining a prediction loss according to the random noise and the target noise at the corresponding time step;

[0043] updating the parameters of the image generation model according to the prediction loss.

[0044] In some implementations, the sample image further has a label text describing the sample image.

[0045] The predicting of the target noise at the current time step according to the sample feature vectors and the current time step comprises:

[0046] The predicting of the target noise according to the sample feature vectors, the text encoding representation of the label text, and the current time step.

[0047] The third aspect of the present specification provides a computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method according to the first aspect or the second aspect.

[0048] The technical scheme of the image generation and the training of the image generation model provided in the present embodiment can utilize a pre-constructed semantic dictionary, introduce a semantic basis vector as a restriction condition and guiding information in the image generation process, and can solve the problem of unreasonable composition or content distortion in the image generation process due to an excessively large search space, and can generate a target image consistent with the semantics of a template image, and improve the authenticity and content diversity of the generated target image. On the other hand, the process of determining the target feature vector is simple and efficient, which improves the image generation effect without affecting the efficiency of image generation. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present specification, the drawings required in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the present specification, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0050] Figure 1 is a technical concept diagram of the image generation method based on semantic features in an embodiment of the present specification;

[0051] Figure 2 is a flowchart of an image generation method according to an embodiment of the present specification;

[0052] Figure 3 is a structural diagram of a template feature according to an embodiment of the present specification;

[0053] Figure 4 is a process diagram of determining an updated template feature according to an embodiment of the present specification;

[0054] Figure 5 is a flowchart of a method for constructing a semantic dictionary according to an embodiment of the present specification;

[0055] Figure 6 is a process diagram of multi-round denoising according to an embodiment of the present specification;

[0056] Figure 7 is a flowchart of a method for training an image generation model according to an embodiment of the present specification. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present specification will be clearly and completely described below with reference to the drawings in the embodiments of the present specification. Obviously, the described embodiments are only a part of the embodiments of the present specification, rather than all the embodiments. Based on the embodiments in the present specification, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present specification.

[0058] The existing image generation technology is extremely dependent on the data set used in the process of training the image generation model. On the one hand, when the data set only contains a specific type of image (here, the type refers to the content category of the image, such as a portrait image or a landscape image), the generated image generated by the trained image generation model is extremely poor when generating images of other types; on the other hand, when the data set contains multiple types of images, the generated images generated by the trained image generation model are difficult to reflect the unique semantic features of different types of images (for example, the body structure in a portrait image or the perspective relationship in a landscape image) when generating images of each type. Therefore, the present specification provides an image generation scheme, which enhances the extraction and expression ability of semantic features to generate high-quality images that are more consistent with the user input semantics.

[0059] Figure 1 The technical concept of the image generation method based on semantic features is shown. As shown in FIG. 1, the image generation method based on semantic features includes the following steps: Figure 1As shown, the overall model architecture for image generation includes an image generation part and a semantic feature matching part. Among them, the image generation part can be designed based on common image generation models, and here only the diffusion model is taken as an example for introduction. In addition, based on the diffusion model for image generation can be divided into unconditional image generation and conditional image generation, here only conditional image generation is discussed.

[0060] Conditional image generation, that is, given a certain preset condition of the image generation model, the image generation model generates an image that meets the preset condition. Usually, the preset condition can be a description text, such as "a running tiger", so that the image generation model generates an image corresponding to the description text; the preset condition can also be a template image, so that the image generation model generates a target image after style conversion or data enhancement according to the template image. In the data enhancement task, the prior art usually extracts features from the template image, and inputs the template features extracted from the template image and a random noise image into the image generation model, so that the image generation model generates a target image. In this process, since the template features contain the semantic features of the template image, the generated target image can have certain similarity with the target image. However, the template features also contain a lot of feature information irrelevant to the semantic features. Such irrelevant feature information may affect the expression of semantic features and thus affect the authenticity of image generation; on the other hand, it also brings unnecessary restrictions to the generation of the target image, affecting the diversity of image generation.

[0061] Therefore, as shown in the semantic feature matching part, in the technical concept as shown in Figure 1 , the template features are not directly used, but after extracting the image features of the template image, the template features are matched with the semantic basis vectors in the pre-established semantic dictionary, and the matched semantic basis vectors are used as the target feature vectors corresponding to the template image. The combination of the target feature vectors is represented as the target semantic features in the figure. Then, as shown in the image generation part, the matched target feature vectors are used to generate a target image with a noise image. Among them, the semantic dictionary can be pre-constructed using an image sample set, and the feature vectors of the image features of each first sample image in the image sample set are clustered to obtain each semantic basis vector in the semantic dictionary. The semantic basis vectors thus determined only contain the common features of a class of feature vectors, that is, semantic features, and do not contain feature information irrelevant to the semantic features. Using the technical concept as shown in Figure 1 avoids the disadvantages of directly inputting the template features into the image generation model, and improves the authenticity and diversity of image generation.

[0062] Figure 2A flowchart of an embodiment of an image generation method is shown, which can be executed by a computing device with computing capability. For convenience, the following describes the image generation method only with the first computing device as the execution subject. The method comprises:

[0063] Step S201: determining a template feature corresponding to the template image by using a feature extraction model, wherein the template feature comprises a plurality of feature vectors.

[0064] Firstly, the first computing device acquires a template image provided by a user. The template image can be pre-stored in a built-in or external storage device of the first computing device, which is not limited in the present specification. After acquiring the template image, the template image is input into a feature extraction model in the first computing device to determine a template feature corresponding to the template image.

[0065] It should be noted that in actual application scenarios, the template image can be provided by the user according to the user's personal requirements for image generation. The template image can represent the user's expectations for the type and content of the target image generated by the model.

[0066] In some implementations, the feature extraction model can be a feature extraction layer in a pre-trained machine learning model for image feature extraction. For example, it can be a feature extraction layer in a Contrastive Language-Image Pretraining (CLIP) model, a Residual Network (ResNet) model, etc.

[0067] Thus, the template feature extracted by the feature extraction model can contain semantic information in the template image. However, since the template feature has not been further processed, each feature vector in the template feature cannot directly represent the semantic information in the template feature.

[0068] In some implementations, the structure of the template feature can be as shown in Figure 3 The template feature is a three-dimensional tensor with dimensions (H*W*D) in the figure, and the template feature contains H*W feature vectors, and the length of each feature vector is D. Combining each feature vector as shown in the figure can obtain the template feature.

[0069] Step S203: determining a plurality of semantic basis vectors matching the plurality of feature vectors in a pre-constructed semantic dictionary as each target feature vector corresponding to the template image; the semantic dictionary is constructed based on an image sample set and comprises a plurality of semantic basis vectors, and each semantic basis vector represents features of a semantic category.

[0070] The semantic dictionary may be pre-stored in a built-in or external storage device of the first computing device, and this specification does not impose any limitation thereto. The semantic dictionary is constructed based on an image sample set, which may include a plurality of first sample images, each of which may include features of a plurality of semantic categories. Feature extraction and further processing of each first sample image may yield semantic basis vectors representing different semantic categories.

[0071] In some implementations, before executing step S203, a feature extraction model can be used by any computing device to determine each first sample feature corresponding to each first sample image in the image sample set, each first sample feature including a plurality of first feature vectors; and each semantic basis vector is determined based on each first feature vector. To determine each semantic basis vector based on each first feature vector, various common clustering methods can be used, such as the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm, the K-Means algorithm, the Agglomerative Nesting (AGNES) algorithm, etc., which are not limited in this specification.

[0072] Specifically, the semantic category represented by each semantic basis vector can represent a type of association between elements in the image. For example, a semantic category corresponding to a landscape image might be "elements in the image have a perspective relationship of near and far." Of course, it should be noted that in the semantic dictionary, there is no need to provide an exact textual description of the semantic category represented by each semantic basis vector; accordingly, in the aforementioned semantic dictionary construction process, there is no need to annotate each first sample image with the label information corresponding to each semantic category.

[0073] After determining the template features corresponding to the template image, the first computing device may determine, in a pre-constructed semantic dictionary, a number of semantic base vectors that match the number of feature vectors as target feature vectors corresponding to the template image.

[0074] In some implementations, each semantic basis vector and the aforementioned feature vector may have the same dimensions, and a difference threshold may be preset. For any feature vector, a semantic basis vector in the semantic dictionary whose difference with the feature vector is less than the difference threshold is used as the target feature vector corresponding to the feature vector. That is, for any feature vector, if there is no target feature vector whose difference with the feature vector is less than the difference threshold, then the feature vector has no corresponding semantic basis vector; if there are multiple semantic basis vectors whose difference with the feature vector is less than the difference threshold, then the feature vector corresponds to multiple target feature vectors.

[0075] Therefore, the determination of the target feature vector is achieved through matching of the semantic base vector and the feature vector without additional training or adjustment of the feature extraction model, which is highly efficient.

[0076] Step S205: generating a target image according to the target feature vectors by using an image generation model.

[0077] After the target feature vectors are determined, the first computing device can input the target feature vectors into the image generation model to generate a target image. In an input space adapted to the image generation model, the computing device can perform at least one of common fusion processing such as splicing, expansion, compression, and mapping on the target feature vectors as input of the image generation model, which is not limited in the specification.

[0078] Since the target feature vectors are not obtained by deforming the template feature, the target feature vectors only contain feature information of the corresponding semantic category and do not contain interference information irrelevant to the semantic category in the template image, and the image generation is performed by using the target feature vectors, the image generation model can focus on the feature information of the semantic category, and the authenticity when generating different types of images is improved.

[0079] In some implementations, the image generation model can be constructed based on a generator structure of a generative adversarial network (GAN), a variational autoencoder (VAE), a diffusion model, etc., which is not limited in the specification.

[0080] It should be noted that the image generation model needs to be pre-trained, so that the image generation model can have the ability to generate images corresponding to each semantic category. The specific process of pre-training the image generation model can refer to steps S203-S205 and the model structure of the image generation model, which is not limited in the specification.

[0081] As shown in an image generation method, Figure 2 by using a pre-constructed semantic dictionary, introducing a semantic base vector as a restriction condition and guide information in the image generation process can solve the problem of unreasonable composition or content distortion due to a too large search space in the image generation process, can generate a target image consistent with the semantics of the template image, and improve the authenticity and content diversity of the generated target image. On the other hand, the process of determining the target feature vector is simple and efficient, which improves the image generation effect without affecting the efficiency of image generation.

[0082] In addition, in some implementations, as shown in Figure 2In step S203, for any feature vector, a semantic base vector with the smallest difference from the feature vector is determined in the semantic dictionary as a matched semantic base vector.

[0083] Thus, for any feature vector in the original template feature, there is only one semantic base vector matched with the feature vector. When the template feature corresponding to the template image includes n feature vectors, step S203 can obtain n target feature vectors corresponding to the template image.

[0084] Figure 4 A process diagram for determining the updated template feature is shown. For any feature vector in the original template feature, after determining the target feature vector matched with the feature vector, the matched target feature vector can be used to replace the feature vector in the template feature, so as to obtain an updated template feature in which each feature vector is replaced by a target feature vector. Then, in step S205, the target image is generated according to the updated template feature.

[0085] Figure 5 A flowchart of a method for constructing a semantic dictionary according to an embodiment of the present specification is shown. The method for constructing the semantic dictionary can be executed by a computing device with computing capability. For convenience, the image generation method is described below only with the second computing device as the execution subject. The method includes:

[0086] Step S501A: initializing a plurality of semantic base vectors.

[0087] First, the second computing device can be initialized to obtain a plurality of semantic base vectors. The number of the initialized semantic base vectors can be set by the user. It should be noted that the number of the initialized semantic base vectors is the same as the number of the semantic base vectors in the constructed semantic dictionary.

[0088] The process of initializing the plurality of semantic base vectors can be a completely random generation process, or a generation process with reference to certain restriction rules, such as a restriction rule for the distance between any two initialized semantic base vectors, which is not limited in the present specification.

[0089] In some implementations, the dimension of the initialized semantic base vector can be determined with reference to the dimension of the feature vector (or the first feature vector) in the feature extraction model output result.

[0090] Step S501B: determining, by using the feature extraction model, each first sample feature corresponding to each first sample image in the image sample set, each first sample feature including a plurality of first feature vectors.

[0091] In another aspect, the second computing device can determine, by using a feature extraction model in the second computing device, each first sample feature corresponding to each first sample image in the image sample set. Each first sample feature includes a plurality of first feature vectors corresponding to the template features. Specifically, when the image sample set includes N first sample images, each first sample image corresponds to a first sample feature, and each first sample feature includes H*W first feature vectors, the step S501B can determine N*H*W first feature vectors in total.

[0092] It should be noted that the model structure and the model parameters of the feature extraction model used in the step S503 are the same as those used in the step S201. Thus, the dimensions of each template feature and each first sample feature are also the same. Further, the characteristics of the first sample features can refer to the description of the template features in the foregoing text.

[0093] Step S503: updating the plurality of semantic base vectors according to the distances between each first feature vector and the plurality of semantic base vectors.

[0094] After determining each first feature vector, the second computing device can update each semantic base vector according to the distance between each first feature vector and each semantic base vector initialized in the step S501. Then, the constructed semantic dictionary is obtained according to the updated semantic base vectors.

[0095] Further, in the step S501B as shown in the following, the initialized plurality of semantic base vectors are taken as initial clustering centers, and each first feature vector is clustered through multiple iterations, wherein each iteration includes updating the clustering centers according to the distances between each first feature vector and the plurality of semantic base vectors. Further, when a preset completion condition is met, each clustering center that meets the completion condition is determined as a semantic base vector in the semantic dictionary. Figure 5

[0096] The completion condition can be that the number of iterations reaches a certain number, or that the update amplitude of the clustering center meets a certain limit, which is not limited in the present specification.

[0097] Specifically, in one iteration process, first, the clustering centers of the current iteration are determined. Then, for any first feature vector, the clustering center closest to the first feature vector is determined as the clustering center matched with the first feature vector. Then, for any clustering center, the clustering center is updated so that the sum of the distances between the updated clustering center and the first feature vectors matched with the clustering center is minimized. Specifically, the following formula can be referred to:

[0098]

[0099] ​Among them, e k Can be any cluster center, is the first eigenvector that matches the cluster center, K is the total number of first eigenvectors that match the cluster center, e' k is the updated cluster center, Indicates Get the minimum value of e k The value of .

[0100] Thus, through multiple rounds of iteration, each cluster center can gradually become independent of each other, and each cluster center has the ability to represent a specific type of first feature vector. After multiple rounds of iteration, each cluster center is determined as a plurality of semantic basis vectors in the constructed semantic dictionary. On the one hand, each semantic basis vector can be capable of representing the feature information of a class of semantic categories, and on the other hand, each semantic basis vector can represent the feature information of different semantic categories. When the semantic categories contained in each first sample image in the image sample set are relatively comprehensive, and the number of semantic basis vectors in the semantic dictionary is set to be sufficient, the semantic dictionary can have the ability to represent the feature information of various semantic categories.

[0101] because Figure 5 The method shown is the same as Figure 2 The methods shown can be performed independently, using Figure 5 The method shown can improve the performance of the image generation model or feature extraction model without any additional modification or training. Figure 2 The diversity and accuracy of the images generated by the shown methods.

[0102] In some implementations, the image generation model is a diffusion model; Figure 2 In step S205 shown, multiple rounds of denoising operations corresponding to multiple time steps are performed, and the output image of the last round is used as the target image, wherein the denoising operation of any round includes: predicting the target noise of this round based on the target feature vectors and the current time step, performing denoising processing based on the input image of this round and the target noise to obtain the denoised image of this round; wherein the denoised image of a non-last round is output as the input image of the next round; and the input image of the first round is a random noise image.

[0103] The diffusion model treats the random noise image as a target image with added noise. Through multiple rounds of denoising, the noise is gradually removed from the random noise image, and the resulting target image is "restored." In step S205, the total number of rounds of denoising can be determined based on the total number of rounds set during the training of the image generation model. Hereinafter, the total number of rounds of denoising is denoted as T.

[0104] Figure 6The process diagram of multiple rounds of denoising is shown. In any round, the first computing device takes each target feature vector and the time label corresponding to the current time step as the model input of the image generation model, obtains the target noise of the current round as the model output, then uses the target noise to denoise the input image to obtain the denoised image of the current round, and if the current round is not the last round, the denoised image of the current round is taken as the input image of the next round; if the current round is the last round, the output image of the current round, the denoised image, is taken as the target image.

[0105] The input image of the first round can be a random noise image, which can be collected from a Gaussian distribution, and the parameters of the Gaussian distribution can be set by the user according to the parameters of the generated image required; the input image can be directly removed from the target noise, or the target noise can be removed from the input image according to the noise weight corresponding to the current round, and the target noise weighted according to the noise weight. The noise weight corresponding to each round can be set by the user, and only the noise weight corresponding to each round is the same as when the image generation model is trained, which is not limited in this specification. The noise weight can simulate the interference degree when noise is added to the image, and by setting the gradient noise weight of each time step, the image generation model can have the ability to gradually "recover" the target image from the noise.

[0106] For any round, the denoising process combined with the noise weight can refer to the following formula:

[0107]

[0108] wherein x t is the input image of the current round, t represents the time label corresponding to the current time step, is each target feature vector, θ is the model parameter of the image generation model, β t is the noise weight of the current round, i.e., the time step t corresponding round, ∈ t is the target noise of the current round, x t-1 is the input image of the next round, i.e., the time step t-1 corresponding round, after removing the target noise. Thus, the random noise image x T is denoised for T rounds to obtain the target image x0.

[0109] It should be noted that when a total of T rounds of denoising are required, the round corresponding to the time step T is the first round, and the input image of the first round is the random noise image x T , and the round corresponding to the time step "1" is the last round, and the output image of the last round is the target image x0, which can also be regarded as corresponding to the time step "0".

[0110] In another aspect, in some implementations, after the description text is introduced, in a step S205 as shown in Figure 2 The description text can be determined by the user, representing the user's content limitation on the target image to be generated. For example, the description text can be "a tiger in a zoo", "a happy child running", etc.

[0111] The description text can be determined by the user, representing the user's content limitation on the target image to be generated. For example, the description text can be "a tiger in a zoo", "a happy child running", etc.

[0112] The description text can be determined by the user, representing the user's content limitation on the target image to be generated. For example, the description text can be "a tiger in a zoo", "a happy child running", etc.

[0113] After the description text is introduced, if the image generation model is a diffusion model, in some implementations, in a step S205 as shown in Figure 2 The description text can be determined by the user, representing the user's content limitation on the target image to be generated. For example, the description text can be "a tiger in a zoo", "a happy child running", etc.

[0114] After the description text is introduced, the denoising process can refer to the following formula:

[0115]

[0116] wherein E represents the text encoding representation, and the meanings of other variables are the same as those in formula (2).

[0117] It should be noted that in a one-time image generation process as shown in Figure 2 The text encoding representation input into the image generation model in different rounds is the same.

[0118] Therefore, the introduction of the description text further controls the generation process of the target image, so that the content of the generated target image is more in line with the user's needs.

[0119] Figure 7 The flowchart shows a method for training an image generation model according to an embodiment of the present specification. The image generation model trained by using the method can perform the steps as shown in Figure 2Step S205 in the method shown: generating a target image according to each target feature vector. The method for training the image generation model can be executed by a computing device with computing capability. For the sake of convenience, the following description of the image generation method is based on the third computing device as the execution subject. The method includes:

[0120] Step S701: determining sample features corresponding to a sample image by using a feature extraction model, wherein the sample features include a plurality of feature vectors.

[0121] The sample image can be derived from an image sample set used when the method shown in the foregoing is executed, or from other sample sets other than the image sample set, which is not limited in the present specification. Figure 5 The sample image can be derived from an image sample set used when the method shown in the foregoing is executed, or from other sample sets other than the image sample set, which is not limited in the present specification.

[0122] Other implementation details of step S701 can refer to the foregoing description of step S201. It should be noted that the same feature extraction model as that used in step S201 is required in step S701.

[0123] Step S703: determining a plurality of semantic base vectors matching the plurality of feature vectors in a pre-constructed semantic dictionary as a plurality of sample feature vectors corresponding to the sample image; the semantic dictionary is constructed based on an image sample set and includes a plurality of semantic base vectors, each of which represents features of a semantic category.

[0124] Implementation details of step S703 can refer to step S203. It should be noted that the same semantic dictionary as that used in step S203 is required in step S703.

[0125] Step S705: generating a predicted image according to the plurality of sample feature vectors by using an image generation model.

[0126] Implementation details of step S705 can refer to step S205.

[0127] Step S707: updating parameters of the image generation model according to sample image information corresponding to the sample image and predicted image information corresponding to the predicted image.

[0128] Specifically, the parameters of the image generation model can be updated by using an updating method corresponding to the structure of the image generation model. Further, according to different updating methods, the sample image information can be the sample image itself or related information determined according to the sample image; and the predicted image information can be the predicted image itself or related information determined according to the predicted image.

[0129] For example, when the image generation model is constructed based on the structure of a generative adversarial network, the image generation model can be a generator in the generative adversarial network. In step S707, the third computing device can input the predicted image and the sample image into a discriminator corresponding to the generator to determine the discrimination results of the predicted image and the sample image, i.e., whether the two are images generated by the generator. The sample image information can be the discrimination result and the authenticity label corresponding to the sample image, and the predicted image information can be the discrimination result and the authenticity label corresponding to the predicted image. According to the difference between the discrimination results and the authenticity labels corresponding to the sample image and the predicted image, the parameters of the image generation model and the discriminator are updated by using a preset loss function.

[0130] It should be noted that when the image generation model is constructed based on the structure of a generative adversarial network, only the generator in the generative adversarial network is used when the method shown in step S205 is performed by using the trained image generation model. Figure 2

[0131] The image generation model trained by the method shown in step S205 can introduce the semantic basis vector in the semantic dictionary as a restriction condition and guiding information in the image generation process, so that the unreasonable composition or content problem caused by a too large search space in the image generation process can be solved, and the content diversity and authenticity of the generated target image can be improved. Figure 7

[0132] In some implementations, the image generation model is a diffusion model, and before step S705 shown in step S705, the method further includes adding random noise based on the sample image at multiple time steps to obtain a sample noise image; and in step S705 shown in step S705, a plurality of rounds of denoising operations corresponding to the multiple time steps are performed, and an output image of the last round is taken as the predicted image. Any round of denoising operation includes predicting a target noise of the current round according to the sample feature vectors and the current time step, and performing denoising processing on an input image of the current round and the target noise to obtain a denoised image of the current round. The denoised image of the non-final round is output as the input image of the next round. The input image of the first round is the sample noise image. Figure 7 Figure 7 It should be noted that when the image generation model is constructed based on the structure of a generative adversarial network, only the generator in the generative adversarial network is used when the method shown in step S205 is performed by using the trained image generation model.

[0133] It should be noted that when the image generation model is constructed based on the structure of a generative adversarial network, only the generator in the generative adversarial network is used when the method shown in step S205 is performed by using the trained image generation model.

[0134] The sample noise image obtained by adding random noise can be obtained according to the following formula:

[0135] ​​​

[0136] The adding random noise process represented by formula (4) is exactly the reverse process of the denoising process represented by the aforementioned formula (2). Wherein, x' t represents the noisy image corresponding to time step t, x' t-1 represents the noisy image corresponding to time step t-1, β t is the noise weight corresponding to time step t, ∈' t is the random noise corresponding to time step t, which can also be collected from a Gaussian distribution. Thus, by performing the operation of adding random noise for T rounds on the noisy image corresponding to time step "0", i.e. the sample image x'0, the noisy image corresponding to time step T, i.e. the sample noise image x' T .

[0137] For the multiple-round denoising operation in step S705, reference can be made to the introduction of formula (2) above, and the present specification will not be repeated here.

[0138] Further, in some implementations, in step S707 as shown in Figure 7 , the noisy image corresponding to each time step is obtained as the sample image information, and the denoised image corresponding to each time step is obtained as the predicted image information; the prediction loss is determined according to the noisy image and the denoised image corresponding to the time step; and the parameters of the image generation model are updated according to the prediction loss.

[0139] In addition, in some implementations, in step S707 as shown in Figure 7 , the random noise added at each time step is obtained as the sample image information; the target noise predicted at each time step is obtained as the predicted image information, the prediction loss is determined according to the random noise and the target noise corresponding to the time step, and the parameters of the image generation model are updated according to the prediction loss.

[0140] Wherein, the prediction loss can be determined by using common loss functions such as mean square error loss function, absolute error loss function, and cosine distance loss function, which are not limited in the present specification.

[0141] Since there is a corresponding prediction loss for each time step, and the parameters of the image generation model are adjusted according to the prediction loss, it can be ensured that the image generation model has a step-by-step and accurate denoising capability in the training process as shown in Figure 7 .

[0142] In some implementations, a number of target time steps can be determined in a plurality of time steps, and the prediction loss is determined according to the difference between the random noise and the target noise of each target time step.

[0143] The more target time steps are, the more the computing power is consumed in a training process, and the faster the convergence rate of the corresponding model parameters is. Therefore, in the training process, the user can set the number of target time steps according to the computing power condition of the user, adjust the training efficiency of the method as shown in Figure 7 .

[0144] On the other hand, the sample image also has a label text describing it; in step S705 as shown in Figure 7 , according to the sample feature vector, the text encoding representation of the label text and the current time step, the target noise is predicted.

[0145] The content and form of the label text can refer to the description text of the template image described above. The method of text encoding of the label text needs to be the same as the method of text encoding of the description text.

[0146] Therefore, the image generation model trained by the method as shown in Figure 7 can use images and texts as model inputs together, and can more finely meet the image generation needs of the user.

[0147] In some implementations, if the image generation model is a diffusion model, in step S705 as shown in Figure 1 , multiple rounds of denoising operations corresponding to multiple time steps are performed, and the output image of the last round is taken as the predicted image, wherein the denoising operation of any round includes: predicting the target noise of the current round according to the sample feature vector, the text encoding representation of the label text and the current time step, performing denoising processing on the input image of the current round and the target noise to obtain the denoised image of the current round, wherein the denoised image of the non-final round is output as the input image of the next round; the input image of the first round is the sample noise image.

[0148] In the 1990s, it was quite obvious to distinguish whether an improvement in a technology was in hardware (e.g., improvement in circuit structures of diodes, transistors, switches, etc.) or in software (improvement in method flow). However, as technology has evolved, many improvements in method flow today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flow into hardware circuits. Therefore, it cannot be said that an improvement in a method flow cannot be implemented by hardware entity modules. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a PLD by the designer programming it, rather than by asking a chip manufacturer to design and fabricate a custom integrated circuit chip. Moreover, instead of manually fabricating integrated circuit chips, this programming is now mostly implemented by "logic compiler" software, which is similar to software compilers used in program development, and the original code to be compiled is written in a specific programming language, which is called a hardware description language (HDL), and there are many such languages, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should be aware that, as long as the method flow is logically programmed in the above-mentioned hardware description languages and programmed into an integrated circuit, a hardware circuit implementing the logical method flow can be easily obtained.

[0149] The controller can be implemented in any suitable way, for example, the controller can take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to being implemented in pure computer readable program code form, the controller can perfectly well be implemented to perform the same functions in the form of logic gates, switches, an application specific integrated circuit, a programmable logic controller and an embedded microcontroller, etc. by means of logical programming of the method steps. Such a controller can thus be considered a hardware component, and the means comprised therein for performing the various functions can also be considered structures within the hardware component. Alternatively, or even, the means for performing the various functions can be considered both software modules implementing the method and structures within the hardware component.

[0150] The systems, apparatuses, modules or units illustrated by the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, the present application does not rule out that with the development of future computer technology, computers implementing the functions of the above embodiments can be personal computers, laptop computers, vehicle human-computer interaction devices, cellular phones, camera phones, smart phones, personal digital assistants, media players, navigation devices, email devices, game consoles, tablet computers, wearable devices, or combinations of any of these devices.

[0151] Although the method operations of the embodiments of the present specification are described in sequential order, some of the operations can in practical applications be performed concurrently, in parallel, or in a different order. The order of execution of the operations of various embodiments can be changed, and various other operations described, can be deleted or added. Additionally, those of ordinary skill in the art will recognize that the mechanisms of the present disclosure can be implemented using a variety of computing systems and devices, and that the computing systems and devices need not use the same mechanisms or be of the same type. For example, some devices used to implement the mechanisms of the present disclosure can be handheld devices, while other devices can be general purpose computers or servers. Some mechanisms of the present disclosure are described in terms of a general purpose computer, without limitation, and it should be understood that those skilled in the art recognize that the mechanisms of the present disclosure can be implemented using any computer that can store and execute program instructions.

[0152] For the sake of description, the above apparatuses are described in functional modules for convenience. Certainly, when implementing one or more of the present specification, the functions of the modules can be implemented in one or more software and / or hardware, or the modules implementing the same function can be combined or combined by a plurality of sub-modules or sub-units. The apparatus embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0153] The present application is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions of a flow or multiple flows and / or blocks Figure 1 The apparatuses that implement the functions specified in a flow or multiple flows and / or blocks.

[0154] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0155] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0156] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0157] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory. The memory is an example of computer-readable media.

[0158] Computer-readable media includes permanent and non-permanent, moveable and non- moveable media that can be implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, graphene storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to computing devices. According to the definition herein, computer-readable media does not include transitory media, such as modulated data signals and carrier waves.

[0159] Those skilled in the art will appreciate that the one or more embodiments described herein can be provided as a method, a system or a computer program product. Accordingly, the one or more embodiments described herein can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the one or more embodiments described herein can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable code.

[0160] The one or more embodiments described herein can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The one or more embodiments described herein can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0161] The various embodiments described in this specification can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The various embodiments described in this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0162] The above description is only some embodiments of the one or more embodiments described in this specification and is not intended to limit the one or more embodiments described in this specification. Those skilled in the art can make various modifications and changes to the one or more embodiments described in this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the one or more embodiments described in this specification shall be included in the scope of the claims.

Claims

1. A method for generating an image, comprising: Using a feature extraction model, determine the template features corresponding to the template image, including several feature vectors; In a pre-constructed semantic dictionary, a plurality of semantic base vectors matching the plurality of feature vectors are determined as target feature vectors corresponding to the template image; the semantic dictionary is constructed based on an image sample set and includes a plurality of semantic base vectors, each semantic base vector representing a feature of a semantic category; An image generation model is used to generate a target image according to each target feature vector.

2. The method according to claim 1, wherein In the pre-built semantic dictionary, a number of semantic basis vectors matching the number of feature vectors are determined, specifically including: For any feature vector, a semantic base vector having the smallest difference with the feature vector is determined in the semantic dictionary as the matching semantic base vector.

3. The method according to claim 1, wherein The semantic dictionary is constructed according to the following steps: Initialize multiple semantic basis vectors; Determine, by using the feature extraction model, first sample features corresponding to the first sample images in the image sample set, each first sample feature including a plurality of first feature vectors; The multiple semantic base vectors are updated according to the distance between each first feature vector and the multiple semantic base vectors.

4. The method according to claim 3, wherein: Updating the plurality of semantic base vectors according to the distance between each first feature vector and the plurality of semantic base vectors includes: Using the initialized multiple semantic basis vectors as initial cluster centers, clustering the first eigenvectors through multiple rounds of iterations, wherein each round of iteration includes updating the cluster center according to the distance between each first eigenvector and the multiple semantic basis vectors; The cluster centers determined after the multiple rounds of iterations are determined as multiple semantic basis vectors in the semantic dictionary.

5. The method according to claim 1, wherein The image generation model is a diffusion model; and the generation of a target image based on each target feature vector using the image generation model specifically includes: Perform multiple rounds of denoising operations corresponding to multiple time steps, and use the output image of the last round as the target image, wherein the denoising operation of any round includes: Predicting the target noise of this round based on the target feature vectors and the current time step; Denoising is performed based on the input image of this round and the target noise to obtain the denoised image of this round; wherein, the denoised image output of the non-final round is used as the input image of the next round; the input image of the first round is a random noise image.

6. The method according to claim 1, wherein generating a target image according to each target feature vector comprises: A target image is generated according to the target feature vectors and the text encoding representation corresponding to the description text of the template image.

7. The method according to claim 6, wherein: The image generation model is a diffusion model; and the generation of a target image based on each target feature vector using the image generation model specifically includes: Perform multiple rounds of denoising operations corresponding to multiple time steps, and use the output image of the last round as the target image, wherein the denoising operation of any round includes: Predicting the target noise of this round based on the target feature vectors, the text encoding representation and the current time step; Denoising is performed based on the input image of this round and the target noise to obtain the denoised image of this round; wherein, the denoised image output of the non-final round is used as the input image of the next round; the input image of the first round is a random noise image.

8. A method for training an image generation model, comprising: Using a feature extraction model, determine the sample features corresponding to the sample image, including several feature vectors; In a pre-constructed semantic dictionary, a plurality of semantic base vectors matching the plurality of feature vectors are determined as the sample feature vectors corresponding to the sample images; the semantic dictionary is constructed based on the image sample set and includes a plurality of semantic base vectors, each semantic base vector representing a feature of a semantic category; Generate a predicted image based on the feature vectors of each sample using an image generation model; The parameters of the image generation model are updated according to the sample image information corresponding to the sample image and the predicted image information corresponding to the predicted image.

9. The method of claim 8, wherein: The image generation model is a diffusion model; the method further includes: Adding random noise to the sample image in multiple time steps to obtain a sample noise image; The generating of the predicted image using the image generation model according to each sample feature vector specifically includes: Perform multiple rounds of denoising operations corresponding to multiple time steps, and use the output image of the last round as the predicted image, wherein the denoising operation of any round includes: Predicting the target noise of this round based on the feature vectors of each sample and the current time step; Denoising is performed based on the input image of this round and the target noise to obtain the denoised image of this round; wherein, the denoised image output of the non-final round is used as the input image of the next round; the input image of the first round is the sample noise image.

10. The method of claim 9, wherein: Updating parameters of the image generation model according to sample image information corresponding to the sample image and predicted image information corresponding to the predicted image includes: Obtaining the random noise added at each time step as the sample image information; obtaining the target noise predicted at each time step as the predicted image information; Determine the prediction loss based on the random noise and target noise at the corresponding time step; Based on the prediction loss, the parameters of the image generation model are updated.

11. The method of claim 9, wherein: The sample image also has label text describing it; The predicting of the target noise of this round according to the feature vectors of each sample and the current time step specifically includes: The target noise is predicted according to the feature vectors of each sample, the text encoding representation of the label text and the current time step.

12. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 11 is implemented.

Images