Image generation method and device based on cross-modal style learning

Abstract

The present disclosure relates to an image generation method and device based on cross-modal style learning, the method comprising: generating a target generated image through a pre-trained generative adversarial network, and performing image enhancement processing on the target training image; inputting the target generated image and the processed target training image into a discriminator of the adversarial network, inputting the target features output by the discriminator into a trained prototype space, and determining the exchange prediction loss according to the output of the prototype space; obtaining the original adversarial loss between the target training image and the target generated image, the contrastive learning loss and the text guidance loss between the source generated image and the target generated image, and determining the fine-tuning loss according to the losses; fine-tuning the adversarial network according to the fine-tuning loss to obtain a fine-tuned image generation model, and generating an image consistent with the training image through the image generation model, which can solve the problem of poor target domain style learning under the current very small number of samples.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image generation method and device based on cross-modal style learning. Background Art

[0002] With the continuous development of artificial intelligence, it is becoming easier to generate diverse and high-quality images, such as facial images, protein molecule diagrams, and paintings of designated artists. This technology also has strong research and application value in fields such as virtual reality. However, traditional generative models require a large amount of high-quality data to generate the images expected by users. For example, the StyleGAN model requires thousands to tens of thousands of high-quality images for training. But in some cases, it is difficult to obtain data sets, such as obtaining images of certain artists or medical images. There are only 10 works for each artist in the Artistic-Faces data set. In the case of very little training data (less than 10 samples), this type of generative model with a huge amount of parameters is difficult to generate high-quality results similar to the training data.

[0003] In order to reduce the demand for training data, many researchers have explored generating high-quality images with few samples. The mainstream few-shot generation method uses a generative model pre-trained on a large-scale dataset. Through various fine-tuning techniques, the fine-tuned generative model can generate images consistent with the training target, while maintaining the model's strong generation ability and achieving personalized few-shot generation. The TGAN model proves that fine-tuning a pre-trained generative model on a small amount of training data can effectively generate images similar to the training data. This process only requires the use of traditional adversarial loss functions. Since then, research on fine-tuning pre-trained models on a small amount of datasets to generate new target domain images has become more and more in-depth. In related technologies, a data augmentation method increases the semantic invariance of training data without introducing data noise, thereby preventing the discriminator from deviating to the training samples, and has good results in training sets of thousands of data. In other related technologies, the weights of the pre-trained model are mapped to an orthogonal parameter space, and the changes brought by the training samples are modified in the new space. This method can still guarantee the generation ability when the training data is less than 100. Some other related technologies have found that the low-level discriminator learns the general features of the image, while the high-level learns the distinguishing features of the image, which can be applied to image classification. Based on this discovery, the low-level parameters of the discriminator are fixed and only the high-level is fine-tuned, which achieves good few-sample generation capabilities. In other related technologies, a new network MineGAN model is proposed. This method first determines the parameter layer in the pre-trained model that is closest to the target training domain, and adjusts the parameters in the subsequent fine-tuning process to guide the generation of the target domain image. The above methods have laid a research foundation for few-sample generation.

[0004] However, their defects are also very obvious. When the number of samples is extremely small, such as less than 10 training images, the above methods can only generate images that are completely consistent with the target training samples, and it is difficult to generate diverse data in a personalized way. This is because when there are few training samples, the model will overfit to the training set and lose its original generation ability.

[0005] To address this problem, the CDC method (cross-domain correspondence) uses contrastive learning to constrain the similarity between the pre-training domain and the target domain, thereby avoiding model overfitting. In related technologies, it was found that the CDC method lost the basic information of the source image generated by the pre-training model during the learning process. In order to ensure that the model can generate images with the same domain features as the training samples and similar to the source images generated by the pre-training model, the RSSA model establishes a high-dimensional mapping from the target domain to the source domain, and uses multi-scale contrastive learning methods to keep the generated image similar to the source image. In related technologies, there is also the problem of bias in the generation of few samples. Previous methods over-focused on the diversity of generation, which may affect the quality of generation. They used the generator and discriminator as two feature extractors to extract features at different levels and proposed a dual contrastive learning method to maximize the mutual information between domains, ensuring the quality of generation.

[0006] To sum up the above methods, in order to avoid the model generating images that are only consistent with the target training samples, early work usually constrained the model, edited the key layers of the model or regularized the model parameters; the work in recent years mainly keeps the generated images similar to the source images, uses contrastive learning methods to improve the understanding of a small number of samples, and retains the source information of the original pre-trained model as much as possible.

[0007] However, these methods do not construct an explicit representation of the target domain information in the generative model, resulting in insufficient learning of domain style information, and they ignore the style features contained in the text information of the target sample, resulting in insufficient expression of the domain style in the generated images. In addition, these methods do not maintain the content of the source image well. In the case of very few training samples, the details generated by these methods are not well maintained, and in the case of single-sample training, these methods cannot generate results similar to the identity of the source image. Summary of the invention

[0008] In order to solve the above technical problem or at least partially solve the above technical problem, an embodiment of the present disclosure provides an image generation method and device based on cross-modal style learning.

[0009] In a first aspect, an embodiment of the present disclosure provides an image generation method based on cross-modal style learning, the method comprising:

[0010] Selecting a target training image from a plurality of preset training images, generating the target generated image through a pre-trained generative adversarial network, and performing image enhancement processing on the target training image, wherein the input of the pre-trained generative adversarial network is noise in a preset noise space;

[0011] Input the target generated image and the processed target training image into the discriminator of the pre-trained generative adversarial network, input the target features corresponding to the target generated image and the processed target training image output by the discriminator into the pre-trained prototype space, and determine the exchange prediction loss of maintaining the target style at the image level according to the output of the pre-trained prototype space;

[0012] Obtaining the original adversarial loss between the target training image and the target generated image, the contrastive learning loss and the text-guided loss between the source generated image and the target generated image, and determining the fine-tuning loss according to the original adversarial loss, the exchange prediction loss, the text-guided loss and the contrastive learning loss, wherein the source generated image is generated by a pre-trained generative adversarial network, and the inputs of the source generated image and the target generated image are noise sampled from the same noise space;

[0013] The pre-trained generative adversarial network is fine-tuned according to the fine-tuning loss to obtain a fine-tuned image generation model, so as to generate an image consistent with a preset training image based on the fine-tuned image generation model.

[0014] In a possible implementation manner, the prototype space is obtained by training through the following steps:

[0015] Perform image enhancement on each preset training image to obtain a target training image set, select two images from the target training image set, and input them into the discriminator of the pre-trained generative adversarial network, where the discriminator does not include the last linear layer;

[0016] The target features corresponding to the two images selected according to the discriminator output are respectively input into the preset prototype space, and the exchange prediction loss of maintaining the target style at the image level is determined according to the output of the preset prototype space;

[0017] The parameters of the preset prototype space are adjusted according to the exchange prediction loss until the exchange prediction loss is less than a preset threshold.

[0018] In a possible implementation, inputting the target features corresponding to the target generated image and the processed target training image output by the discriminator into a pre-trained prototype space, and determining the exchange prediction loss for maintaining the target style at the image level according to the output of the pre-trained prototype space, includes:

[0019] Using the mapping modules of two fully connected layers of the pre-trained prototype space, the target features corresponding to the target generated image and the processed target training image are mapped to the prototype space scale, and the mapping vectors corresponding to the two images are obtained;

[0020] Embedding the mapping vector into the prototype space through a preset linear layer;

[0021] Inputting the vectors taken out from the prototype space into the preset linear layer respectively, and aggregating and normalizing the vectors respectively output by the preset linear layer and the corresponding mapping vector;

[0022] The image-level target-style-preserving exchange prediction loss of the target generated image and the processed target training image is calculated based on the normalized results.

[0023] In one possible implementation, the image-level target-style-preserving exchange prediction loss of the target generated image and the processed target training image is calculated based on the normalized result by the following expression:

[0024] L swap (z i , z j )=l(z i ,q j )+l(z j ,q i )

[0025]

[0026]

[0027] Among them, L swap The image-level target-style-preserving exchange prediction loss for the target generated image and the processed target training image, z i and z j are the mapping vectors corresponding to the two images, q i and q j are the normalized results corresponding to the mapping vectors, τ is the temperature coefficient used to determine the importance of the parameters, c k is the number of prototypes in the prototype space.

[0028] In a possible implementation, obtaining the text guidance loss between the source generated image and the target generated image includes:

[0029] Input the text labels corresponding to the source generated image and the target generated image into the text encoder of the CLIP model to obtain a first output result of the text encoder;

[0030] Input the text label corresponding to the target generated image into the large language generation model to obtain the enhanced text label corresponding to the target generated image;

[0031] The enhanced text label corresponding to the target generated image is input into the text encoder of the CLIP model to obtain the second output result of the text encoder;

[0032] Input the source generated image and the target generated image into the image encoder of the CLIP model respectively to obtain the output result of the image encoder;

[0033] A text guidance loss between a source generated image and a target generated image is calculated based on a first output result and a second output result of the text encoder and an output result of the image encoder.

[0034] In one possible implementation, the text guidance loss between the source generated image and the target generated image is calculated according to the first output result and the second output result of the text encoder and the output result of the image encoder by the following expression:

[0035] ΔT i =E T (T t )-E T (T s )

[0036] ΔI=E I (I gen )-E I (I s )

[0037] L td =1-CosSim(ΔT i , ΔI)

[0038] ΔT s =E T (T t )-E T (T ts )

[0039] L tsd =1-CosSim(ΔT s , ΔI)

[0040] Among them, I gen and I s are the target generated image and the source generated image respectively, Cos Sim(·,·) is the cosine similarity of the two vectors, L td and L tsd is the text-guided loss between the target generated image and the source generated image, T t and T sThe text labels corresponding to the target generated image and the source generated image, E T and E I is the text encoder and image encoder, T ts Generate enhanced text labels corresponding to the target images.

[0041] In a possible implementation, obtaining a contrastive learning loss between a source generated image and a target generated image includes:

[0042] Select patches of different scales on the feature maps of the source generated image and the target generated image respectively, where each patch selects a specified number of anchor points as the center of the region and determines a square region with a specified length as the side length;

[0043] The contrastive learning loss is determined according to the difference between the anchor point m and the jth boundary pixel corresponding to the feature maps of the source generated image and the target generated image.

[0044] In a possible implementation, the contrastive learning loss is determined according to the difference between the anchor point m and the j-th boundary pixel corresponding to the feature maps of the source generated image and the target generated image by the following expression:

[0045]

[0046]

[0047]

[0048]

[0049]

[0050]

[0051]

[0052] in, and are the difference values ​​between the anchor point m and the jth boundary pixel corresponding to the source generated image and the target generated image, respectively. and a m are points on the square boundary and anchor points respectively, f s and f t are the feature maps of the source generated image and the target generated image respectively, The pixel difference values ​​of the source generated image and the target generated image with the same position are positive sample pairs. The pixel difference values ​​of the source generated image and the target generated image with inconsistent positions are negative sample pairs, L sdais the contrastive learning loss, N and n are the number of patches, and M and m are the number of anchor points.

[0053] In one possible implementation, the fine-tuning loss is determined based on the original adversarial loss, the exchange prediction loss, the contextual guidance loss, and the contrastive learning loss by the following expression:

[0054]

[0055]

[0056] in, is the fine-tuning loss, L swap To exchange prediction loss, L td and L tsd is the text-guided loss, is the original adversarial loss, L sda is the contrastive learning loss, λ s , d , t and λ p To adjust the coefficient parameter of the learning rate, G t For and D t The generator and discriminator of the fine-tuned image generation model, x and I t For the target training images, z and p z is the sampled noise and noise space.

[0057] In a second aspect, an embodiment of the present disclosure provides an image generation device based on cross-modal style learning, comprising:

[0058] An enhancement module, used for selecting a target training image from a plurality of preset training images, generating the target generated image through a pre-trained generative adversarial network, and performing image enhancement processing on the target training image, wherein the input of the pre-trained generative adversarial network is noise in a preset noise space;

[0059] An input module is used to input the target generated image and the processed target training image into the discriminator of the pre-trained generative adversarial network, input the target features corresponding to the target generated image and the processed target training image output by the discriminator into the pre-trained prototype space, and determine the exchange prediction loss of maintaining the target style at the image level according to the output of the pre-trained prototype space;

[0060] A determination module is used to obtain an original adversarial loss between a target training image and a target generated image, a contrastive learning loss and a text-guided loss between a source generated image and a target generated image, and determine a fine-tuning loss according to the original adversarial loss, the exchange prediction loss, the text-guided loss and the contrastive learning loss, wherein the source generated image is generated by a pre-trained generative adversarial network, and the inputs of the source generated image and the target generated image are noises sampled from the same noise space;

[0061] A fine-tuning module is used to fine-tune the pre-trained generative adversarial network according to the fine-tuning loss to obtain a fine-tuned image generation model, so as to generate an image consistent with a preset training image based on the fine-tuned image generation model.

[0062] In a third aspect, an embodiment of the present disclosure provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0063] Memory, used to store computer programs;

[0064] The processor is used to implement the above-mentioned image generation method based on cross-modal style learning when executing the program stored in the memory.

[0065] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the above-mentioned image generation method based on cross-modal style learning is implemented.

[0066] The above technical solution provided by the embodiment of the present disclosure has at least some or all of the following advantages compared with the prior art:

[0067] The image generation method based on cross-modal style learning described in the embodiment of the present disclosure learns the style features of the target sample from both the image and text domains. In the image domain, a style space is constructed from limited training samples using prototype learning. Since the training data after image enhancement still maintains the target style features regardless of the number, this consistent invariance is maintained in the prototype space, and in subsequent training, the generated image is constrained to be as similar as possible to the target image in the prototype space. In addition, in the text domain, since the CLIP model is fully trained on a large number of paired image-text pairs, its latent space has rich image-text alignment information. This feature is used to assist the style prototype space in constraining the style similarity between the generated image and the target image, which solves the problem of poor target domain style learning in the case of extremely small numbers of samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0070] Figure 1 The following schematically shows a flow chart of an image generation method based on cross-modal style learning according to an embodiment of the present disclosure;

[0071] Figure 2 A schematic diagram of a method for obtaining exchange prediction loss according to an embodiment of the present disclosure is shown schematically;

[0072] Figure 3 The following schematically shows a flow chart of a method for obtaining text guidance loss according to an embodiment of the present disclosure;

[0073] Figure 4 A schematic diagram of a process flow of a method for obtaining contrastive learning loss according to an embodiment of the present disclosure is shown;

[0074] Figure 5 A structural block diagram of an image generation device based on cross-modal style learning according to an embodiment of the present disclosure is schematically shown; and

[0075] Figure 6 The structural block diagram of an electronic device according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0077] See also Figure 1 , an embodiment of the present disclosure provides an image generation method based on cross-modal style learning, the method comprising:

[0078] S1, selecting a target training image from a plurality of preset training images, generating the target generated image through a pre-trained generative adversarial network, and performing image enhancement processing on the target training image, wherein the input of the pre-trained generative adversarial network is noise in a preset noise space.

[0079] In some embodiments, the image enhancement processing includes but is not limited to random cropping, erasing, affine and flipping, etc.

[0080] S2: Input the target generated image and the processed target training image into the discriminator of the pre-trained generative adversarial network, input the target features corresponding to the target generated image and the processed target training image output by the discriminator into the pre-trained prototype space, and determine the exchange prediction loss for maintaining the target style at the image level based on the output of the pre-trained prototype space.

[0081] S3, obtaining the original adversarial loss between the target training image and the target generated image, the contrastive learning loss and the text-guided loss between the source generated image and the target generated image, and determining the fine-tuning loss according to the original adversarial loss, the exchange prediction loss, the text-guided loss and the contrastive learning loss, wherein the source generated image is generated by a pre-trained generative adversarial network, and the inputs of the source generated image and the target generated image are noise sampled from the same noise space.

[0082] S4, fine-tuning the pre-trained generative adversarial network according to the fine-tuning loss to obtain a fine-tuned image generation model, so as to generate an image consistent with a preset training image based on the fine-tuned image generation model.

[0083] In this embodiment, in step S2, the prototype space is trained by the following steps:

[0084] Perform image enhancement on each preset training image to obtain a target training image set, select two images from the target training image set, and input them into the discriminator of the pre-trained generative adversarial network, where the discriminator does not include the last linear layer;

[0085] The target features corresponding to the two images selected according to the discriminator output are respectively input into the preset prototype space, and the exchange prediction loss of maintaining the target style at the image level is determined according to the output of the preset prototype space;

[0086] The parameters of the preset prototype space are adjusted according to the exchange prediction loss until the exchange prediction loss is less than a preset threshold.

[0087] In some embodiments, see Figure 2In order to learn the target attribute characteristics in limited training data, the data enhancement does not change the characteristics of the domain to which the data belongs (this characteristic is also applicable to single sample generation). However, since the distribution of the enhanced data is inconsistent with the original data distribution only in the discriminator's features, a prototype space is constructed for the enhanced data. In order to learn the prototype space, the last linear layer of the discriminator is discarded, and the target features output by the discriminator are mapped to the prototype space scale using the mapping module of two fully connected layers.

[0088] In some embodiments, the vectors taken out from the prototype space are respectively input into the preset linear layers, and the vectors respectively output by the preset linear layers and the corresponding mapping vectors are aggregated and normalized, so that the prototype space can be optimized to include as much information of the target domain as possible.

[0089] In this embodiment, in step S2, the target features corresponding to the target generated image and the processed target training image output by the discriminator are respectively input into the pre-trained prototype space, and the exchange prediction loss for maintaining the target style at the image level is determined according to the output of the pre-trained prototype space, including:

[0090] Using the mapping modules of two fully connected layers of the pre-trained prototype space, the target features corresponding to the target generated image and the processed target training image are mapped to the prototype space scale, and the mapping vectors corresponding to the two images are obtained;

[0091] Embedding the mapping vector into the prototype space through a preset linear layer;

[0092] Inputting the vectors taken out from the prototype space into the preset linear layer respectively, and aggregating and normalizing the vectors respectively output by the preset linear layer and the corresponding mapping vector;

[0093] The image-level target-style-preserving exchange prediction loss of the target generated image and the processed target training image is calculated based on the normalized results.

[0094] In this embodiment, the image-level target style-preserving exchange prediction loss of the target generated image and the processed target training image is calculated according to the normalized result by the following expression:

[0095] L swap (z i , z j )=l(z i ,q j )+l(z j ,q i )

[0096]

[0097]

[0098] Among them, L swap The image-level target-style-preserving exchange prediction loss for the target generated image and the processed target training image, z i and z j are the mapping vectors corresponding to the two images, q i and q j are the normalized results corresponding to the mapping vectors, τ is the temperature coefficient used to determine the importance of the parameters, c k is the number of prototypes in the prototype space.

[0099] In some embodiments, q i and q j Calculated using the Sinkhom-Knopp algorithm.

[0100] In some embodiments, see Figure 3 , the prototype space helps the overall style learning between target samples, but images in the same dataset may have completely different styles, which may hinder domain style learning. Text can represent more general style information. The CLIP model is trained on massive data, and its latent space contains rich information. In addition, simple and limited text descriptions contain limited information, which is not conducive to diverse and detailed style learning. The GPT-3 large language generation model can be used to describe the style of the target text in a way that the CLIP model can understand, and generate the text label corresponding to the target generated image. After being enhanced by GPT-3, the style text usually includes color, shape, texture and other aspects. Therefore, in this embodiment, in step S3, the text guidance loss between the source generated image and the target generated image is obtained, including:

[0101] Input the text labels corresponding to the source generated image and the target generated image into the text encoder of the CLIP model to obtain a first output result of the text encoder;

[0102] Input the text label corresponding to the target generated image into the large language generation model to obtain the enhanced text label corresponding to the target generated image;

[0103] The enhanced text label corresponding to the target generated image is input into the text encoder of the CLIP model to obtain the second output result of the text encoder;

[0104] Input the source generated image and the target generated image into the image encoder of the CLIP model respectively to obtain the output result of the image encoder;

[0105] A text guidance loss between a source generated image and a target generated image is calculated based on a first output result and a second output result of the text encoder and an output result of the image encoder.

[0106] In this embodiment, considering that simple text contains limited style features, the present disclosure uses the large language model GPT-3 to generate an enhanced style text description for guiding more diverse and robust style information. The style feature learning method of the present disclosure shows that the style information of the target training samples is constructed, and at the text level, the image-text information aligned in the latent space of the CLIP model is used to constrain the style features from a text perspective, thereby achieving a more accurate, diverse and robust representation of few-sample style features.

[0107] In this embodiment, the text guidance loss between the source generated image and the target generated image is calculated according to the first output result and the second output result of the text encoder and the output result of the image encoder by the following expression:

[0108] ΔT i =E T (T t )-E T (T s )

[0109] ΔI=E I (I gen )-E I (I s )

[0110] L td =1-CosSim(ΔT i , ΔI)

[0111] ΔT s =E T (T t )-E T (T ts )

[0112] L tsd =1-CoSSim(ΔT s , ΔI)

[0113] Among them, I gen and I s are the target generated image and the source generated image respectively, Cos Sim(·,·) is the cosine similarity of the two vectors, L td and L tsd is the text-guided loss between the target generated image and the source generated image, T t and T s The text labels corresponding to the target generated image and the source generated image, E T and E I is the text encoder and image encoder, T ts Generate enhanced text labels corresponding to the target images.

[0114] In some embodiments, see Figure 4 In order to keep the generated image similar to the source image in terms of intuitiveness and detail structure, different scales of patches can be selected for the feature maps of the target image and the source image obtained by the target generator and the source generator. Each patch selects m anchor points as the center of the region, and a square region is determined with a side length of 2r. The eight points b on the boundary of the square are m j (j∈(0, 1, ..., 7)) and anchor point a in sequence m Calculate the pixel difference, and for the selected positive and negative sample pairs, use the InfoNCE loss function contrast loss. Therefore, in this embodiment, in step S3, the contrast learning loss between the source generated image and the target generated image is obtained, including:

[0115] Select patches of different scales on the feature maps of the source generated image and the target generated image respectively, where each patch selects a specified number of anchor points as the center of the region and determines a square region with a specified length as the side length;

[0116] The contrastive learning loss is determined according to the difference between the anchor point m and the jth boundary pixel corresponding to the feature maps of the source generated image and the target generated image.

[0117] In this embodiment, the contrastive learning loss is determined according to the difference between the anchor point m and the jth boundary pixel corresponding to the feature maps of the source generated image and the target generated image by the following expression:

[0118]

[0119]

[0120]

[0121]

[0122]

[0123]

[0124]

[0125] in, and are the difference values ​​between the anchor point m and the jth boundary pixel corresponding to the source generated image and the target generated image, respectively. and a m are points on the square boundary and anchor points respectively, f s and f tare the feature maps of the source generated image and the target generated image respectively, The pixel difference values ​​of the source generated image and the target generated image with the same position are positive sample pairs. The pixel difference values ​​of the source generated image and the target generated image with inconsistent positions are negative sample pairs, L sda is the contrastive learning loss, N and n are the number of patches, and M and m are the number of anchor points.

[0126] In order to generate an image that is intuitively similar to the source image (i.e., similar in identity information), the present invention focuses on the consistency of multi-scale pixel differences at the feature image level, and learns the overall and local difference structures in multi-scale regions. This can solve the current problem of poor preservation of source image identity and detail information under few-sample generation, and has better ability to preserve global identity information and local detail information.

[0127] In addition, in some embodiments, in order to smooth the loss, the value of the anchor point is calculated by the following expression:

[0128]

[0129] in, is the value of the anchor point, neighbor(f {s，t} (a m )) is the neighborhood of the anchor point (pixel is 1), and Avg(·) is the average value of ·.

[0130] In this embodiment, the present disclosure obtains the structural association between pixels at different scales by calculating the contrastive learning loss, thereby ensuring better global identity preservation and structural detail preservation capabilities.

[0131] In this embodiment, in step S4, the fine-tuning loss is determined according to the original adversarial loss, the exchange prediction loss, the text guidance loss, and the contrastive learning loss through the following expression:

[0132]

[0133]

[0134] in, is the fine-tuning loss, L swap To exchange prediction loss, L td and L tsd is the text-guided loss, is the original adversarial loss, L sda is the contrastive learning loss, λ s , d , t and λ p To adjust the coefficient parameter of the learning rate, G t For and Dt The generator and discriminator of the fine-tuned image generation model, x and I t For the target training images, z and p z is the sampled noise and noise space.

[0135] See also Figure 5 , an embodiment of the present disclosure provides an image generation device based on cross-modal style learning, comprising:

[0136] An enhancement module 11 is used to select a target training image from a plurality of preset training images, generate the target generated image through a pre-trained generative adversarial network, and perform image enhancement processing on the target training image, wherein the input of the pre-trained generative adversarial network is noise in a preset noise space;

[0137] An input module 12 is used to input the target generated image and the processed target training image into the discriminator of the pre-trained generative adversarial network, input the target features corresponding to the target generated image and the processed target training image output by the discriminator into the pre-trained prototype space, and determine the exchange prediction loss of maintaining the target style at the image level according to the output of the pre-trained prototype space;

[0138] A determination module 13 is used to obtain an original adversarial loss between a target training image and a target generated image, a contrastive learning loss and a text-guided loss between a source generated image and a target generated image, and determine a fine-tuning loss according to the original adversarial loss, the exchange prediction loss, the text-guided loss and the contrastive learning loss, wherein the source generated image is generated by a pre-trained generative adversarial network, and the inputs of the source generated image and the target generated image are noise sampled from the same noise space;

[0139] The fine-tuning module 14 is used to fine-tune the pre-trained generative adversarial network according to the fine-tuning loss to obtain a fine-tuned image generation model, so as to generate an image consistent with a preset training image based on the fine-tuned image generation model.

[0140] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0141] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0142] In the above embodiment, any multiple of the enhancement module 11, the input module 12, the determination module 13 and the fine-tuning module 14 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. At least one of the enhancement module 11, the input module 12, the determination module 13 and the fine-tuning module 14 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the enhancement module 11, the input module 12, the determination module 13 and the fine-tuning module 14 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0143] See also Figure 6 , an electronic device provided by an embodiment of the present disclosure includes a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 1140;

[0144] Memory 1130, used for storing computer programs;

[0145] The processor 1110 is configured to implement the following image generation method based on cross-modal style learning when executing the program stored in the memory 1130:

[0146] Selecting a target training image from a plurality of preset training images, generating the target generated image through a pre-trained generative adversarial network, and performing image enhancement processing on the target training image, wherein the input of the pre-trained generative adversarial network is noise in a preset noise space;

[0147] Input the target generated image and the processed target training image into the discriminator of the pre-trained generative adversarial network, input the target features corresponding to the target generated image and the processed target training image output by the discriminator into the pre-trained prototype space, and determine the exchange prediction loss of maintaining the target style at the image level according to the output of the pre-trained prototype space;

[0148] Obtaining the original adversarial loss between the target training image and the target generated image, the contrastive learning loss and the text-guided loss between the source generated image and the target generated image, and determining the fine-tuning loss according to the original adversarial loss, the exchange prediction loss, the text-guided loss and the contrastive learning loss, wherein the source generated image is generated by a pre-trained generative adversarial network, and the inputs of the source generated image and the target generated image are noise sampled from the same noise space;

[0149] The pre-trained generative adversarial network is fine-tuned according to the fine-tuning loss to obtain a fine-tuned image generation model, so as to generate an image consistent with a preset training image based on the fine-tuned image generation model.

[0150] The communication bus 1140 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus 1140 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0151] The communication interface 1120 is used for communication between the above electronic device and other devices.

[0152] The memory 1130 may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory 1130 may also be at least one storage device located away from the processor 1110.

[0153] The above-mentioned processor 1110 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0154] The embodiment of the present disclosure further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image generation method based on cross-modal style learning as described above is implemented.

[0155] The computer-readable storage medium may be included in the device / apparatus described in the above embodiment; or it may exist independently without being assembled into the device / apparatus. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the image generation method based on cross-modal style learning according to the embodiment of the present disclosure is implemented.

[0156] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0157] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0158] The foregoing is merely a specific embodiment of the present disclosure, which enables those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features applied herein.

Claims

1. An image generation method based on cross-modal style learning, characterized in that: The method comprises: Selecting a target training image from a plurality of preset training images, generating the target generated image through a pre-trained generative adversarial network, and performing image enhancement processing on the target training image, wherein the input of the pre-trained generative adversarial network is noise in a preset noise space; Input the target generated image and the processed target training image into the discriminator of the pre-trained generative adversarial network, input the target features corresponding to the target generated image and the processed target training image output by the discriminator into the pre-trained prototype space, and determine the exchange prediction loss of maintaining the target style at the image level according to the output of the pre-trained prototype space; Obtaining the original adversarial loss between the target training image and the target generated image, the contrastive learning loss and the text-guided loss between the source generated image and the target generated image, and determining the fine-tuning loss according to the original adversarial loss, the exchange prediction loss, the text-guided loss and the contrastive learning loss, wherein the source generated image is generated by a pre-trained generative adversarial network, and the inputs of the source generated image and the target generated image are noise sampled from the same noise space; The pre-trained generative adversarial network is fine-tuned according to the fine-tuning loss to obtain a fine-tuned image generation model, so as to generate an image consistent with a preset training image based on the fine-tuned image generation model.

2. The method according to claim 1, characterized in that The prototype space is trained by the following steps: Perform image enhancement on each preset training image to obtain a target training image set, select two images from the target training image set, and input them into the discriminator of the pre-trained generative adversarial network, where the discriminator does not include the last linear layer; The target features corresponding to the two images selected according to the discriminator output are respectively input into the preset prototype space, and the exchange prediction loss of maintaining the target style at the image level is determined according to the output of the preset prototype space; The parameters of the preset prototype space are adjusted according to the exchange prediction loss until the exchange prediction loss is less than a preset threshold.

3. The method according to claim 1, characterized in that The target features corresponding to the target generated image output by the discriminator and the processed target training image are input into the pre-trained prototype space, and the exchange prediction loss for maintaining the target style at the image level is determined according to the output of the pre-trained prototype space, including: Using the mapping modules of two fully connected layers of the pre-trained prototype space, the target features corresponding to the target generated image and the processed target training image are mapped to the prototype space scale, and the mapping vectors corresponding to the two images are obtained; Embedding the mapping vector into the prototype space through a preset linear layer; Inputting the vectors taken out from the prototype space into the preset linear layer respectively, and aggregating and normalizing the vectors respectively output by the preset linear layer and the corresponding mapping vector; The image-level target-style-preserving exchange prediction loss of the target generated image and the processed target training image is calculated based on the normalized results.

4. The method according to claim 1, characterized in that: The obtaining of the text guidance loss between the source generated image and the target generated image includes: Input the text labels corresponding to the source generated image and the target generated image into the text encoder of the CLIP model to obtain a first output result of the text encoder; Input the text label corresponding to the target generated image into the large language generation model to obtain the enhanced text label corresponding to the target generated image; The enhanced text label corresponding to the target generated image is input into the text encoder of the CLIP model to obtain the second output result of the text encoder; Input the source generated image and the target generated image into the image encoder of the CLIP model respectively to obtain the output result of the image encoder; A text guidance loss between a source generated image and a target generated image is calculated based on a first output result and a second output result of the text encoder and an output result of the image encoder.

5. The method according to claim 1, characterized in that: The obtaining of the contrastive learning loss between the source generated image and the target generated image includes: Select patches of different scales on the feature maps of the source generated image and the target generated image respectively, where each patch selects a specified number of anchor points as the center of the region and determines a square region with a specified length as the side length; The contrastive learning loss is determined according to the difference between the anchor point m and the jth boundary pixel corresponding to the feature maps of the source generated image and the target generated image.

6. The method according to claim 5, characterized in that The contrastive learning loss is determined by the following expression based on the difference between the anchor point m and the jth boundary pixel corresponding to the feature maps of the source generated image and the target generated image: in, and are the difference values ​​between the anchor point m and the jth boundary pixel corresponding to the source generated image and the target generated image, respectively. and a m are points on the square boundary and anchor points respectively, f s and f t are the feature maps of the source generated image and the target generated image respectively, The pixel difference values ​​of the source generated image and the target generated image with the same position are positive sample pairs. The pixel difference values ​​of the source generated image and the target generated image with inconsistent positions are negative sample pairs, L sda is the contrastive learning loss, N and n are the number of patches, M and m are the number of anchor points, and τ is the temperature coefficient used to determine the importance of the parameters.

7. The method according to claim 1, characterized in that The fine-tuning loss is determined from the original adversarial loss, exchange prediction loss, context-guided loss, and contrastive learning loss by the following expression: in, is the fine-tuning loss, L swap To exchange prediction loss, L td and L tsd is the text-guided loss, is the original adversarial loss, L sda is the contrastive learning loss, λ s , d , t and λ p To adjust the coefficient parameter of the learning rate, G t For and D t The generator and discriminator of the fine-tuned image generation model, x and I t For target training images, z and p z is the sampled noise and noise space, is the expected value of the loss function of the fine-tuned image generation model.

8. An image generation device based on cross-modal style learning, characterized in that: include: An enhancement module, used for selecting a target training image from a plurality of preset training images, generating the target generated image through a pre-trained generative adversarial network, and performing image enhancement processing on the target training image, wherein the input of the pre-trained generative adversarial network is noise in a preset noise space; An input module is used to input the target generated image and the processed target training image into the discriminator of the pre-trained generative adversarial network, input the target features corresponding to the target generated image and the processed target training image output by the discriminator into the pre-trained prototype space, and determine the exchange prediction loss of maintaining the target style at the image level according to the output of the pre-trained prototype space; A determination module is used to obtain an original adversarial loss between a target training image and a target generated image, a contrastive learning loss and a text-guided loss between a source generated image and a target generated image, and determine a fine-tuning loss according to the original adversarial loss, the exchange prediction loss, the text-guided loss and the contrastive learning loss, wherein the source generated image is generated by a pre-trained generative adversarial network, and the inputs of the source generated image and the target generated image are noises sampled from the same noise space; A fine-tuning module is used to fine-tune the pre-trained generative adversarial network according to the fine-tuning loss to obtain a fine-tuned image generation model, so as to generate an image consistent with a preset training image based on the fine-tuned image generation model.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; A processor, for implementing the image generation method based on cross-modal style learning according to any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image generation method based on cross-modal style learning described in any one of claims 1 to 7 is implemented.

Images