A single-sample multi-style domain adaptive image generation method and system

Abstract

The present application discloses a single-sample multi-style domain adaptive image generation method and system, which relate to the image generation field and transfer learning field of computer vision. The method comprises obtaining a source domain image; generating a target domain image according to the source domain image using a trained single-sample multi-target domain adaptive image generation model; the trained single-sample multi-target domain adaptive image generation model comprises: a source domain generator, a target domain generator, a target domain style feature super network, a dynamic kernel modulation module, a heterogeneous semantic feature extraction network, a texture difference feature extraction network and a CLIP model; the present application can more accurately mine the knowledge transfer relationship during cross-domain feature learning, and has more advantages in establishing image domain mapping relationships of multiple categories at the same time. While maximizing the retention of the source domain image structure information, it can faithfully reflect the unique attributes of target domains of different styles, thereby generating higher quality target domain images.

Description

Technical Field

[0001] The present application relates to the field of image generation and transfer learning of computer vision, and in particular to a single-sample multi-style domain adaptive image generation method and system. Background Art

[0002] In recent years, deep generative models in the image field have provided increasingly rich technical support for image synthesis tasks. However, current methods based on deep generative models rely heavily on a large amount of training data and must be trained for a long time to achieve state-of-the-art image generation results. This makes it difficult to train deep generative models from scratch for many specific image domains in real life where it is difficult to collect sufficient training data.

[0003] In order to make better use of these deep generative models that have been trained on data-rich source domains, people began to use transfer learning methods to adapt them to some data-scarce target domains. This method is inspired by the success of domain adaptation techniques in various computer vision tasks. Image domain adaptation methods fine-tune the model by fixing some convolutional layers in the model and learning a mapping network between the potential distribution of the source domain and the target domain. Usually, this is done by retraining the target data with an appropriate loss function and fine-tuning from a single pre-trained generator.

[0004] Early methods used model fine-tuning to achieve good generation results on 100 training images. Subsequent studies have shown that domain adaptation methods can be applied even when using less than 10 training images. Recent studies have attempted to achieve cross-domain generation from source domain images to target domain images by fine-tuning the pre-trained model using only one target image. With the success of the Contrastive Language-Image Pre-training (CLIP) model, existing work uses the difference between the CLIP spatial embeddings of the source and target domains to guide the domain adaptation process of cross-domain features to more accurately adapt to the generation of target domain images.

[0005] Although these methods reduce the training burden of data-intensive generative models, they assume that the training images have similar appearance. When the model is faced with training images of different style image domains at the same time, it can only achieve the goal by learning a separate model for each style image domain, which depends on the number of image styles in the target domain, but this requires huge memory and computing time. In addition, separating each style image domain for separate training will seriously ignore the shared knowledge between the target domains, such as the common structural combinations of different artists in portrait painting, which greatly limits the feature expression ability of the domain adaptation model.

[0006] From the above, we can see that the existing image domain adaptation methods use a small number of target domain images, and retrain the target data with appropriate loss functions based on the source domain pre-trained model to adapt to the target domain style for image generation. Among them, the StyleGAN-nada method uses the model fine-tuning method of the generative adversarial network; DynaGAN discards the discriminator and adversarial loss function in the former technology, and introduces the CLIP image space to guide the domain adaptation process of the model. In contrast, DynaGAN can better distinguish the image styles of different target domains through the hyper-network modulation strategy during the fine-tuning process of the model. The One-shot CLIP method further explores the operation strategy of the CLIP space to improve the quality of image domain adaptation generation.

[0007] A common feature of the above methods is that when using pre-trained models for single-sample image domain adaptation, each style of image domain is always separated for training, ignoring the shared knowledge between target domains; and in the process of model optimization, only the gradient of the objective function to the model parameters during training is considered, and this is used as the only criterion for convolution kernel optimization, which greatly limits the feature expression ability of the domain adaptation model, resulting in confusion of style and semantic structure when the model generates images for different style image domains. And when selecting source domain knowledge, the adaptation relationship with different target domains is not considered, which makes them doubt the adaptability between the source domain and different target domains.

[0008] Therefore, based on the above problems, it is urgent to provide a new domain adaptive image generation method that can make full use of the shared knowledge between image domains and more accurately mine the knowledge transfer relationship during cross-domain feature learning. By simultaneously establishing image domain mapping relationships of multiple categories, the structural information of the source domain image can be maximized in a more efficient way, and the unique attributes of the target domains of different styles can be faithfully reflected, thereby generating higher quality target domain images. Summary of the invention

[0009] The purpose of this application is to provide a single-sample multi-style domain adaptive image generation method and system, which can more accurately mine the knowledge transfer relationship during cross-domain feature learning, and thus generate higher quality target domain images.

[0010] To achieve the above objectives, this application provides the following solutions:

[0011] In a first aspect, the present application provides a single-sample multi-style domain adaptive image generation method, the single-sample multi-style domain adaptive image generation method comprising:

[0012] Obtain source domain image;

[0013] According to the source domain image, a target domain image is generated using a trained single-sample multi-target domain adaptive image generation model; the trained single-sample multi-target domain adaptive image generation model includes: a source domain generator, a target domain generator, a target domain style feature super network, a dynamic kernel modulation module, a heterogeneous semantic feature extraction network, a texture difference feature extraction network and a CLIP model;

[0014] The training process of the single-sample multi-target domain adaptive image generation model is:

[0015] S1: Use multiple source domain images and multiple 512-dimensional vectors randomly sampled from a multi-dimensional Gaussian distribution to train the pre-trained source domain generator to obtain the source domain generator G s ; Using the source domain generator G s The weights of the target domain generator are initialized to obtain the initial target domain generator G t0 , and select multiple target domain training images of different styles as the initial target domain generator G t0 The target domain guidance image I t ;

[0016] S2: One-hot vector encoding is performed on multiple target domain training images of different styles to obtain the corresponding target domain style control conditions c ; At the same time, multiple target domain guide images of different styles I t Input into the target domain style feature super network to obtain the corresponding target domain style feature convolution kernel ;

[0017] S3: Fixed Source Domain Generator G s The network structure is randomly sampled from a multidimensional Gaussian distribution. B A 512-dimensional vector, after the mapping network Convert to intermediate latent vector w , the intermediate latent vector w Input to the source domain generator to generate B Zhang source domain synthetic image G s ( w );

[0018] S4: The fixed source domain generator G s Convolution kernels in each convolution layer Convolution kernel with target domain style feature Input to the dynamic kernel modulation module based on cross-domain feature knowledge similarity to generate a new modulated convolution kernel The dynamic kernel modulation module is used to combine the hypernetwork technology, use the similarity between cross-domain convolution kernel encodings to guide the correlation of feature learning, and generate a new target domain modulated convolution kernel ;

[0019] S5: Modulating the convolution kernel using a new target domain For the initial target domain generator G t0 Fine-tune the corresponding convolutional layer; repeat S4 to complete the fine-tuning of all convolutional layers of the target domain generator to obtain the fine-tuned target domain generator G t ;

[0020] S6: Utilization and B 512-dimensional intermediate latent vector w and the corresponding target domain style control condition c , input to the modulated target domain generator G t , obtain the domain adaptive generated image corresponding to the style of the target domain training image G t ( w );

[0021] S7: Generate images based on domain adaptation G t ( w ), source domain synthetic image G s ( w ) and target domain training images I t , a heterogeneous semantic feature extraction network is used to extract the semantic distribution of image blocks in the semantic structure space, and the semantic query samples are obtained respectively. s g , semantic positive samples s u and semantic negative samples s v ; Generate images by domain adaptation G t ( w ), corresponding to the target style training image I t0 And other style training images I t1 Input into the texture difference feature extraction network, extract the style feature differences of each image in the texture style space, and obtain style query samples respectively t g , style positive sample t pand style negative samples t n ; Generate images by domain adaptation G t ( w ), target domain training images I t , Target domain style control conditions c and the intermediate latent vector w Input into the CLIP model together to determine the CLIP loss function value ;

[0022] S8: Query samples based on semantics s g , semantic positive samples s u and semantic negative samples s v Determining the heterogeneous semantic contrast loss function The loss value of ; query samples according to style t g , style positive sample t p and style negative samples t n Determine the style difference contrast loss function The loss value of heterogeneous semantic comparison loss function Contrasting loss function with style difference Sum and obtain the cross-domain comparison loss function value ; Compare the cross-domain loss function value And CLIP loss function value Sum and get the total loss function value Loss ;

[0023] S9: Determine the current total loss function value Loss Is it less than the preset threshold? If so, the current total loss function value is Loss As the optimal total loss value of the fine-tuned target domain generator, the trained single-sample multi-target domain adaptive image generation model is obtained; otherwise, the current total loss function value is used Loss Perform optimization updates to train the fine-tuned target domain generator and return to S2 until the current total loss function value Loss Is it less than the preset threshold?

[0024] Optionally, the target domain style feature super network includes: an image encoder and a multi-layer fully connected layer structure;

[0025] The image encoder is used to convert multiple target domain training images of different styles I tConvert to image vector; and correspond the image vector to the target domain style control condition of the style c Input into the multi-layer fully connected layer structure;

[0026] The multi-layer fully connected layer structure is used to adopt a rank 1 tensor decomposition method to obtain the one-dimensional vectors of the three style features respectively, and obtain the corresponding target domain style feature convolution kernel according to the one-dimensional vectors of the three style features .

[0027] Optionally, the source domain generator is fixed in sequence G s Convolution kernels in each convolution layer Convolution kernel with target domain style feature Input to the dynamic kernel modulation module based on cross-domain feature knowledge similarity to generate a new modulated convolution kernel , specifically including:

[0028] The source domain generator will be fixed G s Convolution kernels in each convolution layer Convolution kernel with target domain style feature Reshape into 2D matrices respectively;

[0029] The reshaped 2D matrices are multiplied to determine the kernel similarity matrix; and the kernel similarity matrix is ​​processed by the softmax function to obtain the global similarity attention matrix;

[0030] Multiply the 2D matrix of the reshaped convolutional layer kernel with the global similarity attention matrix to obtain a new modulated convolution kernel modulated by the similarity of cross-domain feature knowledge. .

[0031] Optionally, the extraction process of the heterogeneous semantic feature extraction network is:

[0032] Generate images based on domain adaptation G t ( w ), source domain synthetic image G s ( w ) and target domain training images I t ,Use the pre-trained VGG-16 network to extract image feature maps;

[0033] Extract semantic query samples according to image feature maps s g , semantic positive samples s u and semantic negative samples s v ;

[0034] Query samples based on semantics s g , semantic positive samples s u and semantic negative samples s v Determine semantic similarity distribution and heterogeneous semantic similarity distribution within corresponding image domains;

[0035] According to the heterogeneous semantic similarity distribution, the von Mises-Fisher distribution is used to guide the strong semantic negative sample sampling strategy.

[0036] Optionally, the adopting von Mises-Fisher distribution to guide the strong semantic negative sample sampling strategy according to the heterogeneous semantic similarity distribution specifically includes:

[0037] Using the formula Determining strong semantic negative samples ;

[0038] in, N q is the normalization constant, To determine each negative sample The importance weight of is a semantic positive sample, the superscript T is transposed, is a single negative sample, is the importance weight in heterogeneous semantic similarity distribution, Generating images for domain adaptation G t ( w ) is the embedding vector in the semantic space, is a strong semantic negative sample distribution.

[0039] Optionally, the extraction process of the texture difference feature extraction network specifically includes:

[0040] Generate images based on domain adaptation G t ( w ), source domain synthetic image G s ( w ) and target domain training images I t , using the pre-trained VGG-16 network to output style feature encoding;

[0041] Get style query samples according to style feature encoding t g , style positive sample t p and style negative samples tn .

[0042] In a second aspect, the present application provides a single-sample multi-style domain adaptive image generation system, the single-sample multi-style domain adaptive image generation system comprising:

[0043] A source domain image acquisition module, used to acquire a source domain image;

[0044] A domain adaptive image generation module is used to generate a target domain image based on a source domain image using a trained single-sample multi-target domain adaptive image generation model; the trained single-sample multi-target domain adaptive image generation model includes: a source domain generator, a target domain generator, a target domain style feature super network, a dynamic kernel modulation module, a heterogeneous semantic feature extraction network, a texture difference feature extraction network and a CLIP model;

[0045] The training process of the single-sample multi-target domain adaptive image generation model is:

[0046] S1: Use multiple source domain images and multiple 512-dimensional vectors randomly sampled from a multi-dimensional Gaussian distribution to train the pre-trained source domain generator to obtain the source domain generator G s ; Using the source domain generator G s The weights of the target domain generator are initialized to obtain the initial target domain generator G t0 , and select multiple target domain training images of different styles as the initial target domain generator G t0 The target domain guidance image I t ;

[0047] S2: One-hot vector encoding is performed on multiple target domain training images of different styles to obtain the corresponding target domain style control conditions c ; At the same time, multiple target domain guide images of different styles I t Input into the target domain style feature super network to obtain the corresponding target domain style feature convolution kernel ;

[0048] S3: Fixed Source Domain Generator G s The network structure is randomly sampled from a multidimensional Gaussian distribution. B A 512-dimensional vector, after the mapping network M Convert to intermediate latent vector w , the intermediate latent vector w Input to the source domain generator to generate B Zhang source domain synthetic imageG s ( w );

[0049] S4: The fixed source domain generator G s Convolution kernels in each convolution layer Convolution kernel with target domain style feature Input to the dynamic kernel modulation module based on cross-domain feature knowledge similarity to generate a new modulated convolution kernel The dynamic kernel modulation module is used to combine the hypernetwork technology, use the similarity between cross-domain convolution kernel encodings to guide the correlation of feature learning, and generate a new target domain modulated convolution kernel ;

[0050] S5: Modulating the convolution kernel using a new target domain For the initial target domain generator G t0 Fine-tune the corresponding convolutional layer; repeat S4 to complete the fine-tuning of all convolutional layers of the target domain generator to obtain the fine-tuned target domain generator G t ;

[0051] S6: Utilization and B 512-dimensional intermediate latent vector w and the corresponding target domain style control condition c , input to the modulated target domain generator G t , obtain the domain adaptive generated image corresponding to the style of the target domain training image G t ( w );

[0052] S7: Generate images based on domain adaptation G t ( w ), source domain synthetic image G s ( w ) and target domain training images I t , a heterogeneous semantic feature extraction network is used to extract the semantic distribution of image blocks in the semantic structure space, and the semantic query samples are obtained respectively. s g , semantic positive samples s u and semantic negative samples s v ; Generate images by domain adaptation G t ( w ), corresponding to the target style training image It0 And other style training images I t1 Input into the texture difference feature extraction network, extract the style feature differences of each image in the texture style space, and obtain style query samples respectively t g , style positive sample t p and style negative samples t n ; Generate images by domain adaptation G t ( w ), target domain training images I t , Target domain style control conditions c and the intermediate latent vector w Input into the CLIP model together to determine the CLIP loss function value ;

[0053] S8: Query samples based on semantics s g , semantic positive samples s u and semantic negative samples s v Determining the heterogeneous semantic contrast loss function The loss value of ; query samples according to style t g , Style Positive Sample t p and style negative samples t n Determine the style difference contrast loss function The loss value of heterogeneous semantic comparison loss function Contrasting loss function with style difference Sum and obtain the cross-domain comparison loss function value ; Compare the cross-domain loss function value And CLIP loss function value Sum and get the total loss function value Loss ;

[0054] S9: Determine the current total loss function value Loss Is it less than the preset threshold? If so, the current total loss function value is Loss As the optimal total loss value of the fine-tuned target domain generator, the trained single-sample multi-target domain adaptive image generation model is obtained; otherwise, the current total loss function value is used Loss Perform optimization updates to train the fine-tuned target domain generator and return to S2 until the current total loss function value LossIs it less than the preset threshold?

[0055] According to the specific embodiments provided in this application, this application has the following technical effects:

[0056] The present application provides a single-sample multi-style domain adaptive image generation method and system. The single-sample multi-target domain adaptive image generation model is constructed based on the one-shot multiple image domain adaptive image generation (One-shot Multiple Image Domain Adaptation, OMiDa) method of image domain knowledge similarity dynamic kernel modulation. The dynamic kernel modulation module based on cross-domain feature knowledge similarity (Kernel Dynamic Modulation Module based on Cross-domain Feature Knowledge Similarity, CFKS module) uses the similarity information of cross-domain knowledge to guide the correlation of model feature learning, which can more accurately control the transfer of knowledge related to the target task and prevent the transmission of other irrelevant information; by querying samples according to semantics s g , semantic positive samples s u and semantic negative samples s v Determining the heterogeneous semantic contrast loss function The loss value of ; query samples according to style t g , Style Positive Sample t p and style negative samples t n Determine the style difference contrast loss function The loss value of heterogeneous semantic comparison loss function Contrasting loss function with style difference Sum and obtain the cross-domain comparison loss function value ; Compare the cross-domain loss function value And CLIP loss function value Sum and get the total loss function value Loss, we obtain the cross-domain contrastive loss of structure and style (SSCCL) based on dual constraints of structure and style, which can more accurately mine the knowledge transfer relationship between target domain images of different styles and source domain images, and achieve knowledge transfer from source domain to target domain images through only one model training, and generate diversified target domain images; this application mines strong negative samples based on similar knowledge between multiple image domains, and maximizes the retention of structural information of source domain images in a more efficient way, and reflects the unique attributes of target domains of different styles. This application has more advantages in establishing image domain mapping relationships of multiple categories at the same time, and can more accurately mine the knowledge transfer relationship during cross-domain feature learning, thereby generating higher quality target domain images. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 This is a schematic diagram of a single-sample multi-style domain adaptive image generation method in an embodiment of the present application;

[0059] Figure 2 This is a schematic diagram of the structure of a single-sample multi-target domain adaptive image generation model in one embodiment of the present application;

[0060] Figure 3 It is a schematic diagram of the structure of the dynamic core modulation module;

[0061] Figure 4 Schematic diagram of using similarity distribution to measure the semantic relationship between cross-domain heterogeneous images;

[0062] Figure 5 Schematic diagram of mining strong negative samples using heterogeneous semantic relations in multi-style image domains;

[0063] Figure 6 A schematic diagram comparing the results generated by this application with other methods in the Church2Painting experimental task;

[0064] Figure 7 A schematic diagram showing the comparison between the results generated by this application and other methods in the Real2MetFaces experimental task;

[0065] Figure 8A schematic diagram comparing the results generated by this application with other methods in the Real2MetFaces experimental task. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0067] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0068] In an exemplary embodiment, a single-sample multi-style domain adaptive image generation method is provided, the method comprising the following S101 to S102. Wherein:

[0069] S101, obtaining a source domain image;

[0070] S102, based on the source domain image, using the trained single-sample multi-target domain adaptive image generation model to generate a target domain image; Figure 2 As shown, the trained single-sample multi-target domain adaptive image generation model includes: a source domain generator, a target domain generator, a target domain style feature super network, a dynamic kernel modulation module, a heterogeneous semantic feature extraction network, a texture difference feature extraction network and a CLIP model; Figure 2 Part (a) is the input target domain training image latent conditional vector; Figure 2 Part (b) is the target domain style feature super network, which converts the latent conditional vector of the target image into the target domain style feature convolution kernel and inputs it into the generation network for dynamic weight modulation. FC represents the fully connected layer. Figure 2 Part (c) is the generation network, which includes the source domain generator and the target domain generator, as well as the dynamic kernel modulation module; Figure 2 Part (d) and Figure 2 Part (e) represents the generated source domain images and the corresponding target domain generated images of different styles.

[0071] like Figure 1 As shown in Figure 2, the training process of the single-sample multi-target domain adaptive image generation model is:

[0072] S1: Use multiple source domain images and multiple 512-dimensional vectors randomly sampled from a multi-dimensional Gaussian distribution to train the pre-trained source domain generator to obtain the source domain generator G s ; Using the source domain generatorG s The weights of the target domain generator are initialized to obtain the initial target domain generator G t0 , and select multiple target domain training images of different styles as the initial target domain generator G t0 The target domain guidance image I t ;

[0073] S2: One-hot vector encoding is performed on multiple target domain training images of different styles to obtain the corresponding target domain style control conditions c ; At the same time, multiple target domain guide images of different styles I t Input into the target domain style feature super network to obtain the corresponding target domain style feature convolution kernel ;

[0074] The target domain style feature super network includes: an image encoder and a multi-layer fully connected layer structure;

[0075] The image encoder is used to convert multiple target domain training images of different styles I t Convert to image vector; and correspond the image vector to the target domain style control condition of the style c Input into the multi-layer fully connected layer structure;

[0076] The multi-layer fully connected layer structure is used to adopt a rank 1 tensor decomposition method to obtain one-dimensional vectors of three style features respectively. , and ; And according to the one-dimensional vector of the three style features, the corresponding target domain style feature convolution kernel is obtained , the calculation expression is:

[0077] (1)

[0078] The rank 1 tensor decomposition method specifically includes:

[0079] In order to achieve more efficient weight estimation and optimization, from the perspective of tensor decomposition, we explore an effective and compact parameter space to fine-tune the weights of the pre-trained model;

[0080] Fixed source domain generator convolution kernel is a four-dimensional tensor, so the target domain style feature convolution kernel used for modulation It is also a four-dimensional tensor. When the width and height dimensions of the convolution kernel are combined, it can be regarded as a three-dimensional tensor, whose dimensions correspond to the input and output channels of the convolution kernel and the number of parameters of the convolution kernel respectively.

[0081] If the network weight matrix (i.e., convolution kernel) is regarded as a full-rank matrix, then the dense full-rank matrix can be expressed as a combination of several low-rank matrices, and the low-rank matrix can be decomposed into the product of small-scale matrices. Therefore, the rank-1 tensor decomposition method can be used to multiply three one-dimensional vectors (Formula 1) to express the three-dimensional tensor (i.e., the target domain style feature convolution kernel );

[0082] By combining matrix decomposition with rank-1 tensor decomposition, the high-dimensional weight convolution kernel is transformed into an estimated three one-dimensional vectors, making the style dynamic modulation module in the kernel space simpler and reducing the model burden;

[0083] S3: Fixed Source Domain Generator G s The network structure is randomly sampled from a multidimensional Gaussian distribution. B A 512-dimensional vector, after the mapping network Convert to intermediate latent vector w , the intermediate latent vector w Input to the source domain generator to generate B Zhang source domain synthetic image G s ( w );

[0084] S4: The fixed source domain generator G s Convolution kernels in each convolution layer Convolution kernel with target domain style feature Input to the dynamic kernel modulation module based on cross-domain feature knowledge similarity to generate a new modulated convolution kernel ;like Figure 3 As shown, the dynamic kernel modulation module is used to combine the hypernetwork technology, use the similarity between cross-domain convolution kernel encodings to guide the correlation of feature learning, and generate a new target domain modulated convolution kernel ;in, Indicates the number of input channels of the convolution kernel; Indicates the number of output channels of the convolution kernel; k Represents the convolution kernel size parameter;

[0085] S4 specifically includes:

[0086] The fixed source domain generator G s Convolution kernels in each convolution layer Convolution kernel with target domain style feature Reshape into 2D matrices respectively; where the source domain convolution kernel Reshape as well as , target domain style feature convolution kernel Reshape ;

[0087] Multiply the reshaped 2D matrices to determine the kernel similarity matrix ( V and K ); and the kernel similarity matrix is ​​processed by softmax function to obtain the global similarity attention matrix ;

[0088] Multiply the 2D matrix of the reshaped convolutional layer kernel with the global similarity attention matrix to obtain a new modulated convolution kernel modulated by the similarity of cross-domain feature knowledge. , the calculation expression is:

[0089] (2)

[0090] In the formula, Both represent operations that reshape the structure of a tensor.

[0091] When performing cross-domain style transfer, some style features of the target domain image are not suitable for the semantics of the source domain image, and they need to be selectively retained according to the semantic structure of the image during transfer. Therefore, in the CFKS module, our goal is to obtain the target domain image style convolution kernel. With pre-trained source domain convolution kernel , combined with the calculation of the similarity between the convolution kernels, the convolution kernel is fine-tuned according to the similarity matrix to obtain a new modulated convolution kernel ; In the implementation method, a quantitative value is defined for each potential convolution kernel, which reflects the importance of the feature pattern of each convolution kernel. A quadratic attention matrix is ​​used to compare each convolution kernel with all other convolution kernels to accurately reflect the similarity between them;

[0092] S5: Modulating the convolution kernel using a new target domain For the initial target domain generator G t0 Fine-tune the corresponding convolutional layer; repeat S4 to complete the fine-tuning of all convolutional layers of the target domain generator to obtain the fine-tuned target domain generator G t ;

[0093] The corresponding convolution layer is fine-tuned by using the modulation and demodulation operation mode. The weight value of the convolution layer is changed by using additional parameters without changing the original convolution layer structure, so as to control the style domain type of the generated image, adapt to the different convolution kernels output by the style feature super network, and complete the domain adaptation of multiple target image domain styles in one training process.

[0094] According to the introduced modulation parameters: and , the modulation expression of the convolutional layer is as follows:

[0095] (3)

[0096] In the formula, Represents the convolution kernel scale factor, which is used to modulate the demodulation output, that is, , i , j and k Represent the indexes of input, output channels and convolution kernel spatial positions respectively; is the residual weight added to the original model convolution kernel weight, which means the adjustment of the original convolution kernel weight based on the style parameters; Formula (3) l In the convolution layer, the input feature map of the previous layer is Convert to output features . and Only the convolutional layers in the model are adjusted, and nothing to do with other parts of the structure. , are all convolutional layer parameters, () is the convolution layer function.

[0097] The convolutional layer is fine-tuned as:

[0098] (4)

[0099] S6: Utilization and B 512-dimensional intermediate latent vector w and the corresponding target domain style control condition c , input to the modulated target domain generator G t , obtain the domain adaptive generated image corresponding to the style of the target domain training image G t ( w );

[0100] S7: Generate images based on domain adaptation G t ( w ), source domain synthetic image G s (w ) and target domain training images I t , a heterogeneous semantic feature extraction network is used to extract the semantic distribution of image blocks in the semantic structure space, and the semantic query samples are obtained respectively. s g , semantic positive samples s u and semantic negative samples s v ; Generate images by domain adaptation G t ( w ), corresponding to the target style training image I t0 And other style training images I t1 Input into the texture difference feature extraction network, extract the style feature differences of each image in the texture style space, and obtain style query samples respectively t g , style positive sample t p and style negative samples t n ; Generate images by domain adaptation G t ( w ), target domain training images I t , Target domain style control conditions c and the intermediate latent vector w Input into the CLIP model together to determine the CLIP loss function value ;

[0101] In images of different style domains, image patches at different positions have different semantic relationships, for example Figure 4 As shown in part (a) of the figure, some image patches from different target domains represent the nose of an animal, while others represent backgrounds or parts such as ears and eyes that are not related to the nose of an animal. In addition, even image patches of fox images have special semantic relationships. Therefore, when facing multiple image domains, different semantic relationships should be considered and retained to achieve accurate image domain adaptation tasks;

[0102] set up Represents the input source domain synthetic image G s(w) , corresponding to the target domain training image I t The embedding vector of the image patch in the semantic space, n Represents the location of the image patch. Generates images by fine-tuning the domain adaptation model G t(w)After that, it is encoded into the semantic space and It means that it is encouraged to generate images G t(w) The image structure is close to the positive sample of the source domain image, and far away from the structural features of the negative sample image;

[0103] According to the instance classification idea of ​​contrastive learning (i.e. assigning label 1 to positive samples and label 0 to negative samples), semantic similarity is measured by probability distribution. Therefore, it can be regarded as a soft label to quantify the semantic relationship between samples. Figure 4 As shown in part (b) of , for each given image, after selecting image blocks for semantic encoding, the similarity probability distribution with other image blocks is defined using softmax in each image domain:

[0104] (5)

[0105] In the formula, Respectively represent the first i Position and n The semantic similarity distribution between image patches at different locations; N Indicates the number of semantic image blocks in each image; Represents the input source domain synthetic image G s ( w )’s i-th image block, j-th image block, and n-th image block’s embedding vector in the semantic space; Represents the corresponding target domain training image I t The embedding vectors of the i-th image block, the j-th image block, and the n-th image block in the semantic space; Represents the generated image after fine-tuning the domain adaptation model G t ( w )’s i-th image block, j-th image block, and n-th image block’s embedding vectors in the semantic space.

[0106] When a query sample is given in the semantic image patch of the generated image Then, the corresponding positive sample semantic image block is embedded At this time, in the negative sample semantic space composed of other image blocks, there is also a cross-domain semantic similarity probability distribution, which is called heterogeneous semantic similarity distribution. ,like Figure 4 Part (c) shows:

[0107] (6)

[0108] In the formula, Represents the total number of negative samples; Represents a single negative sample. Similarly, in the negative sample space excluding the query sample and the positive sample, there is also a similar heterogeneous semantic similarity distribution :

[0109] (7)

[0110] in, represents negative samples;

[0111] The extraction process of the heterogeneous semantic feature extraction network is:

[0112] Generate images based on domain adaptation G t ( w ), source domain synthetic image G s ( w ) and target domain training images I t ,Use the pre-trained VGG-16 network to extract image feature maps;

[0113] Extract semantic query samples according to image feature maps s g , semantic positive samples s u and semantic negative samples s v ;

[0114] Query samples based on semantics s g , semantic positive samples s u and semantic negative samples s v Determine the semantic similarity distribution and heterogeneous semantic similarity distribution in the corresponding image domain and ;

[0115] like Figure 5 As shown in Figure 2, the von Mises-Fisher distribution is used to guide the strong semantic negative sample sampling strategy based on the heterogeneous semantic similarity distribution.

[0116] Using the formula Determining strong semantic negative samples ;

[0117] in, N q is the normalization constant, To determine each negative sample The importance weight of is a semantic positive sample, the superscript T is transposed, is a single negative sample, is the importance weight in heterogeneous semantic similarity distribution, Generating images for domain adaptation G t ( w ) is the embedding vector in the semantic space, is a strong semantic negative sample distribution.

[0118] In order to avoid style crossover and confusion in the multi-image style domain adaptation task, the overall style of the generated image is constrained. A feature space consisting of texture style correlation is constructed, and the feature correlation of texture style is measured in the form of Gram matrix:

[0119] (8)

[0120] In the formula, It is l Feature map in layer i and j The inner product of k Represents the length of the vector. The Gram matrix can be regarded as a quantitative description of the potential image features. Respectively represent l Feature map in layer i and j In the k The elements at position

[0121] During the style adaptation process, the generated image G t ( w ) and style negative samples t n The distance between them should be much larger than the style positive sample t p The distance between them, that is:

[0122] (9)

[0123] It is calculated by cosine distance, that is:

[0124] (10)

[0125] The extraction process of the texture difference feature extraction network specifically includes:

[0126] Generate images based on domain adaptation G t ( w ), source domain synthetic image Gs ( w ) and target domain training images I t , using the pre-trained VGG-16 network, from {1, 2, …, L} layer outputs style feature encoding;

[0127] Calculate the Gram matrix of each image { D 1 , D 2 ,…, D L}, get the style query sample according to the style feature encoding t g , Style Positive Sample t p and style negative samples t n .

[0128] In order to better transfer style features from target domain images under few-sample conditions, the CLIP model training loss is used to constrain feature learning in the model transfer process;

[0129] S8: Query samples based on semantics s g , semantic positive samples s u and semantic negative samples s v Determining the heterogeneous semantic contrast loss function The loss value of ; query samples according to style t g , Style Positive Sample t p and style negative samples t n Determine the style difference contrast loss function The loss value of heterogeneous semantic comparison loss function Contrasting loss function with style difference Sum and obtain the cross-domain comparison loss function value ; Compare the cross-domain loss function value And CLIP loss function value Sum and get the total loss function value Loss ;

[0130] The CLIP loss function includes intra-domain loss function, cross-domain loss function and two image reconstruction loss functions. , the calculation expression is:

[0131] (11)

[0132] In the formula, are weight hyperparameters, which are set to 10, 30, 1 and 0.5 respectively in the experiments.

[0133] Determining the heterogeneous semantic contrast loss function , and its calculation formula is:

[0134] (12)

[0135] in, is the joint distribution mean of the images, is the mean of the distribution of strong semantic negative samples guided by von Mises-Fisher distribution, is the temperature control coefficient, set to 0.07.

[0136] Using importance sampling to approximate the von Mises-Fisher distribution to express , the expression is:

[0137] (13)

[0138] Determining the heterogeneous semantic contrast loss function , and its calculation formula is:

[0139] (14)

[0140] In the formula, α It is a hyperparameter in the triplet loss function and is set to 0.04 in this embodiment.

[0141] Cross-domain comparison loss function value The calculation formula is:

[0142] (15)

[0143] In the formula, are weight coefficients, which are set to 1 and 0.5 respectively in this embodiment.

[0144] Total loss function value Loss The calculation formula is:

[0145] (16)

[0146] In the formula, is the weight coefficient, which is set to 1 in this embodiment.

[0147] S9: Determine the current total loss function value Loss Is it less than the preset threshold? If so, the current total loss function value is LossAs the optimal total loss value of the fine-tuned target domain generator, the trained single-sample multi-target domain adaptive image generation model is obtained; otherwise, the current total loss function value is used Loss Perform optimization updates to train the fine-tuned target domain generator and return to S2 until the current total loss function value Loss Is it less than the preset threshold?

[0148] Based on the idea of ​​dynamic kernel modulation and the constraints of cross-domain contrastive learning, this application constructs a single-sample multi-target domain adaptive image generation model, which can effectively suppress the overfitting problem of the deep generation model under the training conditions of single-sample multi-target domains and generate new images of different style domains at one time; effectively solving the problems of large computing resource usage and high training time cost of current methods when facing such tasks

[0149] This application constructs a dynamic kernel modulation module based on cross-domain feature knowledge similarity, combined with hypernetwork technology, to guide the fine-tuning process of the model with the similarity between cross-domain convolution kernel encodings, and more accurately mine the knowledge transfer ability when learning features between cross-domain images; This application prevents other information irrelevant to the domain adaptation task from being transmitted to the domain adaptation process of the image, and more effectively solves the overfitting problem of deep generative models when the number of training samples is severely limited;

[0150] This application uses a cross-domain contrast loss function based on dual constraints of structure and style, utilizes the idea of ​​contrastive learning, and combines the heterogeneous semantic relationship of images with the style and texture differences to jointly constrain the feature consistency of the generated image, effectively avoiding the feature information confusion problem caused by the joint learning of multiple categories of target domains; thereby reducing the artifacts and distortion problems in the generated images of existing methods when facing such tasks, and further improving the generation quality of domain adaptive images.

[0151] This application has achieved state-of-the-art results in both quantitative and qualitative evaluations of domain adaptation tasks for single-sample multi-category image generation models, demonstrating the advancement and effectiveness of this application.

[0152] In the fine-tuning process of the model, the Adam optimizer is used as the optimization algorithm for gradient descent (the Momentum parameters are 0.0 and 0.99), the learning rate is set to 0.002, and the batch size is set to 4. In the process of calculating the loss function, "ViT-B / 16" and "ViT-B / 32" are used as the encoders for image embedding, respectively, to encode the image into a vector in the CLIP space.

[0153] In the model inference phase, consistent with existing methods, style mixing techniques are used in the target domain generator to better reflect the domain style attributes unique to different target domain images. The style mixing technique uses the latent vectors estimated from the target domain training images for the fine layers of the generator network (the first 10 layers of the generator network in the experiment). w c , while for other network layer structures, the generator is used to map the potential vector in the network w . w c The GAN inversion technology transforms the given target domain image I t Converted to represent the style information of the image.

[0154] The experimental datasets include: Church2Painting, Cat2Animals, and Real2MetFaces; Church2Painting contains 9 painting-style images (3 images each of Paul Cézanne, Claude Monet, and Van Gogh), and the resolution of each image is 256×256. This task can verify the domain adaptive generation ability of the method in this application for typical style transfer tasks. Cat2Animals contains 10 faces of animals of different species (male lion, female lion, white tiger, tiger, snow leopard, cheetah, leopard, snow fox, red fox, wolf), all images are from the AFHQ Wild dataset, and the resolution of each image is 512×512. This task can verify the domain adaptive generation ability of the method in this application when the semantic differences between the target domains are large. For the quantitative evaluation of the generated images, 500 images of the AFHQ Wild test dataset are used as the reference real dataset. The pre-trained source domain is the StyleGAN2 model trained by the AFHQ Cat dataset, and the number of iterations of all methods is set to 1200. Real2MetFaces consists of 10 facial images of different styles from the MetFace dataset (including sculptures, murals, black and white photos, etc.), and the resolution of all images is 1024×1024. For the quantitative evaluation of the generated images, we use 1336 images from the MetFace dataset as the reference real dataset. In the experiment, the pre-trained source domain is the StyleGAN2 model trained with the FFHQ dataset, and the number of iterations for all methods is set to 1500.

[0155] For the quality evaluation of generated images, this application uses Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) as evaluation criteria for the authenticity of generated images, and uses InceptionScore (IS) as an evaluation indicator for the diversity of generated images. FID and KID quantify the similarity between the distributions of real images and generated images, and the lower the calculated value, the higher the similarity. Among them, FID introduces bias estimation, while KID is an unbiased measure of distance estimation, which can ensure the objectivity of the evaluation results. At the same time, the InceptionNet-v3 network is used to evaluate the IS score directly on the generated images, so that there is no need for a test image dataset for comparison. The higher the IS calculation result, the better the diversity of the generated images.

[0156] Figure 6 Comparison of the results generated by this application with other methods in the Church2Painting experimental task; Figure 7 Comparison of the results generated by this application method with other methods in the Cat2Animals experimental task; Figure 8 Table 1 shows the quantitative comparison results of this application with other methods in the Cat2Animals and Real2MetFaces tasks. It can be seen that this application maintains a leading advantage in all indicators.

[0157] Table 1

[0158]

[0159] Based on the same inventive concept, the embodiment of the present application also provides a domain adaptive image generation system for implementing the domain adaptive image generation method involved above. The implementation solution provided by the system to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more domain adaptive image generation system embodiments provided below can refer to the limitations of the domain adaptive image generation method above, and will not be repeated here.

[0160] In an exemplary embodiment, a domain adaptive image generation system is provided, comprising:

[0161] A source domain image acquisition module, used to acquire a source domain image;

[0162] A domain adaptive image generation module is used to generate a target domain image based on a source domain image using a trained single-sample multi-target domain adaptive image generation model; the trained single-sample multi-target domain adaptive image generation model includes: a source domain generator, a target domain generator, a target domain style feature super network, a dynamic kernel modulation module, a heterogeneous semantic feature extraction network, a texture difference feature extraction network and a CLIP model;

[0163] The training process of the single-sample multi-target domain adaptive image generation model is:

[0164] S1: Use multiple source domain images and multiple 512-dimensional vectors randomly sampled from a multi-dimensional Gaussian distribution to train the pre-trained source domain generator to obtain the source domain generator G s ; Using the source domain generator G s The weights of the target domain generator are initialized to obtain the initial target domain generator G t0 , and select multiple target domain training images of different styles as the initial target domain generator G t0 The target domain guidance image I t ;

[0165] S2: One-hot vector encoding is performed on multiple target domain training images of different styles to obtain the corresponding target domain style control conditions c ; At the same time, multiple target domain guide images of different styles I t Input into the target domain style feature super network to obtain the corresponding target domain style feature convolution kernel ;

[0166] S3: Fixed Source Domain Generator G s The network structure is randomly sampled from a multidimensional Gaussian distribution. B A 512-dimensional vector, after the mapping network M Convert to intermediate latent vector w , the intermediate latent vector w Input to the source domain generator to generate B Zhang source domain synthetic image G s ( w );

[0167] S4: The fixed source domain generator G s Convolution kernels in each convolution layer Convolution kernel with target domain style feature Input to the dynamic kernel modulation module based on cross-domain feature knowledge similarity to generate a new modulated convolution kernel The dynamic kernel modulation module is used to combine the hypernetwork technology, use the similarity between cross-domain convolution kernel encodings to guide the correlation of feature learning, and generate a new target domain modulated convolution kernel ;

[0168] S5: Modulating the convolution kernel using a new target domain For the initial target domain generator G t0 Fine-tune the corresponding convolutional layer; repeat S4 to complete the fine-tuning of all convolutional layers of the target domain generator to obtain the fine-tuned target domain generator G t ;

[0169] S6: Utilization and B 512-dimensional intermediate latent vector w and the corresponding target domain style control condition c , input to the modulated target domain generator G t , obtain the domain adaptive generated image corresponding to the style of the target domain training image G t ( w );

[0170] S7: Generate images based on domain adaptation G t ( w ), source domain synthetic image G s ( w ) and target domain training images I t , a heterogeneous semantic feature extraction network is used to extract the semantic distribution of image blocks in the semantic structure space, and the semantic query samples are obtained respectively. s g , semantic positive samples s u and semantic negative samples s v ; Generate images by domain adaptation G t ( w ), corresponding to the target style training image I t0 And other style training images I t1 Input into the texture difference feature extraction network, extract the style feature differences of each image in the texture style space, and obtain style query samples respectively t g , style positive sample t pand style negative samples t n ; Generate images by domain adaptation G t ( w ), target domain training images I t , Target domain style control conditions c and the intermediate latent vector w Input into the CLIP model together to determine the CLIP loss function value ;

[0171] S8: Query samples based on semantics s g , semantic positive samples s u and semantic negative samples s v Determining the heterogeneous semantic contrast loss function The loss value of ; query samples according to style t g , Style Positive Sample t p and style negative samples t n Determine the style difference contrast loss function The loss value of heterogeneous semantic comparison loss function Contrasting loss function with style difference Sum and obtain the cross-domain comparison loss function value ; Compare the cross-domain loss function value And CLIP loss function value Sum and get the total loss function value Loss ;

[0172] S9: Determine the current total loss function value Loss Is it less than the preset threshold? If so, the current total loss function value is Loss As the optimal total loss value of the fine-tuned target domain generator, the trained single-sample multi-target domain adaptive image generation model is obtained; otherwise, the current total loss function value is used Loss Perform optimization updates to train the fine-tuned target domain generator and return to S2 until the current total loss function value Loss Is it less than the preset threshold?

[0173] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0174] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0175] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0176] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0177] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0178] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A single-sample multi-style domain adaptive image generation method, characterized in that: The single-sample multi-style domain adaptive image generation method comprises: Obtain source domain image; According to the source domain image, a target domain image is generated using a trained single-sample multi-target domain adaptive image generation model; the trained single-sample multi-target domain adaptive image generation model includes: a source domain generator, a target domain generator, a target domain style feature super network, a dynamic kernel modulation module, a heterogeneous semantic feature extraction network, a texture difference feature extraction network and a CLIP model; The training process of the single-sample multi-target domain adaptive image generation model is: S1: Use multiple source domain images and multiple 512-dimensional vectors randomly sampled from a multi-dimensional Gaussian distribution to train the pre-trained source domain generator to obtain the source domain generator G s ; Using the source domain generator G s The weights of the target domain generator are initialized to obtain the initial target domain generator G t0 , and select multiple target domain training images of different styles as the initial target domain generator G t0 The target domain training images I t ; S2: One-hot vector encoding is performed on multiple target domain training images of different styles to obtain the corresponding target domain style control conditions c ; At the same time, multiple target domain training images of different styles I t Input into the target domain style feature super network to obtain the corresponding target domain style feature convolution kernel ; S3: Fixed Source Domain Generator G s The network structure is randomly sampled from a multidimensional Gaussian distribution. B A 512-dimensional vector, after the mapping network M Convert to intermediate latent vector w , the intermediate latent vector w Input to the source domain generator to generate B Zhang source domain synthetic image G s ( w ); S4: The fixed source domain generator G s Convolution kernels in each convolution layer Convolution kernel with target domain style feature Input to the dynamic kernel modulation module based on cross-domain feature knowledge similarity to generate a new target domain modulated convolution kernel The dynamic kernel modulation module is used to combine the hypernetwork technology, use the similarity between cross-domain convolution kernel encodings to guide the correlation of feature learning, and generate a new target domain modulated convolution kernel ; S5: Modulating the convolution kernel using a new target domain For the initial target domain generator G t0 Fine-tune the corresponding convolutional layer; repeat S4 to complete the fine-tuning of all convolutional layers of the target domain generator to obtain the fine-tuned target domain generator G t ; S6: Utilization and B 512-dimensional intermediate latent vector w and the corresponding target domain style control condition c , input to the fine-tuned target domain generator G t , obtain the domain adaptive generated image corresponding to the style of the target domain training image G t ( w ); S7: Generate images based on domain adaptation G t ( w ), source domain synthetic image G s ( w ) and target domain training images I t , a heterogeneous semantic feature extraction network is used to extract the semantic distribution of image blocks in the semantic structure space, and the semantic query samples are obtained respectively. s g , semantic positive samples s u and semantic negative samples s v ; Generate images by domain adaptation G t ( w ), corresponding to the target style training image I t0 And other style training images I t1 Input into the texture difference feature extraction network, extract the style feature differences of each image in the texture style space, and obtain style query samples respectively t g , Style Positive Sample t p and style negative samples t n ; Generate images by domain adaptation G t ( w ), target domain training images I t , Target domain style control conditions c and the intermediate latent vector w Input into the CLIP model together to determine the CLIP loss function value ; S8: Query samples based on semantics s g , semantic positive samples s u and semantic negative samples s v Determining the heterogeneous semantic contrast loss function The loss value of ; query samples according to style t g , Style Positive Sample t p and style negative samples t n Determine the style difference contrast loss function The loss value of heterogeneous semantic comparison loss function Comparative loss function with style difference Sum and obtain the cross-domain comparison loss function value ; Compare the cross-domain loss function value And CLIP loss function value Sum and get the total loss function value Loss ; S9: Determine the current total loss function value Loss Is it less than the preset threshold? If so, the current total loss function value is Loss As the optimal total loss value of the fine-tuned target domain generator, the trained single-sample multi-target domain adaptive image generation model is obtained; otherwise, the current total loss function value is used Loss Perform optimization updates to train the fine-tuned target domain generator and return to S2 until the current total loss function value Loss Is it less than the preset threshold? 2. The single-sample multi-style domain adaptive image generation method according to claim 1, characterized in that: The target domain style feature super network includes: an image encoder and a multi-layer fully connected layer structure; The image encoder is used to convert multiple target domain training images of different styles I t Convert to image vector; and correspond the image vector to the target domain style control condition of the style c Input into the multi-layer fully connected layer structure; The multi-layer fully connected layer structure is used to adopt a rank 1 tensor decomposition method to obtain the one-dimensional vectors of the three style features respectively, and obtain the corresponding target domain style feature convolution kernel according to the one-dimensional vectors of the three style features .

3. The single-sample multi-style domain adaptive image generation method according to claim 1, characterized in that: The source domain generator is fixed in turn G s Convolution kernels in each convolution layer Convolution kernel with target domain style feature Input to the dynamic kernel modulation module based on cross-domain feature knowledge similarity to generate a new target domain modulated convolution kernel , specifically including: The source domain generator will be fixed G s Convolution kernels in each convolution layer Convolution kernel with target domain style feature Reshape into 2D matrices respectively; The reshaped 2D matrices are multiplied to determine the kernel similarity matrix; and the kernel similarity matrix is ​​processed by the softmax function to obtain the global similarity attention matrix; Multiply the 2D matrix of the reshaped convolutional layer kernel with the global similarity attention matrix to obtain a new target domain modulated convolution kernel modulated by the similarity of cross-domain feature knowledge .

4. The single-sample multi-style domain adaptive image generation method according to claim 1, characterized in that: The extraction process of the heterogeneous semantic feature extraction network is: Generate images based on domain adaptation G t ( w ), source domain synthetic image G s ( w ) and target domain training images I t ,Use the pre-trained VGG-16 network to extract image feature maps; Extract semantic query samples according to image feature maps s g , semantic positive samples s u and semantic negative samples s v ; Query samples based on semantics s g , semantic positive samples s u and semantic negative samples s v Determine semantic similarity distribution and heterogeneous semantic similarity distribution within corresponding image domains; According to the heterogeneous semantic similarity distribution, the von Mises-Fisher distribution is used to guide the semantic negative sample sampling strategy.

5. The single-sample multi-style domain adaptive image generation method according to claim 4, characterized in that: The strategy of guiding semantic negative sample sampling by using von Mises-Fisher distribution according to heterogeneous semantic similarity distribution specifically includes: Using the formula Determine the i-th semantic negative sample ; in, N q is the normalization constant, To determine each negative sample The importance weight of is a semantic positive sample, the superscript T is transposed, is a single negative sample, is the importance weight in heterogeneous semantic similarity distribution, Generating images for domain adaptation G t ( w ) is the embedding vector in the semantic space, is a semantic negative sample distribution.

6. The single-sample multi-style domain adaptive image generation method according to claim 1, characterized in that: The extraction process of the texture difference feature extraction network specifically includes: Generate images based on domain adaptation G t ( w ), source domain synthetic image G s ( w ) and target domain training images I t , using the pre-trained VGG-16 network to output style feature encoding; Get style query samples according to style feature encoding t g , Style Positive Sample t p and style negative samples t n .

7. A single-sample multi-style domain adaptive image generation system, characterized in that: The single-sample multi-style domain adaptive image generation system comprises: A source domain image acquisition module, used to acquire a source domain image; A domain adaptive image generation module is used to generate a target domain image based on a source domain image using a trained single-sample multi-target domain adaptive image generation model; the trained single-sample multi-target domain adaptive image generation model includes: a source domain generator, a target domain generator, a target domain style feature super network, a dynamic kernel modulation module, a heterogeneous semantic feature extraction network, a texture difference feature extraction network and a CLIP model; The training process of the single-sample multi-target domain adaptive image generation model is: S1: Use multiple source domain images and multiple 512-dimensional vectors randomly sampled from a multi-dimensional Gaussian distribution to train the pre-trained source domain generator to obtain the source domain generator G s ; Using the source domain generator G s The weights of the target domain generator are initialized to obtain the initial target domain generator G t0 , and select multiple target domain training images of different styles as the initial target domain generator G t0 The target domain training images I t ; S2: One-hot vector encoding is performed on multiple target domain training images of different styles to obtain the corresponding target domain style control conditions c ; At the same time, multiple target domain training images of different styles I t Input into the target domain style feature super network to obtain the corresponding target domain style feature convolution kernel ; S3: Fixed Source Domain Generator G s The network structure is randomly sampled from a multidimensional Gaussian distribution. B A 512-dimensional vector, after the mapping network M Convert to intermediate latent vector w , the intermediate latent vector w Input to the source domain generator to generate B Zhang source domain synthetic image G s ( w ); S4: The fixed source domain generator G s Convolution kernels in each convolution layer Convolution kernel with target domain style feature Input to the dynamic kernel modulation module based on cross-domain feature knowledge similarity to generate a new target domain modulated convolution kernel The dynamic kernel modulation module is used to combine the hypernetwork technology, use the similarity between cross-domain convolution kernel encodings to guide the correlation of feature learning, and generate a new target domain modulated convolution kernel ; S5: Modulating the convolution kernel using a new target domain For the initial target domain generator G t0 Fine-tune the corresponding convolutional layer; repeat S4 to complete the fine-tuning of all convolutional layers of the target domain generator to obtain the fine-tuned target domain generator G t ; S6: Utilization and B 512-dimensional intermediate latent vector w and the corresponding target domain style control condition c , input to the fine-tuned target domain generator G t , obtain the domain adaptive generated image corresponding to the style of the target domain training image G t ( w ); S7: Generate images based on domain adaptation G t ( w ), source domain synthetic image G s ( w ) and target domain training images I t , a heterogeneous semantic feature extraction network is used to extract the semantic distribution of image blocks in the semantic structure space, and the semantic query samples are obtained respectively. s g , semantic positive samples s u and semantic negative samples s v ; Generate images by domain adaptation G t ( w ), corresponding to the target style training image I t0 And other style training images I t1 Input into the texture difference feature extraction network, extract the style feature differences of each image in the texture style space, and obtain style query samples respectively t g , Style Positive Sample t p and style negative samples t n ; Generate images by domain adaptation G t ( w ), target domain training images I t , Target domain style control conditions c and the intermediate latent vector w Input into the CLIP model together to determine the CLIP loss function value ; S8: Query samples based on semantics s g , semantic positive samples s u and semantic negative samples s v Determining the heterogeneous semantic contrast loss function The loss value of ; query samples according to style t g , Style Positive Sample t p and style negative samples t n Determine the style difference contrast loss function The loss value of heterogeneous semantic comparison loss function Comparative loss function with style difference Sum and obtain the cross-domain comparison loss function value ; Compare the cross-domain loss function value And CLIP loss function value Sum and get the total loss function value Loss ; S9: Determine the current total loss function value Loss Is it less than the preset threshold? If so, the current total loss function value is Loss As the optimal total loss value of the fine-tuned target domain generator, the trained single-sample multi-target domain adaptive image generation model is obtained; otherwise, the current total loss function value is used Loss Perform optimization updates to train the fine-tuned target domain generator and return to S2 until the current total loss function value Loss Is it less than the preset threshold?

Images