An adversarial fine-tuning and generation method and device for single-source domain generalization

Abstract

The application provides an adversarial fine-tuning and generation method and device for single-source domain generalization, and relates to the technical field of domain generalization image classification. The method comprises the following steps: obtaining a data set for a domain generalization task; constructing a domain generalization model, wherein the domain generalization model comprises a text encoder, a self-encoder and a conditional denoising module connected in sequence, and the text encoder is used for adding a class prompt and a domain style prompt to a plurality of sub-data sets; inputting the plurality of sub-data sets into the domain generalization model to train the domain generalization model by using a progressive adversarial learning strategy, and obtaining a trained domain generalization model; inputting a single-source domain data set into the trained domain generalization model to make the trained domain generalization model perform single-source domain generalization, and outputting target images with different types and different domain styles. When the target domain features are too abstract, image samples corresponding to the domain can be generated, and diversified images of multiple domain styles and multiple classes can also be generated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of domain generalization image classification, and particularly relates to an adversarial fine-tuning and generation method and device for single-source domain generalization. BACKGROUND

[0002] In recent years, machine learning models (such as deep neural networks) have shown excellent performance in computer vision, natural language processing and other tasks, but their success relies on the key assumption that the training data and the test data satisfy the independent and identically distributed assumption. However, in real-world scenarios, domain shift is ubiquitous, for example, the test data may come from an unknown domain (i.e. out-of-distribution data) that is significantly different from the training data distribution, and such distribution difference will cause a significant decrease in model performance. To deal with the problem of domain shift, traditional methods mainly focus on domain adaptation and domain generalization. However, domain adaptation and domain generalization rely on the availability of target domain data and can only obtain single-labeled source domain data in practical applications, which is difficult to apply in scenarios where data privacy is limited or annotation cost is high. Therefore, single-domain generalization has become a more challenging and practical research direction, which aims to train a model through single-source domain data so that it performs well in multiple unknown and distributionally shifted target domains. Current single-domain generalization methods mainly expand the data distribution of the source domain through data augmentation techniques, or use adaptive data normalization methods. However, the sample diversity of these two methods is limited by the inherent distribution characteristics of the source domain, making it difficult to cover the complex and diverse domain shift patterns in real-world scenarios; and lacking theoretical constraints on the distribution difference between generated samples and potential target domains, resulting in a deviation between the distribution expansion of generated samples and the needs of real target domains.

[0003] Currently, the breakthrough of text-to-image base models provides a new idea for single-domain generalization. Text-to-image base models have strong ability to generate images from text descriptions. Since text itself contains rich semantic information, text-to-image base models can generate diverse images across semantic domains based on text prompts. In theory, by designing diverse text prompts, we can generate data covering multiple domain features, thereby expanding the source domain distribution. However, in practical applications, when the target domain features are too abstract and difficult to accurately describe in language, existing text-to-image base models have difficulty in accurately capturing and generating image samples of the corresponding domain; existing text-to-image models rely on manually designed prompts and lack an automatic generation mechanism, limiting the domain coverage of generated data, resulting in insufficient diversity of generated image samples in a single domain. SUMMARY

[0004] The embodiment of the present application aims to provide an adversarial fine-tuning and generation method and device for single-source domain generalization, solve the problems that the existing technology is difficult to generate image samples of the corresponding field when the characteristics of the target field are too abstract and difficult to accurately describe in language, and the image samples generated by the existing technology are single field and lack diversity.

[0005] To solve the above technical problems, the embodiment of the present application provides the following technical solutions:

[0006] The first aspect of the present application provides an adversarial fine-tuning and generation method for single-source domain generalization, comprising:

[0007] Obtain a data set for a domain generalization task, the data set comprising a plurality of sub-data sets, each sub-data set having different domain styles and the same number of categories;

[0008] Construct a domain generalization model, the domain generalization model comprising a text encoder, an autoencoder and a conditional denoising module connected in sequence, the text encoder being used to add category prompts and domain style prompts to the plurality of sub-data sets;

[0009] Input the plurality of sub-data sets into the domain generalization model to train the domain generalization model using a progressive adversarial learning strategy, and obtain a trained domain generalization model;

[0010] Input a single-source domain data set into the trained domain generalization model to make the trained domain generalization model perform single-source domain generalization, and output a target image, the target image being an image having different types and different domain styles.

[0011] The second aspect of the present application provides an adversarial fine-tuning and generation device for single-source domain generalization, comprising:

[0012] An obtaining module is configured to obtain a data set for a domain generalization task, the data set comprising a plurality of sub-data sets, each sub-data set having different domain styles and the same number of categories;

[0013] A constructing module is configured to construct a domain generalization model, the domain generalization model comprising a text encoder, an autoencoder and a conditional denoising module connected in sequence, the text encoder being used to add category prompts and domain style prompts to the plurality of sub-data sets;

[0014] A training module is configured to input the plurality of sub-data sets into the domain generalization model to train the domain generalization model using a progressive adversarial learning strategy, and obtain a trained domain generalization model;

[0015] A single-source domain generalization module is configured to input a single-source domain data set into the trained domain generalization model to make the trained domain generalization model perform single-source domain generalization, and output a target image, the target image being an image having different types and different domain styles.

[0016] Compared with the prior art, the method and device for single-source domain generalization provided by the application obtain a data set for a domain generalization task, the data set includes multiple sub data sets, the domain styles of each sub data set are different and the number of categories is the same; a domain generalization model is constructed, the domain generalization model includes a text encoder, a self-encoder and a conditional denoising module connected in sequence, the text encoder is used to add category prompts and domain style prompts to the multiple sub data sets; the multiple sub data sets are input into the domain generalization model, so as to train the domain generalization model by using a progressive adversarial learning strategy, and obtain a trained domain generalization model; a single-source domain data set is input into the trained domain generalization model, so that the trained domain generalization model performs single-source domain generalization, and outputs a target image, the target image is an image with different types and different domain styles. In this way, through the text encoder of the domain generalization model, the added category prompts and domain style prompts can be learned, and abstract prompts can be learned, so that when the target domain features are too abstract and difficult to accurately describe in language, image samples of the corresponding domain can be generated, and through the added category prompts and domain style prompts, multiple categories and domain styles can be learned, so that the generation can perform single-source domain generalization on the single-source domain data set, and diversified images of multiple categories and multiple domain styles can be generated. BRIEF DESCRIPTION OF DRAWINGS

[0017] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which a number of embodiments of the application are shown by way of example, and in which similar elements are indicated by like reference characters. While the application will be described in conjunction with the preferred embodiments, it will be understood that they are given by way of example only and that the application is not intended to be limited to such embodiments unless specifically set forth.

[0018] Figure 1 A flowchart of the method for single-source domain generalization is schematically shown;

[0019] Figure 2 A flowchart of obtaining the trained domain generalization model is schematically shown;

[0020] Figure 3 A structural diagram of the device for single-source domain generalization is schematically shown. DETAILED DESCRIPTION

[0021] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it is understood that the present application can be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thoroughly and completely understood, and so that the scope of the present application will be completely conveyed to those skilled in the art.

[0022] It should be noted that, unless otherwise specified, the technical or scientific terms used in the present invention should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0023] The method in the embodiment of the present invention is described in detail below.

[0024] Figure 1 The flowchart of the adversarial fine-tuning and generation method for single-source domain generalization in an embodiment of the present invention is schematically shown. Figure 1 As shown, the adversarial fine-tuning and generation method for single-source domain generalization may include:

[0025] S101. Obtain a dataset for domain generalization tasks.

[0026] The dataset includes multiple sub-datasets, each of which has a different domain style and the same number of categories. The multiple sub-datasets each include multiple preset texts and multiple preset images corresponding to the multiple preset texts, and the multiple preset texts are used to express category names.

[0027] The samples in each sub-dataset are disjoint. A sub-dataset has only one domain style, and the domain styles of multiple sub-datasets are different.

[0028] Multiple subdatasets It can be expressed as in, is the first sub-dataset, For the second sub-dataset, For the third sub-dataset, is the m+1th sub-dataset, For the Sub-datasets. As the source field, As multiple target areas Multiple target areas Among them, the second sub-dataset As the first target area, the third sub-dataset As the second target domain, the m+1th sub-dataset As the mth target area, Sub-datasets As the first target areas, is the total number of multiple target fields. Each sub-dataset has a label, that is, the source field and multiple target fields are all labeled. The label is a plurality of preset texts used to express the category name. Expressed as Among them, x i′y is the i'th preset image of the first sub-data set, i.e., a sample i′ N is preset text corresponding to the i'th preset image of the first sub-data set, i.e., a label, s K is the number of preset images of the first sub-data set.

[0029] The number of categories of each sub-data set is K, and is represented as where x i″ is the x i″ th preset image of the m+1th sub-data set, y i″ is the x i″ th preset image of the m+1th sub-data set, and K is the number of preset images of the m+1th sub-data set. The goal of the domain generalization task is to solve the domain distribution shift problem by generating diversified domain style images, thereby constructing a model that can perform well on all unseen target domains, i.e., the domain generalization model in the following step S102.

[0030] S102, constructing a domain generalization model.

[0031] The domain generalization model includes a text encoder, an autoencoder, and a conditional denoising module connected in sequence. The text encoder is used to add category hints and domain style hints to the multiple sub-data sets.

[0032] Specifically, the text encoder is a CLIP (Contrastive Language-Image Pre-training) model-based encoder. The autoencoder includes an encoder and a decoder. The encoder is used to map the input multiple preset images to a latent space to convert the input multiple preset images into latent representations, and the decoder is used to reconstruct the images from the latent representations. The conditional denoising module is a U-Net network based on a cross-attention layer. The U-Net network based on the cross-attention layer is a U-Net network with a cross-attention layer introduced. The conditional denoising module is used to add noise to the latent representations converted by the encoder and also used to predict and denoise the noise.

[0033] S103, inputting the multiple sub-data sets into the domain generalization model to train the domain generalization model using a progressive adversarial learning strategy to obtain a trained domain generalization model.

[0034] Specifically, Figure 2 A flowchart of obtaining the trained domain generalization model is schematically shown, see Figure 2As shown, the plurality of sub-data sets are input into the domain generalization model to train the domain generalization model by using a progressive adversarial learning strategy to obtain a trained domain generalization model, including:

[0035] Step S1031: inputting the plurality of preset texts into the text encoder to add a class prompt in the plurality of preset texts by using the text encoder, so that the text encoder outputs a first text embedding representation feature.

[0036] Specifically, step S1031 includes:

[0037] Step A1: performing word segmentation on the plurality of preset texts by using the first input string to obtain a plurality of first word pieces.

[0038] The expression of the input string is:

[0039]

[0040] wherein, is the first input string, * is the first placeholder, k is the kth class, and K is the number of classes of each sub-data set. The first placeholder is used to indicate the position of the class prompt in the first input string.

[0041] Step A2: converting each first word piece into a first embedding vector.

[0042] wherein, the first embedding vector includes the first placeholder.

[0043] Suppose M c is the length of the word piece corresponding to the first input string except the first placeholder, i.e., the length of the first word piece, then the corresponding first embedding vector is defined as: wherein, is the first embedding vector, is the first embedding vector corresponding to the 1st first word piece length of the kth class, is the first embedding vector corresponding to the jth first word piece length of the kth class, is the first embedding vector corresponding to the M c th first word piece length of the kth class.

[0044] Step A3: adding a class prompt at the first placeholder according to the embedding function corresponding to the plurality of first word pieces to obtain an intermediate feature with a class prompt.

[0045] The class prompt added at the first placeholder is a learnable class prompt.

[0046] Specifically, the expression of the intermediate feature with the class prompt is:

[0047]

[0048] in, is the intermediate feature with category prompt of the k-th category, ε(·) is the embedding function corresponding to multiple first word units, is the first word corresponding to the first word length of the k-th category, is the first word corresponding to the jth first word length of the kth category, is the Mth class of the kth class c The first word corresponding to the first word length, is the first embedding vector corresponding to the first word length of the k-th category, is the first embedding vector corresponding to the j-th first word length of the k-th category, is the Mth class of the kth class c The first embedding vector corresponding to the first word length, is the embedding function of the first embedding vector corresponding to the first word length of the k-th category, is the embedding function of the first embedding vector corresponding to the j-th first word length of the k-th category, is the Mth class of the kth class c The embedding function of the first embedding vector corresponding to the first word length, is the category hint for the k-th category.

[0049] Step A4: Input the intermediate features with category hints into the feature extractor so that the feature extractor outputs the first text embedding representation features.

[0050] The feature extractor is the extractor in the text encoder based on the CLIP model. The feature extractor is represented as Φ(·), and the first text embedding representation feature is represented as

[0051] Step S1032: Input a plurality of preset images into the autoencoder, so that the autoencoder outputs a first latent representation.

[0052] The expression of the first potential representation is: z0 = f E (X), where z0 is the first potential representation, X is a plurality of preset images, and f E (·) is the encoder in the autoencoder.

[0053] Step S1033: inputting the first text embedding representation feature and the first latent representation into a conditional denoising module, so that the conditional denoising module outputs predicted noise.

[0054] Specifically, step S1033 includes:

[0055] Step B1: add Gaussian noise to the first latent representation to obtain a noisy first latent representation.

[0056] The amount of added Gaussian noise varies according to the training round t, following a variance schedule where T' is the total training round, β t is the variance schedule of the t-th training round. Gaussian noise ∈ is added to the first latent representation z0, and the expression of the noisy first latent representation is:

[0057]

[0058] where z t is the noisy first latent representation, α t = 1-β t , α t is the intermediate parameter of the t-th training round, is the average value of the total intermediate parameter corresponding to the total training round, z0 is the first latent representation, ∈ is Gaussian noise, and N(0, 1) is a normal distribution with mean 0 and standard deviation 1.

[0059] Step B2: determine the predicted noise according to the noisy first latent representation, the first text embedding representation feature, and the training round.

[0060] The first text embedding representation feature is passed to the cross-attention layer ∈ θ (·) in the conditional denoising module, and the prediction of the predicted noise is performed based on the noisy first latent representation and the t-th training round. The expression of the predicted noise is:

[0061]

[0062] where ∈ is the predicted noise, ∈ θ (·) is the cross-attention layer, z t is the noisy first latent representation, t is the t-th training round, is the first text embedding representation feature, is the intermediate feature with class prompt of the k-th class, and Φ(·) is the feature extractor.

[0063] Step S1034: based on the predicted noise and the Gaussian noise, determine the updated class prompt using the backpropagation algorithm.

[0064] Specifically, step S1034 includes:

[0065] Step C1: determine the latent space diffusion model loss according to the predicted noise and the Gaussian noise.

[0066] The expression of the latent diffusion model (LDM) loss is as follows:

[0067]

[0068] wherein, is the latent diffusion model loss, E ∈,t is the expectation under Gaussian noise at the t-th training round, and ||·||2 is the L2 norm, is the predicted noise, and ∈ is the Gaussian noise.

[0069] Step C2: using the back propagation algorithm, the latent diffusion model loss is back propagated to obtain an updated class prompt.

[0070] After T' total training rounds, K class prompts can be learned, the K class prompts include the updated class prompt, and each class prompt corresponds to a class.

[0071] The above steps S1031 to S1034 can be referred to as class prompt optimization, and the class information that is invariant to the domain style can be captured from the sub-data set corresponding to the source domain. In actual scenarios, some concepts or classes are difficult to be understood by existing diffusion models through a single class label or name. Therefore, the present application uses a domain generalization model to train a class prompt to represent each class, and the class prompt is used as an abstract representation of the corresponding class.

[0072] Step S1035: inputting the updated class prompt into the text encoder to add a domain style prompt in the plurality of preset texts by using the text encoder, so that the text encoder outputs a second text embedding representation feature.

[0073] Specifically, step S1035 includes:

[0074] Step D1: performing word segmentation on the updated class prompt by using the second input string to obtain a plurality of second wordpieces.

[0075] The expression of the second input string is as follows:

[0076]

[0077] wherein, is the second input string, is the second placeholder, i is the i-th domain style, and N' is the total number of domain styles.

[0078] Step D2: converting each second wordpiece into a second embedding vector.

[0079] wherein, the second embedding vector includes the second placeholder.

[0080] Suppose Md is the length of the second token corresponding to the second input string except the second placeholder, i.e., the second token length, then the corresponding second embedding vector is defined as: wherein, is the second embedding vector, is the second embedding vector corresponding to the first second token length of the i-th domain style, is the second embedding vector corresponding to the j'-th second token length of the i-th domain style, is the second embedding vector corresponding to the M-th second token length of the i-th domain style. d is the second embedding vector corresponding to the M-th second token length of the i-th domain style.

[0081] Step D3: Adding the domain style prompt at the second placeholder according to the embedding function corresponding to the plurality of second tokens to obtain an intermediate feature with a domain style prompt.

[0082] wherein the domain style prompt is the initialized prompt in the memory bank.

[0083] Specifically, the expression of the intermediate feature with the domain style prompt is:

[0084]

[0085] wherein, is the intermediate feature with the domain style prompt of the i-th domain style, and ε(·) is the embedding function corresponding to the plurality of second tokens, is the second token corresponding to the first second token length of the i-th domain style, is the second token corresponding to the j'-th second token length of the i-th domain style, is the second token corresponding to the M-th second token length of the i-th domain style, d is the second token corresponding to the M-th second token length of the i-th domain style, is the second embedding vector corresponding to the first second token length of the i-th domain style, is the second embedding vector corresponding to the j'-th second token length of the i-th domain style, is the second embedding vector corresponding to the M-th second token length of the i-th domain style, d is the second embedding vector corresponding to the M-th second token length of the i-th domain style, is the embedding function of the second embedding vector corresponding to the first second token length of the i-th domain style, is the embedding function of the second embedding vector corresponding to the j'-th second token length of the i-th domain style, is the embedding function of the second embedding vector corresponding to the M-th second token length of the i-th domain style, d is the embedding function of the second embedding vector corresponding to the M-th second token length of the i-th domain style, is the domain style prompt of the i-th domain style.

[0086] Step D4: inputting the intermediate feature with the domain style prompt into the feature extractor, so that the feature extractor outputs a second text embedding representation feature.

[0087] The feature extractor is an extractor in the text encoder based on the CLIP model. The feature extractor is denoted as Φ(·), and the second text embedding representation feature is denoted as

[0088] Step S1036: inputting the second text embedding representation feature into the conditional denoising module, so as to denoise the randomly generated noise by using the conditional denoising module, and output a second latent representation.

[0089] Since random noise is generated in the process of determining the second text embedding representation feature, it is necessary to remove the generated random noise. The second text embedding representation feature is passed to the cross-attention layer ∈ θ (·) in the conditional denoising module, and based on the first latent representation after adding noise, the t-th training round, the second latent representation is made to be as close as possible to the first latent representation after adding noise z t to realize denoising of the randomly generated noise, so as to determine the second latent representation.

[0090] The expression of the second latent representation is:

[0091]

[0092] Wherein, z0′ is the second latent representation, z t is the first latent representation after adding noise, α t = 1-β t , α t is the intermediate parameter of the t-th training round, β t is the variance schedule of the t-th training round, is the average value of the total intermediate parameter corresponding to the total training round, T′ is the total training round, ∈ θ (·) is the cross-attention layer, is the second text embedding representation feature, is the intermediate feature with the domain style prompt of the i-th domain style.

[0093] Step S1037: based on the second latent representation and the second text embedding representation feature, determining an updated domain style prompt by using a back propagation algorithm.

[0094] Specifically, step S1037 includes:

[0095] Step E1: determining the adversarial domain prompt tuning loss according to the second latent representation, the second text embedding representation feature and the hyperparameter.

[0096] First, determining the probability that the second latent representation belongs to the i-th domain style according to the second latent representation, the second text embedding representation feature and the hyperparameter; then, determining the adversarial domain loss according to the probability that the second latent representation belongs to the i-th domain style; then, determining the category consistency loss according to the second latent representation and the first text embedding representation feature; finally, determining the adversarial domain prompt tuning loss according to the adversarial domain loss and the category consistency loss.

[0097] The expression of the probability that the second latent representation belongs to the i-th domain style is:

[0098]

[0099] wherein, is the probability that the second latent representation belongs to the i-th domain style, z0' is the second latent representation, is the second text embedding representation feature, τ is the hyperparameter for controlling the sharpness of the output, <·,·> is the dot product operation, is the i-th domain style intermediate feature with the domain style prompt, i is the i-th domain style, and N' is the total number of domain styles.

[0100] The adversarial domain loss is used to minimize the similarity between the second latent representation z0' and all previous domain styles in the memory bank , and the expression of the adversarial domain loss is:

[0101]

[0102] wherein, is the adversarial domain loss, i is the i-th domain style, and N' is the total number of domain styles, is the probability that the second latent representation belongs to the i-th domain style, z0' is the second latent representation, is the second text embedding representation feature.

[0103] The category consistency loss is used to maximize the semantic consistency between the second latent representation z0' and the k-th category intermediate feature with the category prompt , so as to ensure that the second latent representation z0' can maintain the category semantics after the adversarial domain loss is optimized, and the expression of the category consistency loss is:

[0104]

[0105] wherein, is the category consistency loss, for the probability that the second latent representation belongs to the k-th class, z0' is the second latent representation, for the first text embedding representation feature, for the k-th class intermediate feature with the class hint.

[0106] The expression of the adversarial domain hint tuning loss is:

[0107]

[0108] wherein, is the adversarial domain hint tuning loss, is the class consistency loss, is the adversarial domain loss, and λ is a target hyperparameter for keeping a trade-off between the updated domain style hint and its corresponding updated class hint.

[0109] Step E2: The adversarial domain hint tuning loss is back-propagated by using a back-propagation algorithm to obtain an updated domain style hint.

[0110] The above steps S1035 to S1037 can be referred to as domain style hint tuning, and the entire domain style hint tuning step realizes learning of the domain style hint by using a progressive adversarial learning strategy, ensuring that the updated domain style hint is different from the domain style stored in the memory bank .

[0111] Step S1038: The updated domain style hint is stored in the memory bank to obtain a trained domain generalization model.

[0112] By storing the updated domain style hint in the memory bank , the memory bank is updated. The domain style hint is obtained in each training round until the preset training round is reached, and the iteration is stopped at this time. At this time, the updated memory bank stores the domain style hint obtained in each round, that is, the updated memory bank includes the domain style hint of the last round and the updated domain style hint of the new round. The preset training round can be 50 times.

[0113] The expression of the domain style hint finally stored in the memory bank is: wherein, is the updated memory bank, is the last memory bank, which is used to store the domain style hint of the last round, is the updated domain style hint of the new round.

[0114] When the updated domain style hint is determined, the updated domain style hint The domain style prompt added before, that is, the i-th domain style prompt The distance in the memory bank. In this way, the updated domain style prompt is different from the domain style prompt added before in distribution, and a more challenging abstract domain style can be learned.

[0115] The updated domain style prompt is stored in the memory bank until a preset training round is reached, and a trained domain generalization model can be obtained.

[0116] S104, input the single-source domain data set into the trained domain generalization model to make the trained domain generalization model perform single-source domain generalization, and output a target image.

[0117] Wherein, the target image is an image with different types and different domain styles.

[0118] The simulation experiment of the application uses a batch size of 64 and a learning rate of 0.001 for training for 50 training rounds. The Stochastic Gradient Descent (SGD) optimizer is used, and the training is performed on a single NVIDIA RTX3090 GPU. The total number of domain styles N' of the application is set to 5, and the target hyperparameter λ is set to 10. In order to ensure the reliability of the adversarial fine-tuning and generation method for single-source domain generalization of the application, all simulation experiments are repeated five times independently, and the average result of the five simulation experiment results is taken. The Visual Learning Challenge (VLCS) dataset commonly used in domain generalization tasks is used for evaluation, which includes 4 domain styles, 10,729 samples, and 5 categories. The 4 domain styles are natural pictures, living objects, animals and vehicles, and the 5 categories are birds, cars, chairs, dogs and humans. Table 1 is the average result of the adversarial fine-tuning and generation method for single-source domain generalization of the application and existing methods, including the Augmix method, the Error Rate Minimizatio (ERM) method, the Parametric Adaptive Instance Normalization (pAdaIn) method, the MixStyle method, the Energy-based Feature Distribution Mixing (MixStyle) method, the Database Storage Unit (DSU) method, the cross-view feature distribution mixing method based on attention, and the cross-view feature distribution mixing method based on attention. Figure 1The Attention-based Cross-View Consistency (ACVC) method and the Mean Absolute Deviation (MAD) method. Column 1 in Table 1 is the method, and column 2 is the venue, which includes the International Conference on Learning Representations 2020 (ICLR'20), the International Conference on Learning Representations 2021 (ICLR'21), the IEEE Conference on Computer Vision and Pattern Recognition 2021 (CVPR'21), the IEEE Conference on Computer Vision and Pattern Recognition 2022 (CVPR'22), the International Conference on Learning Representations 2022 (ICLR'22), the Computer Vision and Pattern Recognition 2022 Workshops (CVPRW*22), and the IEEE Conference on Computer Vision and Pattern Recognition 2023 (CVPR'23). As can be seen from the average results (Avg) of the classification accuracy of the test data of the 5 simulation experiments in column 7 of Table 1, the average results of the adversarial fine-tuning and generation method for single-source domain generalization of the present application are the best, and better domain generalization can be achieved.

[0119] Table 1 Average results of classification accuracy of test data of simulation experiments

[0120] Method Venue Natural pictures Living things Animals Transportation Avg (%) Augmix ICLR'20 75.25 59.52 45.90 57.43 59.53 ERM ICLR'21 76.72 58.86 44.95 57.71 59.56 pAdaIn CVPR'21 76.03 65.21 43.17 57.94 60.59 Mixstyle ICLR'21 75.73 61.29 44.66 56.57 59.56 EFDMix CVPR'22 72.35 61.41 52.34 63.28 62.33 DSU ICLR'22 76.93 69.20 46.54 58.36 62.76 ACVC CVPRW*22 76.15 61.23 47.43 60.18 61.25 MAD CVPR'23 76.15 69.36 48.04 61.74 63.82 Invention 73.16 74.69 69.66 75.96 73.37

[0121] Based on the above Figure 1As can be seen from the implementation mode, the method for single-source domain generalization of the adversarial fine-tuning and generation of the embodiment of the application obtains a data set for a domain generalization task, the data set includes multiple sub data sets, the domain styles of each sub data set are different and the number of categories is the same; a domain generalization model is constructed, the domain generalization model includes a text encoder, a self-encoder and a conditional denoising module connected in sequence, the text encoder is used to add category prompts and domain style prompts to the multiple sub data sets; the multiple sub data sets are input into the domain generalization model, so as to train the domain generalization model by using a progressive adversarial learning strategy, and a trained domain generalization model is obtained; the single-source domain data set is input into the trained domain generalization model, so that the trained domain generalization model performs single-source domain generalization, and outputs a target image, the target image is an image with different types and different domain styles. In this way, through the text encoder of the domain generalization model, the added category prompts and domain style prompts can be learned, an abstract prompt can be learned, so that when the target domain features are too abstract and difficult to be accurately described by language, an image sample of the corresponding domain can be generated, through the added category prompts and domain style prompts, multiple categories and domain styles can be learned, so that the generation can perform single-source domain generalization on the single-source domain data set, and diversified images of multiple categories and multiple domain styles can be generated.

[0122] Based on the same inventive concept, as an implementation of the above-mentioned method for single-source domain generalization of the adversarial fine-tuning and generation, the embodiment of the application further provides a device for single-source domain generalization of the adversarial fine-tuning and generation. Figure 3 The structural diagram of the device for single-source domain generalization of the adversarial fine-tuning and generation in the embodiment of the application is shown in Figure 3 The device for single-source domain generalization of the adversarial fine-tuning and generation can include:

[0123] The obtaining module 301 is configured to obtain a data set for a domain generalization task, the data set including multiple sub data sets, the domain styles of each sub data set being different and the number of categories being the same;

[0124] The constructing module 302 is configured to construct a domain generalization model, the domain generalization model including a text encoder, a self-encoder and a conditional denoising module connected in sequence, the text encoder being used to add category prompts and domain style prompts to the multiple sub data sets;

[0125] The training module 303 is configured to input the multiple sub data sets into the domain generalization model, so as to train the domain generalization model by using a progressive adversarial learning strategy, and obtain a trained domain generalization model;

[0126] The single-source domain generalization module 304 is configured to input the single-source domain dataset into the trained domain generalization model, so that the trained domain generalization model performs single-source domain generalization, and outputs a target image. The target image is an image with different types and different domain styles.

[0127] It should be noted that the above description of the embodiment of the device for single-source domain generalization and generation is similar to the above description of the embodiment of the method for single-source domain generalization and generation. The device for single-source domain generalization and generation has similar beneficial effects to the method for single-source domain generalization and generation. For technical details of the embodiment of the device for single-source domain generalization and generation of the present application that are not disclosed, please refer to the description of the embodiment of the method for single-source domain generalization and generation of the present application.

[0128] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for adversarial fine-tuning and generation for single-source domain generalization, characterized by: include: Obtain a dataset for a domain generalization task, wherein the dataset includes multiple sub-datasets, each sub-dataset has a different domain style and the same number of categories; Constructing a domain generalization model, the domain generalization model comprising a text encoder, an autoencoder, and a conditional denoising module connected in sequence, the text encoder being used to add category hints and domain style hints to the multiple sub-datasets; Inputting the multiple sub-datasets into the domain generalization model to train the domain generalization model using a progressive adversarial learning strategy to obtain a trained domain generalization model; Inputting a single-source domain dataset into the trained domain generalization model, so that the trained domain generalization model performs single-source domain generalization and outputs a target image, wherein the target image is an image of different types and different domain styles; The plurality of sub-data sets each include a plurality of preset texts and a plurality of preset images corresponding to the plurality of preset texts, wherein the plurality of preset texts are used to express a category name; Inputting the multiple sub-datasets into the domain generalization model to train the domain generalization model using a progressive adversarial learning strategy to obtain a trained domain generalization model includes: Inputting the plurality of preset texts into the text encoder, and using the text encoder to add the category prompt to the plurality of preset texts, so that the text encoder outputs a first text embedding representation feature; Inputting the plurality of preset images into the autoencoder so that the autoencoder outputs a first latent representation; Inputting the first text embedding representation feature and the first latent representation into the conditional denoising module so that the conditional denoising module outputs predicted noise; Determining updated category cues using a back propagation algorithm based on the predicted noise and the Gaussian noise; Inputting the updated category hint into the text encoder, so as to add the domain style hint to the plurality of preset texts by using the text encoder, so that the text encoder outputs a second text embedding representation feature; Inputting the second text embedding representation feature into the conditional denoising module, so as to denoise the randomly generated noise using the conditional denoising module and output a second latent representation; Determining updated domain style hints based on the second latent representation and the second text embedding representation features using a back-propagation algorithm; The updated domain style hint is stored in a memory bank to obtain the trained domain generalization model.

2. The adversarial fine-tuning and generation method for single-source domain generalization according to claim 1, characterized in that The step of inputting the plurality of preset texts into the text encoder to add category hints to the plurality of preset texts by using the text encoder, so that the text encoder outputs a first text embedding representation feature, includes: Using the first input character string to segment the plurality of preset texts to obtain a plurality of first word units; Converting each first word into a first embedding vector, wherein the first embedding vector includes a first placeholder; adding the category prompt at the first placeholder according to the embedding functions corresponding to the plurality of first word-grams to obtain an intermediate feature with the category prompt; The intermediate features with category hints are input into a feature extractor, so that the feature extractor outputs the first text embedding representation features.

3. The adversarial fine-tuning and generation method for single-source domain generalization according to claim 1, characterized in that Inputting the first text embedding representation feature and the first latent representation into the conditional denoising module so that the conditional denoising module outputs predicted noise includes: adding the Gaussian noise to the first latent representation to obtain a noisy first latent representation; The prediction noise is determined according to the noisy first latent representation, the first text embedding representation feature, and the training round.

4. The adversarial fine-tuning and generation method for single-source domain generalization according to claim 1, characterized in that Determining an updated category prompt based on the predicted noise and Gaussian noise using a back propagation algorithm includes: determining a latent space diffusion model loss based on the predicted noise and the Gaussian noise; The latent space diffusion model loss is back-propagated using the back-propagation algorithm to obtain the updated category hint.

5. The adversarial fine-tuning and generation method for single-source domain generalization according to claim 1, characterized in that Inputting the updated category hint into the text encoder to add the domain style hint to the plurality of preset texts by using the text encoder, so that the text encoder outputs a second text embedding representation feature, includes: Using a second input character string to segment the updated category prompt to obtain a plurality of second word units; Converting each second word into a second embedding vector, wherein the second embedding vector includes a second placeholder; adding the domain style hint at the second placeholder according to the embedding functions corresponding to the plurality of second word-grams to obtain an intermediate feature having the domain style hint, wherein the domain style hint is a hint initialized in a memory bank; The intermediate features with domain style hints are input into a feature extractor so that the feature extractor outputs the second text embedding representation features.

6. The adversarial fine-tuning and generation method for single-source domain generalization according to claim 1, characterized in that The step of determining an updated domain style hint based on the second latent representation and the second text embedding representation features by using a back-propagation algorithm includes: determining an adversarial domain hint tuning loss based on the second latent representation, the second text embedding representation features, and hyperparameters; The adversarial domain hint tuning loss is back-propagated using the back-propagation algorithm to obtain the updated domain style hint.

7. The adversarial fine-tuning and generation method for single-source domain generalization according to claim 2, characterized in that The expression of the intermediate feature with category prompt is: ； in, For the The intermediate features with category hints for categories, is the embedding function corresponding to the multiple first word units, For the The first word corresponding to the first word length of the first category, For the Category The first word corresponding to the first word length, For the Category The first word corresponding to the first word length, For the The first embedding vector corresponding to the first word length of the first category, For the Category The first embedding vector corresponding to the first word length, For the Category The first embedding vector corresponding to the first word length, For the The embedding function of the first embedding vector corresponding to the first word length of the first category, For the Category The embedding function of the first embedding vector corresponding to the first word length, For the Category The embedding function of the first embedding vector corresponding to the first word length, For the Category hints for each category.

8. The adversarial fine-tuning and generation method for single-source domain generalization according to claim 5, characterized in that The expression of the intermediate feature with domain style hints is: ； in, For the The intermediate features with domain style hints of domain style, is the embedding function corresponding to the plurality of second word-grams, For the The second word corresponding to the first second word length of the domain style, For the The style of the field The second word corresponding to the second word length, For the The style of the field The second word corresponding to the second word length, For the The second embedding vector corresponding to the first second word length of the domain style, For the The style of the field The second embedding vector corresponding to the second word length, For the The style of the field The second embedding vector corresponding to the second word length, For the The embedding function of the second embedding vector corresponding to the first second word length of the domain style, For the The style of the field The embedding function of the second embedding vector corresponding to the second word length, For the The style of the field The embedding function of the second embedding vector corresponding to the second word length, For the Field style prompts for each field style.

9. An adversarial fine-tuning and generation device for single-source domain generalization, characterized in that include: an acquisition module, configured to acquire a dataset for a domain generalization task, the dataset comprising a plurality of sub-datasets, each sub-dataset having a different domain style and the same number of categories, the plurality of sub-datasets each comprising a plurality of preset texts and a plurality of preset images corresponding to the plurality of preset texts, the plurality of preset texts being used to express category names; A construction module, configured to construct a domain generalization model, wherein the domain generalization model includes a text encoder, an autoencoder, and a conditional denoising module connected in sequence, wherein the text encoder is configured to add category hints and domain style hints to the multiple sub-datasets; A training module, configured to input the plurality of sub-datasets into the domain generalization model to train the domain generalization model using a progressive adversarial learning strategy to obtain a trained domain generalization model; a single-source domain generalization module, configured to input a single-source domain dataset into the trained domain generalization model, so that the trained domain generalization model performs single-source domain generalization and outputs a target image, wherein the target image is an image of different types and different domain styles; The training module is specifically configured to input the plurality of preset texts into the text encoder, so as to use the text encoder to add the category prompt to the plurality of preset texts, so that the text encoder outputs a first text embedding representation feature; input the plurality of preset images into the autoencoder, so as to use the autoencoder to output a first latent representation; input the first text embedding representation feature and the first latent representation into the conditional denoising module, so as to use the conditional denoising module to output predicted noise; determine an updated category prompt based on the predicted noise and Gaussian noise using a back propagation algorithm; input the updated category prompt into the text encoder, so as to use the text encoder to add the domain style prompt to the plurality of preset texts, so as to use the text encoder to output a second text embedding representation feature; Inputting the second text embedding representation feature into the conditional denoising module to denoise the randomly generated noise using the conditional denoising module and output a second latent representation; determining an updated domain style hint based on the second latent representation and the second text embedding representation feature using a backpropagation algorithm; The updated domain style hint is stored in a memory bank to obtain the trained domain generalization model.

Images