A few-shot image generation and hybrid domain transfer method based on a pre-trained image encoder
By optimizing the generator using a pre-trained image encoder and a directional subspace loss function, the problem of discriminator overfitting in mixed domain transfer is solved, generating high-quality, cross-domain consistent images with comprehensive properties.
Patent Information
- Application Number
- CN202311398718.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-26
- Publication Date
- 2025-12-02
- Estimated Expiration
- 2043-10-26
AI Technical Summary
Existing generative domain transfer methods lack real image samples in hybrid domain transfer, leading to the discriminator overfitting easily learned features and making it difficult to generate images with comprehensive attributes. Furthermore, traditional methods are deficient in terms of generation quality and cross-domain consistency.
A pre-trained image encoder is used to project images from different domains into independent embedding subspaces, and the generator is optimized by the directional subspace loss function to achieve mixed domain transfer and avoid the use of a discriminator.
It achieves the generation of high-quality, cross-domain consistent images with few samples, preserves source domain features, and generates images with comprehensive attributes in mixed domains, outperforming existing methods.
Smart Images
Figure CN117314784B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image generation in computer vision, and in particular relates to a few-shot image generation hybrid domain transfer method based on a pre-trained image encoder. Background Technology
[0002] Few-shot generative domain adaptation aims to adapt a pre-trained image generator from a source domain to a new target domain using a small number of reference images, such as adversarial generative models. Existing methods typically seek to achieve realistic and diverse generation, capturing salient features of the target domain while preserving variations learned from the source domain. For example, the 2021 paper "Few-shot image generation via cross-domain correspondence" published at the IEEE International Conference on Computer Vision proposed an instance distance consistency loss to maintain the distance between different instances in the source domain; and the 2022 paper "Few-shot generative model adaptation via relaxed spatial structural alignment" published at the top international computer vision conference IEEE Conference on Computer Vision and Pattern Recognition proposed a relaxed spatial structural alignment method to preserve the spatial structural information of the source domain.
[0003] Building on previous progress in generative domain transfer, training a transferable model from a small number of real images obtained from the sketching domain is straightforward. However, this problem remains unsolved when generating images with combined attributes of sketching, smiling, and baby characteristics is required, but only real images from each domain are available. Furthermore, in real-world scenarios, images from mixed domains are often more difficult to collect than those from a single domain (e.g., smiling babies with a sketching style). In such cases, traditional generative domain transfer methods become less feasible.
[0004] Compared to traditional generative domain migration, hybrid domain migration has significant problems in two aspects:
[0005] (1) Existing generative domain transfer methods typically use discriminators to determine whether the generated image belongs to the target domain. However, hybrid domain transfer only has images sampled from each domain and lacks real images of the hybrid domain, which poses a challenge to designing a discriminator-based transfer framework.
[0006] (2) Due to the scarcity of reference images, the discriminator is prone to overfitting to some easy-to-learn features in the target domain, resulting in the omission of features from other target domains in the generator. Summary of the Invention
[0007] To address the shortcomings of existing technologies, this invention proposes a mixed domain transfer method for few-shot image generation based on a pre-trained image encoder. This method utilizes a pre-trained image encoder to project images from different domains into independent embedding subspaces, and then reduces the generator's directional subspace loss function during training to complete the mixed domain transfer.
[0008] The specific technical solution of the present invention is as follows:
[0009] (1) Randomly initialize a Gaussian noise, and input the Gaussian noise into the pre-trained original generator and target generator respectively to obtain two images, which are used as the original domain generated image and the target domain generated image respectively;
[0010] The original generator is pre-trained based on the image data of the original domain and is not updated thereafter; the target generator is pre-trained based on the image data of the original domain, but it needs to be continuously updated thereafter.
[0011] The original domain refers to a set of images with a large amount of data, while the target domain refers to a set of images with a smaller amount of data.
[0012] (2) Input both images generated in step (1) into the pre-trained image encoder to obtain their respective embedding vectors;
[0013] (3) Input the real image set of the target domain into the pre-trained image encoder to obtain their respective embedding vectors. All the embedding vectors corresponding to the real image set of each domain form a linear plane [and the linear planes of different domains form separate linear planes].
[0014] (4) Use the linear planes obtained from different target domains in step (3) and the embedding vectors obtained in step (2) to process and obtain the directional subspace loss;
[0015] (5) Use the directional subspace loss to backpropagate the gradient and update the target generator;
[0016] (6) Repeat steps (1) to (5) above to train, update and optimize. After reaching the preset number of training times, end the training and finally use the trained target generator to generate an image and add it to the target domain.
[0017] The target domain's real image set is a few-sample real image set, which refers to a set of real images with a quantity of 10.
[0018] The specific process of step (1) is as follows: randomly initialize Gaussian noise z, and input the Gaussian noise z into the original generator G. S and target generator G T The original domain generated image Y is obtained. s Image Y generated from the target domain t .
[0019] In step (2), the original domain is used to generate the image Y. s Image Y generated from the target domain t The input is fed into an image encoder E pre-trained on a publicly known classification dataset to obtain the corresponding original domain embedding vector f and target domain embedding vector f. t .
[0020] In step (3), the few-sample real image set is input into the pre-trained image encoder to obtain their respective embedding vectors, which are derived from all real images Y in the i-th domain. i The corresponding embedding vectors form the linear plane P of the i-th domain. i The linear planes of all domains form separate linear planes.
[0021] Each domain has a set of real images, and each domain has a linear plane.
[0022] In step (4), the linear plane P corresponding to different target domains is obtained. i and the original domain embedding vector f s Target domain embedding vector f t The directional subspace loss function is calculated as follows:
[0023] (4-1) For all real images Y in each target domain i Each vector is input into a pre-trained image encoder to obtain its own embedding vector, which then forms the embedding vector set for the target domain.
[0024] (4-2) Perform SVD decomposition on the embedding vector set of each target domain to obtain the linear plane P corresponding to that target domain. i The projection matrix M i ;
[0025] (4-2) Using the projection matrix M i The target domain embedding vector f is calculated using the following formula. t To the linear plane P i The projection point f i :
[0026]
[0027] (4-3) Based on the projection points f of all target domains i Calculate the target domain embedding vector ft to projection point f i Distance L dist :
[0028] L dist =∑ i |f i -f t |
[0029] Where i represents the ordinal number of the field;
[0030] Simultaneously, based on the projection points f of all target domains i The directional distance L is calculated from the embedding vector obtained in step (2). direct :
[0031] L direct =1-cos(∑ i (f i -f t ),f t -f s )
[0032] (4-3) Based on distance L dist and distance L direct The directional subspace loss is obtained by summation:
[0033] L = L dist +L direct .
[0034] In step (5), the gradient descent method is used to update the target generator G based on minimizing the directional subspace loss L. T The parameters in the original generator remain unchanged.
[0035] In step (6), steps (1) to (5) are repeated until the preset number of training iterations are reached, at which point the training ends, and the trained model is used to generate an image.
[0036] In step (6), the image generated by the trained target generator and added to the target domain is specifically generated by processing the randomly generated noise using the trained target generator.
[0037] The domain mentioned in this invention refers to a data set with a unified, single feature, such as a set of image data with a unified feature of an infant; the hybrid domain refers to a data set with multiple features, such as a set of image data with infant features and sketch features.
[0038] In practice, the hybrid domain can consist of multiple target domains.
[0039] The method of this invention can be used to expand and migrate data when the sample size is small and the database size is relatively small.
[0040] Compared with the prior art, the present invention has the following beneficial effects:
[0041] 1. This invention proposes a novel discriminator-free framework that uses directional subspace loss for generative hybrid domain transfer. Compared to other methods, this invention achieves fast and general transfer without requiring model training on multiple independent domains.
[0042] 2. For both single-domain and mixed-domain transfers, this invention outperforms existing methods in terms of semantic similarity to the target domain, image quality, and cross-domain consistency. Qualitative and quantitative results demonstrate the effectiveness of this invention. Attached Figure Description
[0043] Figure 1 This is a schematic diagram illustrating the effect of hybrid domain migration in this invention;
[0044] Figure 2 This is the overall framework proposed in this invention;
[0045] Figure 3 This is a comparison chart of the qualitative effects of the present invention on mixed domains and existing methods;
[0046] Figure 4 This is a comparison chart showing the qualitative effects of the present invention on a single domain compared to existing methods. Detailed Implementation
[0047] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.
[0048] The specific embodiments and implementation process of the present invention are as follows:
[0049] (1) Randomly initialize a Gaussian noise, and input the Gaussian noise into the pre-trained original generator and target generator respectively to obtain two images, which are used as the original domain generated image and the target domain generated image respectively;
[0050] Specifically, Gaussian noise z is randomly initialized, [z∈R] C R C [Representing a real number field of dimension C,] Gaussian noise z is input into the original generator G. S and target generator G T The original domain generated image Y is obtained. s Image Y generated from the target domain t .
[0051] (2) Input both images generated in step (1) into the pre-trained image encoder to obtain their respective embedding vectors;
[0052] Specifically, the original domain is used to generate the image Y. s Image Y generated from the target domain t The input is fed into an image encoder E pre-trained on a publicly known classification dataset to obtain the corresponding original domain embedding vector f. s and target domain embedding vector f t .
[0053] (3) Input the real image set of the target domain into the pre-trained image encoder to obtain their respective embedding vectors. All the embedding vectors corresponding to the real image set of each domain form a linear plane.
[0054] The few-shot set of real images is input into the pre-trained image encoder to obtain their respective embedding vectors, which are derived from all the real images Y in the i-th domain. i The corresponding embedding vectors form the linear plane P of the i-th domain. i .
[0055] (4) Use the linear planes obtained from different target domains in step (3) and the embedding vectors obtained in step (2) to process and obtain the directional subspace loss;
[0056] In step (4), the linear plane P corresponding to different target domains is obtained. i and the original domain embedding vector f s Target domain embedding vector f t The directional subspace loss function is calculated as follows:
[0057] (4-1) For all real images Y in each target domain i Each vector is input into a pre-trained image encoder to obtain its own embedding vector, which then forms the embedding vector set for the target domain.
[0058] (4-2) Perform SVD decomposition on the embedding vector set of each target domain to obtain the linear plane P corresponding to that target domain. i The projection matrix M i ;
[0059] (4-2) Using the projection matrix M i The target domain embedding vector f is calculated using the following formula. t To the linear plane P i The projection point f i :
[0060]
[0061] (4-3) Based on the projection points f of all target domains iCalculate the target domain embedding vector f t to projection point f i Distance L dist :
[0062] L dist =∑ i |f i -f t |
[0063] Where i represents the ordinal number of the field;
[0064] Simultaneously, based on the projection points f of all target domains i The directional distance L is calculated from the embedding vector obtained in step (2). direct :
[0065] L direct =1-cos(∑ i (f i -f t ),f t -f s )
[0066] (4-3) Based on distance L dist and distance L direct The directional subspace loss is obtained by summation:
[0067] L = L dist +L dirt .
[0068] (5) Use the directional subspace loss to backpropagate the gradient and update the target generator;
[0069] Based on minimizing the directional subspace loss L, gradient descent is used to update the target generator G. T The parameters in the original generator remain unchanged.
[0070] (6) Repeat steps (1) to (5) above to train, update and optimize. After reaching the preset number of training times, end the training. Finally, use the trained target generator to generate an image and add it to the target domain. Specifically, the trained target generator is used to process the randomly generated noise and generate an image to be added to the target domain.
[0071] like Figure 1The diagram illustrates the hybrid domain transfer effect achieved by this invention. The middle section shows how, given an original generator pre-trained on a large-scale dataset, this invention proposes transferring it to a new target domain. The left side shows how traditional domain transfer tasks transfer the generator from the source domain (sketched portrait) to the target domain. The transferred model uses a very small number of reference samples to capture the target distribution. The right side shows the hybrid domain transfer achieved by this invention, where, given a very small number of reference samples from multiple independent domains, the target generator can generate images that possess attributes of sketching, smiling, and being a baby.
[0072] like Figure 2 As shown, Y s and Y t These are images generated by a fixed original generator and a target generator during training, respectively. A fixed image encoder extracts features to construct the subspace P. i Next, the target embedding vector f is... t Projected onto subspace P i Obtain the projection point f on i Then minimize the target embedding vector f. t and projection point f i The distance between them. Simultaneously, minimize f. i -f t and f t -f s The angle between them, to ensure the target embedding vector f t Along the projection point f s To subspace P i The vertical line moves.
[0073] like Figure 3 As shown, the qualitative results of this invention in mixed-domain transfer surpass those of existing methods. Specifically, simple sequential learning methods suffer from catastrophic forgetting problems, tending to generate images of the last adapted domain instead of the desired image. DoRM methods are unstable in generating mixed-domain images because simple inference-time interpolation latent codes are insufficient to semantically align multiple domains. NADA methods can generate images with integrated properties to some extent, but they exhibit significant biases from CLIP models. For example, the sketch domain deviates stylistically from the reference image and fails to learn the sunglasses domain. In contrast, the results of this invention demonstrate its stable ability to generate mixed-domain target images while preserving source domain features.
[0074] like Figure 4As shown, the qualitative results of this invention in mixed-domain transfer surpass those of existing methods. Specifically, previous methods (such as CDC) severely overfit a small number of reference images, failing to preserve source domain attributes well, while generating relatively poor image quality. Although the NADA method exhibits cross-domain consistency to some extent, it shows significant bias from the CLIP encoder. In contrast, this invention can accurately acquire the visual attributes of the target domain while maintaining strong generation quality and cross-domain consistency.
[0075] The method proposed in this invention will be applied to a practical example below, and compared with other similar methods to demonstrate the technical effects and advantages of this invention.
[0076] This invention was tested on publicly available datasets. The Flickr-Faces-HQ face dataset was selected as the source domain, and the following combinations of few-sample data were used as the target domain: (a) FFHQ-baby (b) sketch (c) FFHQ-smile (d) FFHQ-sunglasses. Consistent with previous methods, 10 images were randomly sampled for each target domain, with both the source and target domains having a resolution of 256.
[0077] This invention employs the following performance evaluation metrics: (a) CLIP score, used to measure the compatibility of image-text pairs, can be considered as semantic similarity to the target domain. (b) Inception Score (IS), used to evaluate image quality, is more practical in few-sample settings. (c) Recognition Similarity (ID), used to measure cross-domain consistency, evaluates the consistency of images in terms of domain invariance such as pose and identity.
[0078] As shown in Table 1, for mixed domain transfer, this invention achieved the best scores across all metrics. Particularly in the CLIP score, this invention significantly outperformed previous methods, indicating that the generated image effectively integrates multiple attributes from different domains. Furthermore, this invention achieved higher IS and ID scores, suggesting higher image quality and preservation of more features from the source domain.
[0079] Table 1
[0080]
[0081] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A few-shot image generation and hybrid domain transfer method based on a pre-trained image encoder, characterized in that: The methods include: (1) Randomly initialize a Gaussian noise, and input the Gaussian noise into the pre-trained original generator and target generator respectively to obtain two images, which are used as the original domain generated image and the target domain generated image respectively; (2) Input both images generated in step (1) into the pre-trained image encoder to obtain their respective embedding vectors; (3) Input the real image set of the target domain into the pre-trained image encoder to obtain their respective embedding vectors. All the embedding vectors corresponding to the real image set of each domain form a linear plane. (4) Use the linear planes obtained from different target domains in step (3) and the embedding vectors obtained in step (2) to process and obtain the directional subspace loss; (5) Use the directional subspace loss to backpropagate the gradient and update the target generator; (6) Repeat steps (1) to (5) above to train, update and optimize. After reaching the preset number of training times, end the training and finally use the trained target generator to generate an image and add it to the target domain.
2. The method for generating mixed domain transfer from few-shot images based on a pre-trained image encoder according to claim 1, characterized in that: The specific process of step (1) is as follows: randomly initialize Gaussian noise z, and input the Gaussian noise z into the original generator G. S and target generator G t The original domain generated image Y is obtained. s Image Y generated from the target domain t .
3. The method for generating mixed domain transfer from few-shot images based on a pre-trained image encoder according to claim 1, characterized in that: In step (2), the original domain is used to generate the image Y. s Image Y generated from the target domain t The input is fed into an image encoder E pre-trained on a classification dataset to obtain the corresponding original domain embedding vector f. s and target domain embedding vector f t .
4. The method for generating mixed domain transfer from few-shot images based on a pre-trained image encoder according to claim 1, characterized in that: In step (3), the few-sample real image set is input into the pre-trained image encoder to obtain their respective embedding vectors, which are derived from all real images Y in the i-th domain. i The corresponding embedding vectors form the linear plane P of the i-th domain. i .
5. The method for generating mixed domain transfer from few-shot images based on a pre-trained image encoder according to claim 1, characterized in that: In step (4), the linear plane P corresponding to different target domains is obtained. i and the original domain embedding vector f s Target domain embedding vector f t The directional subspace loss function is calculated as follows: (4-1) For all real images Y in each target domain i Each vector is input into a pre-trained image encoder to obtain its own embedding vector, which then forms the embedding vector set for the target domain. (4-2) Perform SVD decomposition on the embedding vector set of each target domain to obtain the linear plane P corresponding to that target domain. i The projection matrix M i ; (4-2) Using the projection matrix M i The target domain embedding vector f is calculated using the following formula. t To the linear plane P i The projection point f i : (4-3) Based on the projection points f of all target domains i Calculate the target domain embedding vector f t to projection point f i Distance L dist : L dist =∑ i |f i -f t | Where i represents the ordinal number of the field; Simultaneously, based on the projection points f of all target domains i The directional distance L is calculated from the embedding vector obtained in step (2). direct : L direct =1-cos(∑ i (f i -f t ),f t -f s ) (4-3) Based on distance L dist and distance L direct The directional subspace loss is obtained by summation: L=L dist +L direct 。 6. The method for few-shot image generation and hybrid domain transfer based on a pre-trained image encoder according to claim 1, characterized in that: In step (5), the gradient descent method is used to update the target generator G based on minimizing the directional subspace loss L. T The parameters in.
7. The method for generating mixed domain transfer from few-shot images based on a pre-trained image encoder according to claim 1, characterized in that: In step (6), the image generated by the trained target generator and added to the target domain is specifically generated by processing the randomly generated noise using the trained target generator.