Method and system for generating 2D image under limited sample
Through the codebook initialization method enhanced by intermediate latent spatial quantization technology and knowledge, the pattern crash and overfitting problems of GAN models under finite samples are solved, and high-quality image generation under data scarcity is achieved.
Patent Information
- Application Number
- CN202510277793.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-10
- Publication Date
- 2025-06-27
AI Technical Summary
In the fields of medical care, justice, etc., due to the high cost of data acquisition, privacy protection or scarcity of special scenarios, the training samples are extremely limited, resulting in GAN models being prone to pattern crashes, overfitting and other problems.
Using intermediate latent space quantization technology, by segmenting and quantizing intermediate variables in style space into learnable codebooks, a compact proxy space is built, the consistency regularization of discriminators is optimized, and a knowledge-enhanced codebook initialization method is introduced to capture the inherent semantic structure of training data.
Under the condition of limited data, high-quality target domain images are generated at low cost and quickly, avoiding pattern crashes and overfitting, and improving the training stability and generation quality of the generative adversarial network.
Smart Images

Figure CN120219537A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image processing, and particularly relates to a 2D image generation method and system under limited samples. Background Art
[0002] Generative models have made remarkable progress in creating human-like natural language, high-quality images, and diverse human speech and music. The key factor behind these achievements is the need for a large amount of training data to capture the distribution of real data. The traditional GAN framework highly depends on large-scale high-quality training data, and its training stability and generation quality are often positively correlated with the number of samples. However, due to issues related to themes, privacy, or copyright, data collection can be challenging. Especially in practical application scenarios such as medical image analysis and industrial defect detection, researchers often face the dilemma of severely insufficient training samples (usually less than 1000 images) due to data acquisition costs, privacy protection, or the scarcity of special scenarios. Under the condition of small samples, GAN training is prone to unstable states such as mode collapse and vanishing gradients. At the same time, the discriminator is prone to overfitting, resulting in the generator being unable to effectively learn the data distribution characteristics. This contradiction has given rise to the need for in-depth research on GAN training techniques under limited samples.
[0003] Facing the bottleneck of data scarcity, researchers have proposed multiple solutions. A core problem they aim to solve is the overfitting problem of the discriminator during training with limited data. The most direct solution is data augmentation, including traditional, differentiable, and adaptive methods to augment real or generated data. However, these methods require customized manual design for different datasets, lack scalability, and the types of augmentation methods are also limited to a finite range. In addition, augmentation may distort the semantics of the original image, resulting in a decline in generation quality. As a supplement, model regularization techniques address the discriminator degradation problem by enhancing the robustness of the discriminator representation and stabilizing the learning dynamics during training. Among these techniques, the robust representation of the discriminator is crucial. Consistency regularization (CR) methods provide an alternative perspective, encouraging the discriminator to capture more robust features. These methods introduce an auxiliary term that forces the discriminator to generate consistent scores for input images (including generated and real ones) and their augmented versions. In particular, samples are generated from two adjacent latent codes z and their perturbed versions z+∈ (∈ represents the additional Gaussian noise added, the so-called "perturbation" operation), and they are used as implicit augmentations in the prior space Z to enforce consistency constraints. This strategy circumvents the limitations of manual augmentation method design and has achieved impressive results. However, effectively exploring the Z space under limited data conditions is very challenging. This deficiency may lead to significant differences in the authenticity of the generated images (generated from z and z+∈). If the discriminator is forced to evaluate their consistency, it may weaken its effectiveness. Summary of the Invention
[0004] In view of the above deficiencies in the prior art, a 2D image generation method and system under limited samples provided by the present invention solve the problems that in fields such as medical and judicial, due to privacy, copyright, or acquisition cost limitations, the available data volume is extremely limited (usually less than a thousand cases), resulting in problems such as mode collapse and overfitting of the model.
[0005] To achieve the above objectives, the technical solution adopted by the present invention is: a 2D image generation method under limited samples, including the following steps:
[0006] S1. Obtain a limited dataset in the target domain and perform privacy desensitization processing;
[0007] S2. Preprocess the limited dataset in the target domain after privacy desensitization processing;
[0008] S3. Initialize an unpre-trained StyleGAN generator, construct a native latent space and a quantization codebook, where the encodings in the codebook represent different 2D image features;
[0009] S4. Initialize the codebook by referring to external knowledge based on the native latent space, and train the generator network of the target domain;
[0010] S5. Input the randomly sampled noise into the trained generator network to obtain the data of the target domain, and complete the generation of 2D images, where the data of the target domain includes 2D images.
[0011] The beneficial effects of the present invention are as follows: The present invention proposes an intermediate latent space quantization technique, aiming to optimize the consistency regularization of the discriminator, enhance the robustness of its feature extraction, and alleviate the phenomenon of training degradation of the generative adversarial network (GAN) in the limited sample scenario. In addition, the present invention introduces a novel codebook initialization method, which is crucial for capturing the intrinsic semantic structure of the training data. The present invention can generate a large number of high-quality target domain images at low cost and quickly (the inference time for a single image is less than 1 s) under the condition of limited data. In data-scarce scenarios such as medical image reconstruction and cultural heritage restoration, traditional GAN often results in unstable generation quality due to insufficient exploration of the latent space, and the images generated by adjacent latent variables are significantly different. Although existing methods alleviate overfitting through data augmentation or regularization, they still face problems such as semantic distortion and limited generation diversity. The present invention proposes to reconstruct the latent expression of GAN through style space quantization to provide a structured solution for small-sample generation tasks.
[0012] Further, the S3 includes the following steps:
[0013] S301. Initialize the untrained StyleGAN generator;
[0014] S302. In the architecture of the initialized StyleGAN generator, randomly sample Gaussian noise from the Gaussian distribution N(0,1) and map it through a non-linear mapping function to the intermediate variable w in the native style space W, complete the construction of the native latent space, and split each intermediate variable w into s sub-vectors so as to match the encoding in the codebook, represents the set of real numbers, f w represents the mapping network, d z represents the dimension of the Gaussian noise z, d w represents the dimension of the mapped intermediate variable w ∈ W;
[0015] S303. Based on the splitting result, correspond the subset of the latent variables in the native style space W to the entries of the codebook, and complete the quantization of the style space features with the help of the codebook.
[0016] Still further, the S302 is specifically:
[0017] A1. In the architecture of the initialized StyleGAN generator, Gaussian noise is randomly sampled from the Gaussian distribution N(0, 1).
[0018] A2. Through a non - linear mapping function, the sampled Gaussian noise is converted into an intermediate variable w in the style space W, and each intermediate variable w is split into s sub - vectors.
[0019] A3. Each sub - vector is quantized to the closest entry in the learnable codebook C to obtain the quantized sub - vector. And the sub - vectors are concatenated to obtain the latent code w. q , completing the conversion of the proxy space. Among them, the quantized encoding after codebook matching has the following expression:
[0020]
[0021] The beneficial effect of the above - mentioned further solution is that through the above design, the present invention constructs a compact, structured and discrete proxy space, which can convert the style vector into a more representative quantized encoding.
[0022] Furthermore, the loss function of the generation network in the target domain includes the loss function of the generator and the loss function of the discriminator:
[0023] The expression of the loss function of the generator is as follows:
[0024] L(g,f w ,C,P)=L adv (g)+λ sq L sq (f w ,C)+λ uf L uf (C,P)
[0025] The expression of the loss function of the discriminator is as follows:
[0026] L(f D )=L adv (f D )+λ qcr L qcr (f D )
[0027] Where:
[0028]
[0029] L qcr (f D )=E[||fD (g(w q )) - f D (g(w' q )) || 2 )
[0030] L adv (g) = E[1 - log f D (g(z))]
[0031] L adv (f D ) = -E[log f D (x)] - E[log(1 - fD(g(z)))]
[0032] Among them, L(g, f w , C, P) represents the loss function of the generator, g represents the generator, f w represents the mapping network, C represents the codebook, P represents the learnable projection matrix, L adv (g) and L adv (f D ) both represent adversarial losses, λ sq , λ uf and λ qcr all represent hyperparameters, L sq (f w , C) represents the style quantization loss function, L uf (C, P) represents the uniformity constraint, L(f D ) represents the loss function of the discriminator, f D represents the discriminator, z represents Gaussian noise, x represents the real sample, L qcr (f D ) represents the quantization-based consistency regularization constraint, f D represents the discriminator network, sg() represents the stop gradient operator, represents the s sub-vectors obtained by splitting each intermediate variable w, c represents the entry in the codebook, β represents the hyperparameter for balancing the two values in the style quantization loss, w represents the intermediate variable in the style space W, E(·) represents the mathematical expectation, t represents a fixed scale scaling hyperparameter, c i and c j both represent the entries in the codebook, w q represents the intermediate vector formed by splicing the quantized sub-vectors , w' q represents the intermediate variable obtained by perturbing, mapping, and quantizing the sampled noise.
[0033] The beneficial effects of the above further solution are as follows: With the help of the loss function, the present invention completes the training of each component of the model, and with the help of the constructed proxy space, it can better adapt to the limited data set, and achieve more refined control of the feature space and capture more detailed information.
[0034] Furthermore, the specific content of S4 is as follows:
[0035] B1. Based on the native latent space, using the semantic knowledge embedded in the base model, perform encoding operations on the 2D images in the limited target domain data set after preprocessing and the encodings in the codebook respectively, and map them to the 2D image feature space and the encoding feature space one by one respectively to initialize the codebook;
[0036] B2. Use the initialized codebook to train the generation network of the target domain.
[0037] The beneficial effects of the above further solution are as follows: The present invention aligns the codes in the codebook with the features extracted from the training data, thereby embedding external knowledge into the codebook, which helps to accelerate the model convergence and promote the model to learn more meaningful image features.
[0038] Furthermore, the specific content of the encoding operation on the 2D images in the limited target domain data set and the encodings in the codebook is as follows:
[0039] C1. Given several real 2D images from the limited target domain data set;
[0040] C2. Use the CLIP visual encoder φ of the deep learning model to extract the feature f of each 2D image i ;
[0041] C3. According to the feature f of each 2D image extracted i , obtain a set of 2D image features F;
[0042] C4. Use the fully connected layer to convert the quantized quantization code into a variable with the same embedding dimension as the input token;
[0043] C5. According to the variable after being converted by C4, use the CLIP text encoder of the deep learning model to process it to obtain the encoded embedding feature T;
[0044] C6. According to the marginal distribution p of the 2D image feature F and the marginal distribution q of the encoded embedding feature T, as well as the cost matrix, obtain the optimal transport plan γ* by solving the optimal transport formula;
[0045] C7. According to the optimal transport plan γ*, define the optimal transport loss of the 2D image feature F and the encoded embedding feature T to achieve feature alignment and complete the encoding operation.
[0046] The beneficial effects of the above further solution are as follows: By means of the optimal transport technology, the present invention aligns the codebook features and the image features of the base model.
[0047] Furthermore, the optimal transport formula is as follows:
[0048]
[0049] Subject to:
[0050]
[0051] where D(p, q|C') represents the optimal transport distance, n represents the size of the support set of the marginal distribution p, m represents the size of the support set of the marginal distribution q, γ represents the transport matrix, represents the set of real numbers, C' represents the cost matrix, i represents the index variable of the marginal distribution p, j represents the index variable of the marginal distribution q, γ ij represents the probability of transferring from the i-th point of the marginal distribution p to the j-th point of the marginal distribution q, c represents the cost function, f i x represents the feature vector corresponding to the index i in the marginal distribution p, represents the feature vector corresponding to the index j in the marginal distribution q, s represents the number of sub-vectors into which each intermediate vector w is divided, l represents the sequence length of the image tokens, p i represents the probability value of the i-th point, and each p i represents the probability starting from a point in the marginal distribution p, q j represents the probability value of the j-th point, and each q j represents the probability starting from a point in the marginal distribution q;
[0052] The expression of the cost matrix is as follows:
[0053] C j,k = c(j, k):= d(t .j , f .k ) 1 ≤ j ≤ s, 1 ≤ k ≤ l
[0054] where C j,k represents the cost matrix, c(j, k): represents the element in the cost matrix, d(·, ·) represents the distance function for measuring the transport cost between t .j and f .k , t .j represents the j-th component of all features in the encoded embedding feature set T, f .kThe k-th component of all features in the feature set F, j represents the index of the features in the visual feature set F, s represents the number of sub-vectors corresponding to each intermediate vector w, and k represents the index of the features in the feature set.
[0055] The beneficial effect of the above further solution is that: by solving the optimal transport problem, the present invention obtains an optimal transport scheme for aligning the codebook features and the real data.
[0056] Furthermore, the expression of the optimal transport loss is as follows:
[0057] L ot (f w , C) = E[d(T, F)·γ *
[0058] where, L ot represents the optimal transport loss, f w represents the mapping network, C represents the codebook, E(·) represents the mathematical expectation, d(·) represents the distance function for measuring the transport cost between the feature sets T and F, F represents the visual feature set obtained by encoding the image, and T represents the encoded embedding feature set obtained by encoding the quantized discrete intermediate variables.
[0059] The beneficial effect of the above further solution is that: the present invention defines the optimal transport loss between two feature sets F and T, and optimizes this loss function to achieve feature alignment.
[0060] Furthermore, the expression of the loss function for initializing the codebook is as follows:
[0061] L(f w , C, P) = L sq (f w , C) + L uf (C, P) + λ ot ·L ot (f w , C)
[0062] where, L(f w , C, P) represents the loss function for initializing the codebook, L sq represents the style quantization loss in the feature extraction stage, L uf represents the uniform constraint loss, λ ot represents the weight coefficient for balancing various losses, L ot represents the optimal transport loss, f w represents the mapping network, C represents the codebook, and P represents the projection matrix.
[0063] The beneficial effects of the above further solution are as follows: The present invention utilizes basic models such as the deep learning model CLIP to extract image semantic features, aligns the codebook encoding with real data through the Optimal Transport distance, and constructs a "visual dictionary" rich in semantics.
[0064] The present invention also provides a 2D image generation system under limited samples. The 2D image generation system is used for a 2D image generation method, including:
[0065] A first processing module, configured to obtain a limited data set in the target domain and perform privacy desensitization processing;
[0066] A second processing module, configured to preprocess the limited data set in the target domain after privacy desensitization processing;
[0067] A third processing module, configured to initialize an untrained StyleGAN generator, construct a native latent space and a quantization codebook, wherein the encodings in the codebook represent different 2D image features;
[0068] A fourth processing module, configured to initialize the codebook by referring to external knowledge and train the generation network in the target domain;
[0069] A fifth processing module, configured to input randomly sampled noise into the trained generation network to obtain data in the target domain, and complete the generation of 2D images, wherein the data in the target domain includes 2D images.
[0070] The beneficial effects of the present invention:
[0071] (1) The present invention proposes a strategy to enhance GAN training under limited data by introducing a novel style space quantization method. A more compact proxy space is constructed using the codebook, and the latent space is quantized by improving discriminator consistency regularization (CR) to capture more robust features under limited data conditions.
[0072] (2) The present invention initializes the codebook features by introducing external knowledge of the base model, and pre - establishes a semantic - rich vocabulary for the codebook. Using the optimal transport distance to align its codebook features with the image features in the target domain is the key technology to inject the knowledge of the base model into the codebook. Description of the Drawings
[0073] Figure 1 It is a flowchart of the method of the present invention.
[0074] Figure 2 It is a schematic structural diagram of the system of the present invention. Detailed Embodiments
[0075] The following describes the specific embodiments of the present invention to facilitate those skilled in the art of this technology to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art of this technology, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions created using the concept of the present invention are within the scope of protection.
[0076] Before explaining the present invention, the following abbreviations and key terms are first defined and explained:
[0077] StyleGAN (Style Generative Adversarial Network): An advanced deep learning model dedicated to generating high-quality and high-resolution images. By introducing the concept of style transfer and an improved generative adversarial network architecture, it can control the image generation process at different levels and achieve fine-tuning of image style and content. StyleGAN has demonstrated excellent generative capabilities and flexibility in fields such as face generation, artistic creation, and data augmentation.
[0078] Discrete Representation Learning: A machine learning method that transforms data into finite discrete symbols (such as tokens or categorical variables) to extract key features. The generated representations are interpretable and structured, suitable for tasks that require clear boundaries or symbolic reasoning. Typical methods include vector quantization techniques, such as VQ-VAE, which enhance the model's generalization and interpretability through a discrete latent space.
[0079] VQ (Vector Quantization): A data compression and representation method widely used in fields such as signal processing, image processing, and natural language processing. VQ reduces the complexity of data by mapping continuous input data to a discrete set of vectors. Specifically, VQ divides the data into multiple small vectors and maps each vector to the closest discrete code (i.e., a vector in the dictionary) through a lookup table. This method can effectively reduce storage space and improve efficiency in information transmission and data compression. In image generation and neural network training, VQ is often used to maintain an efficient feature representation of images through low-dimensional representations.
[0080] Codebook Learning: A machine learning method aimed at quantifying continuous features into discrete representations by learning a codebook containing discrete codes, thereby improving the interpretability and sparsity of the model. This method introduces a vector quantization bottleneck in each layer of the neural network to map continuous features to discrete codes in a larger codebook, making the hidden states sparse and discrete.
[0081] Transformer: A deep learning architecture that realizes global dependency modeling of sequential data based on the self-attention mechanism, abandons the traditional recurrent neural network structure, and forms the core module by stacking multi-head attention layers and feed-forward networks. It has the advantage of parallel computing and is good at capturing long-range semantic associations. It innovatively decouples position encoding and content encoding, and demonstrates strong context modeling capabilities in cross-modal tasks such as machine translation and image generation.
[0082] Token: A concept used to represent the smallest unit in a text in natural language processing tasks. It can usually be a word, character, or sub-word. Tokenization is the process of splitting the input text into these smallest units, which helps the model understand and analyze the structure, grammar, and semantic information of the text.
[0083] CLIP: A deep learning model that can simultaneously understand text and images and establish connections between them, with strong feature extraction and understanding capabilities.
[0084] FID (Fréchet Inception Distance): A metric used to evaluate the performance of generative models, which is used to compare the similarity between the real data distribution and the generated data distribution. The lower the FID, the closer the images generated by the generative model are to the real image distribution and the higher the quality.
[0085] Optimal transport: A mathematical theory aiming to find the most economical and effective method to transform one probability distribution into another. This theory determines the optimal mapping scheme between two distributions by minimizing the "cost" or "distance" generated during the transfer process. Optimal transport has a wide range of applications in computer vision, image processing, machine learning, etc. For example, in shape analysis, image style transfer, and domain adaptation, it provides a powerful tool to measure and manipulate the differences between data distributions.
[0086] FFHQ (Flickr-Faces-HQ): A high-quality face dataset containing 70,000 PNG-format face images with a resolution of 1024×1024. These images have rich diversity in terms of age, race, gender, expression, hairstyle, etc., and are widely used for the training and evaluation of machine learning models such as generative adversarial networks (GANs). The data of FFHQ comes from Flickr and has been strictly screened and processed to ensure the high quality and diversity of the data.
[0087] Oxford-IIIT Pet: A pet image dataset containing images of 37 pet breeds, with approximately 200 pictures in each category. These images have significant variations in scale, pose, and lighting, and are widely used for computer vision tasks such as image classification and segmentation.
[0088] MetFaces: A dataset containing face images extracted from artworks. This dataset contains 1336 high-quality PNG images with a resolution of 1024×1024 and is widely used for training models such as generative adversarial networks (GANs), especially in training research under limited data conditions.
[0089] BreCaHAD (Breast Cancer Histopathological Annotation and Diagnosis): A dataset of breast cancer histopathological images, containing 162 microscopic breast cancer section images. These images are divided into 17 different cases, each case containing a different number of images. It is widely used in research on automatic detection and diagnosis of breast cancer, supporting the development and evaluation of computer vision and machine learning methods.
[0090] CR (Consistency Regularization): Consistency regularization, which is a commonly used regularization method in semi-supervised learning. The core idea is that for the same sample, even with a small perturbation, the model's prediction results should remain consistent. By applying different perturbations (such as data augmentation, adversarial perturbations, etc.) to unlabeled data during training and constraining the prediction consistency of the model under these perturbations, CR helps improve the generalization ability and robustness of the model.
[0091] As Figure 1 shown, the present invention provides a 2D image generation method under limited samples, and its implementation method is as follows:
[0092] S1. Obtain a limited dataset in the target domain and perform privacy desensitization processing;
[0093] In this embodiment, a limited dataset in the target domain (such as sensitive fields like medicine / law, etc.) is obtained, and privacy desensitization processing (erasing identity identification information in metadata) is performed. The limited dataset in the target domain may involve issues such as privacy and copyright, resulting in a relatively small dataset size.
[0094] S2. Preprocess the limited dataset in the target domain after privacy desensitization processing;
[0095] In this embodiment, data preprocessing is performed on the obtained limited dataset in the target domain, including but not limited to operations such as image normalization, resolution standardization, and data augmentation.
[0096] S3. Initialize an unpre-trained StyleGAN generator, construct a native latent space and a quantization codebook, where the encodings in the codebook represent different 2D image features, and its implementation method is as follows:
[0097] S301. Initialize an unpre-trained StyleGAN generator;
[0098] S302. In the architecture of the initialized StyleGAN generator, randomly sample Gaussian noise from the Gaussian distribution N(0, 1) and map it through a non-linear mapping function to the intermediate variable w in the native style space W, completing the construction of the native latent space, and splitting each intermediate variable w into s sub-vectors for matching with the encodings in the codebook. represents the set of real numbers, f w represents the mapping network, d z represents the dimension of the Gaussian noise z, d w represents the dimension of the mapped intermediate variable w ∈ W, and its implementation method is as follows:
[0099] A1. In the architecture of the initialized StyleGAN generator, randomly sample Gaussian noise from the Gaussian distribution N(0, 1)
[0100] A2. Through the non-linear mapping function, convert the sampled Gaussian noise into the intermediate variable w in the style space W, and split each intermediate variable w into s sub-vectors
[0101] A3. Quantize each sub-vector to the closest entry in the learnable codebook C to obtain the quantized sub-vector and splice the sub-vectors to obtain the latent code w q , completing the conversion of the proxy space. Among them, the quantized encoding after codebook matching has the following expression:
[0102]
[0103] S303. Based on the splitting result, map the subsets of the latent variables in the native style space W to the entries of the codebook, and complete the quantization of the style space features with the help of the codebook.
[0104] In this embodiment, the native latent space is the native style space of the StyleGN model, also called the W space. The training processes in the subsequent S4 and S5 steps both require mapping and conversion of the intermediate latent variables through this W space. The specific flow and conversion process of the latent variables is: z (random noise, located in the Z space) -> w (latent variable, located in the W space, i.e., the native latent space of StyleGAN) -> w q (latent variable, the compact proxy space W constructed with the help of the codebook q, which has been quantized by the codebook) -> x (generate samples, the target domain data generated by the decoder).
[0105] In this embodiment, an untrained StyleGAN generator is initialized, and Gaussian noise is randomly sampled from the Gaussian distribution N(0,1) and mapped to the native style space W through the mapping network. The style space W is converted into a compact, structured, and discrete proxy space W by the method of vector quantization q , and the quantized proxy space W q in the latent variables The partitioned subsets correspond to the entries in the codebook.
[0106] S4. Initialize the codebook by referring to external knowledge and train the generation network of the target domain;
[0107] In this embodiment, the knowledge of the pre-trained base model is introduced to optionally initialize the codebook, so that the codebook initially establishes an abstract semantic description of the target domain dataset. The implementation method is as follows:
[0108] B1. Based on the native latent space, using the semantic knowledge embedded in the base model, encode the 2D images in the limited target domain dataset after preprocessing and the encodings in the codebook respectively, and map them to the 2D image feature space and the encoding feature space one by one to initialize the codebook;
[0109] The encoding operations for the 2D images in the limited target domain dataset and the encodings in the codebook are as follows:
[0110] C1. Given several real 2D images from the limited target domain dataset;
[0111] C2. Use the CLIP visual encoder φ of the deep learning model to extract the features f of each 2D image i ;
[0112] C3. According to the features f of each 2D image extracted i , obtain a set of 2D image features F;
[0113] C4. Use the fully connected layer to convert the quantized encoding into a variable with the same embedding dimension as the input token token;
[0114] C5. According to the variable converted in C4, use the CLIP text encoder of the deep learning model for processing to obtain the encoded embedding feature T;
[0115] C6. According to the marginal distribution p of the 2D image feature F, the marginal distribution q of the encoded embedding feature T, and the cost matrix, an optimal transport solution γ* is obtained by solving the optimal transport formula;
[0116] C7. According to the optimal transport solution γ*, define the optimal transport loss of the 2D image feature F and the encoded embedding feature T to achieve feature alignment and complete the encoding operation;
[0117] B2. Use the initialized codebook to train the generation network of the target domain;
[0118] S5. Input the randomly sampled noise into the trained generation network to obtain the data of the target domain, and complete the generation of the 2D image, where the data of the target domain includes the 2D image.
[0119] In this embodiment, after the generation network is trained, the randomly sampled noise can be used to generate any new data that conforms to the target domain distribution.
[0120] In this embodiment, the framework of the present invention mainly includes two components: style space quantization and knowledge-enhanced codebook initialization, as well as related constraints and components specifically designed inside.
[0121] In this embodiment, for a batch of intermediate variables w ∈ W, the framework designed by the present invention can apply the intermediate latent space quantization technology and use a hyperspherical codebook to segment and quantize it. The concatenated discrete codes are directly input into the generator for image synthesis.
[0122] In this embodiment, the conversion process from the style space W to the discrete proxy space W q is as follows:
[0123] In the generator architecture, the input Gaussian noise z ∈ Z is mapped to a relatively decoupled intermediate latent space W through a non-linear mapping function This process is similar to the StyleGAN generator. Among them, d z represents the dimension of the Gaussian noise z, and d w represents the dimension of the mapped intermediate variable w ∈ W. The intermediate variable w affects each layer of the subsequent synthesis network g through an affine transformation, and actually acts as a "style" role, controlling various high-level attributes of the generated image. z represents the sampled Gaussian noise, and Z generally refers to the noise space where the Gaussian noise is located.
[0124] In order to establish an ideal mapping between the style space W and the data points, the present invention proposes a novel style space quantization method called Style Quantization Generative Adversarial Network (SQ-GAN) to construct a compact, structured and discrete proxy space, so as to better adapt to the limited data set.
[0125] In this embodiment, in order to achieve more precise control over the feature space and capture more detailed information, the framework divides each intermediate variable w into s sub-vectors Each subvector is optimized to manage specific details or properties. Then, each subvector Quantized to a learnable codebook The closest entry in , thus obtaining the quantized sub-vector
[0126]
[0127] This method converts each original w i Represented as a set of s quantized discrete representations Finally, the synthesis network g is derived from the latent code w q ∈W q Generate high-quality images Among them, the potential code w q By quantizing the sub-vector Splicing to get:
[0128]
[0129] in, Represents each subvector Quantize to the closest entry in the learnable codebook C to obtain the quantized subvector, Indicates a d w dimensional real space, d w Represents the dimension of the intermediate variable w∈W after mapping.
[0130] Therefore, the proxy space W q is defined as the Cartesian product of s codebooks, namely W q =C 1 ×C 2 ×,...,×C s , where C s represents the sth codebook.
[0131] In this embodiment, the present invention defines the following loss function for end-to-end training of the generated network:
[0132] L(g,f w ,C,P)=L adv (g)+λ sq L sq (f w ,C)+λ uf L uf (C,P)
[0133] L(f D )=Ladv (f D ) + λ qcr L qcr (f D )
[0134] where λ sq , λ uf and λ qcr are all hyperparameters used to balance the contributions of various loss functions.
[0135] The adversarial losses L adv (g) and L adv (f D ) are designed to optimize the generator g and the discriminator f D . The style quantization loss L sq (g) is optimized by minimizing the distance between the segmented latent variables and the corresponding codebook representations :
[0136]
[0137] where sg() is the stop gradient operator, and the second term (scaled by β) is the so-called "commitment loss" used to prevent the encodings learned in the codebook C from growing infinitely. To handle the non-differentiable operations in the style quantization loss, the present invention uses a straight-through gradient estimator that allows the generator network g to directly pass the gradient to the mapping network f during backpropagation w .
[0138] To prevent the codebook from collapsing in the case of limited data volume, the present invention imposes a uniformity constraint L uf (C, P) on the codebook C. The present invention first projects the codebook C onto the unit hypersphere using a learnable projection matrix P and l2 regularization, such that represents the codebook entry after being mapped to the hypersphere and processed by l2 regularization, P represents the projection matrix, and c i represents the i-th entry in the codebook. Among them, the uniformity problem on the unit hypersphere is characterized by a radial basis function (RBF) kernel and minimizing all pairwise potentials:
[0139]
[0140] where t > 0 is a fixed scale parameter. This regularization helps to promote a uniform distribution of feature encodings on the unit hypersphere, ensuring diverse and well-defined mean encodings in the codebook.
[0141] In quantization-based CR, the present invention enforces that the discriminator is invariant to the quantization of the intermediate variable w qThe predictions of the generated image and its perturbed version are consistent. The perturbed intermediate variable is obtained by adding Gaussian noise ∈ (the same as the CR method) to the original input latent code z, and then mapping it to the intermediate variable w' = f w (z + ò), and quantizing it through the codebook C to obtain w' q . The quantization-based consistency regularization loss L qcr is defined as:
[0142] L qcr (f D ) = E[||f D (g(w q )) - f D (g(w' q ))|| 2
[0143] The adversarial losses for training the generator and discriminator are given separately below:
[0144] L adv (g) = E[1 - logf D (g(z))]
[0145] L adv (f D ) = -E[logf D (x)] - E[log(1 - fD(g(z)))]
[0146] where L(g, f w , C, P) represents the loss function of the generator, g represents the generator, f w represents the mapping network, C represents the codebook, P represents the learnable projection matrix, L adv (g) and L adv (f D ) both represent adversarial losses, λ sq , λ uf and λ qcr all represent hyperparameters, L sq (f w , C) represents the style quantization loss function, L uf (C, P) represents the uniformity constraint, L(f D ) represents the loss function of the discriminator, f D represents the discriminator, z represents Gaussian noise, x represents the real sample, L qcr (f D ) represents the quantization-based consistency regularization constraint (in the quantization-based consistency regularization (qCR) of the quantization-based method, the present invention enforces that the discriminator pairs the quantized intermediate variable w q The image predictions generated by it and the intermediate variable w' of the perturbed version are consistent. The perturbed intermediate variable w' is obtained by adding Gaussian noise ∈ to the original input latent variable z, and then mapping it to the intermediate latent variable w' = f w (z + ò), and obtaining w' through quantization by the codebook C q . This form of regularization ensures that even if small perturbations in the latent space result in different quantization representations, they can be evaluated consistently by the discriminator), f D represents the discriminator network, sg() represents the stop gradient operator, represents s sub-vectors obtained by splitting each intermediate variable w, c represents the entries in the codebook, β represents the hyperparameter that balances the two values in the style quantization loss, w represents the intermediate variable in the style space W, E(·) represents the mathematical expectation, t represents a fixed scale scaling hyperparameter, c i and c j both represent the entries in the codebook, w q represents the sub-vector after quantization The intermediate vector formed by splicing, w' q represents the intermediate variable obtained after the sampling noise is perturbed, mapped, and quantized.
[0147] This form of regularization ensures that even if small perturbations in the latent space result in different quantization representations, the discriminator can still make consistent evaluations.
[0148] In this embodiment, the knowledge-enhanced codebook initialization process is as follows: The present invention aims to ensure that the encodings in the codebook can appropriately represent different image features and correspond to specific data points through combinations of these encodings. However, deriving such codes from a limited training dataset is not an easy task. To solve this problem, the present invention proposes an alignment strategy based on the optimal transport distance for codebook initialization. This method utilizes the semantic knowledge embedded in the base model to construct a richer vocabulary for the training dataset.
[0149] 1. Feature extraction
[0150] The present invention first uses the base model to perform encoding operations on the images in the training dataset and the encodings (quantized sub-vectors) in the codebook, mapping them to their respective feature spaces. Since the initialization of the codebook is essentially a pairing task, the present invention adopts the visual encoder of the deep learning model CLIP, a widely used large-scale vision-language pre-training model. The extensive pre-training of the visual encoder of the deep learning model CLIP can help explore the semantic correspondence between visual and language modalities.
[0151] In this embodiment, Foundation models refer to large, multi-functional models trained on large-scale datasets. They are typically based on deep learning, especially the Transformer architecture (such as BERT, GPT, etc.). These models can be pre-trained on a large amount of text or other types of data to learn extensive knowledge and complex language structures. The key feature of the foundation model is its strong generalization ability. It can adapt to various downstream tasks through fine-tuning with a small amount of task-specific data, thus greatly reducing the need to develop specialized models for each specific application. In addition, such models also demonstrate excellent performance in multiple fields such as natural language processing and computer vision. The pre-trained deep learning model CLIP is used in the present invention.
[0152] During the image encoding process, the present invention directly uses the visual encoder of the deep learning model CLIP to extract image features. Specifically, given n real images from a limited dataset The present invention uses the CLIP visual encoder φ to extract the features of each image x i represents the i-th real image, l represents the length of the image token sequence, and d e represents the embedding dimension of the image features, and i represents the index number. Finally, a set of image features is obtained
[0153] Consider m latent variables sampled from the prior distribution First, these latent variables are mapped to the style space W and then quantized into the discrete proxy space W q . To align with the Transformer architecture, the quantized sub-vectors are input into a trainable fully connected layer (i.e., the projector), and through this layer, they are transformed into variables with the same embedding dimension as the input tokens. Then, these transformed variables will be processed by a specific Transformer structure. The present invention uses the text encoder of the deep learning model CLIP to implement this structural design, and finally, the feature set t i represents the i-th encoded embedding feature. The present invention regards these discrete encodings as similar to text encodings composed of words in the vocabulary. Therefore, the initialized codebook can be regarded as a vocabulary, abstractly describing the image features in the dataset.
[0154] (2) Feature alignment
[0155] Since the images and latent variables in each batch are unpaired, and considering the differences in semantic structure and interpretation between the image features F and the encoded embedding features T, the present invention adopts the optimal transport (OT) distance as an effective metric to quantify the differences between them.
[0156] According to the discrete encoded embedding features and the visual features of the marginal distributions p and q, and the corresponding cost matrix C, by solving the optimal transport formula:
[0157]
[0158] subject to:
[0159] the optimal transport plan γ* (minimizing the total transport cost of mapping one feature distribution to another) can be obtained. To simplify the calculation, the marginal distributions p and q are initialized to uniform distributions, that is:
[0160]
[0161] where D(p,q|C') represents the optimal transport distance, n represents the size of the support set of the marginal distribution p, m represents the size of the support set of the marginal distribution q, γ represents the transport matrix, represents the set of real numbers, C' represents the cost matrix, i represents the index variable of the marginal distribution p, j represents the index variable of the marginal distribution q, γ ij represents the probability of transferring from the i-th point of the marginal distribution p to the j-th point of the marginal distribution q, c represents the cost function, f i x represents the feature vector corresponding to the index i in the marginal distribution p, represents the feature vector corresponding to the index j in the marginal distribution q, s represents the number of sub-vectors into which each intermediate vector w is divided, l represents the sequence length of the image tokens, p i represents the probability value of the i-th point, and each p i represents the probability starting from a point in the marginal distribution p, q j represents the probability value of the j-th point, and each q j represents the probability starting from a point in the marginal distribution q.
[0162] In this embodiment, the transport matrix γ describes how to transport the mass in one marginal distribution p to another marginal distribution q. Each element γ in the transport matrix ij represents the mass or probability of transporting from the i-th point in the marginal distribution p to the j-th point in the marginal distribution q (that is, transferring from the distribution of the quantized encoded embedding features T to the distribution of the real image features F).
[0163] In this embodiment, p i and q j respectively represent the probability masses of the source distribution and the target distribution at specific points. The element γ ij of the transport matrix γ describes the mass transported from the ith point in the source distribution to the jth point in the target distribution. These transports must satisfy the above two constraints, that is, the sum of the masses transported from each point in the source distribution is equal to the probability mass p i of that point, and the sum of the masses transported to each point in the target distribution is also equal to the probability mass q j of that point. In this way, the transport matrix γ realizes the mass transfer from one probability distribution to another while keeping the total mass of both unchanged. Note that p i and q j are not directly related. They belong to two different distributions respectively, and the transport matrix establishes a connection between these two distributions.
[0164] The cost matrix C is calculated as follows:
[0165] C j,k = c(j,k):= d(t .j , f .k ) 1 ≤ j ≤ s, 1 ≤ k ≤ l
[0166] where C j,k represents the cost matrix, c(j,k): represents the element in the cost matrix, d(·,·) represents the distance function used to measure the transport cost between t .j and f .k , t .j represents the jth component of all features in the encoded embedding feature set T, f .k represents the kth component of all features in the feature set F, j represents the index of the features in the visual feature set F, s represents the number of sub-vectors corresponding to each intermediate vector w, and k represents the index of the features in the feature set.
[0167] In this embodiment, c is the element in the cost matrix, which forms the cost matrix C'. Each element c is obtained by calculating the value with the distance metric function.
[0168] After calculating γ*, the present invention defines the optimal transport loss between two features T and F to achieve feature alignment:
[0169] L ot (f w , C) = E[d(T, F)·γ * )
[0170] where L ot represents the optimal transport loss, fw Let \(G\) denote the mapping network, \(C\) denote the codebook, \(E(\cdot)\) denote the mathematical expectation, \(d(\cdot)\) denote the distance function used to measure the transmission cost between the feature sets \(T\) and \(F\), \(F\) denote the visual feature set obtained by encoding the image, and \(T\) denote the encoded embedding feature set obtained by encoding the quantized discrete intermediate variable.
[0171] In the codebook initialization stage, the present invention optimizes the optimal transport loss, as well as the weighted sum of two additional loss terms corresponding to the style quantization loss and the uniformity constraint involved in the feature extraction stage. The final total loss function in this stage is defined as follows:
[0172] \(L(f\) w , C, P) = \(L\) sq (f w , C) + \(L\) uf (C, P) + \(\lambda\) ot ·\(L\) ot (f w , C)
[0173] where \(L(f\) w , C, P) represents the loss function for initializing the codebook, \(L\) sq represents the style quantization loss in the feature extraction stage, \(L\) uf represents the uniformity constraint loss, \(\lambda\) ot represents the weight coefficient for balancing the losses, \(L\) ot represents the optimal transport loss, \(f\) w represents the mapping network, \(C\) represents the codebook, and \(P\) represents the projection matrix.
[0174] Through the above design, the model obtained by the present invention can generate a large number of high-quality target domain 2D images at low cost and quickly (the inference time for a single image is less than 1 s) under the condition of limited data.
[0175] In summary, the present invention provides a more compact and structured input latent space for the generative adversarial network (GAN), enabling it to more feasibly and effectively utilize the input latent space under limited data conditions, thereby improving the performance of consistency regularization (CR) (in the original consistency regularization (CR) method setting, the generated samples need to rely on the perturbation of the latent space. However, the sparsity of the input latent space may lead to a significant difference in the realism between the generated images \(g(z)\) and \(g(z + \delta)\)). It solves the problems such as mode collapse and overfitting that are prone to occur in the medical, judicial, and other fields due to privacy, copyright, or acquisition cost limitations, resulting in extremely limited available data volume (usually less than a thousand cases). The intermediate latent space quantization technology proposed by the present invention segments the intermediate variables and quantizes them into a learnable codebook, constructing the discrete proxy space \(W\) qAs a high-level abstraction of the native style space W, it provides a more compact representation for image generation, enabling more realistic and higher-quality image generation. In addition, the knowledge-enhanced codebook initialization method proposed in the present invention combines the external semantic knowledge of the base model, thereby constructing a semantically rich and perceptually diverse codebook. The introduction of the uniformity constraint avoids representation collapse and ensures that each codebook entry clearly controls different variation factors in image generation. The present invention can generate a large number of high-quality target domain images at low cost and quickly (inference time for a single image is less than 1 s) under the condition of limited data.
[0176] Embodiment 2
[0177] As Figure 2 shown, the present invention provides a 2D image generation system under limited samples. The 2D image generation system is used to execute the 2D image generation method described in Embodiment 1, and includes:
[0178] A first processing module, configured to obtain a limited data set of the target domain and perform privacy desensitization processing;
[0179] A second processing module, configured to preprocess the limited data set of the target domain after privacy desensitization processing;
[0180] A third processing module, configured to initialize an untrained StyleGAN generator, construct a native latent space and a quantization codebook, wherein the encodings in the codebook represent different 2D image features;
[0181] A fourth processing module, configured to initialize the codebook by referring to external knowledge and train the generation network of the target domain;
[0182] A fifth processing module, configured to input randomly sampled noise into the trained generation network to obtain data of the target domain, and complete the generation of 2D images, wherein the data of the target domain includes 2D images.
[0183] As Figure 2 shown, the 2D image generation system provided in the embodiment can execute the technical solutions shown in the above method embodiment of the 2D image generation method. Its implementation principle and beneficial effects are similar, and will not be elaborated here.
[0184] In this embodiment, the present application can divide functional units according to the 2D image generation method. For example, each function can be divided into each functional unit, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. It should be noted that the division of units in the present invention is illustrative, only a logical division, and there may be other division methods in actual implementation.
[0185] In this embodiment, in order to implement the principle and beneficial effects of the 2D image generation method, the 2D image generation system includes the corresponding hardware structure and / or software module for executing each function. Those skilled in the art should easily realize that, combining the various schematic units and algorithm steps described in the embodiments disclosed by the present invention, the present invention can be implemented in the form of hardware and / or the combination of hardware and computer software. Whether a certain function is executed in the way driven by hardware or computer software depends on the specific application and design constraints of the technical solution. Different methods can be used for each specific application to implement the described function, but such implementation should not be considered to exceed the scope of this application.
[0186] In summary, the present invention proposes an intermediate latent space quantization technique, aiming to optimize the consistency regularization of the discriminator, enhance the robustness of its feature extraction, and alleviate the phenomenon of the degradation of the training of the generative adversarial network (GAN) in the limited sample scenario. Specifically, this framework embeds the input latent variable z into an intermediate latent space W. The intermediate variable w ∈ W is usually interpreted as a less entangled "style", controlling various high-level attributes of the generated image. Each intermediate variable w is split into s sub-vectors, corresponding to the attributes (or subsets of attributes) of different aspects, and then these split latent variables are quantized to the nearest neighbors in the learnable codebook, so that each quantization code can control different variation factors. During the image generation process, the quantization codes are re-spliced ( represents the nth quantization code, i = 1, 2, 3,..., n, n represents the number of quantization codes, i represents the ith quantization code, and W q represents the spliced quantization code and is input into the synthesis network. In the limited data setting, exploring this smaller and more compact proxy space W q is more feasible than directly exploring the intermediate latent space W. This design effectively forces the intermediate variable w to recombine from a finite set of different discrete sub-vectors so that the generator can assign clear meanings to each sub-vector. In addition, this framework introduces a novel codebook initialization method, which is crucial for capturing the intrinsic semantic structure of the training data. This method uses the base model to map the encodings of the codebook and the images in the training dataset to the feature space, and aligns these two sets of features based on the optimal transport distance to initialize the codebook in a way that abstractly describes the training dataset. This alignment process embeds external knowledge into the codebook, thus pre-establishing a semantically rich vocabulary about the training dataset.
[0187] In summary, the present invention provides a more compact and structured input latent space for the generative adversarial network (GAN), enabling it to more feasibly and effectively utilize the input latent space under limited data conditions, thereby enhancing the performance of consistency regularization (CR). (In the original CR method setting, the generated samples need to rely on the perturbation of the latent space. However, the sparsity of the input latent space may lead to significant differences in the realism of the generated images.) Subsequently, it solves the problems such as mode collapse and overfitting that are prone to occur in models due to extremely limited available data volume (usually less than a thousand cases) caused by privacy, copyright, or acquisition cost restrictions in fields such as healthcare and justice.
Claims
1. A 2D image generation method under finite samples, characterized in that: The following steps are involved: S1. Obtain a limited dataset of the target domain and perform privacy masking. S2, preprocessing the target domain limited data set after privacy masking; S3, initialize the unpre-trained StyleGAN generator, construct the native latent space and quantization codebook, where the codes in the codebook represent different 2D image features; S4. Based on the native latent space, external knowledge is referenced to initialize the codebook and train the generative network of the target domain; S5. Input the randomly sampled noise into the trained generation network to obtain the data of the target domain, and complete the generation of the 2D image, wherein the data of the target domain includes the 2D image.
2. The 2D image generation method under limited samples according to claim 1, characterized in that: The S3 comprises the following steps: S301, initializing the unpre-trained StyleGAN generator; S302: In the initialized StyleGAN generator architecture, randomly sample Gaussian noise from Gaussian distribution N(0,1) And map it through nonlinear mapping function To the intermediate variable w of the native style space W, complete the native latent space construction, and split each intermediate variable w into s sub-vectors In order to match the encoding in the codebook, represents the set of real numbers, f w represents the mapping network, d z represents the dimension of Gaussian noise z, d w Represents the dimension of the intermediate variable w∈W after mapping; S303: Based on the splitting result, the partitioned subsets of the latent variables in the native style space W are mapped to the entries of the codebook, and the quantization of the style space features is completed with the help of the codebook.
3. The 2D image generation method under limited samples according to claim 2, characterized in that: The S302 is specifically as follows: A1. In the architecture of the initialized StyleGAN generator, randomly sample Gaussian noise from the Gaussian distribution N(0,1) A2. Through the nonlinear mapping function, the sampled Gaussian noise is converted into an intermediate variable w in the style space W, and each intermediate variable w is divided into s sub-vectors A3. Each sub-vector Quantize to the closest entry in the learnable codebook C to obtain the quantized subvector And the sub-vector Concatenate to get the potential code w q , complete the conversion of the proxy space, where the quantization coding after codebook matching The expression is as follows:
4. The 2D image generation method under limited samples according to claim 3, characterized in that: The loss function of the target domain generation network includes the loss function of the generator and the loss function of the discriminator: The expression of the generator's loss function is as follows: L(g,f w ,C,P)=L adv (g)+λ sq L sq (f w ,C)+λ uf L uf (C,P) The loss function of the discriminator is expressed as follows: L(f D )=L adv (f D )+λ qcr L qcr (f D ) in: L qcr (f D )=E[||f D (g(w q ))-f D (g(w' q ))|| 2 ] L adv (g)=E[1-logf D (g(z))] L adv (f D )=-E[logf D (x)]-E[log(1-fD(g(z)))] Among them, L(g,f w ,C,P) represents the loss function of the generator, g represents the generator, f w represents the mapping network, C represents the codebook, P represents the learnable projection matrix, and L adv (g) and L adv (f D ) denotes adversarial loss, λ sq , uf and λ qcr Both represent hyperparameters, L sq (f w ,C) represents the style quantization loss function, L uf (C,P) represents the uniformity constraint, L(f D ) represents the loss function of the discriminator, f D represents the discriminator, z represents Gaussian noise, x represents the real sample, L qcr (f D ) represents the quantization-based consistency regularization constraint, f D represents the discriminator network, sg() represents the stop gradient operator, represents the s sub-vectors of each intermediate variable w, c represents the entry in the codebook, β represents the hyperparameter of the two values in the balanced style quantization loss, w represents the intermediate variable in the style space W, E(·) represents the mathematical expectation, t represents a fixed scale hyperparameter, c i and c j Both represent entries in the codebook, w q Represents the quantized sub-vector The concatenated intermediate vector, w' q Represents the intermediate variable obtained after the sampling noise is disturbed, mapped and quantized.
5. The 2D image generation method under limited samples according to claim 2, characterized in that: The S4 is specifically: B1. Based on the native latent space, using the semantic knowledge embedded in the basic model, the 2D image in the preprocessed target domain finite data set and the code in the codebook are encoded, and mapped one-to-one to the 2D image feature space and the encoding feature space respectively to initialize the codebook; B2. Use the initialized codebook to train the target domain’s generative network.
6. The 2D image generation method under limited samples according to claim 5, characterized in that: The encoding operation is performed on the 2D image in the target domain limited data set and the encoding in the codebook, which is specifically: C1. Given several real 2D images from a limited dataset in the target domain; C2. Use the deep learning model CLIP visual encoder φ to extract the features f of each 2D image i ; C3, based on the features f of each 2D image extracted i , get a set of 2D image features F; C4, use the fully connected layer to encode the quantized Convert to a variable with the same embedding dimension as the input token; C5. Based on the variables converted by C4, use the deep learning model CLIP text encoder Processing is performed to obtain the encoded embedding feature T; C6. According to the edge distribution p of the 2D image feature F and the edge distribution q of the encoded embedded feature T, as well as the cost matrix, the optimal transmission scheme γ* is obtained by solving the optimal transmission formula; C7. According to the optimal transmission scheme γ*, define the optimal transmission loss of the 2D image feature F and the encoded embedded feature T to achieve feature alignment and complete the encoding operation.
7. The 2D image generation method under limited samples according to claim 6, characterized in that: The optimal transmission formula is as follows: obey: Where D(p,q|C') represents the optimal transmission distance, n represents the support set size of the edge distribution p, m represents the support set size of the edge distribution q, γ represents the transportation matrix, represents a set of real numbers, C' represents a cost matrix, i represents the index variable of the marginal distribution p, j represents the index variable of the marginal distribution q, γ ij represents the probability of transferring from the i-th point of marginal distribution p to the j-th point of marginal distribution q, c represents the cost function, and f i x represents the eigenvector corresponding to index i in the marginal distribution p, represents the feature vector corresponding to index j in the marginal distribution q, s represents the number of sub-vectors divided by each intermediate vector w, l represents the sequence length of the image token, and p i Represents the probability value of the i-th point, each p i represents the probability of starting from a point in the marginal distribution p, q j Represents the probability value of the jth point, each q j represents the probability of starting from a point in the marginal distribution q; The expression of the cost matrix is as follows: C j,k =c(j,k):=d(t .j ,f .k )1≤j≤s,1≤k≤l Among them, C j,k represents the cost matrix, c(j,k): represents the elements in the cost matrix, and d(·,·) represents the cost matrix used to measure t .j and f .k The distance function of the transmission cost between .j represents the jth component of all features in the encoding embedding feature set T, f .k represents the kth component of all features in the feature set F, j represents the index of the feature in the visual feature set F, s represents the number of subvectors corresponding to each intermediate vector w, and k represents the index of the feature in the feature set.
8. The 2D image generation method under limited samples according to claim 6, characterized in that: The expression of the optimal transmission loss is as follows: L ot (f w ,C)=E[d(T,F)·γ*] Among them, L ot represents the optimal transmission loss, f w represents the mapping network, C represents the codebook, E(·) represents the mathematical expectation, d(·) represents the distance function used to measure the transmission cost between feature sets T and F, F represents the visual feature set obtained by encoding the image, and T represents the encoded embedded feature set obtained by encoding the quantized discrete intermediate variables.
9. The 2D image generation method under limited samples according to claim 1, characterized in that: The expression of the loss function for initializing the codebook is as follows: L(f w ,C,P)=L sq (f w ,C)+L uf (C,P)+λ ot ·L ot (f w ,C) Among them, L(f w ,C,P) represents the loss function for initializing the codebook, L sq represents the style quantization loss in the feature extraction stage, L uf represents uniform constraint loss, λ ot Represents the weight coefficient for balancing various losses, L ot represents the optimal transmission loss, f w represents the mapping network, C represents the codebook, and P represents the projection matrix.
10. A 2D image generation system under finite samples, characterized in that: The 2D image generation system is used to execute the 2D image generation method according to any one of claims 1 to 9, comprising: The first processing module is used to obtain a limited data set of the target domain and perform privacy desensitization processing; The second processing module is used to pre-process the target domain limited data set after the privacy desensitization processing; The third processing module is used to initialize the untrained StyleGAN generator and construct a native latent space and quantization codebook, where the codes in the codebook represent different 2D image features; The fourth processing module is used to initialize the codebook by referencing external knowledge and train the generative network of the target domain; The fifth processing module is used to input the randomly sampled noise into the trained generation network to obtain the data of the target domain and complete the generation of the 2D image, wherein the data of the target domain includes the 2D image.
Citation Information
Cited By
High-fidelity low-illumination image enhancement method and system based on one-step diffusion
CN121660909A
High-fidelity low-light image enhancement method and system based on one-step diffusion
CN121660909B
