A few-shot image generation method and system based on category-level latent representation semantic editing

Abstract

The present application relates to the field of image generation and computer vision, and proposes a few-shot image generation method and system based on class-level latent representation semantic editing. The method comprises the following steps: S1, loading a pre-trained image encoding model and a generation model, mapping a few target class sample images to a latent feature space, obtaining corresponding instance-level latent representations, and completing the mapping alignment between images and the latent space; S2, by introducing a latent adapter and combining a metric learning strategy, aggregating multiple instance-level latent representations of the same target class to construct a class-level latent representation representing the overall semantic features of the target class; S3, on the basis of the class-level latent representation, according to semantic description information and introducing target region constraints, optimizing the latent features in the semantic direction to obtain a semantic editing direction focusing on the target region; S4, interpolating the class-level latent representation along the semantic editing direction, and generating diversified and structurally consistent target class images through the generation model. Through the above technical scheme, the consistency of the generated image semantics and structure is controlled under the condition of few samples, the stability and controllability of image generation are improved, and the complexity of model training and parameter adjustment is reduced.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and generative artificial intelligence. Specifically, it relates to a method and system for generating few-shot images based on category-level latent representation semantic editing. It is suitable for the synthesis of diverse, high-fidelity images in scenarios with scarce samples (such as 1-shot and 5-shot) and can be applied to fields such as image data enhancement, digital content creation, and assistance in the restoration of cultural relics. Background Technology

[0002] Performance improvements in deep vision models heavily rely on large-scale labeled data. However, in real-world scenarios, many categories face challenges in obtaining sufficient training samples due to high acquisition costs, privacy restrictions, and sample scarcity, posing a serious challenge to the model's generalization ability. Few-shot image generation technology, as an effective solution to the data scarcity problem, aims to synthesize images with semantic consistency and diversity using a very small number of new category samples, providing data expansion support for downstream tasks. Its core value lies in breaking through data dependency bottlenecks and expanding the application boundaries of models.

[0003] Existing few-shot image generation methods are mainly divided into four categories: meta-learning, fusion modeling, transform transfer, and latent space editing. Each method has obvious limitations: meta-learning methods improve generalization ability through cross-task training, but have insufficient control over image quality and diversity, and are only applicable to simple datasets; fusion modeling methods enhance representation ability through feature fusion, but are prone to introducing structural artifacts and semantic shifts, affecting image realism; transform transfer methods have a certain flexibility in class transformation, but lack semantic interpretability and structural consistency guarantees; latent space editing methods rely on pre-trained generators to discover semantic directions, which have some controllability, but depend on human prior knowledge and lack structural awareness, which can easily lead to image deformation or inconsistency in intra-class structure.

[0004] Meanwhile, existing latent space editing and semantic guidance technologies either lack category-level shared representation learning, making it difficult to overcome the diversity bottleneck of instance-level encoding; or semantic direction discovery lacks regional constraints, is easily affected by background interference, and has insufficient editing accuracy.

[0005] In summary, existing technologies cannot simultaneously achieve structural consistency, semantic controllability, and generation diversity. There is an urgent need for a modular, efficient, and controllable few-sample image generation paradigm to adapt to the practical application needs in complex scenarios. Summary of the Invention

[0006] To achieve the above objectives, this invention adopts a two-stage modular technical solution, built upon a pre-trained generative model. Through category-level latent representation learning and CLIP-guided semantic editing with region constraints, it achieves efficient and controllable few-shot image generation. The specific technical solution is as follows: a few-shot image generation method based on category-level latent representation semantic editing; a few-shot image generation system based on category-level latent representation semantic editing.

[0007] The few-shot image generation method based on category-level latent representation semantic editing includes the following steps:

[0008] Step S1: Pre-trained loading and image inverse encoding: Load the pre-trained pSp encoder and StyleGAN2 generator, and input a small number of new category image samples. Reverse encoding to StyleGAN2 using a pSp encoder The latent space yields instance-level latent encodings. ,in ; via StyleGAN2 generator Image reconstructed based on this encoding This completes the mapping and alignment of the image with the latent space.

[0009] Step S2: Category-level latent representation learning.

[0010] 1) Introduce a lightweight latent adapter (LA), and use an episode-based metric learning strategy to construct category-aware constraints, guiding instance-level encodings to aggregate towards a category-level shared distribution. The specific process is as follows: Construct a subset of episode tasks, where each episode contains... There are 10 categories, and each category corresponds to a support set. (including) (a number of real image samples) and query set (including) (a number of generated image samples), denoted as , ,in For the corresponding category labels;

[0011] 2) Latent encoding adaptation and transformation: The instance-level encoding of the query set samples is transformed through a latent adapter (a 4-layer MLP structure with input and output dimensions of 18×512, activation function of ReLU, and dropout probability of 0.1) to obtain the category-level latent encoding. Based on this encoding, reconstruct category-aligned images ;

[0012] 3) Optimize category-aware loss by using a pre-trained metric learning model. Output sample category classification score The class aggregation constraint is achieved by minimizing the mean squared loss, and the loss function is:

[0013]

[0014] This allows similar samples to form consistent clusters in the latent space.

[0015] Step S3: Region-constrained CLIP-guided semantic direction discovery.

[0016] 1) Based on SAM, target region masks are automatically generated. Combined with CLIP's cross-modal capabilities, semantic editing direction is optimized to achieve fine-grained semantic control. The specific steps are as follows: Automated region mask generation: The image is aligned to categories using Segment Anything Model (SAM). Perform segmentation and retain the candidate set of high-confidence region masks. Target region mask is selected based on area heuristic rules.

[0017]

[0018] Obtain the binary target region mask ;

[0019] 2) CLIP-guided semantic direction optimization: based on target text description Guided by the concepts of "red petals" and "sideways pose," a mask-constrained CLIP loss function is constructed.

[0020]

[0021] in For CLIP encoding functions, To optimize the semantic direction of the focused target region through pixel-by-pixel multiplication.

[0022]

[0023] 3) Semantic interpolation generates diverse images: along semantic directions Perform linear interpolation by adjusting the interpolation coefficients. (like Generate new images with continuous semantic changes This achieves a synergy between preserving category structure and changing semantic attributes.

[0024] Step S4: Joint loss training optimization and semantic interpolation generation, fixing the parameters of the pSp encoder and StyleGAN2 generator, optimizing only the latent adapter. With metric model Introducing perceived loss and the loss of combat A joint loss function is constructed to balance class consistency, image fidelity, and generation diversity, as follows:

[0025] Potential adapter losses:

[0026]

[0027] in

[0028]

[0029] For VGG feature extractor, , For loss weights;

[0030] Measuring model loss:

[0031]

[0032] in For gradient regularization, As regularization weights, adversarial training is used to improve the realism of generated images; after training, linear interpolation is performed along the optimized semantic direction to generate diverse and structurally consistent images.

[0033] The few-shot image generation system based on category-level latent representation semantic editing adopts a modular architecture, divided from top to bottom into an application layer, a business layer, and a data layer. Each layer collaborates to achieve few-shot image generation and semantic editing functions, specifically including:

[0034] 1) Application layer: As a user interaction interface, it provides functions such as image import, parameter configuration, semantic editing, result display and export. It supports text description input (semantic target setting), interpolation coefficient adjustment (semantic change intensity control) and mask area visualization. The interface adopts a responsive design to adapt to different terminal operation scenarios and meet users' flexible control needs for the generation process.

[0035] 2) Business Layer: This is the core processing layer of the system, containing four major functional modules. It realizes the entire process from image encoding to semantic generation. The functions of each module are as follows: Pre-trained model loading module: It is responsible for loading the pre-trained weights of the pSp encoder, StyleGAN2 generator, CLIP model and SAM model, initializing the model parameters and keeping the pre-trained generator parameters fixed to reduce training overhead.

[0036] 3) Category-level representation learning module: Integrates latent adapter and metric learning model, and realizes the transformation from instance-level encoding to category-level encoding through episode-style training and category-aware loss optimization, and constructs a category-consistent latent space distribution;

[0037] 4) Semantic editing and generation module: Linking the SAM mask generation unit and the CLIP semantic guidance unit, it sequentially completes the automatic generation of target region mask, semantic direction optimization and interpolation generation, and outputs diverse images;

[0038] 5) Model training optimization module: Construct a joint loss function, use the Adam optimizer (initial learning rate 0.001, batch size 4) for adversarial training, and improve the model's few-shot generalization ability through episode-style data organization.

[0039] 6) Data Layer: Responsible for data storage and management. It uses a MySQL database to store structured data (category labels, training parameters, semantic descriptions, and generation logs) and a local file system to store unstructured data (input sample images, generated result images, and pre-trained model weights). Through data encryption, regular backups, and caching mechanisms, it ensures data integrity and access efficiency, and supports fast retrieval of generated results by category and semantic label.

[0040] As can be seen from the above technical solution, the invention has the following beneficial effects compared with the prior art:

[0041] 1) Synergistic improvement of structural consistency and diversity: By using the episode-style category-aware loss constraint described in claim 2, combined with the encoding transformation of the lightweight latent adapter (claim 5), similar samples form a clustered distribution in the latent space, effectively solving the problems of image deformation and structural inconsistency generated by existing methods; combined with the region-constrained semantic editing of claim 3, the intra-class semantic changes are expanded while preserving the category structure. Experiments show that the LPIPS index reaches 0.6825 and 0.7306 on the 102Flowers and Animal Faces datasets, respectively, and the diversity is significantly better than baseline methods such as AGE and HAE.

[0042] 2) Excellent semantic controllability and editing accuracy: As described in claim 3, the target region mask is automatically generated by SAM and high-confidence results are selected. With CLIP cross-modal guidance to construct the mask constraint loss function, the semantic direction optimization is strictly limited to the target region, effectively avoiding background interference and realizing fine-grained semantic control. Compared with the full-image semantic editing method, the semantic consistency between the generated image and the target text is improved by more than 30%, and the editing accuracy is significantly improved.

[0043] 3) Low training cost and strong generalization ability: As described in claim 4, the present invention fixes the pre-trained generator parameters and only optimizes the lightweight latent adapter and metric learning model, which shortens the training time by more than 40% and significantly reduces the computational cost; at the same time, relying on the episode-style training strategy of claim 2, it adapts to the low-sample scenario and shows good generalization ability on unseen categories. The FID index of the VGGFace dataset reaches 30.43, which is better than most baseline methods that require fine-tuning of the generator.

[0044] 4) Modular design and wide applicability: The modules are decoupled, supporting the replacement of different pre-trained generators (such as StyleGAN3, diffusion model) and segmentation models to meet the needs of multiple scenarios; it can be applied to fields such as data augmentation, digital creation, and cultural relic restoration, providing effective data support for visual tasks in scenarios with scarce samples. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0046] Figure 1 This is an overall flowchart of the few-sample image generation method of the present invention;

[0047] Figure 2 This is a schematic diagram of the structure of the category-level latent representation learning module of the present invention;

[0048] Figure 3 This is a schematic diagram of the CLIP-guided semantic editing module structure for region constraints in this invention;

[0049] Figure 4 This is a diagram of the overall system architecture of the present invention;

[0050] Figure 5 Generate a qualitative comparison chart of the results for each method. Detailed Implementation

[0051] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0052] 1) Experimental environment and dataset setup

[0053] Experimental environment: Operating system: Ubuntu 20.04; CPU: Intel i9-12900K; GPU: NVIDIA RTX4090 (24GB VRAM); RAM: 64GB; Deep learning framework: PyTorch 1.12 + CUDA 11.8; Dependencies: OpenCV, TorchVision, CLIP, SAM official library; Optimizer: Adam; Initial learning rate: 0.001; Batch size: 4.

[0054] Dataset setup: Three benchmark datasets were used: 102Flowers, Animal Faces, and VGGFaces. The training / test classes were divided as follows: 102Flowers (85 training classes / 17 testing classes), Animal Faces (119 training classes / 30 testing classes), and VGGFaces (1802 training classes / 572 testing classes). The test classes were completely invisible during the training phase to evaluate the model's generalization ability. The experiment used a 1-shot setting, with only one real image used as the generation condition for each class, generating 128 images for metric evaluation.

[0055] 2) Model parameters and training details

[0056] Pre-trained model parameters: The pSp encoder and StyleGAN2 generator are pre-trained on the FFHQ dataset, and the StyleGAN2 output image resolution is 256×256; the CLIP model uses the ViT-B / 32 version, the SAM model uses the vit_h version, and the metric learning model is pre-trained on the visible categories of the training set for category semantic modeling.

[0057] Potential adapter design: A 4-layer MLP structure is adopted, with both input and output dimensions of 18×512. The activation function is ReLU, and the dropout probability is set to 0.1 to avoid overfitting.

[0058] Training process: Each episode contains 1-shot samples from 4 classes, with 20,000 training steps and loss weights set to... , , During training, the pre-trained model parameters are fixed, and only the potential adapter and metric learning model are optimized. Training data is constructed by randomly sampling episodes to improve the model's ability to adapt to small samples.

[0059] 3) System operation and generation process

[0060] Data import and preprocessing: The user imports a target category image through the application layer. The system automatically performs size normalization (256×256) and pixel value standardization, and inputs it into the pSp encoder to obtain the instance-level latent code.

[0061] Category-level representation learning: The latent adapter transforms the instance-level encoding, combines a metric learning model with a category-aware loss, optimizes the class-level latent encoding, and generates a class-aligned image.

[0062] Semantic editing parameter configuration: The user inputs semantic description text (such as "yellow petals" or "frontal pose"), the system automatically generates a target region mask through SAM, the CLIP model constructs a semantic guidance target, and optimizes to obtain a semantic direction vector.

[0063] Diverse image generation: Users can adjust the range of interpolation coefficients (e.g., [-3,3]), and the system will linearly interpolate along the semantic direction to generate 128 images, calculate FID, LPIPS, and CLIP similarity indices, filter high-quality generated results and display them, and support users to export the images they need.

Claims

1. A few-shot image generation method based on category-level latent representation semantic editing, characterized in that, Includes the following steps: Step S1: Pre-trained model loading and image inverse encoding: Load the pre-trained pSp encoder and StyleGAN2 generator, and inverse encode a small number of target category image samples into the StyleGAN2 generator using the pSp encoder. The latent space is used to obtain instance-level latent encodings. The StyleGAN2 generator then reconstructs the image based on these encodings, thus completing the mapping and alignment between the image and the latent space. Step S2: Category-level latent representation learning: Introduce a lightweight latent adapter, use an episode-style metric learning strategy to construct category-aware constraints, guide instance-level encodings to aggregate towards a category-level shared distribution, obtain category-level latent encodings, and reconstruct category-aligned images; Step S3: Region-constrained CLIP-guided semantic direction discovery: Target region masks are automatically generated through SAM and high-confidence masks are selected. Guided by the target text description, a mask constraint loss function is constructed by combining CLIP's cross-modal capabilities to optimize and obtain semantic directions that focus on the target region. Step S4: Joint Loss Training Optimization and Semantic Interpolation Generation: Fix the parameters of the pSp encoder and StyleGAN2 generator, optimize only the latent adapter and metric learning model, introduce perceptual loss and adversarial loss to construct a joint loss function, and linearly interpolate along the optimized semantic direction to generate diverse and structurally consistent images.

2. The method according to claim 1, characterized in that, The episode-based training paradigm in the category-level latent representation learning, where each episode contains... There are 10 categories, each category contains 100 items. The support set and contents of a real image sample A query set of generated image samples is used to construct a mean squared loss function by outputting sample category classification scores through a metric learning model. Implement category aggregation constraints to ensure that samples of the same type form consistent clusters in the latent space.

3. The method according to claim 1, characterized in that, The implementation of the region constraint is as follows: the category-aligned image is segmented using the SegmentAnything Model (SAM) to obtain a high-confidence region mask candidate set, and the target region mask is selected based on the area heuristic rule. Semantic direction optimization is limited to the target region corresponding to the mask to avoid background interference.

4. The method according to claim 1, characterized in that, The joint loss function includes category-aware loss. VGG perceived loss and the loss of combat The formula for potential adapter loss is: The formula for measuring model loss is: Reduce training costs by fixing the parameters of the pre-trained generator.

5. The method according to claim 1, characterized in that, The lightweight latent adapter adopts a 4-layer MLP structure with input and output dimensions of 18×512, activation function of ReLU, and dropout probability of 0.1, achieving efficient conversion from instance-level encoding to class-level encoding while avoiding overfitting.

6. A few-shot image generation system based on category-level latent representation semantic editing, characterized in that, A top-down, layered, modular architecture is adopted, including an application layer, a business layer, and a data layer. Each layer works together to implement the function of any one of the methods described in claims 1-5. The business layer is the core processing layer, which includes a pre-trained model loading module, a category-level representation learning module, a semantic editing and generation module, and a model training optimization module.

7. The system according to claim 6, characterized in that, The pre-trained model loading module is responsible for loading the pre-trained weights of the pSp encoder, StyleGAN2 generator, CLIP model, and SAM model, initializing the parameters, and keeping the pre-trained generator parameters fixed. The semantic editing and generation module works in conjunction with the SAM mask generation unit and the CLIP semantic guidance unit to complete the entire process of mask generation, semantic direction optimization, and interpolation generation.

8. The system according to claim 6, characterized in that, The application layer supports text description input, interpolation coefficient adjustment, and mask area visualization. The data layer uses a MySQL database to store structured data and a local file system to store unstructured data. Data security and access efficiency are ensured through encryption, backup, and caching mechanisms.

Images