A data-free knowledge distillation method based on latent space modeling

Abstract

The application discloses a kind of data-free knowledge distillation methods based on latent space modeling, including S1, obtaining teacher model, student model and target category set, under the condition of not accessing original training data, category conditional text prompt is constructed to each target category, and corresponding category semantic embedding is extracted by frozen text encoder;S2, the category semantic embedding is input into latent variable generator, after projection mapping, noise layer disturbance and convolution generation processing, output target latent variable tensor and the like steps.The application can significantly reduce the inverse mapping dimension of high-dimensional pixel space from discriminant signal under high-resolution scene, fundamentally alleviate the ill-posed problem of inversion, effectively avoid semantic drift, texture artifact and mode collapse, improve the authenticity and diversity of pseudo samples.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, model compression and image generation technology, and particularly relates to a data-free knowledge distillation method based on latent space modeling. Background Technology

[0002] Knowledge distillation is an effective model compression technique that significantly reduces model parameters, inference latency, and deployment costs while maintaining high performance by transferring knowledge from a large teacher model to a small student model. Traditional knowledge distillation methods typically rely on raw training data as the medium for knowledge transfer, with the student model learning hidden knowledge by mimicking the output distribution of the teacher model on real samples. However, in many practical applications, raw training data is often unavailable due to privacy regulations (such as GDPR), commercial copyright restrictions, data sensitivity, or the inability to revert data (e.g., data cannot be uploaded from edge devices). This makes traditional distillation methods that rely on raw data unsuitable, leading to the emergence of data-free knowledge distillation techniques as a research hotspot.

[0003] Existing data-free knowledge distillation methods can be broadly categorized into two types. One type directly synthesizes pseudo-samples in pixel space based on generative adversarial networks or variational autoencoders, and then uses signals such as batch normalized statistics, class prediction confidence, and prediction differences between teacher and student models stored in the teacher model to retrieve training samples. This type of method has achieved certain results in low-resolution tasks, but in high-resolution image classification scenarios, the inverse problem of retrieving high-dimensional pixel space from low-dimensional discrimination signals suffers from severe ill-posedness. This easily leads to problems such as semantic drift, texture artifacts, pattern collapse, and loss of class diversity in pseudo-samples, which in turn affect the stability of the distillation process and the accuracy of the final student model.

[0004] Another approach utilizes a pre-trained diffusion model to generate pseudo-samples, which are then used for distillation training. This type of approach typically employs a two-stage process: first, a large number of pseudo-samples are generated independently, and then distillation is performed. This decoupling approach results in a lack of dynamic collaboration between the generation and distillation objectives. The generation process struggles to adaptively adjust the sample distribution based on the student model's current learning state, and the stepwise sampling inference cost of the diffusion model is high, leading to overall low training efficiency. Furthermore, existing techniques largely focus on the visual realism of the pseudo-samples, paying less attention to whether they can consistently expose the knowledge differences between the teacher and student models. If the generated samples only satisfy visual realism but lack information near the class decision boundaries, the student model still struggles to fully learn the implicit knowledge contained in the teacher model. This problem is particularly pronounced in high-resolution, fine-grained classification tasks with a large number of classes and subtle inter-class differences.

[0005] Therefore, how to construct an efficient, stable, and dynamically adaptable data-free knowledge distillation method that can simultaneously consider the natural image priors of pseudo-samples, category semantic consistency, latent variable distribution stability, and the effectiveness of distillation training without accessing the original training data has become a pressing technical challenge in this field. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing data-free knowledge distillation methods in high-resolution scenarios, such as poor pseudo-sample authenticity, weak semantic consistency, and difficulty in continuously exposing knowledge differences between models when directly inverting samples in pixel space. This invention provides a data-free knowledge distillation method based on latent space modeling.

[0007] To achieve the above objectives, the present invention employs the following technical solution: a data-free knowledge distillation method based on latent space modeling, comprising the following steps: S1. Obtain the teacher model, student model, and target category set. Without accessing the original training data, construct category conditional text prompts for each target category and extract the corresponding category semantic embeddings through a frozen text encoder. S2. Embed the category semantics into the input latent variable generator, and after projection mapping, noise layer perturbation and convolution generation processing, output the target latent variable tensor. S3. Input the target latent variable tensor into the frozen variational autoencoder decoder to decode and obtain the pseudo-sample image; S4. Input the pseudo-sample images into the teacher model and the student model respectively to construct the inversion loss; S5. Construct fractional distillation sampling semantic guidance loss and latent variable distribution regularization loss based on the target latent variable tensor, and combine fractional distillation sampling semantic guidance loss and latent variable distribution regularization loss with the inversion loss as the optimization objective to update the latent variable generator; S6. Continuously generate pseudo-sample images using the optimized latent variable generator, and use the output of the teacher model on the pseudo-sample images as a supervision signal to train the student model through knowledge distillation.

[0008] Furthermore, the "constructing category conditional text prompts" in S1 specifically refers to: generating category conditional text prompts using natural language templates containing category semantic names, and outputting the corresponding text semantic embedding vectors using a frozen cross-modal text encoding network; for the same category, using multiple different text templates and averaging, concatenating, or weighting the semantic embeddings they output to obtain the final conditional vector for that category.

[0009] The "latent variable generator" in step S2 includes: a projection module for mapping the category semantic embedding to a high-dimensional feature space; a noise layer for introducing learnable or random perturbations on the mapped features to enhance generation diversity; and a convolutional generation module for outputting the target latent variable tensor based on the perturbated features. The convolutional generation module includes an upsampling layer, a convolutional layer, a batch normalization layer, and a nonlinear activation layer, and the output does not have an activation function that forcibly truncates the output range.

[0010] During the training phase, the noise layer applies perturbations to the mapped features that satisfy a preset distribution or adjustable intensity, and is jointly trained with the subsequent normalization layer, upsampling layer and convolutional layer, so that semantic embeddings of the same category can form different latent variable responses while maintaining category consistency.

[0011] The "variational autoencoder decoder" mentioned in step S3 is used to decode the target latent variable tensor into a pseudo-sample image of the first resolution, and then adjust it to the input resolution required by the teacher model and student model through interpolation scaling to obtain a pseudo-sample image for knowledge distillation.

[0012] The "construction of fractional distillation sampling semantic guidance loss" in step S5 includes the following steps: S51. Apply Gaussian noise perturbation to the target latent variable tensor and obtain the perturbation latent variable according to the diffusion time step of random sampling; S52. Embed the perturbation latent variables and category semantics into the input-frozen diffusion denoising network to obtain the predicted noise components; S53. Construct a semantically guided gradient based on the difference between the predicted noise component and the actual noise component, and guide the target latent variable tensor to approach the high-density region of the natural image manifold.

[0013] The "latest variable distribution regularization loss" mentioned in step S5 refers to the result obtained by statistically analyzing the mean and variance of each dimension of the mini-batch latent variable tensor and calculating its KL divergence with the standard normal distribution.

[0014] In step S6, both the "latent variable generator" and the "student model" are updated alternately at a preset frequency. The update steps are as follows: within a training round, the student model is fixed first, and the latent variable generator is updated several times according to the joint optimization objective. Then, the latent variable generator is fixed again, and the student model is updated using newly generated pseudo-sample images. Alternatively, the number of updates of the latent variable generator is adaptively adjusted according to the prediction difference between the student model and the teacher model, so that the generated pseudo-sample distribution continuously covers the area that the student model has not yet fully learned.

[0015] The “inversion loss” mentioned in step S4 includes at least batch normalized statistical constraint loss, category prior constraint loss, and prediction difference constraint loss between the teacher model and the student model.

[0016] In step S1, both the teacher model and the student model are image classification models. The target category set is determined by the output category space of the teacher model or by a partial category set specified by the application side. The frozen text encoder is the text encoding branch in the cross-modal pre-trained model.

[0017] Compared with the prior art, the beneficial effects of the present invention are: (1) This invention transfers the sample inversion process from the pixel space to the latent space, uses a latent variable generator to output a compact latent variable tensor, and uses a pre-trained variational autoencoder decoder to decode it into a pseudo sample image. Compared with the scheme that optimizes directly in the pixel space, it significantly reduces the inverse mapping dimension of the high-dimensional pixel space inversion from the discriminant signal in high-resolution scenes, fundamentally alleviates the ill-posedness problem of inversion, effectively avoids semantic drift, texture artifacts and mode collapse, and improves the authenticity and diversity of pseudo samples.

[0018] (2) The present invention uses a pre-trained variational autoencoder decoder as a frozen decoder. It uses the decoding prior learned on large-scale natural images to map the latent variable tensor into pseudo sample images with natural texture and structural consistency. At the same time, it expands the exploration range of the latent space through the controllable perturbation of the noise layer. Compared with the existing pixel-level generation scheme, it reduces the degree of freedom to be optimized and reduces the training difficulty. It also makes the generated pseudo samples have better visual realism and intra-category diversity, thus overcoming the defect of insufficient pseudo sample quality in the existing technology.

[0019] (3) This invention introduces fractional distillation sampling semantic guidance loss and latent variable distribution regularization loss. The former uses the frozen diffusion denoising network to provide gradient constraints on the direction of the natural image manifold, so that the latent variable tensor gradually approaches the high-density region of the natural image. The latter constrains the latent variable output distribution to align with the standard Gaussian prior of the variational autoencoder. The two work together to constrain the generation process from the two levels of semantic manifold and statistical distribution, respectively, effectively reducing the semantic drift and artifact risk of pseudo samples, ensuring the class consistency and generation stability of pseudo samples, and solving the problems of weak semantic consistency and unstable training of pseudo samples in the prior art.

[0020] (4) This invention limits the diffusion model to a semantically guided regularizer rather than a complete data generation engine, avoiding the high inference overhead of stepwise reverse diffusion sampling. At the same time, through the alternating update strategy of the latent variable generator and the student model, the distribution of pseudo samples can be continuously and dynamically adjusted according to the current learning state of the student model, ensuring that the generated pseudo samples continuously cover the discriminative boundary region where there is a knowledge difference between the teacher model and the student model, fully exposing the teacher model's dark knowledge. This overcomes the shortcomings of the existing two-stage method, such as low coupling between the generation target and the distillation target and the lack of discriminative information in the pseudo samples. It significantly improves the effectiveness of distillation training and the accuracy of the final student model without accessing the original training data. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall process of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example

[0023] like Figure 1 As shown, the dataless knowledge distillation method based on latent space modeling described in this embodiment is applicable to high-resolution image classification model distillation scenarios. It can be deployed on servers, workstations, or computing devices with GPU or AI acceleration capabilities, or on training platforms composed of multiple computing nodes.

[0024] The method in this embodiment includes six steps, S1 to S6, which are as follows: S1. Obtain the teacher model, student model, and target category set. Without accessing the original training data, construct category conditional text prompts for each target category and extract the corresponding category semantic embeddings through a frozen text encoder.

[0025] In this embodiment, the teacher model needs to be set to evaluation mode and its parameters frozen to prevent performance drift without original data; simultaneously, the student model needs to be set to training mode to update its parameters based on distillation loss. For the target category set, each category in the target category set needs to be... Generate one or more category-based conditional text hints, and input the text hints into the frozen text encoder to obtain the category semantic embedding. If multiple prompt templates correspond to the same category, the multiple semantic embeddings can be averaged or weighted and fused to obtain the final condition vector for that category.

[0026] That is, the "construction of category conditional text hints" mentioned in this step specifically refers to generating category conditional text hints using natural language templates containing category semantic names, and outputting the corresponding text semantic embedding vectors using a frozen cross-modal text encoding network; for the same category, multiple different text templates are used and their output semantic embeddings are averaged, concatenated, or weighted to obtain the final conditional vector for that category.

[0027] To ensure operational effectiveness, both the teacher model and the student model are image classification models. The target category set is determined by the output category space of the teacher model or by a partial category set specified by the application side. The frozen text encoder is the text encoding branch in the cross-modal pre-trained model.

[0028] S2. Embed the category semantics into the input latent variable generator, and after projection mapping, noise layer perturbation and convolution generation processing, output the target latent variable tensor.

[0029] The latent variable generator includes: a projection module for mapping the category semantic embedding to a high-dimensional feature space; a noise layer for introducing learnable or random perturbations into the mapped features to enhance generation diversity; and a convolutional generation module for outputting the target latent variable tensor based on the perturbated features. The convolutional generation module includes an upsampling layer, a convolutional layer, a batch normalization layer, and a nonlinear activation layer, and its output does not have an activation function that forcibly truncates the output range.

[0030] During runtime, the latent variable generator first performs high-dimensional mapping of the text semantic embedding through the projection module, and perturbs the mapped features through the noise layer to improve the latent space exploration capability; then, it performs multi-layer transformations on the features through the convolution generation module to output the latent variable tensor.

[0031] The noise layer perturbs the mapping features. Specifically, during the training phase, the noise layer applies perturbations to the mapping features that satisfy a preset distribution or adjustable intensity. It is then trained in conjunction with subsequent normalization layers, upsampling layers, and convolutional layers, so that semantic embeddings of the same category can form different latent variable responses while maintaining category consistency.

[0032] S3. Input the target latent variable tensor into the frozen variational autoencoder decoder to decode and obtain the pseudo-sample image.

[0033] The variational autoencoder decoder's role is to decode the target latent variable tensor into a pseudo-sample image of the first resolution, and then adjust it to the input resolution required by the teacher and student models through interpolation and scaling, thus obtaining the pseudo-sample image used for knowledge distillation. Specifically, the target latent variable tensor is input into the frozen variational autoencoder decoder to obtain the pseudo-sample image of the first resolution; then, bilinear interpolation is used to adjust the first-resolution pseudo-sample image to the resolution required for the distillation task, resulting in the target pseudo-sample image input to the teacher and student models.

[0034] S4. Input the pseudo-sample images into the teacher model and the student model respectively to construct the inversion loss. The inversion loss is a composite loss function composed of the weighted sum of the batch normalized statistical constraint loss, the category prior constraint loss, and the prediction difference constraint loss.

[0035] Among them, the batch normalized statistical constraint loss is used to ensure that the mean and variance of the pseudo-sample images in each batch normalization layer of the teacher model are consistent with the pre-stored statistics.

[0036] The category prior constraint loss is used to constrain the teacher model to predict the same category as the target category for pseudo-sample images.

[0037] Predictive difference constraint loss is used to construct hard samples by the output differences between the teacher model and the student model on pseudo-sample images.

[0038] S5. Construct fractional distillation sampling semantic guidance loss and latent variable distribution regularization loss based on the target latent variable tensor, and combine fractional distillation sampling semantic guidance loss and latent variable distribution regularization loss with the inversion loss as the optimization objective to update the latent variable generator.

[0039] The core purpose of constructing the fractional distillation sampling semantic guided loss is to inject the "natural image manifold prior knowledge" contained in the pre-trained diffusion model into the latent variable generator with extremely low computational cost, thereby ensuring that the decoded pseudo samples are visually real and semantically clear, rather than distorted noise.

[0040] The specific steps for constructing the fractional distillation sampling semantic guided loss are as follows: S51. Apply Gaussian noise perturbation to the target latent variable tensor and obtain the perturbation latent variable according to the diffusion time step of random sampling.

[0041] S52. Embed the perturbation latent variables and category semantics into the input-frozen diffusion denoising network to obtain the predicted noise components.

[0042] S53. Construct a semantically guided gradient based on the difference between the predicted noise component and the actual noise component, and guide the target latent variable tensor to approach the high-density region of the natural image manifold.

[0043] The latent variable distribution regularization loss is obtained by statistically analyzing the mean and variance of each dimension of the mini-batch latent variable tensor and calculating its KL divergence with the standard normal distribution.

[0044] S6. Continuously generate pseudo-sample images using the optimized latent variable generator, and use the output of the teacher model on the pseudo-sample images as a supervision signal to train the student model through knowledge distillation.

[0045] The purpose of this step is to use the "high-quality pseudo-samples" generated by the generator as a carrier to transfer the "hidden knowledge" of the teacher model to the student model, ultimately achieving model compression that does not depend on the original data.

[0046] The "latent variable generator" and "student model" are updated alternately at a preset frequency. The update steps are as follows: within a training round, the student model is fixed first, and the latent variable generator is updated several times according to the joint optimization objective. Then, the latent variable generator is fixed again, and the student model is updated using newly generated pseudo-sample images. Alternatively, the number of updates of the latent variable generator is adaptively adjusted according to the prediction difference between the student model and the teacher model, so that the distribution of generated pseudo-samples continuously covers the areas that the student model has not yet fully learned.

[0047] The phrase "adaptively adjusting the number of updates of the latent variable generator based on the prediction difference between the student model and the teacher model" is essentially a dynamic learning scheduling strategy. Its processing logic is as follows: using the prediction difference between the teacher and student models as a feedback signal, the latent variable generator's optimization steps in each iteration are adjusted in real time, so that the latent variable generator can concentrate its computing power on mining the current weaknesses of the student model, thereby solving the problem of "non-steady-state data supply" in data distillation.

[0048] Under the existing alternating training framework, the conventional approach is to fix the student model and update the latent variable generator K times (K is a fixed constant). However, the adaptive adjustment in this embodiment makes this constant dynamic, which can be achieved by using the threshold triggering method or the gradient scaling method.

[0049] The threshold triggering method refers to the following: if the average difference value of the current batch exceeds a preset high threshold, it indicates that the area is a student's "knowledge blind spot". The number of updates of the latent variable generator is automatically increased (e.g., from K=1 to K=3), forcing the latent variable generator to fully optimize and produce more samples for the blind spot. If it is below the low threshold, the number of updates is reduced to save computing power.

[0050] The gradient scaling method refers to mapping the difference value to an adjustment coefficient α, and setting the current round of latent variable generator updates Kt = Kbase × α. The larger the difference, the larger α is, and the more iterations the latent variable generator will linger at this point.

[0051] As described above, the present invention can be well implemented.

Claims

1. A data-free knowledge distillation method based on latent space modeling, characterized in that, Includes the following steps: S1. Obtain the teacher model, student model, and target category set. Without accessing the original training data, construct category conditional text prompts for each target category and extract the corresponding category semantic embeddings through a frozen text encoder. S2. Embed the category semantics into the input latent variable generator, and after projection mapping, noise layer perturbation and convolution generation processing, output the target latent variable tensor. S3. Input the target latent variable tensor into the frozen variational autoencoder decoder to decode and obtain the pseudo-sample image; S4. Input the pseudo-sample images into the teacher model and the student model respectively to construct the inversion loss; S5. Construct fractional distillation sampling semantic guidance loss and latent variable distribution regularization loss based on the target latent variable tensor, and combine fractional distillation sampling semantic guidance loss and latent variable distribution regularization loss with the inversion loss as the optimization objective to update the latent variable generator; S6. Continuously generate pseudo-sample images using the optimized latent variable generator, and use the output of the teacher model on the pseudo-sample images as a supervision signal to train the student model through knowledge distillation.

2. The data-free knowledge distillation method based on latent space modeling according to claim 1, characterized in that, The "constructing category conditional text hints" mentioned in S1 specifically refers to: generating category conditional text hints using a natural language template containing category semantic names, and outputting the corresponding text semantic embedding vector using a frozen cross-modal text encoding network; for the same category, using multiple different text templates and averaging, concatenating, or weighting the semantic embeddings they output to obtain the final conditional vector for that category.

3. The data-free knowledge distillation method based on latent space modeling according to claim 2, characterized in that, The "latent variable generator" in step S2 includes: The projection module is used to map the category semantic embedding to a high-dimensional feature space; A noise layer is used to introduce learnable or random perturbations into the mapped features to enhance generative diversity. The convolution generation module is used to output the target latent variable tensor based on the perturbed features; wherein, the convolution generation module includes an upsampling layer, a convolutional layer, a batch normalization layer and a nonlinear activation layer, and the output end does not have an activation function that forcibly truncates the output range.

4. The data-free knowledge distillation method based on latent space modeling according to claim 3, characterized in that, During the training phase, the noise layer applies perturbations to the mapped features that satisfy a preset distribution or adjustable intensity, and is jointly trained with the subsequent normalization layer, upsampling layer and convolutional layer, so that semantic embeddings of the same category can form different latent variable responses while maintaining category consistency.

5. The data-free knowledge distillation method based on latent space modeling according to claim 1, characterized in that, The "variational autoencoder decoder" in step S3 is used to decode the target latent variable tensor into a pseudo-sample image of the first resolution, and then adjust it to the input resolution required by the teacher model and student model through interpolation and scaling to obtain a pseudo-sample image for knowledge distillation.

6. The data-free knowledge distillation method based on latent space modeling according to claim 1, characterized in that, The "construction of fractional distillation sampling semantic guidance loss" in step S5 includes the following steps: S51. Apply Gaussian noise perturbation to the target latent variable tensor and obtain the perturbation latent variable according to the diffusion time step of random sampling; S52. Embed the perturbation latent variables and category semantics into the input-frozen diffusion denoising network to obtain the predicted noise components; S53. Construct a semantically guided gradient based on the difference between the predicted noise component and the actual noise component, and guide the target latent variable tensor to approach the high-density region of the natural image manifold.

7. The data-free knowledge distillation method based on latent space modeling according to claim 6, characterized in that, The "latest variable distribution regularization loss" mentioned in step S5 refers to the result obtained by statistically analyzing the mean and variance of each dimension of the mini-batch latent variable tensor and calculating its KL divergence with the standard normal distribution.

8. The data-free knowledge distillation method based on latent space modeling according to claim 1, characterized in that, In step S6, both the "latent variable generator" and the "student model" are updated alternately at a preset frequency. The update steps are as follows: within a training round, the student model is fixed first, and the latent variable generator is updated several times according to the joint optimization objective. Then, the latent variable generator is fixed again, and the student model is updated using newly generated pseudo-sample images. Alternatively, the number of updates of the latent variable generator is adaptively adjusted according to the prediction difference between the student model and the teacher model, so that the generated pseudo-sample distribution continuously covers the area that the student model has not yet fully learned.

9. The data-free knowledge distillation method based on latent space modeling according to claim 1, characterized in that, The "inversion loss" mentioned in step S4 includes at least batch normalized statistical constraint loss, category prior constraint loss, and prediction difference constraint loss between the teacher model and the student model.

10. A data-free knowledge distillation method based on latent space modeling according to any one of claims 1 to 9, characterized in that, In step S1, both the teacher model and the student model are image classification models. The target category set is determined by the output category space of the teacher model or by a partial category set specified by the application side. The frozen text encoder is the text encoding branch in the cross-modal pre-trained model.

Images