Video dataset compression method based on differentiable pruning and latent diffusion distillation

Abstract

The application relates to the technical field of computer vision, and discloses a video dataset compression method based on differential pruning and latent diffusion distillation. First, the video dataset is preprocessed to obtain feature space encoding and space-time feature vectors of samples. Then, through a differential pruning module, a learnable importance parameter is set for each sample, and an end-to-end collaborative optimization of pruning and distillation is realized in combination with a simulated annealing mechanism. Then, through a latent diffusion distillation module, hierarchical time sequence modeling and global time sequence alignment are adopted to process video frame time sequence redundancy, and an intermediate synthetic video dataset is generated. Finally, the sample importance parameter is updated by using a meta-learning differential strategy. The application not only realizes adaptive perception of pruning decision requirements of downstream generation models and classifiers, avoids gradient loss accumulation, but also decouples the time sequence redundancy within and between video samples, generates a synthetic dataset with high compression ratio and high quality, and reduces the storage and model training cost of video data.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a video dataset compression method based on differentiable pruning and latent diffusion distillation. Background Technology

[0002] With the explosive growth of applications such as short videos, video surveillance, and autonomous driving, the amount of video data is increasing exponentially. Training high-performance video action recognition models (such as C3D, I3D, and SlowFast) typically requires massive amounts of labeled video data, which not only leads to high storage costs but also results in excessively long model training times and huge consumption of computing resources. Especially for edge computing scenarios with high real-time requirements, the transmission, storage, and repeated training of large-scale video datasets have become key bottlenecks restricting the deployment of technologies. As a result, dataset compression technology has emerged, aiming to synthesize a small, high-information-density synthetic dataset based on key information from the original large-scale data. This allows models trained on the synthetic dataset to achieve performance close to that trained on the original dataset, thereby significantly reducing data storage overhead and model training costs, and providing a feasible path for the lightweight deployment of video understanding models.

[0003] Current dataset compression methods are mainly divided into two categories: dataset pruning and dataset distillation. Dataset pruning methods select a core subset from the original dataset by designing heuristic criteria (such as diversity and representativeness), but this method discards a large number of samples, which may lead to the loss of some key information. Moreover, its pruning decisions are not only usually unlearnable, but also unable to perceive the needs of downstream tasks. Dataset distillation methods learn how to generate a small batch of high-information-density synthetic samples, but existing video distillation methods either rely too much on single-frame discrimination capabilities and have limited modeling of fine-grained motion patterns; or heavily depend on generator quality and introduce additional storage overhead; or are difficult to scale to large-scale datasets due to high computational costs.

[0004] More importantly, existing methods typically separate pruning and distillation into two independent domains or two independent stages within a single method: first, a subset is selected based on diversity or representativeness criteria, and then the generative model is trained using this subset to synthesize new samples. However, this two-stage process has fundamental optimization limitations: 1) The pruning stage cannot perceive the needs of the subsequent generation model and classifier, which may cause the selected subset to be biased towards certain patterns and ignore samples that are crucial to the generation task; sample selection is a discrete operation and is not differentiable, which makes the entire compression process lack adaptive adjustment capability. 2) The inherent temporal continuity of video data (high similarity between frames) and the overlap of content and motion patterns between different video samples further exacerbate the problem of information redundancy. How to effectively decouple and eliminate temporal redundancy in the collaborative process of pruning and distillation is also a major challenge.

[0005] Therefore, integrating the pruning process into a differentiable learning framework to achieve joint optimization of pruning and distillation is key to improving dataset compression performance. Furthermore, decoupling intra- and inter-sample temporal redundancy during the collaborative process of pruning and distillation—preserving crucial temporal dynamic information to support downstream tasks while eliminating redundancy to improve compression ratio and the diversity of synthesized data—is crucial for achieving efficient video dataset compression. Based on these considerations, there is an urgent need to design a dataset compression method that can achieve end-to-end collaborative optimization of pruning and distillation while effectively handling temporal redundancy in video. Summary of the Invention

[0006] To achieve the above objectives, this invention addresses the aforementioned problems by providing a video dataset compression method based on differentiable pruning and latent diffusion distillation. This method can achieve joint optimization of pruning and distillation, enabling pruning decisions to be aware of the needs of downstream generative models and classifiers, while also decoupling temporal redundancy within and between video samples, thereby generating high-compression-ratio, high-quality synthetic video datasets.

[0007] To achieve the above-mentioned objectives, this invention employs the following technical solution: a video dataset compression method based on differentiable pruning and latent diffusion distillation, comprising the following steps: Step 1: Preprocess the original video dataset by using a pre-trained encoder and a 3D convolutional neural network to extract feature codes and spatiotemporal feature vectors from the video samples, respectively. Step 2: Construct a differentiable pruning module. The input is the feature encoding and spatiotemporal feature vector of the original dataset samples, and the output is the pruned core subset and its corresponding sample weights. Step 3: Construct a potential diffusion distillation module, with the pruned core subset and sample weights as inputs and the intermediate synthetic video dataset as output; Step 4: Calculate the pruning-distillation collaborative loss, update the sample weights using the expansion differential strategy in meta-learning, iteratively optimize the pruning and distillation process until the pruning-distillation collaborative loss converges, and obtain the final synthetic video dataset.

[0008] Furthermore, step 1 specifically includes the following steps: Step 11: Process the original video dataset Each video sample ( For the first Category labeling of each video sample (the number of original videos), in seconds Frame (e.g.) Take 15 samples to obtain the video frame sequence. ,in This represents the total number of frames after sampling. and The height and width of the original video frame, where 3 represents the RGB three channels; Step 12: Sample the video frame sequence The input is fed into a 3D encoder (such as a 3D variational autoencoder (3D-VAE), preferably HunyuanVideo-1.5 VAE) pre-trained on a large video dataset to obtain the corresponding sample feature tensor. ,in, and For the spatial dimension of the potential tensor, The number of channels yields the sample feature set. ; Step 13: In the original video dataset Pre-train a 3D convolutional neural network classification model (such as C3D, I3D, etc.), and use its feature extraction layer as a spatiotemporal feature extractor (the spatiotemporal feature extractor can be pre-trained on a large dataset (such as Kinetics-400) and then fine-tuned on the target task dataset); for each video sample Overlay them front and back The frame is divided into For each segment, the corresponding spatiotemporal feature vector is extracted using the obtained spatiotemporal feature extractor. Then, the average of the spatiotemporal feature vectors of all segments is used to obtain the video sample. Corresponding spatiotemporal feature vector ,in For feature dimensions.

[0009] Furthermore, step 2 specifically includes the following steps: Step 21: Construct a differentiable pruning module for each video sample. Initialize a learnable importance parameter The initial value is set to 1; the softmax operation within the category is used to... Convert to sample weights : , in, ; or It is the original video sample. or The corresponding label (or tag); this normalization ensures that the weights within each category sum to 1, facilitating the subsequent generation of a fixed number of synthetic samples by category; the simulated annealing idea used here is: during the high-temperature stage ( Large scale, smooth softmax output, uniform weight distribution, rich gradient information, and optimization algorithm can fully explore the entire sample space; in the low temperature stage ( (Small), the softmax output tends to be sharp, and the weights gradually concentrate on a few samples to make full use of them; through the gradual cooling process, the gradient vanishing caused by premature discretization can be avoided, as well as the failure to converge caused by always being uniformly distributed; Step 22: Within each category, based on sample weights The sample feature tensor in the feature set Sort in descending order of weight value, based on the preset pruning rate. Remove the parts with lower weights and use the remaining tensors to form the pruned sample feature set. ,in This represents the number of feature tensors for the pruned samples.

[0010] Step 2 constructs a differentiable pruning module, sets learnable importance parameters for each original sample, and uses a simulated annealing temperature control mechanism to make the discrete selection of samples by the dataset pruning method continuous, thus achieving synergistic optimization of pruning and distillation.

[0011] Furthermore, step 3 specifically includes the following steps: Step 31: The potential diffusion distillation module introduces hierarchical temporal modeling inside the Transformer block: it uses sparse causal attention to maintain local temporal dependencies at the token level, and combines a State Space Model (SSM) layer to achieve long-range memory with linear complexity. Specifically, for a clean feature tensor sequence in the feature space ( frame) (here) Equivalent to ), diffusion time step Standard Gaussian noise ( ), length of the random sampled noisy prefix The number of frames generated per autoregressive block During training, the prefix length is randomly selected, let (clean), (To be noise-added); Add noise to the part to be noise-added. : , in, For the first The clean feature tensor of the frame; The noise dispatch coefficient is the noise control factor for the forward diffusion process. Final noise prediction network (parameter The input is The time step vector is set to 0 in the clean part; Step 32: Through global temporal alignment, the latent diffusion distillation module uses a global encoder to extract a fixed number of global semantic tokens from historical frames during the generation of each block, and injects them into each denoising step of the current frame block, thereby enabling the noise prediction network to... Gain the ability to perceive the temporal context of the entire video; Specifically, for clean prefixes Spatiotemporal average pooling is performed to obtain an intermediate representation, and then the learnable global token vector is updated through cross-attention. : , in, As a global condition for the current denoising step Initialization global conditions; Step 33: For the attention mechanism, causal sparse attention is adopted: the token sequence of the current block is divided into groups of size 1. The process involves first calculating coarse-grained causal attention among the cubes to select key cubes, and then calculating fine-grained causal token attention within the selected cubes. The calculation formula is as follows: Let the first cube be... The selected cube contains There are 1 token, and the token feature matrix is ​​as follows: ,in It is a noise prediction network The dimension; causal attention output is : , Among them, the query matrix Key matrix Value matrix ; , , All are learnable projection matrices; It is the dimension of key / query. It is the dimension of the value; It is a causal mask matrix, and its elements are defined as follows: ; Step 34: Set up the noise prediction network Total For each block, if the block index is odd, a causal sparse attention block is used, and its output is... If the block index is even, a State Space Model (SSM) block is used. SSM employs a block-based scanning method, dividing the space into independent blocks. Each block runs independently along the time dimension using Mamba2 (Mamba2 is a second-generation state space model that mathematically unifies the Transformer's attention mechanism with the SSM architecture, achieving stronger expressive power and higher training efficiency within this unified framework). Its output... With attention output Gated linear units are used here through gated fusion: , in, This represents element-wise multiplication; It is a learnable gating matrix, with dimension and The same, obtained from input features through linear transformation; In this step, either parallel hybrid or alternating stacking of sparse causal attention and SSM layers is acceptable. In the parallel hybrid approach, SSM and attention are complementary, with SSM capturing long-term trends and attention focusing on local details. The alternating stacking approach can reduce computational cost and flexibly adjust the proportion of long-range memory. Step 35: The loss function in the pre-training phase calculates the predicted noise loss only for the positions of the noisy frames: , in, for In the The entire video tensor after adding noise; For noise prediction network (parameters) ); For the indicator tensor, the position of the noisy frame is 1, and the position of the clean prefix is ​​0; Step 36: In the fine-tuning stage, based on the pruned sample feature set... The pre-trained diffusion model is fine-tuned by combining the current weights of the corresponding samples; the parameters of the first few layers of the pre-trained model are frozen, and only subsequent high-level parameters are allowed to be updated; the weighted reconstruction loss is defined as: , in, For the first Each sample at time step The noisy tensor (here) Different from They are at two different cutoff points, and it is impossible for them to influence each other. For the first Clean prefixes for each sample; during fine-tuning, the gradient of the reconstruction loss on the outer layer weight parameters is calculated every few steps, and the sample importance parameters are updated; Step 37: After fine-tuning, fix the pre-trained diffusion model and generate a preset number of instances per class (IPC) of synthesized videos for each category. During generation, a denoising diffusion implicit model (DDIM) sampling method is used (DDIM is a method to accelerate diffusion model sampling, which can generate high-quality samples with fewer steps through a non-Markov inference process) to gradually denoise from pure noise to obtain clean frames. An overlapping region is set between adjacent blocks, and a smooth transition is achieved through temporal position weighted fusion. Let the length of the overlapping region between two adjacent blocks be... Within the overlapping region, the fused frame is: , in, This indicates the temporal position of the frame within the overlapping region; These are the fused feature frames; For the current block's first Frame; then the fused feature frames replace the corresponding overlapping parts in the historical frames to form a smooth feature video sequence. The resulting feature video sequence is then fused. The input is fed into the decoder of the 3D variational autoencoder used in step 12 to reconstruct the video frame sequence in pixel space. Repeat the above ideas in this step to... Each category generates 10 synthetic samples (IPCs), which together form an intermediate synthetic video dataset. ( , For the first Class 1 (A composite video).

[0012] Step 3 involves constructing a latent diffusion distillation module, employing a strategy of pre-training on a large dataset and fine-tuning on a pruned core subset. This module follows an autoregressive causal generation paradigm (which means that when generating sequence data (such as video frames), the generation at each moment depends only on historical information generated before that moment, and not on future information). During training, a no-noise prefix strategy is adopted (which means that in the input sequence, a completely clean (no noise) historical frame is retained as a prefix, and noise is added only to subsequent frames), forcing the model to be conditioned on real historical frames, and achieving linear inference cost through Key-Value-Cache reuse.

[0013] Furthermore, step 4 specifically includes the following steps: Step 41: The differentiable pruning module described in Step 2 and the potential diffusion distillation module described in Step 3 are jointly optimized using the sample importance parameter to form an end-to-end video dataset compression process; Step 42: Using intermediate synthetic datasets Training classification model Calculate the classification loss on the test subset of the original dataset; classification loss A combination of average category penalty and worst category penalty is used: , in, This represents the total number of categories in the sample. For category Cross-entropy loss; The weight of the worst-case category penalty; The maximum cross-entropy loss is used across all categories. In this step, the classification model can use a 3D-CNN architecture (such as C3D, I3D, SlowFast, etc.), and its training epochs need to be coordinated with those of the generative model. Step 43: From the original dataset and intermediate synthetic datasets The data is sampled in batches, input into the current classification model, features of each layer are extracted, and the layer feature alignment loss is calculated. : , in, For the original video in the first The feature mean vector of the layer, Synthetic video in The feature mean vector of the layer, The number of channels is used; simultaneously, the feature centers of the synthesized samples are used to classify the original samples, and the discriminative loss is calculated. : , in, For the test set The number of samples in the middle; For category The feature centers of the synthetic samples, where ; Step 44: Based on the current sample weights and spatiotemporal eigenvectors (After dimensionality reduction using global average pooling), construct a weighted kernel matrix for each class. ,in Calculate the weighted diversity loss : , in The diagonal matrix formed by the sample weights The matrix determinant solution function is used; this loss encourages high-weight samples to be distributed in the feature space, ensuring the diversity of video samples within a category; Step 45: Pruning-Distillation Synergistic Loss Weighted reconstruction loss Classification loss Feature alignment loss Discriminative loss and diversity loss The composition, and the specific calculation formula are as follows: , in, , , ,and All are the weights of each loss term, and their values ​​range from [value range missing]. ; Step 46: Since the pruning-distillation collaborative loss requires backpropagation of gradients through the computational graphs of the generative and classification models, the unrolled differential strategy from meta-learning needs to be used to update the gradient. ;calculate Update the sample importance parameters using the stochastic gradient descent algorithm: , in, Let be the learning rate for the sample importance parameter, and In this step, to simplify the calculation, a first-order approximation (such as the Reptile style) can also be used to update the weight parameters; Step 47: After each outer layer iteration, anneal the temperature parameters according to the exponential decay strategy: ,in The attenuation coefficient is used; repeat steps 2 to 46 until the pruning-distillation synergistic loss occurs. Convergence or reaching the maximum number of iterations; Step 48: Finally, use the dataset obtained in the last round of pruning to train the weighted diffusion model until convergence, and then generate a synthetic dataset as the compression result based on IPC; thus, the final synthetic video dataset is obtained. This synthetic dataset can be directly used for training downstream task models.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention provides a video dataset compression method based on differentiable pruning and latent diffusion distillation. Utilizing a differentiable pruning module, by setting learnable importance parameters for each sample and combining this with a simulated annealing temperature control mechanism, the discrete sample selection is transformed into a continuous process for evaluating the importance of samples to downstream training, achieving end-to-end collaborative optimization of pruning and distillation. Through the differentiable collaborative optimization framework, pruning decisions can perceive the needs of downstream generative models and classifiers, avoiding the gradient loss accumulation of traditional two-stage methods.

[0015] 2. The present invention provides a video dataset compression method based on differentiable pruning and latent diffusion distillation. Utilizing a latent diffusion distillation module, it decouples intra- and inter-sample temporal redundancy through hierarchical temporal modeling (sparse causal attention + state space model) and global temporal alignment, generating a high-compression-ratio, high-quality synthetic video dataset. Through hierarchical temporal modeling and global temporal alignment, it eliminates inter-frame temporal redundancy while preserving key action semantics and dynamic information. Furthermore, through generative distillation of the feature space, it significantly reduces the storage and training costs of the original dataset. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the workflow of a video dataset compression method based on differentiable pruning and potential diffusion distillation according to the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of this invention.

[0018] Example 1 It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings.

[0020] Conversely, this invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of the invention as defined in the claims. Furthermore, to provide a better understanding of the invention, certain specific details are described in detail below. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0021] This embodiment provides a video dataset compression method based on differentiable pruning and latent diffusion distillation, such as... Figure 1 As shown, it includes the following steps: Step 1: Sample and preprocess the given original video dataset, and use a 3D variational autoencoder and a 3D convolutional neural network to extract the feature codes and spatiotemporal feature vectors of the video samples, respectively; Step 1 specifically includes: Step 11: Process the original video dataset Each video sample at a rate of per second Frames are sampled to obtain a video frame sequence. ,in For the first Category labeling of each video sample The number of original videos, This represents the total number of frames after sampling. and The height and width of the original video frame, where 3 represents the RGB three channels; Step 12: Sample the video frame sequence The input is fed into a 3D encoder pre-trained on a large video dataset to obtain the corresponding sample feature tensor. ,in, and For the spatial dimension of the potential tensor, The number of channels yields the sample feature set. In this embodiment, the three-dimensional encoder is a three-dimensional variational autoencoder (3D-VAE). Step 13: In the original video dataset Pre-train a 3D convolutional neural network classification model (such as C3D, I3D, etc.), and use its feature extraction layer as a spatiotemporal feature extractor (the spatiotemporal feature extractor can be pre-trained on a large dataset (such as Kinetics-400) and then fine-tuned on the target task dataset); for each video sample Overlay them front and back The frame is divided into For each segment, the corresponding spatiotemporal feature vector is extracted using the obtained spatiotemporal feature extractor. Then, the average of the spatiotemporal feature vectors of all segments is used to obtain the video sample. Corresponding spatiotemporal feature vector ,in For feature dimensions.

[0022] Step 2: By constructing a differentiable pruning module, the input is the feature encoding and spatiotemporal feature vector of the original dataset samples. Learnable importance parameters are set for each sample. Combined with the temperature control mechanism of simulated annealing, the core subset after pruning is obtained according to the sample weight and the preset pruning rate. Step 2 specifically includes: Step 21: Construct a differentiable pruning module for each video sample. Initialize a learnable importance parameter The initial value is set to 1; the softmax operation within the category is used to... Convert to sample weights : , in, ; or It is the original video sample. or The corresponding markers or labels; this normalization ensures that the weights within each category sum to 1, facilitating the subsequent generation of a fixed number of synthetic samples by category; the simulated annealing approach used here is: during the high-temperature stage ( Large scale, smooth softmax output, uniform weight distribution, rich gradient information, and optimization algorithm can fully explore the entire sample space; in the low temperature stage ( (Small), the softmax output tends to be sharp, and the weights gradually concentrate on a few samples to make full use of them; through the gradual cooling process, the gradient vanishing caused by premature discretization can be avoided, as well as the failure to converge caused by always being uniformly distributed; Step 22: Within each category, based on sample weights The sample feature tensor in the feature set Sort in descending order of weight value, based on the preset pruning rate. Remove the parts with lower weights and use the remaining tensors to form the pruned sample feature set. ,in This represents the number of feature tensors for the pruned samples.

[0023] Step 2 constructs a differentiable pruning module, sets learnable importance parameters for each original sample, and uses a simulated annealing temperature control mechanism to make the discrete selection of samples by the dataset pruning method continuous, thus achieving synergistic optimization of pruning and distillation.

[0024] Step 3: By constructing a latent diffusion distillation module, the input is the pruned core subset and sample weights. The pre-trained diffusion model is fine-tuned on the pruned core subset. The intermediate synthetic video dataset is generated by using hierarchical temporal modeling (sparse causal attention + state space model) and global temporal alignment to handle temporal redundancy. Step 3 specifically includes: Step 31: The potential diffusion distillation module introduces hierarchical temporal modeling inside the Transformer block: it uses sparse causal attention to maintain local temporal dependencies at the token level, and combines a State Space Model (SSM) layer to achieve long-range memory with linear complexity. Specifically, for a clean feature tensor sequence in the feature space ( frame) (here) Equivalent to ), diffusion time step Standard Gaussian noise ( ), length of the random sampled noisy prefix The number of frames generated per autoregressive block During training, the prefix length is randomly selected, let (clean), (To be noise-added); Add noise to the part to be noise-added. : , in, For the first The clean feature tensor of the frame; The noise dispatch coefficient is the noise control factor for the forward diffusion process. Final noise prediction network (parameter The input is The time step vector is set to 0 in the clean part; Step 32: Through global temporal alignment, the latent diffusion distillation module uses a global encoder to extract a fixed number of global semantic tokens from historical frames during the generation of each block, and injects them into each denoising step of the current frame block, thereby enabling the noise prediction network to... Gain the ability to perceive the temporal context of the entire video; Specifically, for clean prefixes Spatiotemporal average pooling is performed to obtain an intermediate representation, and then the learnable global token vector is updated through cross-attention. : , in, As a global condition for the current denoising step Initialization global conditions; Step 33: For the attention mechanism, causal sparse attention is adopted: the token sequence of the current block is divided into groups of size 1. The process involves first calculating coarse-grained causal attention among the cubes to select key cubes, and then calculating fine-grained causal token attention within the selected cubes. The calculation formula is as follows: Let the first cube be... The selected cube contains There are 1 token, and the token feature matrix is ​​as follows: ,in It is a noise prediction network The dimension; causal attention output is : , Among them, the query matrix Key matrix Value matrix ; , , All are learnable projection matrices; It is the dimension of key / query. It is the dimension of the value; It is a causal mask matrix, and its elements are defined as follows: ; Step 34: Set up the noise prediction network The common For each block, if the block index is odd, a causal sparse attention block is used, and its output is... If the block index is even, a State Space Model (SSM) block is used. SSM employs a block-based scanning method, dividing the space into independent blocks. Each block runs independently along the time dimension using Mamba2 (Mamba2 is a second-generation state space model that mathematically unifies the Transformer's attention mechanism with the SSM architecture, achieving stronger expressive power and higher training efficiency within this unified framework). Its output... With attention output Gated linear units are used here through gated fusion: , in, This represents element-wise multiplication; It is a learnable gating matrix, with dimension and The same, obtained from input features through linear transformation; In this step, either parallel hybrid or alternating stacking of sparse causal attention and SSM layers is acceptable. In the parallel hybrid approach, SSM and attention are complementary, with SSM capturing long-term trends and attention focusing on local details. The alternating stacking approach can reduce computational cost and flexibly adjust the proportion of long-range memory. Step 35: The loss function in the pre-training phase calculates the predicted noise loss only for the positions of the noisy frames: , in, for In the The entire video tensor after adding noise; For noise prediction network (parameters) ); For the indicator tensor, the position of the noisy frame is 1, and the position of the clean prefix is ​​0; Step 36: In the fine-tuning stage, based on the pruned sample feature set... The pre-trained diffusion model is fine-tuned by combining the current weights of the corresponding samples; the parameters of the first few layers of the pre-trained model are frozen, and only subsequent high-level parameters are allowed to be updated; the weighted reconstruction loss is defined as: , in, For the first Each sample at time step The noisy tensor (here) Different from They are at two different cutoff points, and it is impossible for them to influence each other. For the first Clean prefixes for each sample; during fine-tuning, the gradient of the reconstruction loss on the outer layer weight parameters is calculated every few steps, and the sample importance parameters are updated; Step 37: After fine-tuning, fix the pre-trained diffusion model and generate a preset number of instances per class (IPC) of synthesized videos for each category. During generation, a denoising diffusion implicit model (DDIM) sampling method is used (DDIM is a method to accelerate diffusion model sampling, which can generate high-quality samples with fewer steps through a non-Markov inference process) to gradually denoise from pure noise to obtain clean frames. An overlapping region is set between adjacent blocks, and a smooth transition is achieved through temporal position weighted fusion. Let the length of the overlapping region between two adjacent blocks be... Within the overlapping region, the fused frame is: , in, This indicates the temporal position of the frame within the overlapping region; These are the fused feature frames; For the current block's first Frame; then the fused feature frames replace the corresponding overlapping parts in the historical frames to form a smooth feature video sequence. The resulting feature video sequence is then fused. The input is fed into the decoder of the 3D variational autoencoder used in step 12 to reconstruct the video frame sequence in pixel space. Repeat the above ideas in this step to... Each category generates 10 synthetic samples (IPCs), which together form an intermediate synthetic video dataset. ( , For the first Class 1 (A composite video).

[0025] Step 3 involves constructing a latent diffusion distillation module, employing a strategy of pre-training on a large dataset and fine-tuning on a pruned core subset. This module follows an autoregressive causal generation paradigm (which means that when generating sequence data (such as video frames), the generation at each moment depends only on historical information generated before that moment, and not on future information). During training, a no-noise prefix strategy is adopted (which means that in the input sequence, a completely clean (no noise) historical frame is retained as a prefix, and noise is added only to subsequent frames), forcing the model to be conditioned on real historical frames, and achieving linear inference cost through Key-Value-Cache reuse.

[0026] Step 4: Construct a collaborative optimization loss function (including weighted reconstruction loss, classification loss, feature alignment loss, discriminative loss, and diversity loss), calculate the pruning-distillation collaborative loss, update the sample weights using the expansion differential strategy in meta-learning, iteratively optimize the pruning and distillation process until the pruning-distillation collaborative loss converges, and obtain the final synthetic video dataset.

[0027] Step 4 specifically includes: Step 41: The differentiable pruning module described in Step 2 and the potential diffusion distillation module described in Step 3 are jointly optimized using the sample importance parameter to form an end-to-end video dataset compression process; Step 42: Using intermediate synthetic datasets Training classification model Calculate the classification loss on the test subset of the original dataset; classification loss A combination of average category penalty and worst category penalty is used: , in, This represents the total number of categories in the sample. For category Cross-entropy loss; The weight of the worst-case category penalty; The maximum cross-entropy loss is used across all categories. In this step, the classification model can use a 3D-CNN architecture (such as C3D, I3D, SlowFast, etc.), and its training epochs need to be coordinated with those of the generative model. Step 43: From the original dataset and intermediate synthetic datasets The data is sampled in batches, input into the current classification model, features of each layer are extracted, and the layer feature alignment loss is calculated. : , in, For the original video in the first The feature mean vector of the layer, Synthetic video in The feature mean vector of the layer, The number of channels is used; simultaneously, the feature centers of the synthesized samples are used to classify the original samples, and the discriminative loss is calculated. : , in, For the test set The number of samples in the middle; For category The feature centers of the synthetic samples, where ; Step 44: Based on the current sample weights and spatiotemporal eigenvectors (After dimensionality reduction using global average pooling), construct a weighted kernel matrix for each class. ,in Calculate the weighted diversity loss : , in The diagonal matrix formed by the sample weights The matrix determinant solution function is used; this loss encourages high-weight samples to be distributed in the feature space, ensuring the diversity of video samples within a category; Step 45: Pruning-Distillation Synergistic Loss Weighted reconstruction loss Classification loss Feature alignment loss Discriminative loss and diversity loss The composition, and the specific calculation formula are as follows: , in, , , ,and All are the weights of each loss term, and their values ​​range from [value range missing]. ; Step 46: Since the pruning-distillation collaborative loss requires backpropagation of gradients through the computational graphs of the generative and classification models, the unrolled differential strategy from meta-learning needs to be used to update the gradient. ;calculate Update the sample importance parameters using the stochastic gradient descent algorithm: , in, Let be the learning rate for the sample importance parameter, and In this step, to simplify the calculation, a first-order approximation (such as the Reptile style) can also be used to update the weight parameters; Step 47: After each outer layer iteration, anneal the temperature parameters according to the exponential decay strategy: ,in The attenuation coefficient is used; repeat steps 2 to 46 until the pruning-distillation synergistic loss occurs. Convergence or reaching the maximum number of iterations; Step 48: Finally, use the dataset obtained in the last round of pruning to train the weighted diffusion model until convergence, and then generate a synthetic dataset as the compression result based on IPC; thus, the final synthetic video dataset is obtained. This synthetic dataset can be directly used for training downstream task models.

[0028] In one specific embodiment of the present invention, the Kinetics-400 dataset is used as pre-training data, and compression experiments are conducted on the UCF101 or HMDB51 datasets. Specific parameter settings are as follows: video sampling frame rate is 15 fps; the 3D-VAE encoder uses HunyuanVideo-1.5 VAE, and the output feature tensor space dimension is... The spatiotemporal feature extractor uses an R(2+1)D-34 network pre-trained on Kinetics-400, extracting features with a dimension of 512; the initial value of the learnable importance parameter is set to 1, and the temperature parameter... The initial value is 1.0, and the decay coefficient is set in each iteration. Finally annealed to 0.1; pruning rate The values ​​are set according to the dataset size: for large datasets, start with 0.9 and gradually decrease; for small datasets, start with 0.1 and gradually increase. The preset IPC for each class of synthesized video is 10, and the total number of denoising steps for the diffusion model is [not specified]. Autoregressive block length Number of overlapping frames between blocks The weights of each loss term in the pruning-distillation synergistic loss are respectively set as follows: , , , The maximum number of outer iterations is set to 50 rounds, the number of fine-tuning steps for the generative model in each round is set to 200 steps, and the number of training rounds for the classification model is set to 10 rounds; the Adam optimizer is used, and the learning rate is... .

[0029] In the execution of the method of this invention, a 3D-VAE encoder and diffusion distillation module are first pre-trained on Kinetics-400 to master the general video distribution. Then, the target dataset is preprocessed to obtain feature tensors. and spatiotemporal eigenvectors Initialize the learnable importance parameter for each sample. Then, begin outer-layer iterative optimization: based on the current sample weights. and pruning rate Obtain a pruned subset, fine-tune the diffusion model on the pruned subset, generate a synthetic video, train a classification model using the synthetic video and calculate various losses, and finally perform differential updates through meta-learning. and annealing temperature After 50 iterations, the pruning-distillation collaborative loss converged. Finally, the dataset obtained from the last round of pruning was used to train the weighted diffusion model until convergence, generating IPC synthetic videos as the compression result.

[0030] After iterative optimization through the above steps until convergence, the final synthetic video dataset can be obtained. This synthetic dataset retains the key action semantics and temporal dynamic information of the original dataset, while the data volume is only 1% to 10% of the original dataset. It can be directly used for model training in downstream tasks such as classification, significantly reducing storage and computation costs.

[0031] The embodiments and implementation process of the present invention have been described in detail above with reference to the accompanying drawings and tables, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A video dataset compression method based on differentiable pruning and latent diffusion distillation, characterized in that, Includes the following steps, Step 1: Preprocess the original video dataset by using a pre-trained encoder and a 3D convolutional neural network to extract feature codes and spatiotemporal feature vectors from the video samples, respectively. Step 2: Construct a differentiable pruning module. The input is the feature encoding and spatiotemporal feature vector of the original dataset samples, and the output is the pruned core subset and its corresponding sample weights. Step 3: Construct a potential diffusion distillation module, with the pruned core subset and sample weights as inputs and the intermediate synthetic video dataset as output; Step 4: Calculate the pruning-distillation collaborative loss, update the sample weights using the expansion differential strategy in meta-learning, iteratively optimize the pruning and distillation process until the distillation-pruning collaborative loss converges, and obtain the final synthetic video dataset.

2. The video dataset compression method based on differentiable pruning and potential diffusion distillation according to claim 1, characterized in that, Step 1 includes the following steps: Step 11: Process the original video dataset Each video sample at a rate of per second Frames are sampled to obtain a video frame sequence. ;in For the first Category labeling of each video sample The number of original videos, This represents the total number of frames after sampling. and The height and width of the original video frame, where 3 represents the RGB three channels; Step 12: Sample the video frame sequence The input is fed into a 3D encoder pre-trained on a large video dataset to obtain the corresponding sample feature tensor. ,in, and For the spatial dimension of the potential tensor, The number of channels yields the sample feature set. ; Step 13: In the original video dataset A 3D convolutional neural network classification model is pre-trained; for each video sample Overlay them front and back The frame is divided into For each segment, the corresponding spatiotemporal feature vector is extracted using the obtained spatiotemporal feature extractor. Then, the average of the spatiotemporal feature vectors of all segments is used to obtain the video sample. Corresponding spatiotemporal feature vector ,in For feature dimensions.

3. The video dataset compression method based on differentiable pruning and latent diffusion distillation according to claim 2, characterized in that, Step 2 includes the following steps: Step 21: Construct a differentiable pruning module for each video sample. Initialize a learnable importance parameter The initial value is set to 1; the softmax operation within the category is used to... Convert to sample weights : ， in, ; or It is the original video sample. or Corresponding markers; Step 22: Within each category, based on sample weights The sample feature tensor in the feature set Sort in descending order of weight value, based on the preset pruning rate. Remove the parts with lower weights and use the remaining tensors to form the pruned sample feature set. ,in This represents the number of feature tensors for the pruned samples.

4. The video dataset compression method based on differentiable pruning and latent diffusion distillation according to claim 3, characterized in that, Step 3 includes the following steps: Step 31: Construct a potential diffusion distillation module and introduce hierarchical temporal modeling inside the Transformer block: use sparse causal attention to maintain local temporal dependencies at the token level, and combine it with a state space model layer to achieve long-range memory with linear complexity. For a clean feature tensor sequence in the feature space diffusion time step Standard Gaussian noise The length of the random sampled noisy prefix The number of frames generated per autoregressive block During training, the prefix length is randomly selected, let , Add noise to the part to be noised. : ， in, For the first The clean feature tensor of the frame; The noise dispatch coefficient is the noise control factor for the forward diffusion process. Final noise prediction network (parameter The input is The time step vector is set to 0 in the clean part; Step 32: Through global temporal alignment, the latent diffusion distillation module uses a global encoder to extract a fixed number of global semantic tokens from historical frames during the generation of each block, and injects them into each denoising step of the current frame block, thereby enabling the noise prediction network to... Gain the ability to perceive the temporal context of the entire video; clean prefix Spatiotemporal average pooling is performed to obtain an intermediate representation, and then the learnable global token vector is updated through cross-attention. : ， in, As a global condition for the current denoising step Initialization of global conditions.

5. The video dataset compression method based on differentiable pruning and latent diffusion distillation according to claim 4, characterized in that, Step 3 also includes: Step 33: For the attention mechanism, causal sparse attention is adopted: the token sequence of the current block is divided into groups of size 1. The cube is first computed with coarse-grained causal attention between cubes to select key cubes, and then fine-grained causal token attention is computed within the selected cubes. Let the first The selected cube contains There are 1 token, and the token feature matrix is ​​as follows: ,in It is a noise prediction network The dimension; causal attention output is : ， Among them, the query matrix Key matrix Value matrix ; , , All are learnable projection matrices; It is the dimension of key / query. It is the dimension of the value; It is a causal mask matrix, and its elements are defined as follows: ； Step 34: Set up the noise prediction network Total For each block, if the block index is odd, a causal sparse attention block is used, and its output is... If the block index is even, the state-space model block is used. The state-space model adopts a block-based scanning method, dividing the space into independent blocks. Each block runs Mamba2 independently along the time dimension, and its output... With attention output Gated linear units are used here through gated fusion: ， in, This represents element-wise multiplication; It is a learnable gating matrix, with dimension and They are the same, obtained by linear transformation of the input features.

6. The video dataset compression method based on differentiable pruning and latent diffusion distillation according to claim 5, characterized in that, Step 3 also includes: Step 35: The loss function in the pre-training phase calculates the predicted noise loss only for the positions of the noisy frames: ， in, for In the The entire video tensor after adding noise; For noise prediction networks; For the indicator tensor, the position of the noisy frame is 1, and the position of the clean prefix is ​​0; Step 36: In the fine-tuning stage, based on the pruned sample feature set... The pre-trained diffusion model is fine-tuned by combining the current weights of the corresponding samples; the parameters of the first few layers of the pre-trained model are frozen, and only subsequent high-level parameters are allowed to be updated; the weighted reconstruction loss is defined as: ， in, For the first Each sample at time step The noisy tensor; For the first A clean prefix for each sample; during fine-tuning, the gradient of the reconstruction loss on the outer layer weight parameters is calculated every few steps, and the sample importance parameters are updated.

7. The video dataset compression method based on differentiable pruning and latent diffusion distillation according to claim 6, characterized in that, Step 3 also includes: Step 37: After fine-tuning, fix the pre-trained diffusion model and generate a preset number of synthetic videos for each category. During generation, a denoising diffusion implicit model is used to sample and gradually denoise from pure noise to obtain clean frames. Overlapping regions are set between adjacent blocks, and smooth transitions are achieved through temporal position weighted fusion. Let the length of the overlap region between two adjacent blocks be... Within the overlapping region, the fused frame is: ， in, This indicates the temporal position of the frame within the overlapping region; These are the fused feature frames; For the current block's first Frame; then the fused feature frames replace the corresponding overlapping parts in the historical frames to form a smooth feature video sequence. The resulting feature video sequence is then fused. The input is fed into the decoder of the 3D encoder used in step 12 to reconstruct the video frame sequence in pixel space. Repeat the above ideas in this step to... Each category generates 10 synthetic samples (IPCs), which together form an intermediate synthetic video dataset. ,in , For the first Class 1 A composite video.

8. The video dataset compression method based on differentiable pruning and latent diffusion distillation according to claim 7, characterized in that, Step 4 includes the following steps: Step 41: The differentiable pruning module described in Step 2 and the potential diffusion distillation module described in Step 3 are jointly optimized using the sample importance parameter to form an end-to-end video dataset compression process; Step 42: Using intermediate synthetic datasets Training classification model Calculate the classification loss on the test subset of the original video dataset; classification loss A combination of average category penalty and worst category penalty is used: ， in, This represents the total number of categories in the sample. For category Cross-entropy loss; The weight of the worst-case category penalty; It represents the largest cross-entropy loss across all categories; Step 43: From the original video dataset and intermediate synthetic datasets The data is sampled in batches, input into the current classification model, features of each layer are extracted, and the layer feature alignment loss is calculated. : ， in, For the original video in the first The feature mean vector of the layer, Synthetic video in The feature mean vector of the layer, The number of channels is used; simultaneously, the feature centers of the synthesized samples are used to classify the original samples, and the discriminative loss is calculated. : ， in, For the test set The number of samples in the middle; For category The feature centers of the synthetic samples, where .

9. The video dataset compression method based on differentiable pruning and latent diffusion distillation according to claim 8, characterized in that, Step 4 also includes: Step 44: Based on the current sample weights and spatiotemporal eigenvectors Construct a weighted kernel matrix for each class. ,in Calculate the weighted diversity loss : ， in The diagonal matrix formed by the sample weights A function for solving matrix determinants; Step 45: Pruning-Distillation Synergistic Loss Weighted reconstruction loss Classification loss Feature alignment loss Discriminative loss and diversity loss The composition and calculation formula are as follows: ， in, , , ,and All are the weights of each loss term, and their values ​​range from [value range missing]. .

10. The video dataset compression method based on differentiable pruning and latent diffusion distillation according to claim 9, characterized in that, Step 4 also includes: Step 46: Update using the expansion differential strategy in meta-learning ;calculate Update the sample importance parameters using the stochastic gradient descent algorithm: ， in, Let be the learning rate for the sample importance parameter, and ; Step 47: After each outer layer iteration, anneal the temperature parameters according to the exponential decay strategy: ,in The attenuation coefficient is used; repeat steps 2 to 46 until the pruning-distillation synergistic loss occurs. Convergence or reaching the maximum number of iterations; Step 48: Finally, use the dataset obtained in the last round of pruning to train the weighted diffusion model until convergence, and then generate a synthetic dataset as the compression result based on IPC; thus, the final synthetic video dataset is obtained. This synthetic dataset can be directly used for training downstream task models.

Images