Multi-label text data set distillation method and equipment based on distribution matching, and medium

Abstract

The invention discloses a multi-label text data set distillation method, equipment and medium based on distribution matching, and the method comprises the steps: firstly extracting a real feature embedding vector of an original multi-label data set, and combining Gaussian distribution and original label statistical probability distribution to initialize a synthetic data set; and then, constructing a measurement network, and updating measurement network parameters by maximizing divergence between real and synthetic data batches on the premise of fixing the synthetic data. And finally, the updated measurement network is fixed, a Mixup data enhancement strategy is utilized to construct a hybrid embedding vector, the gradient of a loss function to a synthetic feature embedding vector is calculated through back propagation, and then a gradient descent method is adopted to update a synthetic data set. According to the method, the feature distillation of the multi-label text data is effectively realized through an alternate optimization and hybrid enhancement strategy.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, and in particular relates to a method, device, and medium for distilling multi-label text datasets based on distribution matching. Background Technology

[0002] With the rapid development of deep learning and natural language processing technologies, multi-label text classification tasks have been widely applied in various fields, such as academic paper classification, legal case analysis, and multi-dimensional sentiment analysis. A key characteristic of multi-label text data is that each text sample may correspond to multiple labels simultaneously, and these labels often exhibit complex co-occurrence or mutual exclusion relationships (e.g., "machine learning" and "deep learning" often appear together). To capture this complex semantic and structural information, existing multi-label classification methods typically rely on large-scale pre-trained models and massive amounts of labeled data for training. However, large-scale datasets incur significant storage and computational overhead, resulting in high training time costs and making rapid deployment and iteration difficult in resource-constrained scenarios (such as mobile devices and edge computing nodes).

[0003] To reduce data storage and computation costs, dataset distillation (DD) has emerged. Its core idea is to synthesize a "synthetic dataset" much smaller than the original dataset, allowing the model trained on this synthetic set to achieve performance close to that of training on the full dataset. While dataset distillation has made significant progress in image and single-label text classification tasks, existing mainstream techniques (primarily gradient matching methods based on two-layer optimization) face serious challenges when applied to multi-label text scenarios. First, the "class-by-class optimization" paradigm leads to the loss of structural information. Most existing methods follow the paradigm of generating representative samples independently for each class, treating labels as independent entities. This approach ignores the crucial label correlation in multi-label data, failing to effectively model the co-occurrence structure between labels. This results in incomplete semantic information in the synthesized data, limiting the performance ceiling of downstream models.

[0004] Secondly, it suffers from poor cross-model generalization ability. Existing text distillation methods typically operate in a continuous embedding space, and the resulting synthetic embeddings tend to overfit to specific model architectures (such as a specific BERT model used for feature extraction). This leads to a significant performance drop when data distilled on one model is used to train a model with a different architecture, severely limiting the generality and practicality of the synthetic data.

[0005] Finally, the computational and storage overhead is enormous. Mainstream gradient matching-based methods heavily rely on computationally intensive two-layer optimization frameworks, requiring repeated backpropagation in the inner loop (model training) and outer loop (data update). This not only places extremely high demands on GPU memory resources but also makes the distillation process extremely slow, making it difficult to scale to large-scale pre-trained models and datasets used in industrial applications.

[0006] Therefore, there is an urgent need for a multi-label text dataset distillation method that can effectively capture multi-label structural information, has strong cross-model generalization ability, and is computationally efficient, in order to solve the problems of paradigm mismatch, poor generalization and high cost of existing technologies. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a method, device, and medium for distilling multi-label text datasets based on distribution matching.

[0008] In a first aspect, embodiments of the present invention provide a method for distilling multi-label text datasets based on distribution matching, the method comprising: Obtain the original multi-label dataset; extract the true feature embedding vectors of all samples in the original multi-label dataset to form the true feature set; determine the synthetic feature embedding vectors according to the Gaussian distribution; assign an initial synthetic label to each synthetic feature embedding vector according to the statistical label probability distribution of the labels in the original multi-label dataset, thereby obtaining the synthetic dataset; Construct a metric network; The synthetic feature embedding vectors in the synthetic dataset are fixed; a batch of real features and a first batch of synthetic data are sampled from the real feature set and the synthetic dataset, respectively; the parameters of the metric network are updated by maximizing the divergence between the real feature batch and the first batch of synthetic data. The parameters of the updated metric network are fixed; a second batch of synthetic data is sampled from the synthetic dataset, and a hybrid embedding vector is constructed based on the Mixup data augmentation strategy; the hybrid embedding vector is input into the updated metric network, and the loss function is calculated; the gradient of the loss function with respect to the synthetic feature embedding vector is calculated through backpropagation, and the parameters in the synthetic dataset are updated using gradient descent, thereby updating the obtained synthetic data.

[0009] In a second aspect, embodiments of the present invention provide an electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and the at least one computer program is executed by the at least one processor to enable the at least one processor to perform the above-described distribution-matching-based multi-label text dataset distillation method.

[0010] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described multi-label text dataset distillation method based on distribution matching.

[0011] Fourthly, embodiments of the present invention provide a computer program product, including a computer program / instruction, characterized in that the computer program / instruction, when executed by a processor, implements the above-described multi-label text dataset distillation method based on distribution matching.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) The present invention assigns an initial synthetic label to each synthetic feature embedding vector based on the statistical label probability distribution of the labels in the original multi-label dataset. The present invention explicitly preserves the complex co-occurrence and mutual exclusion relationships in the multi-label data by label initialization based on real statistics and global distribution matching, thus solving the problem of semantic loss in synthetic data.

[0013] (2) The present invention constructs a hybrid embedding vector based on the Mixup data augmentation strategy. By effectively regularizing the distillation process, it prevents the synthetic data from overfitting to the inductive bias of a specific model, so that the generated synthetic dataset can perform well on downstream models of different architectures.

[0014] (3) The present invention uses a single-layer adversarial optimization framework to optimize the metric network and performs matching directly in the feature space, avoiding repeated model training and backpropagation, and significantly reducing memory requirements and training time. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating the multi-label text dataset distillation method based on distribution matching provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0019] like Figure 1 As shown, this embodiment of the invention provides a method for distilling multi-label text datasets based on distribution matching, the method comprising the following steps: Step S1: Obtain the original multi-label dataset; extract the true feature embedding vectors of all samples in the original multi-label dataset to form the true feature set T; determine the synthetic feature embedding vector according to the Gaussian distribution; assign an initialized synthetic label to each synthetic feature embedding vector according to the statistical label probability distribution of the labels in the original multi-label dataset, thereby obtaining the synthetic dataset S.

[0020] Specifically, step S1 includes the following sub-steps: Step S101: Obtain the original multi-label dataset ,in, Represents the original text sequence. It is the corresponding multi-hot embedding vector, where C is the total number of labels and N is the total number of samples in the original multi-label dataset.

[0021] Step S102: Process the original multi-label dataset through a text encoder to obtain the feature embedding vectors of all samples in the original multi-label dataset, forming the true feature set T.

[0022] Furthermore, the text encoder employs a pre-trained language model. The pre-trained language model includes, but is not limited to, BERT-base, RoBERTa, or DistilBERT. The feature embedding vectors of all samples in the original multi-label dataset are calculated. Thus constructing a true feature set It should be noted that during the feature extraction process, the pre-trained language model... The parameters are kept frozen and do not participate in subsequent distillation updates to ensure the stability of the feature space.

[0023] Step S103: Determine the synthetic feature embedding vector based on the Gaussian distribution, and assign an initialized synthetic label to each synthetic feature embedding vector according to the statistical label probability distribution of the labels in the original multi-label dataset, thereby obtaining the synthetic dataset. .

[0024] Where M is the total number of samples in the synthetic dataset. Typically, the total number of samples M in the synthetic dataset is much smaller than the total number of samples N in the original multi-label dataset. For example, the total number of samples M in the synthetic dataset is 2% or less of the total number of samples N in the original multi-label dataset.

[0025] Synthetic feature embedding vectors in synthetic datasets The data is randomly initialized using a standard Gaussian distribution N(0,I) with a mean of 0 and a variance of I, where d is the embedding dimension.

[0026] Initialize the synthesis tag The process includes: traversing the original multi-label text dataset D, counting the frequency of occurrence of each unique label combination, and constructing a statistical label probability distribution P(y) for the real label combinations; performing weighted sampling based on this statistical label probability distribution P(y) to embed a vector for each synthetic feature. Assign a synthetic tag It should be noted that this initialization strategy ensures that the synthetic dataset not only maintains consistency with the real data in terms of the number of categories, but more importantly, it preserves the dependencies between labels (e.g., "label A" and "label B" often appear together), which is crucial for the accuracy of multi-label classification tasks.

[0027] Step S2: Construct the metric network.

[0028] Furthermore, in this example, the metric network This is a multilayer perceptron (MLP). The network consists of an input layer, several hidden layers, and an output layer. LeakyReLU or ReLU is used as the activation function between the hidden layers, and layer normalization can be added to improve training stability. The metric network is used to embed the input feature vector (whether it is the true feature embedding vector) into the input feature vector. Or synthesized feature embedding vector This is mapped to a scalar score to measure the distribution of the sample set.

[0029] Step S3: Fix the synthetic feature embedding vector in the synthetic dataset; sample a batch of real features and a first batch of synthetic data from the real feature set and the synthetic dataset respectively; update the parameters of the metric network by maximizing the divergence between the real feature batch and the first batch of synthetic data.

[0030] Specifically, step S3 includes the following steps: Step S301: Freeze the synthetic feature embedding vectors in the synthetic dataset S. This prevents changes to the synthetic data when optimizing the metric network.

[0031] Step S302: Randomly sample a batch of real features of size b from the real feature set T. A first batch of synthetic data of size b is randomly sampled from the synthetic dataset S. .

[0032] Step S303: The parameters of the metric network are updated with the objective of maximizing the divergence between the real feature batch and the first synthetic data batch.

[0033] Specifically, the optimization objective function is set as follows: In the formula, E represents the expectation, and t represents the batch size from the true features. The sampled from the first batch of synthetic data, s represents the sample from the first batch of synthetic data. Samples sampled from the middle.

[0034] It should be noted that this example preferably uses the dual form of the Wasserstein-1 distance as the optimization objective because, compared to KL divergence or JS divergence, the Wasserstein distance can still provide a meaningful gradient when the two distributions do not overlap, avoiding the gradient vanishing problem. The objective function aims to increase the difference between the expected scores of the true samples and the expected scores of the synthetic samples, thereby approximating the Wasserstein distance between the two distributions.

[0035] Furthermore, in this example, gradient ascent or minimization is used. This updates the parameters of the metric network. ,in, It is the learning rate. Meanwhile, to maintain the Lipschitz continuity of the metric network, the parameters can be adjusted. Perform weight truncation or apply gradient penalties. This step is typically repeated k times in each iteration (e.g., k=5) to ensure that the metric network can adequately capture the subtle differences between the current first batch of synthetic data and the true feature batch.

[0036] It should be noted that this invention, by employing this single-layer adversarial optimization framework, avoids the problems of traditional two-layer optimization methods that require lengthy model training trajectories and complex calculations of second-order derivatives. "Single-layer" means that the update gradient of the synthetic dataset is directly derived from the distribution divergence of the current batch, rather than from the validation loss of a surrogate model trained over multiple steps. This mechanism eliminates the dependence of the optimization process on the convergence state of the inner loop; the entire process involves only the calculation of the first-order gradients of the parameters. This technique significantly reduces the complexity of the computational graph and the demand for GPU memory, making efficient distillation of large-scale multi-label text datasets possible on a single ordinary GPU.

[0037] Step S4: Fix the parameters of the updated metric network; sample a second batch of synthetic data from the synthetic dataset and construct a hybrid embedding vector based on the Mixup data augmentation strategy; input the hybrid embedding vector into the updated metric network and calculate the loss function; calculate the gradient of the loss function with respect to the synthetic feature embedding vector through backpropagation, and update the parameters in the synthetic dataset using gradient descent, thereby updating the obtained synthetic data.

[0038] Specifically, step S4 includes the following sub-steps: Step S401: Freeze the metric network updated in step S3. The parameters.

[0039] Step S402: Randomly sample a new second batch of synthetic data of size b from the synthetic dataset S. .

[0040] Step S403, from the second batch of synthesized data The first sample was randomly selected from the middle. Second Sample Construct hybrid embedding vectors The expression is as follows: In the formula, obey , For hyperparameters (e.g., values ​​ranging from 0.2 to 0.5). Hybrid embedding vectors. To form a new batch of synthetic samples .

[0041] It should be noted that synthetic data consists of discrete points in the embedding space. Direct optimization can easily lead these points into local optima, only fitting the feature preferences of a specific model. Through Mixup data augmentation, this invention constructs a continuous linear manifold among synthetic samples, forcing the synthetic data to learn the overall distribution pattern of the feature space rather than discrete points, thereby greatly enhancing the smoothness of the synthetic data and its versatility across model architectures.

[0042] Step S404, mix the embedding vectors Input into the updated metric network to calculate the generator loss. The expression is as follows: In the formula, This indicates a batch of synthesized data that has been processed using the Mixup data augmentation strategy; Calculate the generator loss using backpropagation. The gradient of the synthetic feature embedding vector is used to update the parameters in the synthetic dataset S using gradient descent. , The learning rate for the synthetic data is used to update the resulting synthetic data.

[0043] Furthermore, the method also includes: Step S5 updates the obtained synthetic dataset and uses it to train the downstream multi-label text classification model; the trained downstream multi-label text classification model is used to perform downstream text classification tasks, which include at least long text classification tasks, sentiment analysis tasks, or intent recognition tasks.

[0044] For example, the downstream multi-label text classification models include, but are not limited to, BERT, RoBERTa, etc.

[0045] Example 1 The following provides a detailed explanation of the above steps in conjunction with specific parameters: Step S1: Obtain the original multi-label dataset; the original multi-label dataset is the Arxiv Academic Paper Dataset (AAPD), which contains 53,840 academic paper abstracts covering 54 categories.

[0046] The pre-trained language model bert-base-uncased was used to encode all samples in the Arxiv academic paper dataset. The output vector at the [CLS] position was extracted as a feature embedding to construct the ground feature set T, where each feature vector has a dimension of 768. The pre-trained language model bert-base-uncased contains a 12-layer Transformer encoder with a hidden layer dimension of 768.

[0047] Initialize the synthetic dataset S, set the compression ratio (e.g., retain 1 sample per class, IPC=1), then the total number of synthetic samples M is approximately 54 (the exact number depends on the label combination coverage). Synthetic embeddings. Initialize using a standard Gaussian distribution N(0, I), with the same dimension of 768. For synthetic labels... The frequency of all unique label combinations in the Arxiv academic paper dataset is statistically analyzed. Based on this frequency distribution, samples are generated and a fixed multi-hot label vector is assigned to each synthetic embedding. For example, if the labels "algorithm" and "complexity" frequently co-occur in real data, a corresponding proportion of samples in the initial synthetic data will also possess both labels.

[0048] Step S2, construct the metric network In this example, the metric network A 3-layer Multilayer Perceptron (MLP). The input layer has a dimension of 768, containing two hidden layers with dimensions of 512 and 256 respectively. The activation function used is LeakyReLU (with a negative slope of 0.2). The output layer has a dimension of 1 and outputs a scalar score. Measure the network parameters. Use the Xavier method for initialization.

[0049] Step S3, in the maximization phase of adversarial optimization, sets the batch size b=64. 64 samples are randomly sampled from the real feature set T and 64 samples are randomly sampled from the synthetic dataset S. The two sets of samples are input into the metric network, and the dual form of the Wasserstein-1 distance is calculated as the loss function. The metric network parameters are updated using the Adam optimizer, with a learning rate set to 1e-4. This step is repeated 5 times (k=5) in each outer layer iteration to ensure that the metric network accurately captures distributional differences. During this process, the weights of the metric network are truncated in the range [-0.01, 0.01] to satisfy the Lipschitz continuity constraint.

[0050] Step S4, in the minimization phase of adversarial optimization, fix the metric network parameters. Randomly sample 64 samples from the synthetic dataset S. Apply Mixup data augmentation to this batch of samples: randomly pair samples. and Interpolation coefficients are sampled from the Beta(0.2, 0.2) distribution. Generate hybrid embeddings The generator loss is calculated by embedding the hybrid input metric network. Update the synthetic embedding using the SGD optimizer. The learning rate was set to 1e-2, and the momentum was set to 0.9. This step forces the synthetic data to form a smooth manifold in the embedding space, rather than discrete points.

[0051] Step S5: Repeat steps 3 and 4 for a total of 400 epochs. After training, save the final synthetic dataset S. To verify the results, use the synthetic dataset S to train a new, randomly initialized BERT-base-uncased classifier (downstream model).

[0052] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the distribution-matching-based multi-label text dataset distillation method described above. Figure 2 The diagram shown illustrates a hardware structure of any device with data processing capabilities for implementing the multi-label text dataset distillation method based on distribution matching provided in this embodiment of the invention, except... Figure 2 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0053] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the multi-label text dataset distillation method based on distribution matching as described above. The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0054] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only.

[0055] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for distilling multi-label text datasets based on distribution matching, characterized in that, The method includes: Obtain the original multi-label dataset; extract the true feature embedding vectors of all samples in the original multi-label dataset to form the true feature set; determine the synthetic feature embedding vectors according to the Gaussian distribution; assign an initial synthetic label to each synthetic feature embedding vector according to the statistical label probability distribution of the labels in the original multi-label dataset, thereby obtaining the synthetic dataset; Construct a metric network; The synthetic feature embedding vectors in the synthetic dataset are fixed; a batch of real features and a first batch of synthetic data are sampled from the real feature set and the synthetic dataset, respectively; the parameters of the metric network are updated by maximizing the divergence between the real feature batch and the first batch of synthetic data. The parameters of the updated metric network are fixed; a second batch of synthetic data is sampled from the synthetic dataset, and a hybrid embedding vector is constructed based on the Mixup data augmentation strategy; the hybrid embedding vector is input into the updated metric network, and the loss function is calculated; the gradient of the loss function with respect to the synthetic feature embedding vector is calculated through backpropagation, and the parameters in the synthetic dataset are updated using gradient descent, thereby updating the obtained synthetic data.

2. The method for distilling multi-label text datasets based on distribution matching according to claim 1, characterized in that, The process of extracting the true feature embedding vectors of all samples in the original multi-label dataset to form the true feature set includes: The original multi-label dataset is processed by a text encoder, and the feature embedding vectors of all samples in the original multi-label dataset are calculated to construct the true feature set. The text encoder employs a pre-trained language model, which includes BERT-base, RoBERTa, or DistilBERT. During feature extraction, the parameters of the pre-trained language model remain frozen.

3. The method for distilling multi-label text datasets based on distribution matching according to claim 1, characterized in that, The process of obtaining a synthetic dataset includes: The synthetic feature embedding vector is determined based on the Gaussian distribution; Traverse the original multi-label text dataset, count the frequency of occurrence of each unique label combination, and construct the statistical label probability distribution of the real label combinations; Weighted sampling is performed based on the statistical label probability distribution of the labels in the original multi-label dataset, and an initial synthetic label is assigned to each synthetic feature embedding vector.

4. The method for distilling multi-label text datasets based on distribution matching according to claim 1, characterized in that, The expression for the divergence between the true feature batch and the first synthetic data batch is as follows: ； In the formula, E represents the expectation, and t represents the batch size from the true features. The sampled from the first batch of synthetic data, s represents the sample from the first batch of synthetic data. Samples sampled from the middle, This represents a metric network.

5. The method for distilling multi-label text datasets based on distribution matching according to claim 1, characterized in that, The process of constructing hybrid embedding vectors based on the Mixup data augmentation strategy includes: From the second batch of synthetic data The first sample was randomly selected from the middle. Second Sample Construct hybrid embedding vectors The expression is as follows: ； In the formula, obey , This is a hyperparameter.

6. A method for distilling multi-label text datasets based on distribution matching according to claim 1 or 5, characterized in that, The process of updating the synthetic data by calculating the gradient of the loss function with respect to the synthetic feature embedding vector through backpropagation and then updating the parameters in the synthetic dataset using gradient descent includes: Mixed embedding vectors Input to the updated metric network Calculate generator loss The expression is as follows: ； In the formula, b represents the batch size. This indicates a batch of synthesized data that has been processed using the Mixup data augmentation strategy; Calculate the generator loss using backpropagation. The gradient of the synthetic feature embedding vector is used to update the parameters in the synthetic dataset S using gradient descent. , The learning rate for the synthetic data is used to update the resulting synthetic data.

7. The method for distilling multi-label text datasets based on distribution matching according to claim 1, characterized in that, The method further includes: The final updated synthetic dataset will be used to train the downstream multi-label text classification model. The trained downstream multi-label text classification model is used to perform downstream text classification tasks, which include at least long text classification tasks, sentiment analysis tasks, or intent recognition tasks.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more of the computer programs being executed by the at least one processor to enable the at least one processor to perform the distribution-matching-based multi-label text dataset distillation method as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the multi-label text dataset distillation method based on distribution matching as described in any one of claims 1-7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the multi-label text dataset distillation method based on distribution matching as described in any one of claims 1-7.

Images