Distillation method, device, computer equipment and storage medium for multi-label text dataset

Abstract

The present application relates to a method, apparatus, computer equipment and storage medium for distilling a multi-label text dataset, wherein the method includes: a method, apparatus, computer equipment and storage medium for distilling a multi-label text dataset, by obtaining the first text embedding vector and the shape of the first attention label of the original dataset; the original dataset is a multi-label text dataset; the shape of the first attention label is pruned, and an initial synthetic dataset is constructed based on the pruning result and the first text embedding vector; the initial synthetic dataset and the original dataset are processed using a preset internal and external loop mechanism to obtain a target synthetic dataset. Through this application, the problem of poor performance on minority class samples and reduced recognition accuracy in related technologies is solved; the computational cost of attention labels is reduced by pruning operations, and fine-grained semantic features in the text can be captured, thereby improving the reliability of the dataset.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a distillation method, apparatus, computer equipment, and storage medium for multi-label text datasets. Background Art

[0002] Due to the storage and computational overhead associated with large datasets, dataset distillation methods have been introduced to address this issue. Currently, mainstream dataset distillation methods can be divided into two categories: matching-based methods (such as meta-learning, gradient matching, and distribution matching) and kernel-based methods (such as using kernel ridge regression or neural tangent kernels to transform a two-layer optimization problem into a single-layer optimization problem). However, most of these methods have been successful on image data or single-label tasks, but lack applicability to text data, especially multi-label text classification tasks.

[0003] In the field of multi-label text classification, the current approach is to directly use pre-trained language models to extract global text representations, and then combine them with conventional classifiers for training to address the problems of long-tail labels and class imbalance. However, the disadvantages of this approach are: due to the discrete nature of text data and the complex relationships between labels in multi-label scenarios, methods based on global representations often have difficulty capturing fine-grained label co-occurrence information, resulting in poor performance on minority class samples and reduced dataset reliability.

[0004] There is currently no effective solution to the problem that related technologies perform poorly on minority samples, which reduces the reliability of the dataset. Summary of the Invention

[0005] In this embodiment, a distillation method, apparatus, computer device, and storage medium for a multi-label text dataset are provided to address the problem in related technologies of poor performance on minority class samples and reduced reliability of the dataset.

[0006] First, in this embodiment, a distillation method for a multi-label text dataset is provided, including:

[0007] Obtaining the shape of a first text embedding vector and a first attention label of an original data set; the original data set is a multi-label text data set;

[0008] Pruning the shape of the first attention label, and constructing an initial synthetic dataset based on the pruning result and the first text embedding vector;

[0009] The initial synthetic data set and the original data set are processed using a preset internal and external loop mechanism to obtain a target synthetic data set.

[0010] In some embodiments, obtaining the shape of the first text embedding vector and the first attention label of the original data set includes:

[0011] The obtained original data set is transformed using the pre-trained language model to obtain the first text embedding vector and the first attention label distribution.

[0012] In some embodiments, pruning the shape of the first attention tag and constructing an initial synthetic dataset based on the pruning result and the first text embedding vector include:

[0013] Randomly initialize the first text embedding vector using Gaussian noise to obtain a second text embedding vector;

[0014] Use Gaussian noise to randomly initialize the first attention label and prune the shape of the first attention label to obtain the shape of the second attention label;

[0015] An initial learning rate of the shape of the second text embedding vector and the second attention label is given to construct an initial synthetic dataset.

[0016] In some embodiments, the initial synthetic data set and the original data set are processed using a preset inner and outer loop mechanism to obtain a target synthetic data set, including:

[0017] Based on a preset first loss function, inputting the initial synthetic dataset into an initial multi-label classification model for inner loop training to update model parameters of the initial multi-label classification model; the first loss function includes a first task loss function, a first attention loss function, and a combined total loss;

[0018] Based on the preset first task loss function, the original data set is input into the initial multi-label classification model after inner loop training for outer loop training, and the initial synthetic data set is optimized based on the outer loop training result to obtain the target multi-label classification model and the target synthetic data set.

[0019] In some embodiments, based on a preset first task loss function, the original dataset is input into the initial multi-label classification model after inner loop training for outer loop training, and the initial synthetic dataset is optimized based on the outer loop training result to obtain a target multi-label classification model and the target synthetic dataset, including:

[0020] Inputting the original data set into the initial multi-label classification model after inner loop training for forward propagation, and calculating the first task loss function and the corresponding loss gradient;

[0021] Optimizing the initial synthetic data based on the loss gradient and adjusting the initial learning rate;

[0022] When the outer loop cutoff condition is met, a target multi-label classification model and the target synthetic dataset are obtained.

[0023] In some embodiments, the initial synthetic data set and the original data set are processed using a preset inner and outer loop mechanism to obtain a target synthetic data set, including:

[0024] Processing the initial synthetic dataset and the original dataset according to a preset two-stage loss strategy and a preset internal and external loop mechanism to obtain a target synthetic dataset;

[0025] The two-stage loss strategy includes a first-stage strategy and a second-stage strategy; the first-stage strategy is to use the first task loss function for training; the second-stage strategy is to use the first task loss function, the first attention loss function and the balance loss function for training.

[0026] In some embodiments, the method further comprises:

[0027] Obtain a text dataset corresponding to the downstream task, input the text dataset into the target multi-label classification model, and obtain a classification result of the downstream task.

[0028] In a second aspect, a distillation device for a multi-label text dataset is provided in this embodiment, comprising: an acquisition module, a construction module, and a processing module;

[0029] The acquisition module is used to obtain the shape of the first text embedding vector and the first attention label of the original data set; the original data set is a multi-label text data set;

[0030] The construction module is configured to prune the shape of the first attention label and construct an initial synthetic dataset based on the pruning result and the first text embedding vector;

[0031] The processing module is used to process the initial synthetic data set and the original data set using a preset internal and external loop mechanism to obtain a target synthetic data set.

[0032] In a third aspect, a computer device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the distillation method for a multi-label text dataset described in the first aspect is implemented.

[0033] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored. When the program is executed by a processor, the distillation method of the multi-label text dataset described in the first aspect is implemented.

[0034] Compared with the related art, the distillation method, device, computer equipment and storage medium of the multi-label text dataset provided in this embodiment obtain the accurate first text embedding vector and the shape of the first attention label in the original dataset; the original dataset is a multi-label text dataset; the shape of the first attention label is pruned, and an initial synthetic dataset is constructed based on the pruning result and the first text embedding vector, which can reduce the computational cost of the attention label and capture the fine-grained semantic features in the text; in addition, the initial synthetic dataset and the original dataset are processed using a preset internal and external loop mechanism to effectively capture the feature distribution of the original multi-label text data, thereby obtaining the target synthetic dataset; it solves the problem of poor performance on minority class samples and reduced recognition accuracy in the related art; the computational cost of the attention label is reduced through the pruning operation, and the fine-grained semantic features in the text can be captured, thereby improving the reliability of the dataset.

[0035] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0037] Figure 1 This is a hardware structure block diagram of a terminal device for a multi-label text dataset distillation method provided in one embodiment of the present application;

[0038] Figure 2 This is a flowchart of a distillation method for a multi-label text dataset provided by an embodiment of the present application;

[0039] Figure 3 is a flowchart of step S220;

[0040] Figure 4 is a flowchart of step S230;

[0041] Figure 5 This is a structural block diagram of a distillation device for a multi-label text dataset provided in one embodiment of the present application.

[0042] In the figure: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 210, acquisition module; 220, construction module; 230, processing module. DETAILED DESCRIPTION

[0043] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0044] Unless otherwise defined, technical or scientific terms used in this application shall have the ordinary meanings as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "the," "these," and similar expressions in this application do not denote limitations on quantity and may be singular or plural. The terms "comprise," "include," "have," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include unlisted steps or modules (units) or other steps or modules (units) inherent to the process, method, product, or device. The terms "connected," "connected," "coupled," and similar expressions used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used in this application, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone; A and B exist simultaneously; or B exists alone. Generally, the character " / " indicates that the objects in the preceding and following relationship are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0045] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 This is a hardware block diagram of the terminal of the distillation method of the multi-label text dataset of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 The processor 102 (only one is shown) and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The terminal may also include a transmission device 106 for communication functions and an input / output device 108. It will be understood by those skilled in the art that Figure 1The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0046] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the distillation method of the multi-label text dataset in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0047] Transmission device 106 is used to receive or transmit data via a network. This network may include a wireless network provided by the terminal's communications provider. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0048] In this embodiment, a distillation method for a multi-label text dataset is provided. Figure 2 This is a flow chart of the distillation method for the multi-label text dataset of this embodiment. Figure 2 As shown, the process includes the following steps:

[0049] Step S210, obtaining the shape of the first text embedding vector and the first attention label of the original data set; the original data set is a multi-label text data set;

[0050] Step S220, pruning the shape of the first attention label, and constructing an initial synthetic dataset based on the pruning result and the first text embedding vector;

[0051] Step S230 : Processing the initial synthetic data set and the original data set using a preset internal and external loop mechanism to obtain a target synthetic data set.

[0052] Specifically, the original dataset is an original multi-label text dataset, which refers to a dataset in which each text sample can be associated with multiple category labels at the same time. It is different from traditional single-label classification (a sample belongs to only one category). In multi-label tasks, a sample may belong to a combination of multiple categories. This data structure is closer to the complexity of text content in real-world scenarios. The first text embedding vector is a fixed-length numerical vector (i.e., a vectorized representation) obtained by converting the text (such as words, phrases, sentences, or documents) in the original dataset. These vectors can capture the semantic, grammatical, or contextual information of the text, making it easier for computers to understand and process text data. The shape of the first attention label refers to the dimensional structure of the weight matrix or mask generated in the attention mechanism. Its specific form depends on the task type, input data structure, and attention type.

[0053] In actual application, the embodiment of the present application obtains the first text embedding vector and the shape of the first attention label of the original data set in a manner including but not limited to obtaining from a database pre-stored therein to obtain the first text embedding vector and the shape of the first attention label of the original data set that meet the above requirements; it can also be downloaded from a network platform to download the first text embedding vector and the shape of the first attention label of the original data set that meet the requirements; it can also be generated according to demand to generate the corresponding first text embedding vector and the shape of the first attention label of the original data set, etc. The embodiment of the present application does not limit the method for obtaining the shape of the first text embedding vector and the first attention label of the original data set.

[0054] The shape of the attention label can be determined by the required synthetic dataset size, the number of Transformer layers, the number of attention heads, and the sequence length; the pruning process can be to trim the number of attention heads to retain the key attention heads, thereby reducing the computational cost of the attention label; and the retained attention heads pay more attention to the relationship with other attention heads to capture the fine-grained semantic features in the text. The inner and outer loop mechanism refers to performing inner loop training on the initial synthetic dataset; and then performing outer loop training in combination with the original dataset. That is to say, under the preset inner and outer loop mechanism, the initial synthetic dataset is trained in an inner loop; and then performing outer loop training in combination with the original dataset, so as to effectively capture the feature distribution of the original multi-label text data and obtain the target synthetic dataset. It can be considered that through this embodiment, a highly concentrated target synthetic dataset can be generated at an extremely low data compression ratio, so that the multi-label classification model trained subsequently can significantly reduce the storage and computing resource requirements while retaining the performance of the original data.

[0055] Related technologies use pre-trained language models to extract global text representations, which are then trained in conjunction with conventional classifiers to address the issues of long-tail labels and class imbalance. However, these approaches have the following drawbacks: Due to the discrete nature of text data and the complex relationships between labels in multi-label scenarios, global representation-based methods often struggle to capture fine-grained label co-occurrence information, resulting in poor performance on minority class samples and reduced dataset reliability. In this embodiment, the above steps are used to obtain the accurate first text embedding vector and the shape of the first attention label in the original data set; the original data set is a multi-label text data set; the shape of the first attention label is pruned to construct a compact initial synthetic data set, which can reduce the computational cost of the attention label and capture the fine-grained semantic features in the text; in addition, the initial synthetic data set and the original data set are processed using the preset internal and external loop mechanism to effectively capture the feature distribution of the original multi-label text data, thereby obtaining the target synthetic data set; the problem of poor performance on minority class samples and reduced recognition accuracy in related technologies is solved; the computational cost of the attention label is reduced through the pruning operation, and the fine-grained semantic features in the text can be captured, thereby improving the reliability of the data set and thus improving the recognition accuracy of the subsequent classification model; and it is possible to generate a highly concentrated target synthetic data set at an extremely low data compression ratio, so that the subsequently trained multi-label classification model can retain the performance of the original data while significantly reducing the storage and computing resource requirements.

[0056] The following is a detailed description of each of the above steps:

[0057] In some embodiments, obtaining the first text embedding vector and the shape of the first attention label of the original data set in step S210 includes the following steps:

[0058] Step S210: Using the pre-trained language model, the obtained original data set is converted to obtain a first text embedding vector and a first attention label distribution.

[0059] Specifically, the language model is pre-trained, and its model architecture includes but is not limited to the BERT architecture, the Encoder-Decoder hybrid architecture, and the Switch Transformer sparse expert model architecture. Different architectures can use corresponding training methods to complete pre-training, and there is no restriction on this.

[0060] After the above model architecture is pre-trained, the obtained original dataset is transformed using the pre-trained language model to ensure that the shape of the first text embedding vector and the first attention label are accurately obtained.

[0061] In this embodiment, a pre-trained language model is used to extract core semantic information from the original data set to ensure accurate acquisition of the shape of the first text embedding vector and the first attention label.

[0062] In some of these embodiments, Figure 3 As shown, in step S220, the shape of the first attention label is pruned, and an initial synthetic dataset is constructed based on the pruning result and the first text embedding vector, including:

[0063] Step S221, randomly initializing the first text embedding vector using Gaussian noise to obtain a second text embedding vector;

[0064] Step S222: randomly initialize the first attention label using Gaussian noise, and prune the shape of the first attention label to obtain the shape of the second attention label;

[0065] Step S223, assigning an initial learning rate of the shape of the second text embedding vector and the second attention label to construct an initial synthetic dataset.

[0066] Specifically, the size of the required synthetic dataset can be determined based on the preset number of retained samples per class (IPC). The shape of the first text embedding vector and the first attention label can be initialized by generating independent Gaussian distributed random numbers through a uniform distribution or calling a Gaussian noise layer or initialization function in a deep learning library. The first text embedding vector is randomly initialized with Gaussian noise to obtain the second text embedding vector.

[0067] After random initialization with Gaussian noise, the shape of the first attention tag is set to the attention shape of the pretrained language model (|S|, L, H, N, N), where |S| is the required synthetic dataset size (determined by IPC), L is the number of Transformer layers, H is the number of attention heads, and N is the sequence length. To reduce storage and computational overhead, the number of attention heads H of the attention tag is pruned to H´ (for example, H´ = 2), retaining only the first H´ attention heads. Simultaneously, the second, third, or fourth sequence length dimensions of the attention matrix are pruned, retaining only the first element, focusing only on the attention of the [CLS] token to other words. The pruned attention tag has a shape of (|S|, L, H´, 1, N).

[0068] Finally, an initial learning rate is assigned to the second text embedding vector and the shape of the second attention label, thereby completing the construction of the initial synthetic dataset; the initial synthetic dataset then includes the second text embedding vector, the shape of the second attention label, and the initial learning rate η. The initial learning rate can be 0.05.

[0069] Through this embodiment, it is possible to extract core semantic information from the original data set and construct a compact initial synthetic data set.

[0070] Preferably, the sequence length of the third or fourth dimension of the attention matrix can be trimmed. For example, trimming the sequence length of the third dimension can only retain the content of the first two dimensions (retaining the attention content corresponding to the two heads). Another example is trimming the sequence length of the fourth dimension to focus on the global features of the entire sequence and retain the attention of the [CLS] token to the entire sequence.

[0071] In some of these embodiments, Figure 4 As shown, step S230 uses a preset inner and outer loop mechanism to process the initial synthetic data set and the original data set to obtain a target synthetic data set, including the following steps:

[0072] Step S231: Inputting the initial synthetic dataset into the initial multi-label classification model for inner loop training based on a preset first loss function to update the model parameters of the initial multi-label classification model; the first loss function includes a first task loss function, a first attention loss function, and a combined total loss;

[0073] In step S232, based on the preset first task loss function, the original data set is input into the initial multi-label classification model after inner loop training for outer loop training, and the initial synthetic data set is optimized based on the outer loop training results to obtain the target multi-label classification model and the target synthetic data set.

[0074] Specifically, the internal and external circulation mechanisms can be divided into internal circulation mechanisms and external circulation mechanisms.

[0075] The inner loop mechanism is used to adaptively update the model parameters of the initial multi-label classification model. During this phase, the initial synthetic dataset is used to train the initial multi-label classification model, gradually enabling the initial multi-label classification model to capture the key information in the initial synthetic dataset.

[0076] The specific process may be: inputting the embedded portion of the initial synthetic dataset (the second text embedding vector) into the initial multi-label classification model for forward propagation, calculating the task loss (such as binary cross entropy loss) to match the true labels; simultaneously, calculating the attention loss via KL divergence to ensure that the initial multi-label classification model learns a reasonable attention distribution. After combining the task loss and attention loss with a certain weight, the model parameters are updated via backpropagation, while freezing some layers of the pre-trained model to preserve the existing knowledge. In other embodiments, other approaches may be adopted, and this is not limited to this.

[0077] The outer loop mechanism is used to optimize the synthetic dataset to obtain the target synthetic dataset. After completing the inner and outer loop mechanisms, the target multi-label classification model is finally obtained. During the outer loop mechanism, the evaluation feedback of the original dataset is used to further calibrate and optimize the initial synthetic dataset to make it more realistically reflect the original data distribution. The specific process is described as follows:

[0078] In some embodiments, step S232 inputs the original dataset into the initial multi-label classification model after inner-loop training for outer-loop training based on a preset first task loss function, and optimizes the initial synthetic dataset based on the outer-loop training result to obtain a target multi-label classification model and a target synthetic dataset, including the following steps:

[0079] Input the original data set into the initial multi-label classification model after inner loop training for forward propagation, and calculate the first task loss function and the corresponding loss gradient;

[0080] Optimize the initial synthetic data based on the loss gradient and adjust the initial learning rate;

[0081] When the outer loop cutoff condition is met, the target multi-label classification model and target synthetic dataset are obtained.

[0082] Specifically, the original dataset is fed into the initial multi-label classification model trained in the inner loop for forward propagation to evaluate its performance and calculate the first task loss function and the corresponding loss gradient. The loss gradient is then propagated to the second text embedding vector and the shape of the second attention label in the initial synthetic data, and both are updated separately. A dynamic learning rate scheduling strategy (such as cosine annealing) is used to adjust the initial learning rate to ensure a stable update process. When the outer loop cutoff conditions are met, the target multi-label classification model and target synthetic dataset are obtained.

[0083] The outer loop cutoff condition is that when the number of iterations reaches the predetermined round T2 or the combined total loss converges, the final efficient and compact target synthetic data set S is output, and the training of the initial multi-label classification model is completed to obtain the target multi-label classification model. Each outer loop training can be embedded with an inner loop training, and it can be considered that there is at least one inner loop training in one outer loop training. The predetermined round T2 can be 30 times. In other embodiments, the outer loop can adopt other methods, which are not limited to this.

[0084] Through this embodiment, the initial synthetic data set can be further corrected to make it reflect the original data distribution more realistically.

[0085] In some embodiments, the process of processing the initial synthetic dataset and the original dataset using a preset inner and outer loop mechanism in step S230 to obtain a target synthetic dataset includes the following steps:

[0086] According to the preset two-stage loss strategy and the preset internal and external loop mechanism, the initial synthetic dataset and the original dataset are processed to obtain the target synthetic dataset;

[0087] The two-stage loss strategy includes the first-stage strategy and the second-stage strategy; the first-stage strategy is to use the first-task loss function for training; the second-stage strategy is to use the first-task loss function, the first attention loss function and the balance loss function for training.

[0088] In this embodiment, a two-stage loss strategy is combined with an internal and external loop mechanism to implement training, ensuring that the model fully captures the overall data distribution, and then focuses on optimizing the minority classes, thereby balancing global learning and local reinforcement; making the multi-label classification model perform better in class imbalance scenarios.

[0089] Among them, the two-stage loss strategy includes the first-stage strategy and the second-stage strategy; the first-stage strategy uses the standard first-task loss function L task After training global features and stabilizing the model performance, switch to the second-stage strategy. The second-stage strategy uses the first task loss function, the first attention loss function, and the balance loss function for training. By introducing the category balance factor and the focus parameter to adjust the weights of various types, the recognition ability of minority classes is improved.

[0090] Among them, the balance loss function can be expressed as L CB (Class Balanced Focus Loss); where the class balance factor is set ( is the number of samples in the i-th category, β is the preset coefficient) and the focus parameter γ (for example, γ=2), at this time, the total loss function of the second-stage strategy is updated as:

[0091] , thereby strengthening the learning of minority class features. Among them, L attn represents the first attention loss function.

[0092] In some embodiments, the distillation method for a multi-label text dataset further includes the following steps:

[0093] Obtain the text dataset corresponding to the downstream task, input the text dataset into the target multi-label classification model, and obtain the classification results of the downstream task.

[0094] Specifically, after verification, the target multi-label classification model can be directly applied to actual downstream tasks. For example, a text dataset corresponding to a specific downstream task can be obtained and input into the target multi-label classification model to obtain the classification results of the downstream task.

[0095] This embodiment can be applied to various classification scenarios and provide the applicability of the target multi-label classification model.

[0096] The present embodiment is described and illustrated below through preferred embodiments.

[0097] The steps of this preferred embodiment are:

[0098] If the original dataset T has a sequence length of N; the dimension of the first text embedding vector is d; the number of Transformer layers is L; the number of attention heads in each layer is H, and only H' heads are retained after pruning (expressed as H'<H); IPC samples are retained for each category; therefore, the initial synthetic dataset size is |S|=IPC×C (C is the total number of categories); the initial learning rate is set to η.

[0099] The implementation steps are:

[0100] Through the pre-trained language model, the original dataset T is converted into a d-dimensional first text embedding vector, and the first attention label distribution is obtained.

[0101] Use Gaussian noise to randomly initialize the first text embedding vector and obtain the second text embedding vector, which is expressed as ; Among them, R represents a real number, and E can be considered as a matrix with a shape of (|S|,d), and the elements of the matrix are all real numbers.

[0102] Use Gaussian noise to randomly initialize the first attention label A attn , the shape is (|S|, L, H, N, N); the shape of the first attention label is pruned to obtain the shape of the second attention label:

[0103] .

[0104] Give the second text embedding vector E and the second attention label the shape The initial learning rate η (can be 0.05).

[0105] Based on this, an initial synthetic dataset is constructed, which includes the shape of the second text embedding vector and the second attention label as well as the initial learning rate.

[0106] The following describes inner loop training, which is built into outer loop training.

[0107] The implementation steps of inner loop training are:

[0108] Input the embedded part of the initial synthetic dataset (the second text embedding vector E) into the initial multi-label classification model to obtain the predicted output Y pred ;

[0109] Calculate the first task loss function in the first loss function:

[0110] ;

[0111] in, is the true label, is the predicted probability.

[0112] Calculate the first attention loss function in the first loss function: L attn =KL(P,Q);

[0113] Among them, P is the preset attention distribution (obtained by A´), and Q is the self-attention distribution calculated by the initial multi-label classification model.

[0114] Total portfolio loss: (λ is a weight factor), and its loss gradient is calculated by back propagation.

[0115] The Adam optimizer is used to update the parameters of the initial multi-label classification model, which is expressed as:

[0116] At the same time, the parameters of the first 10 Transformer layers of the model are frozen to preserve the pre-trained knowledge. When the number of iterations reaches the predetermined number of rounds T1 (which can be at least 1), the inner loop training is completed.

[0117] The following describes outer loop training and optimization of synthetic datasets.

[0118] The implementation steps of outer loop training are:

[0119] Input the original data set T into the initial multi-label classification model after inner loop training for forward propagation, and calculate the first task loss function L task And the corresponding loss gradient; the first task loss function L task The loss gradient is passed to the initial synthetic dataset, and the shape of the second text embedding vector E and the second attention label Update them separately, and the update is expressed as:

[0120] ;

[0121] ;

[0122] The initial learning rate is adjusted using cosine annealing or linear decay strategy to ensure a stable update process.

[0123] When the number of iterations reaches the predetermined round T2 (which can be 30) or the combined total loss converges, the final efficient and compact target synthetic dataset S is output, and the training of the initial multi-label classification model is completed to obtain the target multi-label classification model.

[0124] Through the above-mentioned preferred embodiment, by constructing a compact synthetic dataset and collaboratively optimizing the inner and outer loops of meta-learning, a highly concentrated synthetic dataset can be generated at an extremely low data compression ratio, so that the trained target multi-label classification model can retain the performance of the original dataset while significantly reducing the storage and computing resource requirements.

[0125] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0126] In this embodiment, a distillation device for a multi-label text dataset is also provided. The device is used to implement the above-mentioned embodiments and preferred embodiments, and the details that have been described will not be repeated. The terms "module", "unit", "sub-unit", etc. used below can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation by hardware, or a combination of software and hardware, is also possible and conceivable.

[0127] Figure 5 This is a structural block diagram of the distillation device for the multi-label text dataset of this embodiment, such as Figure 5 As shown, the device includes: an acquisition module 210, a construction module 220 and a processing module 230;

[0128] An acquisition module 210 is configured to acquire a first text embedding vector and a shape of a first attention label of an original data set; the original data set is a multi-label text data set;

[0129] A construction module 220 is configured to prune the shape of the first attention label and construct an initial synthetic dataset based on the pruning result and the first text embedding vector;

[0130] The processing module 230 is configured to process the initial synthetic data set and the original data set using a preset internal and external loop mechanism to obtain a target synthetic data set.

[0131] Through the above-mentioned device, the problem of poor performance on minority class samples and reduced recognition accuracy in related technologies is solved; the computational cost of attention labels is reduced by pruning operations, and the fine-grained semantic features in the text can be captured, thereby improving the reliability of the dataset and further improving the recognition accuracy of subsequent classification models; and it is possible to generate a highly concentrated target synthetic dataset at an extremely low data compression ratio, so that the subsequently trained multi-label classification model can retain the performance of the original data while significantly reducing the storage and computing resource requirements.

[0132] In some embodiments, the acquisition module 210 is further configured to transform the acquired original data set using a pre-trained language model to obtain a first text embedding vector and a first attention tag distribution.

[0133] In some embodiments, the construction module 220 is further configured to randomly initialize the first text embedding vector using Gaussian noise to obtain a second text embedding vector;

[0134] Use Gaussian noise to randomly initialize the first attention label and prune the shape of the first attention label to obtain the shape of the second attention label;

[0135] An initial learning rate that gives the shape of the second text embedding vector and the second attention label to construct the initial synthetic dataset.

[0136] In some embodiments, the processing module 230 is further configured to input the initial synthetic data set into the initial multi-label classification model for inner loop training based on a preset first loss function to update model parameters of the initial multi-label classification model; the first loss function includes a first task loss function, a first attention loss function, and a combined total loss;

[0137] Based on the preset first task loss function, the original dataset is input into the initial multi-label classification model after inner loop training for outer loop training, and the initial synthetic dataset is optimized based on the outer loop training results to obtain the target multi-label classification model and target synthetic dataset.

[0138] In some embodiments, the processing module 230 is further configured to input the original data set into the initial multi-label classification model after inner loop training for forward propagation, and calculate the first task loss function and the corresponding loss gradient;

[0139] Optimize the initial synthetic data based on the loss gradient and adjust the initial learning rate;

[0140] When the outer loop cutoff condition is met, the target multi-label classification model and target synthetic dataset are obtained.

[0141] In some embodiments, the processing module 230 is further configured to process the initial synthetic data set and the original data set according to a preset two-stage loss strategy and a preset inner and outer loop mechanism to obtain a target synthetic data set;

[0142] The two-stage loss strategy includes the first-stage strategy and the second-stage strategy; the first-stage strategy is to use the first-task loss function for training; the second-stage strategy is to use the first-task loss function, the first attention loss function and the balance loss function for training.

[0143] In some of these embodiments, the distillation apparatus for a multi-label text dataset further includes: a classification module;

[0144] The classification module is used to obtain the text dataset corresponding to the downstream task, input the text dataset into the target multi-label classification model, and obtain the classification results of the downstream task.

[0145] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0146] This embodiment further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0147] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0148] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0149] S1, obtain the shape of the first text embedding vector and the first attention label of the original dataset; the original dataset is a multi-label text dataset;

[0150] S2, prunes the shape of the first attention label and constructs an initial synthetic dataset based on the pruning result and the first text embedding vector;

[0151] S3, using the preset internal and external loop mechanism, processes the initial synthetic dataset and the original dataset to obtain the target synthetic dataset.

[0152] It should be noted that, for specific examples in this embodiment, reference may be made to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0153] In addition, in conjunction with the multi-label text dataset distillation method provided in the above embodiments, this embodiment may also provide a storage medium for implementation. The storage medium stores a computer program; when executed by a processor, the computer program implements any of the multi-label text dataset distillation methods described in the above embodiments.

[0154] It should be noted that the information and data involved in this application are all authorized by the user or fully authorized by all parties and will be used legally.

[0155] It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0156] Obviously, the accompanying drawings are merely examples or embodiments of the present application. A person skilled in the art can also apply the present application to other similar situations based on these drawings without inventive effort. Furthermore, it is understandable that, although the work involved in this development process may be complex and lengthy, certain design, manufacturing, or production changes based on the technical content disclosed in this application are merely routine technical means for a person skilled in the art and should not be considered to constitute a deficiency in the disclosure of the present application.

[0157] The term "embodiment" as used in this application refers to specific features, structures, or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily mean that the embodiment is the same, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is understood, either explicitly or implicitly, by those skilled in the art that the embodiments described in this application can be combined with other embodiments when there is no conflict.

[0158] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A distillation method for a multi-label text dataset, characterized in that: include: Obtain the shape of the first text embedding vector and the first attention label of the original dataset; the original dataset is a multi-label text dataset; the shape of the first attention label is the dimensional structure of the weight matrix or mask generated in the attention mechanism, the specific shape of which depends on the task type, input data structure, and attention type; the shape of the first attention label is determined by the required synthetic dataset size, the number of Transformer layers, the number of attention heads, and the sequence length; Pruning the shape of the first attention label and constructing an initial synthetic dataset based on the pruning result and the first text embedding vector; the pruning is based on trimming the number of the attention heads to retain key attention heads; The initial synthetic data set and the original data set are processed using a preset internal and external loop mechanism to obtain a target synthetic data set.

2. The distillation method for a multi-label text dataset according to claim 1, characterized in that: Get the shape of the first text embedding vector and the first attention label of the original dataset, including: The obtained original data set is transformed using the pre-trained language model to obtain the first text embedding vector and the first attention label distribution.

3. The distillation method for a multi-label text dataset according to claim 2, characterized in that: Pruning the shape of the first attention label and constructing an initial synthetic dataset based on the pruning result and the first text embedding vector, including: Randomly initialize the first text embedding vector using Gaussian noise to obtain a second text embedding vector; Use Gaussian noise to randomly initialize the first attention label and prune the shape of the first attention label to obtain the shape of the second attention label; An initial learning rate of the shape of the second text embedding vector and the second attention label is given to construct an initial synthetic dataset.

4. The distillation method for a multi-label text dataset according to claim 1, characterized in that: The initial synthetic data set and the original data set are processed using a preset internal and external loop mechanism to obtain a target synthetic data set, including: Based on a preset first loss function, inputting the initial synthetic dataset into an initial multi-label classification model for inner loop training to update model parameters of the initial multi-label classification model; the first loss function includes a first task loss function, a first attention loss function, and a combined total loss; Based on the preset first task loss function, the original data set is input into the initial multi-label classification model after inner loop training for outer loop training, and the initial synthetic data set is optimized based on the outer loop training result to obtain the target multi-label classification model and the target synthetic data set.

5. The distillation method for a multi-label text dataset according to claim 4, characterized in that: Based on a preset first task loss function, the original dataset is input into the initial multi-label classification model after inner loop training for outer loop training, and the initial synthetic dataset is optimized based on the outer loop training result to obtain a target multi-label classification model and the target synthetic dataset, including: Inputting the original data set into the initial multi-label classification model after inner loop training for forward propagation, and calculating the first task loss function and the corresponding loss gradient; Optimizing the initial synthetic data based on the loss gradient and adjusting the initial learning rate; When the outer loop cutoff condition is met, a target multi-label classification model and the target synthetic dataset are obtained.

6. The distillation method for a multi-label text dataset according to claim 1, characterized in that: The initial synthetic data set and the original data set are processed using a preset internal and external loop mechanism to obtain a target synthetic data set, including: Processing the initial synthetic dataset and the original dataset according to a preset two-stage loss strategy and a preset internal and external loop mechanism to obtain a target synthetic dataset; The two-stage loss strategy includes a first-stage strategy and a second-stage strategy; the first-stage strategy is to use the first task loss function for training; the second-stage strategy is to use the first task loss function, the first attention loss function and the balance loss function for training.

7. The distillation method for a multi-label text dataset according to claim 4, characterized in that: The method further comprises: Obtain a text dataset corresponding to the downstream task, input the text dataset into the target multi-label classification model, and obtain a classification result of the downstream task.

8. A distillation device for a multi-label text dataset, characterized in that: include: Acquire modules, build modules, and process modules; The acquisition module is configured to obtain the shape of a first text embedding vector and a first attention label of an original dataset; the original dataset is a multi-label text dataset; the shape of the first attention label is the dimensional structure of a weight matrix or mask generated in the attention mechanism, the specific shape of which depends on the task type, input data structure, and attention type; the shape of the first attention label is determined by the required synthetic dataset size, the number of Transformer layers, the number of attention heads, and the sequence length; The construction module is configured to prune the shape of the first attention label and construct an initial synthetic dataset based on the pruning result and the first text embedding vector; the pruning is performed by trimming the number of the attention heads to retain key attention heads; The processing module is used to process the initial synthetic data set and the original data set using a preset internal and external loop mechanism to obtain a target synthetic data set.

9. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the steps of the distillation method for a multi-label text dataset according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the distillation method of a multi-label text dataset according to any one of claims 1 to 7 are implemented.

Images