Zero-sample distillation system and method for case classification based on pseudo-word sequence generation
By generating pseudo-samples through the word embedding layer of the teacher model, and utilizing multivariate normal distribution sampling and supervised optimization, the problem of text discrete optimization is solved, enabling knowledge transfer and lightweight model deployment in a data privacy context, and improving the distillation effect.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-30
- Publication Date
- 2026-04-03
AI Technical Summary
In existing technologies, the discrete nature of text word generation is difficult to optimize, resulting in pseudo-samples that do not closely approximate the real data distribution, poor distillation performance, and the inability to obtain original training data due to data privacy concerns, making it difficult to achieve knowledge transfer.
Pseudo-samples that closely resemble real data are generated through the word embedding layer of the teacher model. Continuous pseudo-sample vectors are generated using multivariate normal distribution sampling and supervised optimization. The knowledge of the teacher model is then transferred to the lightweight student model through the distillation module.
In a data privacy context, the effective deployment and knowledge transfer of lightweight models were achieved, and the generated pseudo-samples closely approximate the real data distribution, thus improving distillation performance.
Smart Images
Figure CN116011559B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of natural language processing, specifically referring to a zero-sample distillation system and method for case classification based on pseudo-word sequence generation. Background Technology
[0002] Case records, as defined by medical institutions, document instances of a disease, documenting a person's or organism's experience with that illness. They play a crucial role in medical treatment, prevention, teaching, research, and hospital management. However, in reality, much case data is unavailable, and cases often contain diverse categories of information. Manual screening would incur significant manpower costs, and the sheer volume of cases increases the error rate. Case data classification aids in disease diagnosis; for example, determining the type of cancer or diabetes requires considering various hospital examination indicators and the patient's specific condition. However, due to data privacy concerns, some patients' medical and personal characteristics are generally not publicly available. Therefore, in practice, to obtain a lightweight and easily deployable model capable of identifying a patient's disease type, we can generate pseudo-samples from these unpublished data and distill them to obtain the desired lightweight model. In this context, zero-shot knowledge distillation using natural language processing can solve this problem.
[0003] First, let me introduce knowledge distillation. Originally proposed by Hinton, knowledge distillation refers to the process of introducing a well-trained, complex but highly accurate teacher model to induce a simplified, low-complexity student network that is more suitable for inference deployment, thereby achieving knowledge transfer.
[0004] However, in practical distillation, the original training dataset is needed. But due to data privacy concerns, data such as biometric data and patient healthcare information are generally not publicly available, making the original training data unavailable. This led to the development of zero-shot knowledge distillation. Traditional zero-shot knowledge distillation in natural language processing often uses datasets with many samples and few categories, generates pseudo-samples in a discrete manner that is difficult to optimize, and has shortcomings in its pseudo-sample generation methods.
[0005] In view of this, the pseudo-sample generation method based on pseudo-word sequences proposed in this invention can generate pseudo-samples that are close to the word embedding vectors of the original data in the same space through the word embedding layer of the teacher model itself. This is continuous and easy to optimize, thereby improving distillation performance. Furthermore, this invention is also innovative in terms of technology. Because images are continuous and can be directly optimized, they are relatively easy to generate. However, text in natural language is discrete and not easy to optimize directly. Therefore, this invention also proposes a new method for generating text pseudo-samples from a novel perspective. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this invention aims to provide a zero-sample distillation system and method for case classification based on pseudo-word sequence generation. This solves the problem in existing technologies where discrete text word generation is difficult to optimize, leading to pseudo-samples that do not closely approximate the real data distribution and resulting in poor distillation performance. This invention enables a teacher model trained on real data to better transfer and compress its knowledge to a lightweight student model when real data is unavailable due to data privacy or other reasons. This facilitates deployment and achieves the goal of knowledge transfer. Furthermore, this invention does not require a separate generative model; it can directly optimize continuous pseudo-sample vectors through the teacher model's word embedding layer.
[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0008] The zero-sample distillation system and method for case classification based on pseudo-word sequence generation of the present invention is characterized by comprising: a model separation module; a multivariate normal distribution generation module; a pseudo-sample generation module; and a distillation module;
[0009] The model separation module divides the teacher model into a word embedding layer and the remaining encoding layer, and performs sentence segmentation on the publicly available case dataset. In this application, the "publicly available case dataset" refers to case datasets publicly available on the Internet, including case data of diseases such as heart disease and COVID-19 infection, which need to be segmented first.
[0010] The multivariate normal distribution generation module includes: a multivariate normal distribution coefficient calculation module and a simulated label distribution sampling calculation module;
[0011] The multivariate normal distribution coefficient calculation module calculates the correlation coefficient matrix of the multivariate normal distribution based on the pooler layer weights of the trained teacher model, sets the multivariate normal distribution variance matrix as a diagonal matrix, calculates the covariance matrix, calculates the multivariate normal distribution coefficients, and obtains the corresponding multivariate normal distribution.
[0012] The simulated label distribution sampling module randomly samples the obtained multivariate normal distribution and performs softmax calculation on the sampled samples;
[0013] The pseudo-sample generation module includes: a pseudo-sample label generation module and a multivariate normal distribution sampling supervised optimization module;
[0014] The pseudo-sample label distribution generation module performs uniform sampling between a set minimum value and a set maximum value, uses the obtained samples to replace the word indices in the Bert vocabulary and inputs them into the word embedding layer of the teacher model to obtain word embedding vectors, and then inputs them into the teacher model to obtain the pseudo-sample label distribution.
[0015] The multivariate normal distribution sampling supervised optimization module uses multivariate normal distribution sampling as supervision information for pseudo-sample label distribution, optimizes the word embedding layer segmented by the teacher model until a word embedding vector that meets the requirements is obtained, and reinitializes the word embedding layer.
[0016] The distillation module includes: a word embedding space consistency training module and a knowledge transfer module;
[0017] The word embedding space consistency training module transforms the word embedding vectors of the teacher model and the word embedding vectors of the student model into the same space after a linear layer transformation.
[0018] The knowledge transfer module inputs the generated word embedding vectors into the linear layers of the teacher model and the student model respectively, transforms them into word embedding vectors of a set dimension, obtains the label distributions output by the teacher model and the student model, aligns them, and thus transfers the knowledge of the teacher model to the lightweight student model.
[0019] Furthermore, the multivariate normal distribution generation module separates the word embedding layer and the encoding layer of the teacher model that has already been trained on real training data to form a new model and save it;
[0020] The new model includes: a word embedding layer module and an encoder module;
[0021] Word embedding layer module: Transforms the input X, as expressed by the following formula:
[0022] Y = E t (X)+E s (X)+E p (X),
[0023] Among them, E t E is an operator that converts words into fixed-dimensional vector representations. s For the segmentation embedding operator, E p For position embedding operators;
[0024] The transformer encoder module employs an attention mechanism A, which consists of three matrices—Q (Query), K (Key), and V (Value)—originating from the same input, forming the encoder F. a And the corresponding residual mechanism and layer normalization F n The formula is expressed as follows:
[0025]
[0026] The expression for softmax is:
[0027]
[0028] n represents the n values of the softmax input vector.
[0029] The encoder expression is:
[0030] Y = F n ...(F n (F n (X+F a (X))+F a (F n (X+F a (X)))).
[0031] Furthermore, the corresponding multivariate normal distribution is expressed by the following formula:
[0032]
[0033]
[0034] Σ=D·C·D
[0035] C(i,j) is the class similarity coefficient, w i and w j These are the neuron weights at positions i and j in the pooler layer of the teacher model, respectively, where D is the standard deviation of each element in a multivariate normal distribution. The diagonal matrix formed by these two elements, Σ, is the covariance matrix of a multivariate normal distribution.
[0036] Furthermore, the simulated label distribution sampling module incorporates a scaling factor to adjust the probability sharpness of the simulated label distribution for different categories of data during the softmax calculation process.
[0037] Furthermore, the simulated label distribution sampling module samples the noise z from the multivariate normal distribution, inputs it into the last classification linear layer of the teacher model, multiplies it with the weight W to obtain the output, and then divides the output by the scaling factor α before inputting it into the softmax function F.s The simulated label distribution is calculated and expressed by the following formula:
[0038] z~N(μ,Σ)
[0039]
[0040] Furthermore, the pseudo-sample label distribution generation module replaces the word index X in the BERT vocabulary with randomly generated uniformly distributed integers and inputs them into the word embedding layer F of the teacher model. emb The word embedding vectors are obtained and then input into the encoding layer F of the teacher model. ec (θ T The pseudo-sample label distribution is obtained from θ. T The teacher model parameters are expressed by the following formula:
[0041] Y = F ec (F emb (X),θ T ).
[0042] Furthermore, the multivariate normal distribution sampling supervised optimization module uses the sampled multivariate normal distribution samples n as label information to supervise the generation of pseudo-samples and backpropagate to optimize the word embedding layer E. T This process continues until a pseudo-sample that meets the conditions is generated as the final pseudo-sample x. The loss function used is cross-entropy loss L. CE and KL divergence loss L KL Furthermore, a temperature coefficient τ is incorporated to control the sharpness of the label distribution, expressed by the following formula:
[0043]
[0044] Furthermore, the word embedding space consistency training module, by inputting the i-th sentence x from the publicly available case dataset... i Embedded layer E of the teacher model T Embedding layer E of the student model S The word embedding vectors of the teacher model and the student model are transformed by a linear layer and then subjected to KL divergence L. KL The linear layer is processed and optimized to ensure that the word embedding vectors from both sides are in the same space. The formula is expressed as follows:
[0045]
[0046] Furthermore, the knowledge transfer module inputs the generated word embedding vector X into the encoding layer F of the teacher model. T (θ T ), and the linear layer F that transforms the word vector dimensions. fcThe word embeddings are transformed into 768-dimensional word embedding vectors to obtain the label distributions output by the teacher and student models. These distributions are then aligned to transfer the knowledge from the teacher model to the lightweight student model using KL divergence. KL As a loss function, the encoding layer F of the student model is optimized. S (θ S The formula is expressed as follows:
[0047]
[0048] The beneficial effects of this invention are:
[0049] 1. This invention considers data privacy. Case data includes patients' age, gender, past medical history, and various physical indicators. For example, cancer diagnosis classification datasets include indicators such as average tumor smoothness and average radius to jointly determine the type of cancer. This invention uses a pre-trained case classification teacher model to generate pseudo-word sequences for the distillation process, better transferring and compressing the knowledge of the teacher model into a lightweight student model. This allows for the creation of a student model even without the original case dataset, enabling its deployment in relevant case diagnosis scenarios and achieving good diagnostic results.
[0050] 2. This invention simulates the label distribution of real text data by sampling using a multivariate normal distribution, and uses it as supervision information to generate pseudo-word sequences. Furthermore, it uses sentences from a publicly available case dataset as input, and the supervised generation model generates pseudo-samples.
[0051] 3. This invention proposes a novel text generation method that overcomes the difficulty of optimizing discrete text by optimizing and generating continuous word embedding vectors as pseudo-word sequences. Furthermore, it uses the loss of the generation process as a constraint to select pseudo-word sequences that meet the conditions as optimal pseudo-samples, thereby enabling the next step of the distillation process to transfer the knowledge of the teacher model to the student model. Attached Figure Description
[0052] Figure 1 This is a schematic diagram of the module design of the present invention.
[0053] Figure 2 This is a schematic diagram of the separation module of the present invention.
[0054] Figure 3 This is a schematic diagram of the module for generating a multivariate normal distribution.
[0055] Figure 4 This is a schematic diagram of the pseudo-sample generation module.
[0056] Figure 5 This is a schematic diagram of distillation learning. Detailed Implementation
[0057] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and accompanying drawings. The content mentioned in the embodiments is not intended to limit the present invention.
[0058] This invention achieves zero-shot knowledge distillation by generating pseudo-samples of word embedding vectors using a multivariate normal distribution and a teacher model.
[0059] Reference Figure 1 As shown, the zero-sample distillation system and method for case classification based on pseudo-word sequence generation of the present invention includes: a model separation module; a multivariate normal distribution generation module; a pseudo-sample generation module; and a distillation module; wherein,
[0060] Reference Figure 2 As shown, the model separation module divides the teacher model into a word embedding layer and the remaining encoding layer, and performs sentence segmentation on the public case dataset;
[0061] Reference Figure 3 As shown, the multivariate normal distribution generation module includes: a multivariate normal distribution coefficient calculation module and a simulated label distribution sampling calculation module;
[0062] The multivariate normal distribution coefficient calculation module calculates the correlation coefficient matrix of the multivariate normal distribution using the pooler layer weights of the teacher model trained in advance using the original real training dataset, and sets the multivariate normal distribution variance matrix as a diagonal matrix with all diagonal elements being 1 and the mean being 0, thereby calculating the covariance matrix, calculating the multivariate normal distribution coefficient, and obtaining the corresponding multivariate normal distribution.
[0063] The multivariate normal distribution coefficient calculation module calculates the correlation coefficient matrix C of the multivariate normal distribution using the pooler layer weights W of the teacher model trained using the original real training dataset. It also sets the multivariate normal distribution variance matrix D as a diagonal matrix, with each diagonal element representing the standard deviation σ1 of the multivariate normal distribution and a mean μ of 0. This allows for the calculation of the covariance matrix Σ, and the calculation of the multivariate normal distribution coefficients to obtain the corresponding multivariate normal distribution. The formula is as follows:
[0064]
[0065]
[0066] Σ=D·C·D
[0067] The simulated label distribution sampling module samples the noise z from the multivariate normal distribution, inputs it into the last classification linear layer of the teacher model, multiplies it with the weight W to obtain the output, and then divides the output by the scaling factor α before inputting it into the softmax function F. sThe simulated label distribution is calculated and expressed by the following formula:
[0068] z~N(μ,Σ)
[0069]
[0070] Reference Figure 4 As shown, the pseudo-sample label distribution generation module replaces the word indices in the BERT vocabulary with randomly generated uniformly distributed integers and inputs them into the word embedding layer of the teacher model to obtain word embedding vectors, which are then input into the teacher model to obtain the pseudo-sample label distribution.
[0071] The pseudo-sample label distribution generation module replaces the word index X in the Bert vocabulary with randomly generated uniformly distributed integers and inputs them into the word embedding layer F of the teacher model. emb The word embedding vectors are obtained and then input into the encoding layer F of the teacher model. ec (θ T The pseudo-sample label distribution is obtained from the data, and the formula is expressed as follows:
[0072] Y = F ec (F emb (X),θ T )
[0073] The multivariate normal distribution sampling supervised optimization module uses the sampled multivariate normal distribution samples n as label information to supervise the generation of pseudo-samples and backpropagate to optimize the word embedding layer E. T This process continues until a pseudo-sample that meets the conditions is generated as the final pseudo-sample x. The loss function used is cross-entropy loss L. CE and KL divergence loss L KL Furthermore, a temperature coefficient τ is incorporated to control the sharpness of the label distribution, expressed by the following formula:
[0074]
[0075] Reference Figure 5 As shown, the distillation module includes: a word embedding space consistency training module and a knowledge transfer module;
[0076] The word embedding space consistency training module is trained by inputting the i-th sentence x from the publicly available case dataset. i Embedded layer E of the teacher model T Embedding layer E of the student model S The teacher model word embedding vector dimension of 768 and the student model word embedding vector dimension of 256 are transformed by a linear layer and then subjected to KL divergence L. KL The linear layer is processed and optimized to ensure that the word embedding vectors from both sides are in the same space. The formula is expressed as follows:
[0077]
[0078] The knowledge transfer module inputs the generated word embedding vector X into the encoding layer F of the teacher model. T (θ T ), and the linear layer F that transforms the word vector dimensions. fc The word embeddings are transformed into 768-dimensional word embedding vectors to obtain the label distributions output by the teacher and student models. These distributions are then aligned to transfer the knowledge from the teacher model to the lightweight student model using KL divergence. KL As a loss function, the encoding layer F of the student model is optimized. S (θ S The formula is expressed as follows:
[0079]
[0080] The zero-sample distillation system and method for case classification based on pseudo-word sequence generation of the present invention includes the following steps:
[0081] 1) Obtain the teacher model, and segment and save the word embedding layer and encoding layer of the teacher model, as well as obtain the publicly available case dataset after sentence segmentation;
[0082] 2) The covariance matrix of the multivariate normal distribution is calculated using the weights of the pooler layer network in the teacher model, thus obtaining the multivariate normal distribution;
[0083] 3) Sample the obtained multivariate normal distribution and input it into the classification linear layer and softmax layer of the teacher model to obtain the label distribution supervision information of different categories;
[0084] 4) Randomly generate uniformly distributed word indices, input them into the word embedding layer of the teacher model to obtain word embedding vectors, and then input them into the encoding layer of the teacher model. Combine them with the label distribution supervision information obtained in the previous step to optimize the word embedding layer.
[0085] 5) Optimize the word embedding layer until a word embedding vector that meets the conditions is generated, then initialize the word embedding layer and repeat step 4) until the generation is complete;
[0086] 6) Input the sentences from the public case dataset into the embedding layers of the teacher model and the student model to obtain word embedding vectors. Input the word embedding vectors of the student model into an additional linear layer to obtain 768-dimensional word embedding vectors with the same dimension as the teacher model. Perform loss optimization on the linear layer, thereby using the linear layer to map the word embedding vectors of the student model and the teacher model to the same space.
[0087] 7) Input the generated word embedding vector into the encoding layer of the teacher model, then input it into the linear layer in 6) to obtain a 768-dimensional vector, and then input it into the encoding layer of the student model. Use the label distribution obtained from both models as the loss to optimize the student model, so as to achieve knowledge transfer and compression.
[0088] This invention has many specific applications. The above description is only a preferred embodiment of this invention. It should be noted that for those skilled in the art, several improvements can be made without departing from the principle of this invention, and these improvements should also be considered within the scope of protection of this invention.
Claims
1. A zero-sample distillation system and method for case classification based on pseudo-word sequence generation, characterized in that, include: Model separation module; Multivariate normal distribution generation module; Pseudo-sample generation module; Distillation module; The model separation module divides the teacher model into a word embedding layer and the remaining encoding layer, and performs sentence segmentation on the public case dataset; The multivariate normal distribution generation module includes: a multivariate normal distribution coefficient calculation module and a simulated label distribution sampling calculation module; The multivariate normal distribution coefficient calculation module calculates the correlation coefficient matrix of the multivariate normal distribution based on the pooler layer weights of the trained teacher model, sets the multivariate normal distribution variance matrix as a diagonal matrix, calculates the covariance matrix, calculates the multivariate normal distribution coefficients, and obtains the corresponding multivariate normal distribution. The simulated label distribution sampling module randomly samples the obtained multivariate normal distribution and performs softmax calculation on the sampled samples; The pseudo-sample generation module includes: a pseudo-sample label generation module and a multivariate normal distribution sampling supervised optimization module; The pseudo-sample label distribution generation module performs uniform sampling between a set minimum value and a set maximum value, uses the obtained samples to replace the word indices in the Bert vocabulary and inputs them into the word embedding layer of the teacher model to obtain word embedding vectors, and then inputs them into the teacher model to obtain the pseudo-sample label distribution. The multivariate normal distribution sampling supervised optimization module uses multivariate normal distribution sampling as supervision information for pseudo-sample label distribution, optimizes the word embedding layer segmented by the teacher model until a word embedding vector that meets the requirements is obtained, and reinitializes the word embedding layer. The distillation module includes: a word embedding space consistency training module and a knowledge transfer module; The word embedding space consistency training module transforms the word embedding vectors of the teacher model and the word embedding vectors of the student model into the same space after a linear layer transformation. The knowledge transfer module inputs the generated word embedding vectors into the linear layers of the teacher model and the student model respectively, transforms them into word embedding vectors of a set dimension, obtains the label distributions output by the teacher model and the student model, aligns them, and thus transfers the knowledge of the teacher model to the lightweight student model. The pseudo-sample label distribution generation module replaces the word index X in the BERT vocabulary with a randomly generated uniformly distributed integer and inputs it into the word embedding layer Femb of the teacher model to obtain the word embedding vector. Then, it inputs this vector into the encoding layer Fec(θT) of the teacher model to obtain the pseudo-sample label distribution, where θT is the teacher model parameter. The formula is expressed as follows: Y = Fec(Femb(X),θT).
2. The zero-sample distillation system and method for case classification based on pseudo-word sequence generation according to claim 1, characterized in that, The multivariate normal distribution generation module separates the word embedding layer and the encoding layer of the teacher model that has been trained on real training data to form a new model and save it. The new model includes: a word embedding layer module and an encoder module; Word embedding layer module: Transforms the input X, as expressed by the following formula: Y=E t (X)+E s (X)+E p (X), Among them, E t E is an operator that converts words into fixed-dimensional vector representations. s For the segmentation embedding operator, E p For position embedding operators; The transformer encoder module employs an attention mechanism A, which consists of three matrices—Q (Query), K (Key), and V (Value)—originating from the same input, forming the encoder F. a And the corresponding residual mechanism and layer normalization F n The formula is expressed as follows: The expression for softmax is: n represents the n values of the softmax input vector. The encoder expression is: Y=F n ...(F n (F n (X+F a (X))+F a (F n (X+F a (X))))。 3. The zero-sample distillation system and method for case classification based on pseudo-word sequence generation according to claim 1, characterized in that, The corresponding multivariate normal distribution is expressed by the following formula: Σ=D·C·D C(i,j) is the class similarity coefficient, w i and w j These are the neuron weights at positions i and j in the pooler layer of the teacher model, respectively, where D is the standard deviation of each element in a multivariate normal distribution. The diagonal matrix formed by these two elements, Σ, is the covariance matrix of a multivariate normal distribution.
4. The zero-sample distillation system and method for case classification based on pseudo-word sequence generation according to claim 1, characterized in that, The simulated label distribution sampling module incorporates scaling factors during the softmax calculation process to adjust the probability sharpness of the simulated label distribution for different categories of data.
5. The zero-sample distillation system and method for case classification based on pseudo-word sequence generation according to claim 1, characterized in that, The simulated label distribution sampling module samples the noise z from the multivariate normal distribution, inputs it into the last classification linear layer of the teacher model, multiplies it with the weight W to obtain the output, and then divides the output by the scaling factor α before inputting it into the softmax function F. s The simulated label distribution is calculated and expressed by the following formula: z~N(μ,Σ) 6. The zero-sample distillation system and method for case classification based on pseudo-word sequence generation according to claim 1, characterized in that, The multivariate normal distribution sampling supervised optimization module uses the sampled multivariate normal distribution samples n as label information to supervise the generation of pseudo-samples and backpropagate to optimize the word embedding layer E. T This process continues until a pseudo-sample that meets the conditions is generated as the final pseudo-sample x. The loss function used is cross-entropy loss. and KL divergence loss Furthermore, a temperature coefficient τ is incorporated to control the sharpness of the label distribution, expressed by the following formula:
7. The zero-sample distillation system and method for case classification based on pseudo-word sequence generation according to claim 1, characterized in that, The word embedding space consistency training module is trained by inputting the i-th sentence x from the publicly available case dataset. i Embedded layer E of the teacher model T Embedding layer E of the student model S The word embedding vectors of the teacher model and the student model are transformed by a linear layer and then subjected to KL divergence L. KL The linear layer is processed and optimized to ensure that the word embedding vectors from both sides are in the same space. The formula is expressed as follows:
8. The zero-sample distillation system and method for case classification based on pseudo-word sequence generation according to claim 1, characterized in that, The knowledge transfer module inputs the generated word embedding vector X into the encoding layer F of the teacher model. T (θ T ), and the linear layer F that transforms the word vector dimensions. fc The word embeddings are transformed into 768-dimensional word embedding vectors to obtain the label distributions output by the teacher and student models. These distributions are then aligned to transfer the knowledge from the teacher model to the lightweight student model, using KL divergence. KL As a loss function, the encoding layer F of the student model is optimized. S (θ S The formula is expressed as follows:
9. A zero-sample distillation system and method for case classification based on pseudo-word sequence generation, characterized in that, The steps include the following: 1) Obtain the teacher model, divide the teacher model into a word embedding layer and an encoding layer, and perform sentence segmentation on the public case dataset to obtain the off-domain dataset; 2) The covariance matrix of the multivariate normal distribution is calculated using the weights of the pooler layer network in the teacher model, thus obtaining the multivariate normal distribution; 3) Sample the obtained multivariate normal distribution and input it into the classification linear layer and softmax layer of the teacher model to obtain the label distribution supervision information of different categories; 4) Randomly generate uniformly distributed word indices, input them into the word embedding layer of the teacher model to obtain word embedding vectors, and then input them into the encoding layer of the teacher model. Use the label distribution supervision information obtained in step 3) to perform loss optimization on the word embedding layer. 5) Optimize the word embedding layer until a word embedding vector that meets the conditions is generated, then initialize the word embedding layer and repeat step 4) until the generation is complete; 6) Input the sentences from the public case dataset into the embedding layers of the teacher model and the student model to obtain word embedding vectors. Input the word embedding vectors of the student model into an additional linear layer to obtain word embedding vectors with the same dimension as the teacher model. Perform loss optimization on the linear layer, thereby using the linear layer to map the word embedding vectors of the student model and the teacher model to the same space. 7) Input the generated word embedding vector into the encoding layer of the teacher model, input it into the linear layer in step 6) to obtain a 768-dimensional vector, and then input it into the encoding layer of the student model. Calculate the loss between the label distributions obtained from the two models to optimize the student model, so as to achieve knowledge transfer and compression.