A fine-tuning stage distillation method and device for text-based models

Abstract

This specification discloses a distillation method for the fine-tuning stage of a text-based model, comprising: acquiring training set samples and a pre-trained teacher model, the teacher model including multiple hidden layers; selecting at least two specified layers from the multiple hidden layers of the teacher model and aggregating them to obtain aggregated hidden layers; generating a student model based on the aggregated hidden layers and the remaining unspecified layers in the teacher model; distilling the student model using the teacher model; and reusing the distilled student model as the teacher model until it is impossible to select at least two specified layers from the teacher model, at which point the teacher model at this point is used as the target model. Accordingly, this specification also discloses a distillation apparatus for the fine-tuning stage of a text-based model.

Description

Technical Field

[0001] The embodiments of this specification relate to a model optimization method and apparatus, and more particularly to a distillation method and apparatus for the model fine-tuning stage. Background Technology

[0002] With the rapid development of science and technology, data is spreading more widely and faster, leading to increasingly higher demands for data security. To meet the needs of rapidly evolving content risk control scenarios, more and more algorithms and models are being used, but cost issues are becoming increasingly prominent.

[0003] Typically, efforts are made to reduce machine costs through model optimization and model merging. However, due to the diverse application scenarios and broad requirements of content risk control, the models used also have their own characteristics. Among these, methods for "slimming down" the BERT model mainly include pruning, distillation, and quantization.

[0004] Among these methods, quantization offers limited improvement to the model, pruning has an unstable effect, while distillation has better robustness.

[0005] Therefore, the embodiments in this specification aim to propose a new distillation method for the fine-tuning stage of text-based models, which can improve the training effect of the model while better controlling the computational cost. Summary of the Invention

[0006] One of the objectives of the embodiments in this specification is to provide a distillation method for fine-tuning text-based models. This method enables student models to learn the intermediate hidden layer information of teacher models while learning progressively at different training stages. The resulting target model can meet accuracy requirements while reducing resource consumption, and can efficiently complete text classification tasks in scenarios such as content risk control.

[0007] In accordance with the above objectives, embodiments of this specification propose a distillation method for the fine-tuning stage of a text-based model, the method comprising:

[0008] Obtain training set samples;

[0009] Obtain a pre-trained teacher model, which includes multiple hidden layers;

[0010] Select at least two designated layers from the multiple hidden layers of the teacher model;

[0011] Aggregate the selected specified layers to obtain the aggregated hidden layer;

[0012] The student model is generated based on the aggregated hidden layer and the remaining unspecified layers in the teacher model;

[0013] The teacher model is used to distill the student model;

[0014] The distilled student model is reused as the teacher model until at least two specified layers cannot be selected from the teacher model. At this point, the teacher model is used as the target model.

[0015] In one or more embodiments of this specification, the text-based model may include the BERT model. Distillation of the BERT model may include pre-training distillation and fine-tuning distillation. Pre-training distillation requires training on a large unlabeled corpus unrelated to downstream tasks, while fine-tuning distillation trains directly on specific downstream tasks, offering faster training speeds and significant performance improvements, making it more suitable for practical needs. Therefore, one or more embodiments of this specification employ fine-tuning distillation for model optimization.

[0016] In the embodiments of this specification, by aggregating the hidden layers in the teacher model, the structural coupling information between the hidden layers of the teacher model is fully utilized and progressively inherited into the student model during the model training process. This allows the student model to understand semantic knowledge that matches its learning ability at different training stages, thereby continuously strengthening its learning ability and ultimately obtaining the target model. The obtained target model not only meets the accuracy requirements of text classification but also reduces resource consumption, enabling efficient completion of text classification tasks, such as those in content risk control scenarios.

[0017] Furthermore, in one or more embodiments of this specification, the fine-tuning distillation method for text-based models is used for risk control, the method comprising:

[0018] Obtain product information text and / or user review text as training set samples;

[0019] Obtain a pre-trained teacher model, which includes multiple hidden layers; the pre-trained teacher model is used to output risk control results based on the input product information text and / or user comment text.

[0020] Select at least two designated layers from the multiple hidden layers of the teacher model;

[0021] Aggregate the selected specified layers to obtain the aggregated hidden layer;

[0022] Generate a student model based on the aggregated hidden layer and the remaining unspecified layers in the teacher model;

[0023] The teacher model is used to distill the student model;

[0024] The distilled student model is used as the teacher model again until at least two specified layers cannot be selected from the teacher model. At this point, the teacher model is used as the target model.

[0025] The target model is used to output risk control results based on the input product information text and / or user review text.

[0026] The product information text can be various product information from e-commerce platforms, and the risk control result can be a classification result, such as whether the product information text "yes" or "no" contains illegal or non-compliant content, or whether the user comment text "yes" or "no" contains illegal or non-compliant content.

[0027] Based on this, when the fine-tuning distillation method used for text-based models is applied to risk control, it can accurately output risk control results while reducing resource consumption.

[0028] Furthermore, in some other embodiments of this specification, the fine-tuning distillation method for text-based models, used for sentiment attitude classification, includes:

[0029] Obtain user comment texts as training set samples;

[0030] Obtain a pre-trained teacher model, which includes multiple hidden layers; the pre-trained teacher model is used to output user sentiment and attitude classification results for input user comment text.

[0031] Select at least two designated layers from the multiple hidden layers of the teacher model;

[0032] Aggregate the selected specified layers to obtain the aggregated hidden layer;

[0033] Generate a student model based on the aggregated hidden layer and the remaining unspecified layers in the teacher model;

[0034] The teacher model is used to distill the student model;

[0035] The distilled student model is used as the teacher model again until at least two specified layers cannot be selected from the teacher model. At this point, the teacher model is used as the target model.

[0036] The target model is used to output user sentiment and attitude classification results for the input user comment text.

[0037] Using this implementation method, users' positive, negative, and neutral reviews of a product can be distinguished by judging whether the emotions contained in user comments are positive, negative, or neutral. In other words, the output user emotional attitude classification results can include positive, negative, and neutral reviews.

[0038] Based on this, when the fine-tuning distillation method used for text-based models is applied to sentiment and attitude classification, it can accurately output classification results while reducing resource consumption.

[0039] Furthermore, in the embodiments of this specification, the original labels corresponding to the training set samples are also obtained when obtaining the training set samples;

[0040] The distillation of the student model using the teacher model specifically includes:

[0041] The training set samples are input into the student model, and the classification loss is obtained based on the output of the student model and the original labels.

[0042] The training set samples are input into the teacher model, and the distillation loss is obtained based on the output of the teacher model and the output of the student model.

[0043] Calculate the total loss based on the classified loss and the distillation loss;

[0044] The student model is trained based on minimizing the overall loss.

[0045] Furthermore, in the embodiments of this specification, the teacher model further includes an embedding layer, a pooling layer, and a classification layer;

[0046] Training the student model based on minimizing the overall loss specifically includes:

[0047] Fix the parameters of the hidden layer and embedding layer in the student model that have not yet been aggregated, and update the parameters of the hidden layer, pooling layer and classification layer in the student model after this aggregation with the goal of minimizing the overall loss.

[0048] Furthermore, in the embodiments of this specification, selecting at least two designated layers from the plurality of hidden layers of the teacher model includes:

[0049] Select at least two adjacent layers from the multiple hidden layers of the teacher model.

[0050] Furthermore, in the embodiments of this specification, the aggregation of selected specified layers to obtain the aggregated hidden layer specifically includes:

[0051] The aggregation coefficient is calculated based on the original number of hidden layers in the teacher model and the number of hidden layers in the target model. A specified layer with an equal number of aggregation coefficients is selected and aggregated into an aggregated hidden layer.

[0052] Furthermore, in the embodiments of this specification, the aggregation of selected specified layers to obtain the aggregated hidden layer specifically includes:

[0053] Aggregate the selected specified layers in either top-down or bottom-up order.

[0054] Furthermore, in the embodiments of this specification, the aggregation of selected specified layers to obtain the aggregated hidden layer specifically includes:

[0055] Initialize the parameters of any one of the selected specified layers as the initial parameters of the aggregated hidden layer of the student model.

[0056] Furthermore, in the embodiments of this specification, the distillation method for the fine-tuning stage of a text-based model further includes: training the target model using the training set samples, which includes:

[0057] The training set samples are input into the target model, and the classification loss is obtained based on the output of the target model and the original labels.

[0058] The training set samples are input into the previous teacher model that distills the target model, and the distillation loss is obtained based on the output of the teacher model and the output of the target model.

[0059] Calculate the total loss based on the classified loss and the distillation loss;

[0060] The target model is trained based on minimizing the overall loss.

[0061] Furthermore, in the embodiments described in this specification, the target model further includes an embedding layer, a pooling layer, and a classification layer;

[0062] Training the target model based on minimizing the overall loss specifically includes:

[0063] With the parameters of all aggregated hidden layers and embedding layers in the target model fixed, and the overall loss minimized as the training objective, the parameters of the pooling layer and classification layer in the target model are updated.

[0064] Another objective of the embodiments of this specification is to provide a fine-tuning distillation apparatus for text-based models. This apparatus enables student models to learn progressively at different training stages while fully learning the intermediate hidden layer information of teacher models. The resulting target model not only meets accuracy requirements but also reduces resource consumption, and can efficiently complete text classification processing tasks, such as those in content security scenarios.

[0065] In accordance with the above objectives, embodiments of this specification provide a distillation apparatus for the fine-tuning stage of a text-based model, comprising:

[0066] The sample acquisition module acquires training set samples.

[0067] The model preparation module acquires a pre-trained teacher model, which includes multiple hidden layers;

[0068] The distillation module selects at least two specified layers from multiple hidden layers in the teacher model; aggregates the selected specified layers to obtain aggregated hidden layers; generates a student model based on the aggregated hidden layers and the remaining unspecified layers in the teacher model; distills the student model using the teacher model; and uses the distilled student model as the teacher model again until it is impossible to select at least two specified layers from the teacher model, at which point the teacher model at this point is used as the target model.

[0069] Furthermore, in one or more embodiments of this specification, when the fine-tuning distillation apparatus for text-based models is used for risk control:

[0070] The sample acquisition module acquires product information text and / or user comment text as training set samples;

[0071] The model preparation module obtains a pre-trained teacher model, which includes multiple hidden layers; the pre-trained teacher model is used to output risk control results based on the input product information text and / or user comment text.

[0072] The distillation module selects at least two specified layers from the multiple hidden layers of the teacher model; aggregates the selected specified layers to obtain an aggregated hidden layer; generates a student model based on the aggregated hidden layer and the remaining unspecified layers in the teacher model; distills the student model using the teacher model; and uses the distilled student model as the teacher model again until it is impossible to select at least two specified layers from the teacher model, at which point the teacher model is used as the target model; the target model is used to output risk control results based on the input product information text and / or user comment text.

[0073] Furthermore, in some other embodiments of this specification, the fine-tuning distillation apparatus for text-based models is used for sentiment / attitude classification, wherein:

[0074] The sample acquisition module acquires user comment text as training set samples;

[0075] The model preparation module obtains a pre-trained teacher model, which includes multiple hidden layers; the pre-trained teacher model is used to output user sentiment and attitude classification results for the input user comment text.

[0076] The distillation module selects at least two specified layers from the multiple hidden layers of the teacher model; aggregates the selected specified layers to obtain an aggregated hidden layer; generates a student model based on the aggregated hidden layer and the remaining unspecified layers in the teacher model; distills the student model using the teacher model; and uses the distilled student model as the teacher model again until it is impossible to select at least two specified layers from the teacher model, at which point the teacher model at this point is used as the target model; the target model is used to output user sentiment and attitude classification results for the input user comment text.

[0077] Furthermore, in the embodiments of this specification, the sample acquisition module also acquires the original labels corresponding to the training set samples when acquiring the training set samples;

[0078] The distillation module employs a teacher model to distill the student model, specifically including:

[0079] The training set samples are input into the student model, and the classification loss is obtained based on the output of the student model and the original labels.

[0080] The training set samples are input into the teacher model, and the distillation loss is obtained based on the output of the teacher model and the output of the student model.

[0081] Calculate the total loss based on the classified loss and the distillation loss;

[0082] The student model is trained based on minimizing the overall loss.

[0083] Furthermore, in the embodiments described in this specification, the teacher model further includes an embedding layer, a pooling layer, and a classification layer;

[0084] The distillation module trains the student model based on minimizing the overall loss, specifically including:

[0085] Fix the parameters of the hidden layer and embedding layer in the student model that have not yet been aggregated, and update the parameters of the hidden layer, pooling layer and classification layer in the student model after this aggregation with the goal of minimizing the overall loss.

[0086] Furthermore, in the embodiments of this specification, the distillation module selects at least two designated layers from the plurality of hidden layers of the teacher model, including:

[0087] Select at least two adjacent layers from the multiple hidden layers of the teacher model.

[0088] Furthermore, in the embodiments of this specification, the distillation module polymerizes selected designated layers to obtain a polymerized hidden layer, specifically including:

[0089] The aggregation coefficient is calculated based on the original number of hidden layers in the teacher model and the number of hidden layers in the target model. A specified layer with an equal number of aggregation coefficients is selected and aggregated into an aggregated hidden layer.

[0090] Furthermore, in the embodiments of this specification, the distillation module polymerizes selected designated layers to obtain a polymerized hidden layer, specifically including:

[0091] Aggregate the selected specified layers in either top-down or bottom-up order.

[0092] Furthermore, in the embodiments of this specification, the distillation module polymerizes selected designated layers to obtain a polymerized hidden layer, specifically including:

[0093] Initialize the parameters of any one of the selected specified layers as the initial parameters of the aggregated hidden layer of the student model.

[0094] Furthermore, in the embodiments of this specification, a fine-tuning module is also included, wherein the fine-tuning module:

[0095] The training set samples are input into the target model, and the classification loss is obtained based on the output of the target model and the original labels.

[0096] The training set samples are input into the teacher model that distills the target model, and the distillation loss is obtained based on the output of the teacher model and the output of the target model.

[0097] Calculate the total loss based on the classified loss and the distillation loss;

[0098] The target model is trained based on minimizing the overall loss.

[0099] Furthermore, in the embodiments described in this specification, the target model further includes an embedding layer, a pooling layer, and a classification layer;

[0100] The fine-tuning module trains the target model based on minimizing the overall loss, specifically including:

[0101] With the parameters of all aggregated hidden layers and embedding layers in the target model fixed, and the overall loss minimized as the training objective, the parameters of the pooling layer and classification layer in the target model are updated.

[0102] Another object of the embodiments of this specification is to provide a computer-readable storage medium, characterized in that the storage medium stores a computer program, which, when executed by a processor, implements the steps of the distillation method for the fine-tuning stage of a text-based model as described above.

[0103] Another object of the embodiments of this specification is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the steps of the distillation method for fine-tuning a text-based model as described above.

[0104] The distillation method for the fine-tuning stage of text-based models described in the embodiments of this specification has the following beneficial effects:

[0105] In the embodiments of this specification, by aggregating specified hidden layers in the teacher model, the structural coupling information between the intermediate hidden layers of the teacher model is fully utilized and progressively inherited into the student model during the model training process. This allows the student model to understand semantic knowledge that matches its learning ability at different training stages, continuously strengthening its learning ability and ultimately obtaining the target model. The obtained target model not only meets accuracy requirements but also reduces resource consumption, enabling efficient completion of tasks such as text classification in content security scenarios.

[0106] When the fine-tuning distillation method used in text-based models is applied to risk control, it can quickly and accurately output risk control results based on the various complex commodity information input, while also reducing the model's resource consumption and saving costs.

[0107] When the distillation method used in the fine-tuning stage of text-based models is applied to sentiment and attitude classification, it can quickly and accurately extract users' sentiment and attitude based on the various complex text information in the input, while also reducing the model's resource consumption and saving costs.

[0108] The distillation apparatus for fine-tuning the text-based model described in the embodiments of this specification also has the aforementioned beneficial effects. Attached Figure Description

[0109] Figure 1 The principle of the fine-tuning stage distillation method for text-based models described in the embodiments of this specification is illustrated in one implementation.

[0110] Figure 2 An exemplary flowchart of the fine-tuning stage distillation method for text-based models described in this specification is shown in one embodiment.

[0111] Figure 3 The following example illustrates the steps of distilling a student model using a teacher model in one embodiment of the fine-tuning distillation method for text-based models described in this specification.

[0112] Figure 4The steps of the fine-tuning stage distillation method for text-based models described in the embodiments of this specification are illustrated in one implementation.

[0113] Figure 5 An exemplary schematic diagram of the fine-tuning stage distillation apparatus for text-based models described in this specification is shown in one embodiment. Detailed Implementation

[0114] The distillation method and apparatus for fine-tuning text-based models described in this invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. However, this detailed description does not constitute a limitation of the invention.

[0115] The following embodiments mainly use text content risk control as an example to illustrate the solution. However, the methods and devices provided by the present invention are not limited to the field of text content risk control, and the following embodiments are not intended to limit the application scenarios of this solution.

[0116] Knowledge distillation is a model training method in which a teacher model and a student model are set up, and the teacher model is used to guide the training of the student model. The teacher model is the original model in the pre-trained knowledge distillation training task, which is a large model with a complex structure and many parameters. The student model uses the output of the teacher model on the training set as the training target. During the training process, it continuously learns the model structure information of the teacher model and inherits the generalization ability of the teacher model. It is a small model with a simpler structure and fewer parameters built based on the architecture of the teacher model.

[0117] The BERT model is an autoencoder language model used for text classification. Training the BERT model can include a pre-training phase and a fine-tuning phase. The examples in this specification focus on the fine-tuning phase. During fine-tuning, the BERT model is fine-tuned for specific downstream natural language processing tasks, resulting in faster training speed and significant performance improvements, making it more suitable for practical needs.

[0118] Therefore, in some implementations, the teacher model, student model, and target model all include the BERT model.

[0119] In some specific implementations, taking the BERT model used in text content classification as an example, the teacher BERT model can be set with more than 12 layers, while the student BERT model can be set with 3 to 4 layers. Therefore, the resources required for computation and the inference latency are much smaller than those of the teacher BERT model.

[0120] Figure 1 The principle of the fine-tuning stage distillation method for text-based models described in the embodiments of this specification is illustrated in one implementation.

[0121] like Figure 1 As shown, in some implementations, during knowledge distillation, training set samples x are simultaneously input into both the teacher model and the student model, and the prediction results are exponentially normalized using the Softmax function. The Softmax function maps the output value range of each output node of the model to [0,1] and constrains the sum of the output values ​​of each output node of the model to 1. The training set samples obtain the prediction result P through the teacher network. t The prediction result P was obtained through the student network. s According to P t and P s The distillation loss is calculated, and the student model is trained with the goal of minimizing the distillation loss, thereby obtaining the target model.

[0122] Figure 2 An exemplary flowchart of the fine-tuning stage distillation method for text-based models described in this specification is shown in one embodiment.

[0123] like Figure 2 As shown, in this embodiment, the distillation method for the fine-tuning stage of a text-based model may include:

[0124] 100: Obtain training set samples.

[0125] The training set samples are in text form, which can come from publicly available large-scale corpora or from text content collected from the target platform, such as product information text and / or user review text on e-commerce platforms.

[0126] It should be noted that while obtaining training set samples, the original labels corresponding to the training set samples can also be obtained at the same time. The original labels represent the actual risk control results or sentiment attitude classification results of the corresponding training set samples, such as whether the content of product information on e-commerce platforms is illegal, or whether the user's comments are positive, negative or neutral.

[0127] 102: Obtain a pre-trained teacher model, which includes multiple hidden layers.

[0128] In the embodiments of this specification, the teacher model may include a BERT model. In some implementations, the original model trained on unlabeled text data in a large-scale corpus during the pre-training phase is further fine-tuned on downstream tasks to obtain the teacher model required for knowledge distillation. This teacher model already possesses strong text semantic extraction capabilities for the student model to learn from. The pre-training and acquisition processes of the teacher model are not improvements of this solution and will not be described in detail here.

[0129] 104: Select at least two specified layers from the multiple hidden layers of the teacher model.

[0130] During knowledge distillation, the student model learns the vector features of each layer in the teacher model through a mapping function and updates the parameters of its corresponding layers. This can be understood as guiding the student model to learn the specified mapping layer of the teacher model. In some implementations, the student model and the teacher model use a one-to-one mapping from layer to layer; while in other implementations, the layers in the student model and the layers in the teacher model can also use a one-to-many or many-to-one mapping.

[0131] Based on this, in some more specific embodiments, at least two adjacent layers are selected from the multiple hidden layers of the teacher model.

[0132] This is because there is important structural coupling information between the layers of the BERT model. Selecting an adjacent layer as the designated layer can effectively preserve and utilize this structural information for student models to learn, thereby enhancing the effect of knowledge distillation.

[0133] 106: Aggregate the selected specified layers to obtain the aggregated hidden layer.

[0134] As mentioned above, the student model learns the vector features of a specified layer in the teacher model through the mapping relationship function. Since the semantic information of the higher layers of the teacher model is relatively abstract, it is very difficult to learn the abstract semantic information of the teacher model in the early training stage when the student model's learning ability has not been significantly improved. This is one of the reasons why using a larger and stronger teacher model cannot obtain a better student model.

[0135] Based on this, in the embodiments of this specification, the specified layer in the teacher model is directly aggregated into a new hidden layer in the student model, thereby discarding the mapping relationship function, reducing the hyperparameters of the text model, and enhancing the robustness of the text model; at the same time, it can retain the structural coupling information between the layers of the teacher model, so that the student model can learn the complete structural information of the teacher model step by step.

[0136] In addition, the knowledge distillation process is very sensitive to hyperparameters. Therefore, it is preferable to minimize the number of hyperparameters and weaken their impact on the knowledge distillation effect, so as to distill out a target model with better performance.

[0137] In some embodiments, an aggregation coefficient is calculated based on the original number of hidden layers in the teacher model and the number of hidden layers in the target model. A specified layer with an equal number of aggregation coefficients is selected and aggregated into an aggregated hidden layer.

[0138] It should be noted that this aggregation coefficient only needs to be calculated during the first aggregation. The calculated aggregation coefficient is used every time a specified layer is aggregated in subsequent aggregations, and there is no need to recalculate it.

[0139] The number of hidden layers in the target model can be selected and set by those skilled in the art based on the specific downstream task objectives and available computing resources.

[0140] In some embodiments, the selected specified layers can be aggregated in a top-down or bottom-up order.

[0141] More preferably, in some embodiments, the selected designated layers can be aggregated in a top-down order.

[0142] This setup takes into account that the student model's learning ability is limited in the early stages of training, while the teacher model's high-level semantic information is more abstract and difficult to learn. Therefore, the training of the student model requires a gradual process. The BERT model encodes rich linguistic hierarchical information: representational features are in the lower (shallow) layers, syntactic features are in the middle layers, and semantic features are in the higher (deep) layers. In the early stages of training, the student model should focus more on learning semantic information related to the text classification task. As training progresses and the student model's understanding of semantic knowledge gradually improves, the learning difficulty also increases. At this point, the focus can shift to learning the teacher model's lower-level representational abilities, allowing the student model to be trained more comprehensively.

[0143] In some embodiments, the parameters of any one of the selected specified layers are initialized as the initial parameters of the aggregated hidden layer of the student model.

[0144] In some more specific embodiments, the parameters of the highest or lowest layer or other hidden layers among the selected specified layers are initialized as the initial parameters of the hidden layers after the aggregation of the student model, thereby completing a hidden layer aggregation process.

[0145] 108: Generate the student model based on the aggregated hidden layer and the remaining unspecified layers in the teacher model.

[0146] This method of directly replacing layers eliminates the mapping function of the intermediate layers in the teacher model, which simplifies the number of hyperparameters and enhances the robustness of the text model.

[0147] 110: The teacher model is used to distill the student model.

[0148] Figure 3 The following example illustrates the steps of distilling a student model using a teacher model in one embodiment of the fine-tuning distillation method for text-based models described in this specification.

[0149] like Figure 3As shown, in some embodiments, the original labels corresponding to the training set samples are also obtained when obtaining the training set samples; the original labels represent the actual risk control results of the corresponding samples, such as whether the content of the product information on the e-commerce platform is illegal, or whether the user's review is a positive review, a negative review, or a neutral review.

[0150] Distilling the student model using the teacher model can include:

[0151] 200: Input the training set samples into the student model, and obtain the classification loss based on the output of the student model and the original labels.

[0152] In some more specific embodiments, the cross-entropy loss between the student model's output and the original labels can be calculated as a classification loss.

[0153] 202: Input the training set samples into the teacher model, and obtain the distillation loss based on the output of the teacher model and the output of the student model.

[0154] In some more specific embodiments, the KL divergence between the outputs of the teacher model and the student model can be calculated as a distillation loss.

[0155] 204: Calculate the total loss based on the classification loss and distillation loss.

[0156] 206: Train the student model based on minimizing the overall loss.

[0157] In some embodiments, the teacher model further includes an embedding layer, a pooling layer, and a classification layer;

[0158] Therefore, training the student model based on minimizing the overall loss can include:

[0159] Fix the parameters of the hidden and embedding layers in the student model that have not yet been aggregated, and update the parameters of the hidden, pooling, and classification layers in the student model after this aggregation with the goal of minimizing the overall loss.

[0160] During the training of the student model, based on the initialization parameters set after the aggregation of hidden layers, the student model continuously learns the structural coupling information between specified hidden layers in the teacher model that is the predecessor of the aggregated hidden layer while updating the initialization parameters, so that the information of the intermediate layers of the teacher model can be fully utilized.

[0161] In the BERT model, although the other layers besides the hidden layers have a relatively simple structure, they also play a significant role in performing downstream tasks, and therefore also need to be trained and learned.

[0162] In some more specific embodiments, the classification loss based on the student model's output and the original labels can be obtained using the following formula:

[0163]

[0164] Among them, L CE σ is the classification loss; y is the original label; σ is the softmax function; S(x) is the output of the student model; x is the training set sample.

[0165] The distillation loss, based on the outputs of the teacher model and the student model, can be obtained using the following formula:

[0166] L logits =KL(σ(S(x) / T),σ(T(x) / T))

[0167] Among them, L logits σ is the distillation loss; KL() is the KL divergence function; σ is the softmax function; S(x) is the output of the student model; T(x) is the output of the teacher model; T is the temperature coefficient of the softmax function, used to adjust the smoothness of the softmax function curve, thereby adjusting the distribution of output results that are too close or too far apart.

[0168] The total loss can be calculated based on the classification loss and distillation loss using the following formula:

[0169] L = L CE +α·L logits

[0170] Where L is the total loss; L CE For classification loss; L logits α represents the distillation loss; α is the model hyperparameter.

[0171] 112: Reuse the distilled student model as the teacher model until at least two specified layers cannot be selected from the teacher model, then use the teacher model at this point as the target model.

[0172] In some embodiments, the target model is further trained using training set samples, which includes:

[0173] Input the training set samples into the target model, and obtain the classification loss based on the output of the target model and the original labels;

[0174] The training set samples are input into the previous teacher model that distills the target model, and the distillation loss is obtained based on the output of the teacher model and the output of the target model.

[0175] Calculate the total loss based on the classification loss and distillation loss;

[0176] The target model is trained based on minimizing the overall loss.

[0177] In some more specific embodiments, the cross-entropy loss between the student model's output and the original labels can be calculated as the classification loss; the KL divergence between the teacher model's output and the student model's output can be calculated as the distillation loss.

[0178] Since the method for calculating classification loss and distillation loss in this embodiment, and for calculating the overall loss based on classification loss and distillation loss, is similar to the method described above for distilling the student model using the teacher model, it will not be repeated here.

[0179] In some specific embodiments, the target model also includes an embedding layer, a pooling layer, and a classification layer;

[0180] Based on this, training the target model based on minimizing the overall loss can include:

[0181] With the parameters of all aggregated hidden layers and embedding layers in the target model fixed, and the overall loss minimized as the training objective, the parameters of the pooling layer and classification layer in the target model are updated.

[0182] Once all aggregations are complete, if the number of hidden layers in the obtained student model is equal to the number of hidden layers in the set target model, then the student model is taken as the target model.

[0183] Further training of the target model can continuously improve its classification ability and enhance the efficiency of handling content risk control tasks.

[0184] The embodiments in this specification, by gradually replacing the layers of the teacher model with the mapping layers of the student model, not only make full use of the structural coupling information between the intermediate layers of the teacher model, simplify the number of model hyperparameters, and enhance the robustness of the model, but also allow the student model to learn the knowledge of different levels of abstraction contained in the shallow and deep layers of the teacher model in stages during training, thereby learning the representation ability of the teacher model and applying it to downstream tasks.

[0185] Figure 4 The steps of the fine-tuning stage distillation method for text-based models described in the embodiments of this specification are illustrated in one implementation.

[0186] like Figure 4 As shown, in some implementations, assuming the teacher model has m hidden layers and the student model has n hidden layers, the aggregation coefficient K can be obtained by the following formula:

[0187]

[0188] In a more specific example, if the teacher model has 8 hidden layers and the target model has 4 hidden layers, then the aggregation coefficient is 2. This means that each aggregation operation selects two specified layers from the teacher model's hidden layers. Assume the training set samples are x, the maximum sequence length is |x|, the size of each batch of training data is bs, and the dimension of the hidden layers in the teacher or student model is d. h .

[0189] First, the original model is fine-tuned on a downstream task to obtain the teacher model T required for knowledge distillation. The two highest hidden layers (i.e., the 7th and 8th layers) in the teacher model are selected as designated layers and aggregated into a new hidden layer. The parameters of the new hidden layer after aggregation are initialized using the parameters of the lower layer in the designated layer, and this new hidden layer is used as the highest hidden layer in the student model. Optionally, the hidden layer can be constructed based on Transformer.

[0190] Next, the other layers in the teacher model are directly copied into the student model, including hidden layers 1-6, the embedding layer, the classification layer, and the pooling layer. When distilling the student model using the teacher model, the parameters of hidden layers 1-6 and the embedding layer in the student model are fixed, while the parameters in other layers, including the fourth hidden layer, the pooling layer, and the classification layer in the student model, are updated to complete one round of training.

[0191] Following the steps described above, the hidden layers in the teacher model are aggregated pairwise from top to bottom, and the other layers are copied to obtain the student model. The student model is then trained until the number of hidden layers obtained through aggregation in the student model reaches 4, at which point it is used as the target model. Next, the pooling and classification layers in the target model are trained. After the final hidden layer aggregation, when training the student model, the parameters of the 2nd to 4th hidden layers (i.e., the already aggregated hidden layers) are fixed, and the parameters of the newly aggregated hidden layers, as well as the embedding, pooling, and classification layers, are updated.

[0192] In each training round, the classification loss L is obtained based on the student model's output and the original labels. CE ( Figure 4 (Not shown in the diagram), the distillation loss L is obtained based on the outputs of the teacher model and the student model. logits The overall loss (Loss) is calculated based on the classification loss and distillation loss, and the student model is trained based on minimizing the overall loss. Specifically, the classification results are normalized using the softmax function; T is the temperature coefficient of the softmax function, used to adjust the smoothness of the softmax curve, thereby adjusting the distribution of output results that are too close or too far apart.

[0193] In some embodiments, the fine-tuning distillation method for text-based models described in this specification is used for risk control, wherein:

[0194] Obtain product information text or user review text as training set samples;

[0195] Obtain a pre-trained teacher model, which includes multiple hidden layers; the pre-trained teacher model is used to output risk control results based on the input text or user comment text.

[0196] Select at least two specified layers from the multiple hidden layers of the teacher model;

[0197] Aggregate the selected specified layers to obtain the aggregated hidden layer;

[0198] Generate a student model based on the aggregated hidden layer and the remaining unspecified layers in the teacher model;

[0199] The teacher model is used to distill the student model;

[0200] The distilled student model is used as the teacher model again until at least two specified layers cannot be selected from the teacher model. At this point, the teacher model is used as the target model.

[0201] The target model is used to output risk control results based on the input text or user comment text.

[0202] Text content risk control includes identifying and filtering illegal content in text on a target platform to ensure the platform operates legally, compliantly, and healthily. Therefore, in some embodiments of this invention, the distillation method for fine-tuning text models described in this specification can be used to perform content risk control on product information text on e-commerce platforms. The product information text is input into a pre-trained teacher model, which performs semantic extraction. The output risk control result can be a classification result, such as whether the product information text contains illegal content. The target model is obtained through distillation. Inputting the product information text into the target model yields the final risk control result, determining whether the product information text contains illegal content, allowing for further deletion and filtering of product information containing illegal content.

[0203] In some other implementations, the fine-tuning distillation method for text-based models provided in the embodiments of this specification can also be used for content risk control of user comments on e-commerce platforms. For example, the method described above can be used to detect illegal content in user comments, thereby enabling operations such as blocking and muting.

[0204] In other embodiments, the distillation method for fine-tuning text-based models provided in the embodiments of this specification can also be used to determine the emotional attitude of user comments. For example, by determining whether the emotion contained in the user comments is positive, negative, or neutral, the user's positive, negative, and neutral reviews of the product can be distinguished.

[0205] As the types of content risk control applications increase and the demands broaden, more and more algorithms and models are being used. The distillation method for fine-tuning text models provided in the embodiments of this specification can meet the needs of rapidly developing content risk control applications. It can save computing resources while quickly completing text classification tasks and effectively alleviate the increasingly prominent cost problem.

[0206] In another embodiment of this specification, a fine-tuning stage distillation apparatus for text-based models is proposed. Figure 5 An exemplary schematic diagram of the fine-tuning stage distillation apparatus for text-based models described in this specification is shown in one embodiment.

[0207] like Figure 5 As shown, the device includes:

[0208] Sample acquisition module 30, which acquires training set samples;

[0209] Model preparation module 32, which obtains a pre-trained teacher model, which includes multiple hidden layers;

[0210] Distillation module 34 selects at least two specified layers from multiple hidden layers in the teacher model; aggregates the selected specified layers to obtain aggregated hidden layers; generates a student model based on the aggregated hidden layers and the remaining unspecified layers in the teacher model; distills the student model using the teacher model; and uses the distilled student model as the teacher model again until it is impossible to select at least two specified layers from the teacher model, at which point the teacher model at this point is used as the target model.

[0211] In the sample acquisition module, the training set samples are in text format, which can come from publicly available large-scale corpora or from text content collected from the target platform, such as product information text or user review text on e-commerce platforms. It should be noted that while acquiring the training set samples, the original labels corresponding to the samples can also be obtained. These original labels represent the actual risk control results or sentiment attitude classification results for the corresponding training set samples, such as whether the content of product information on the e-commerce platform violates regulations, or whether the user's review is positive, negative, or neutral.

[0212] In the model preparation module, as described in the embodiments of this specification, the teacher model may include a BERT model. The original model, trained on unlabeled text data in a large-scale corpus during the pre-training phase, is further fine-tuned on downstream tasks to obtain the teacher model required for knowledge distillation. This teacher model already possesses strong text semantic extraction capabilities and is used by the student model for learning.

[0213] In the distillation module, the student model learns the vector features of each layer in the teacher model through a mapping function and updates the parameters of its corresponding layers. This can be understood as guiding the student model to learn the specified mapping layers of the teacher model. In some implementations, the student model and the teacher model use a one-to-one mapping between layers; while in other implementations, the layers in the student model and the layers in the teacher model can also use a one-to-many or many-to-one mapping.

[0214] In some embodiments, the distillation module selects at least two adjacent layers from multiple hidden layers of the teacher model for aggregation.

[0215] There is important structural coupling information between the layers of the BERT model. Selecting an adjacent layer as the designated layer can effectively preserve and utilize this structural information for student model learning, thereby enhancing the effect of knowledge distillation.

[0216] As mentioned above, the student model learns the vector features of a specified layer in the teacher model through the mapping relationship function. Since the semantic information of the higher layers of the teacher model is relatively abstract, it is very difficult to learn the abstract semantic information of the teacher model in the early training stage when the student model's learning ability has not been significantly improved. This is one of the reasons why using a larger and stronger teacher model cannot obtain a better student model.

[0217] Based on this, in the embodiments of this specification, the specified layer in the teacher model is directly aggregated into a new hidden layer in the student model, thereby discarding the mapping relationship function, reducing the hyperparameters of the text model, and enhancing the robustness of the text model; at the same time, it can retain the structural coupling information between the layers of the teacher model, so that the student model can learn the complete structural information of the teacher model step by step.

[0218] In some embodiments, the distillation module can calculate the aggregation coefficient based on the original number of hidden layers in the teacher model and the number of hidden layers in the target model, select a specified layer with an amount equal to the aggregation coefficient, and aggregate them into an aggregated hidden layer.

[0219] It should be noted that this aggregation coefficient only needs to be calculated during the first aggregation. The calculated aggregation coefficient is used every time a specified layer is aggregated in subsequent aggregations, and there is no need to recalculate it.

[0220] The number of hidden layers in the target model can be determined by relevant technical personnel based on the specific downstream task objectives and available computing resources.

[0221] In some embodiments, the rectifier module aggregates the selected designated layers in a top-down or bottom-up order.

[0222] In some preferred embodiments, the distillation module polymerizes the selected designated layers in a top-down order.

[0223] The aggregation is performed from top to bottom because student models have limited learning capabilities in the early stages of training, while the high-level semantic information of teacher models is more abstract and difficult to learn. Therefore, the training of student models requires a gradual process. The BERT model encodes rich linguistic hierarchical information: representational features are in the lower (shallow) layers, syntactic features in the middle layers, and semantic features in the higher (deep) layers. In the early stages of training, student models should focus more on learning semantic information relevant to the text classification task. As training progresses and the student model's understanding of semantic knowledge gradually improves, the learning difficulty also increases. At this point, the focus can shift to learning the lower-level representational capabilities of the teacher model, allowing for more comprehensive training of the student model.

[0224] In some embodiments, the distillation module initializes the parameters of any one of the selected specified layers as the initial parameters of the aggregated hidden layer of the student model.

[0225] In some more specific embodiments, the distillation module initializes the parameters of the highest or lowest layer or other hidden layers among the selected specified layers as the initial parameters of the aggregated hidden layers of the student model, thereby completing a hidden layer aggregation.

[0226] In some more specific embodiments, the distillation module initializes the parameters of the lowest hidden layer among the selected specified layers as the initial parameters of the aggregated hidden layer of the student model, thereby completing a hidden layer aggregation.

[0227] This method of directly replacing layers eliminates the mapping function of the intermediate layers in the teacher model, which simplifies the number of hyperparameters and enhances the robustness of the text model.

[0228] In some embodiments, the sample acquisition module may also acquire the original label corresponding to the training set sample when acquiring the training set sample; the original label represents the actual risk control result or sentiment attitude classification result of the corresponding training set sample, such as whether the content of the product information on the e-commerce platform is illegal, or whether the user's review is positive, negative or neutral.

[0229] Therefore, the distillation module, which uses the teacher model to distill the student model, can include:

[0230] The training set samples are input into the student model, and the classification loss is obtained based on the student model's output and the original labels.

[0231] The training set samples are input into the teacher model, and the distillation loss is obtained based on the output of the teacher model and the output of the student model.

[0232] Calculate the total loss based on the classification loss and distillation loss;

[0233] The student model is trained based on minimizing the overall loss.

[0234] In some more specific embodiments, the distillation module calculates the cross-entropy loss between the student model's output and the original labels as the classification loss; and calculates the KL divergence between the teacher model's output and the student model's output as the distillation loss.

[0235] In some embodiments, the teacher model further includes an embedding layer, a pooling layer, and a classification layer;

[0236] Therefore, the distillation module can train the student model based on minimizing the overall loss, which may include:

[0237] Fix the parameters of the hidden and embedding layers in the student model that have not yet been aggregated, and update the parameters of the hidden, pooling, and classification layers in the student model after this aggregation with the goal of minimizing the overall loss.

[0238] During the training of the student model, based on the initialization parameters set after the aggregation of hidden layers, the student model continuously learns the structural coupling information between specified hidden layers in the teacher model that is the predecessor of the aggregated hidden layer while updating the initialization parameters, so that the information of the intermediate layers of the teacher model can be fully utilized.

[0239] In some more specific embodiments, the distillation module obtains the classification loss based on the student model's output and the original labels using the following formula:

[0240]

[0241] Among them, L CE σ is the classification loss; y is the original label; σ is the softmax function; S(x) is the output of the student model; x is the training set sample.

[0242] The distillation module obtains the distillation loss based on the outputs of the teacher model and the student model, which can be achieved using the following formula:

[0243] L logits =KL(σ(S(x) / T),σ(T(x) / T))

[0244] Among them, L logits σ is the distillation loss; KL() is the KL divergence function; σ is the softmax function; S(x) is the output of the student model; T(x) is the output of the teacher model; T is the temperature coefficient of the softmax function, used to adjust the smoothness of the softmax function curve, thereby adjusting the distribution of output results that are too close or too far apart.

[0245] The distillation module can calculate the total loss based on the classification loss and distillation loss using the following formula:

[0246] L = L CE +α·L logits

[0247] Where L is the total loss; L CE For classification loss; L logits α represents the distillation loss; α is the model hyperparameter.

[0248] In some embodiments, the distillation stage for fine-tuning text-based models may further include a fine-tuning module that trains the target model using training set samples, comprising:

[0249] Input the training set samples into the target model, and obtain the classification loss based on the output of the target model and the original labels;

[0250] The training set samples are input into the previous teacher model that distills the target model, and the distillation loss is obtained based on the output of the teacher model and the output of the target model.

[0251] Calculate the total loss based on the classification loss and distillation loss;

[0252] The target model is trained based on minimizing the overall loss.

[0253] Optionally, the cross-entropy loss between the target model's output and the original labels is calculated as the classification loss; the KL divergence between the teacher model's output and the target model's output is calculated as the distillation loss.

[0254] Since the method for calculating classification loss and distillation loss in this embodiment, and for calculating the overall loss based on classification loss and distillation loss, is similar to the method described above for distilling the student model using the teacher model, it will not be repeated here.

[0255] In some specific embodiments, the target model also includes an embedding layer, a pooling layer, and a classification layer;

[0256] Therefore, the fine-tuning module, which trains the target model based on minimizing the overall loss, can include:

[0257] With the parameters of all aggregated hidden layers and embedding layers in the target model fixed, and the overall loss minimized as the training objective, the parameters of the pooling layer and classification layer in the target model are updated.

[0258] Once all aggregations are complete, if the number of hidden layers in the resulting student model is equal to the number of hidden layers in the target model, then this student model is used as the target model. The target model is then further trained to continuously improve its classification ability, thereby enhancing the efficiency of handling content risk control tasks.

[0259] In one embodiment of this specification, a computer-readable storage medium is also provided, characterized in that the storage medium stores a computer program that, when executed by a processor, implements the steps of the fine-tuning distillation method for a text-based model as described above.

[0260] In one embodiment of this specification, an electronic device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the steps of the fine-tuning distillation method for a text-based model as described above.

[0261] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0262] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0263] It should be noted that the above examples are merely specific embodiments of the present invention, and the present invention is obviously not limited to the above embodiments, with many similar variations. All modifications that can be directly derived or conceived by those skilled in the art from the content disclosed in this invention should fall within the protection scope of this invention.

Claims

1. A distillation method for the fine-tuning stage of a text-based model, the method comprising: Obtain training set samples; Obtain a pre-trained teacher model, which includes multiple hidden layers; Select at least two designated layers from the multiple hidden layers of the teacher model; Aggregate the selected specified layers to obtain the aggregated hidden layer; The student model is generated based on the aggregated hidden layer and the remaining unspecified layers in the teacher model; The teacher model is used to distill the student model; The distilled student model is reused as the teacher model until at least two specified layers cannot be selected from the teacher model. At this point, the teacher model is used as the target model.

2. The distillation method for fine-tuning stage of text-based models as described in claim 1, used for risk control, the method comprising: Obtain product information text and / or user review text as training set samples; Obtain a pre-trained teacher model, which includes multiple hidden layers; the pre-trained teacher model is used to output risk control results based on the input product information text and / or user comment text. Select at least two designated layers from the multiple hidden layers of the teacher model; Aggregate the selected specified layers to obtain the aggregated hidden layer; Generate a student model based on the aggregated hidden layer and the remaining unspecified layers in the teacher model; The teacher model is used to distill the student model; The distilled student model is used as the teacher model again until at least two specified layers cannot be selected from the teacher model. At this point, the teacher model is used as the target model. The target model is used to output risk control results based on the input product information text and / or user review text.

3. The distillation method for fine-tuning stage of text-based models as described in claim 1, used for sentiment / attitude classification, the method comprising: Obtain user comment texts as training set samples; Obtain a pre-trained teacher model, which includes multiple hidden layers; The pre-trained teacher model is used to output user sentiment and attitude classification results for the input user comment text; Select at least two designated layers from the multiple hidden layers of the teacher model; Aggregate the selected specified layers to obtain the aggregated hidden layer; Generate a student model based on the aggregated hidden layer and the remaining unspecified layers in the teacher model; The teacher model is used to distill the student model; The distilled student model is used as the teacher model again until at least two specified layers cannot be selected from the teacher model. At this point, the teacher model is used as the target model. The target model is used to output user sentiment and attitude classification results for the input user comment text.

4. The distillation method for fine-tuning stage of text-based models as described in any one of claims 1-3, wherein the original labels corresponding to the training set samples are also obtained when obtaining the training set samples; The distillation of the student model using the teacher model specifically includes: The training set samples are input into the student model, and the classification loss is obtained based on the output of the student model and the original labels. The training set samples are input into the teacher model, and the distillation loss is obtained based on the output of the teacher model and the output of the student model. Calculate the total loss based on the classified loss and the distillation loss; The student model is trained based on minimizing the overall loss.

5. The distillation method for fine-tuning stage of text-based models as described in claim 4, wherein the teacher model further comprises an embedding layer, a pooling layer, and a classification layer; Training the student model based on minimizing the overall loss specifically includes: Fix the parameters of the hidden layer and embedding layer in the student model that have not yet been aggregated, and update the parameters of the hidden layer, pooling layer and classification layer in the student model after this aggregation with the goal of minimizing the overall loss.

6. The distillation method for fine-tuning a text-based model as described in any one of claims 1-3, wherein selecting at least two designated layers from the plurality of hidden layers of the teacher model comprises: Select at least two adjacent layers from the multiple hidden layers of the teacher model.

7. The distillation method for fine-tuning a text-based model as described in any one of claims 1-3, wherein the aggregation of selected specified layers to obtain the aggregated hidden layer specifically includes: The aggregation coefficient is calculated based on the original number of hidden layers in the teacher model and the number of hidden layers in the target model. A specified layer with an equal number of aggregation coefficients is selected and aggregated into an aggregated hidden layer.

8. The distillation method for fine-tuning a text-based model as described in any one of claims 1-3, wherein the aggregation of selected specified layers to obtain the aggregated hidden layer specifically includes: Aggregate the selected specified layers in either top-down or bottom-up order.

9. The distillation method for fine-tuning a text-based model as described in any one of claims 1-3, wherein the aggregation of selected specified layers to obtain the aggregated hidden layer specifically includes: Initialize the parameters of any one of the selected specified layers as the initial parameters of the aggregated hidden layer of the student model.

10. The distillation method for the fine-tuning stage of a text-based model as described in any one of claims 1-3, further comprising: Training the target model using the training set samples includes: The training set samples are input into the target model, and the classification loss is obtained based on the output of the target model and the original labels. The training set samples are input into the previous teacher model that distills the target model, and the distillation loss is obtained based on the output of the teacher model and the output of the target model. Calculate the total loss based on the classified loss and the distillation loss; The target model is trained based on minimizing the overall loss.

11. The distillation method for fine-tuning stage of a text-based model as described in claim 10, wherein the target model further comprises an embedding layer, a pooling layer, and a classification layer; Training the target model based on minimizing the overall loss specifically includes: With the parameters of all aggregated hidden layers and embedding layers in the target model fixed, and the overall loss minimized as the training objective, the parameters of the pooling layer and classification layer in the target model are updated.

12. A distillation apparatus for the fine-tuning stage of a text-based model, comprising: The sample acquisition module acquires training set samples. The model preparation module acquires a pre-trained teacher model, which includes multiple hidden layers; The distillation module selects at least two specified layers from multiple hidden layers in the teacher model; aggregates the selected specified layers to obtain aggregated hidden layers; generates a student model based on the aggregated hidden layers and the remaining unspecified layers in the teacher model; distills the student model using the teacher model; and uses the distilled student model as the teacher model again until it is impossible to select at least two specified layers from the teacher model, at which point the teacher model at this point is used as the target model.

13. The fine-tuning distillation apparatus for text-based models as described in claim 12, wherein it is used for risk control, wherein: The sample acquisition module acquires product information text and / or user comment text as training set samples; The model preparation module obtains a pre-trained teacher model, which includes multiple hidden layers; the pre-trained teacher model is used to output risk control results based on the input product information text and / or user comment text. The distillation module selects at least two specified layers from the multiple hidden layers of the teacher model; aggregates the selected specified layers to obtain an aggregated hidden layer; generates a student model based on the aggregated hidden layer and the remaining unspecified layers in the teacher model; distills the student model using the teacher model; and uses the distilled student model as the teacher model again until it is impossible to select at least two specified layers from the teacher model, at which point the teacher model is used as the target model; the target model is used to output risk control results based on the input product information text and / or user comment text.

14. The fine-tuning distillation apparatus for text-based models as described in claim 13, used for sentiment / attitude classification, wherein: The sample acquisition module acquires user comment text as training set samples; The model preparation module obtains a pre-trained teacher model, which includes multiple hidden layers; the pre-trained teacher model is used to output user sentiment and attitude classification results for the input user comment text. The distillation module selects at least two specified layers from the multiple hidden layers of the teacher model; aggregates the selected specified layers to obtain an aggregated hidden layer; generates a student model based on the aggregated hidden layer and the remaining unspecified layers in the teacher model; distills the student model using the teacher model; and uses the distilled student model as the teacher model again until it is impossible to select at least two specified layers from the teacher model, at which point the teacher model at this point is used as the target model; the target model is used to output user sentiment and attitude classification results for the input user comment text.

15. The fine-tuning distillation apparatus for a text-based model as described in any one of claims 12-14, wherein the sample acquisition module acquires the original labels corresponding to the training set samples when acquiring the training set samples; The distillation module employs a teacher model to distill the student model, specifically including: The training set samples are input into the student model, and the classification loss is obtained based on the output of the student model and the original labels. The training set samples are input into the teacher model, and the distillation loss is obtained based on the output of the teacher model and the output of the student model. Calculate the total loss based on the classified loss and the distillation loss; The student model is trained based on minimizing the overall loss.

16. The fine-tuning distillation apparatus for a text-based model as described in claim 15, wherein the teacher model further comprises an embedding layer, a pooling layer, and a classification layer; The distillation module trains the student model based on minimizing the overall loss, specifically including: Fix the parameters of the hidden layer and embedding layer in the student model that have not yet been aggregated, and update the parameters of the hidden layer, pooling layer and classification layer in the student model after this aggregation with the goal of minimizing the overall loss.

17. The distillation apparatus for fine-tuning a text-based model as described in any one of claims 12-14, wherein the distillation module selects at least two designated layers from the plurality of hidden layers of the teacher model, comprising: Select at least two adjacent layers from the multiple hidden layers of the teacher model.

18. The distillation apparatus for fine-tuning a text-based model as described in any one of claims 12-14, wherein the distillation module aggregates selected specified layers to obtain aggregated hidden layers, specifically including: The aggregation coefficient is calculated based on the original number of hidden layers in the teacher model and the number of hidden layers in the target model. A specified layer with an equal number of aggregation coefficients is selected and aggregated into an aggregated hidden layer.

19. The distillation apparatus for fine-tuning a text-based model as described in any one of claims 12-14, wherein the distillation module aggregates selected specified layers to obtain aggregated hidden layers, specifically including: Aggregate the selected specified layers in either top-down or bottom-up order.

20. The distillation apparatus for fine-tuning a text-based model as described in any one of claims 12-14, wherein the distillation module aggregates selected specified layers to obtain aggregated hidden layers, specifically comprising: Initialize the parameters of any one of the selected specified layers as the initial parameters of the aggregated hidden layer of the student model.

21. The fine-tuning distillation apparatus for a text-based model as described in any one of claims 12-14, further comprising a fine-tuning module, the fine-tuning module being: The training set samples are input into the target model, and the classification loss is obtained based on the output of the target model and the original labels. The training set samples are input into the teacher model that distills the target model, and the distillation loss is obtained based on the output of the teacher model and the output of the target model. Calculate the total loss based on the classified loss and the distillation loss; The target model is trained based on minimizing the overall loss.

22. The fine-tuning distillation apparatus for text-based models as described in claim 21, wherein the target model further comprises an embedding layer, a pooling layer, and a classification layer; The fine-tuning module trains the target model based on minimizing the overall loss, specifically including: With the parameters of all aggregated hidden layers and embedding layers in the target model fixed, and the overall loss minimized as the training objective, the parameters of the pooling layer and classification layer in the target model are updated.

23. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of any of the methods described in claims 1-11.

Images