A model distillation method, device, computer equipment and storage medium

Abstract

Embodiments of the present application disclose a model distillation method and device, computer equipment and a storage medium. Embodiments of the present application can obtain a question sample set and a teacher model used for question recommendation, wherein the question sample set comprises a first question sample subset and a second question sample subset. Data augmentation is performed on question samples in the first question sample subset to obtain augmented question samples. Model augmentation is performed on the teacher model by using the augmented question samples to obtain an augmented teacher model. The augmented teacher model is used to identify question samples in the second question sample subset to obtain reference distribution information of the question samples in the second question sample subset. A student model is obtained, and the student model is used to learn the reference distribution information obtained by the augmented teacher model to obtain a target student model, thereby improving the effect of question recommendation.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a model distillation method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the continuous development of artificial intelligence (AI) technology, many applications have emerged. For example, AI can be applied to education to recommend questions to students. For instance, AI models can be trained to recommend questions. However, question recommendation often prioritizes high efficiency and real-time performance. In pursuit of high efficiency, AI models tend to be complex, and complex models are time-consuming to recommend questions, resulting in suboptimal real-time performance. Current technologies typically address this issue through model distillation, which reduces complex models to simpler ones, thus improving the real-time performance of recommendations. However, this significantly reduces the effectiveness of the recommendation process. Summary of the Invention

[0003] This application proposes a model distillation method, apparatus, computer equipment, and storage medium, which can achieve both real-time performance and quality of question recommendation when using artificial intelligence models for question recommendation.

[0004] This application provides a model distillation method, including:

[0005] Obtain a question sample set and a teacher model for question recommendation, wherein the question sample set includes a first question sample subset and a second question sample subset;

[0006] Data augmentation is performed on the question samples in the first question sample subset to obtain augmented question samples;

[0007] The teacher model is enhanced using the enhanced question samples to obtain the enhanced teacher model.

[0008] The enhanced teacher model is used to identify the question samples in the second question sample subset to obtain the reference distribution information of the question samples in the second question sample subset;

[0009] A student model is obtained, and the reference distribution information identified by the enhanced teacher model is used to learn the target student model.

[0010] Accordingly, this application also provides a model distillation apparatus, including:

[0011] An acquisition unit is used to acquire a question sample set and a teacher model for question recommendation, wherein the question sample set includes a first question sample subset and a second question sample subset;

[0012] The data augmentation unit is used to augment the question samples in the first question sample subset to obtain augmented question samples;

[0013] The model enhancement unit is used to enhance the teacher model using the enhanced question samples to obtain the enhanced teacher model.

[0014] The identification unit is used to identify the question samples in the second question sample subset using the enhanced teacher model, and obtain the reference distribution information of the question samples in the second question sample subset.

[0015] The distillation unit is used to obtain a student model, and then uses the student model to learn the reference distribution information identified by the enhanced teacher model to obtain a target student model.

[0016] In one embodiment, the data enhancement unit may include:

[0017] The first relation identification subunit is used to identify the current question sample in the first question sample subset and obtain the associated question samples in the first question sample subset and the current question sample that have a preset association relationship.

[0018] The second relationship identification subunit is used to identify target question samples that have a preset relationship with the associated question samples in the first question sample subset, wherein the target question samples include other question samples in the first question sample subset other than the current question sample;

[0019] The relationship establishment subunit is used to establish the relationship between the current question sample and the target question sample based on the preset relationship between the current question sample and the associated question sample, and the preset relationship between the associated question sample and the target question sample, so as to obtain the enhanced question sample.

[0020] In one embodiment, the relationship-establishing subunit may include:

[0021] The knowledge point recognition module is used to identify the knowledge point information of the current question sample and the knowledge point information of the target question sample;

[0022] The comparison module is used to compare the knowledge point information of the current question sample with the knowledge point information of the target question sample to obtain the comparison result;

[0023] The relationship establishment module is used to establish the association between the current question sample and the target question sample based on the comparison results, so as to obtain the enhanced question sample.

[0024] In one embodiment, the model enhancement unit may include:

[0025] The training subunit is used to train each teacher model using the enhanced question samples to obtain the trained teacher model corresponding to each teacher model.

[0026] The indicator generation subunit is used to generate a performance measurement indicator of the trained teacher model based on the training results of the trained teacher model, wherein the performance measurement indicator characterizes the quality of the trained teacher model in recommending questions.

[0027] The constituent sub-units are used to combine the trained teacher models into the enhanced teacher model based on the performance metrics of each trained teacher model.

[0028] In one embodiment, the training subunit may include:

[0029] An initialization module is used to initialize the weight coefficients of each augmented question sample, wherein the weight coefficients represent the value of the augmented question sample for training.

[0030] The first training module is used to train the teacher model using the enhanced question samples to obtain the initial trained teacher model and the initial training results.

[0031] An update module is used to update the weight coefficients of the enhanced question samples based on the initial training results, so as to obtain the updated weight coefficients of the enhanced question samples.

[0032] The filtering module is used to filter out target question samples from the enhanced question samples based on the updated weight coefficients of the enhanced question samples.

[0033] The second training module is used to train the initial trained teacher model using the target question samples to obtain the trained teacher model.

[0034] In one embodiment, the constituent subunit may include:

[0035] The coefficient generation module is used to generate combined weight coefficients for each trained teacher model based on the performance metric.

[0036] The model fusion module is used to fuse each trained teacher model according to the combined weight coefficients to obtain a fused teacher model.

[0037] The model construction module is used to construct the enhanced teacher model from the fused teacher model according to the preset constructor.

[0038] In one embodiment, the identification unit may include:

[0039] The information extraction subunit is used to extract information from the question sample to obtain the question information of the question sample;

[0040] The normalization subunit is used to normalize the question information to obtain normalized question information.

[0041] The first identification subunit is used to identify the normalized question information using the enhanced teacher model to obtain the reference distribution information of the question samples in the second question sample subset.

[0042] In one embodiment, the normalization subunit may include:

[0043] The content recognition module is used to identify at least one type of content to be normalized in the question information;

[0044] The method determination module is used to determine the normalization method for the content to be normalized of the current type;

[0045] The content conversion module is used to convert the content to be normalized using the normalization method to obtain the normalized question information.

[0046] In one embodiment, the identification subunit may include:

[0047] The feature extraction module is used to extract features from the normalized question information to obtain the feature information of the normalized question information;

[0048] The pooling module is used to perform pooling operations on the feature information to obtain pooled feature information.

[0049] The distribution fitting module is used to perform distribution fitting on the pooled feature information to obtain the distribution information of the question samples in the second question sample subset.

[0050] In one embodiment, the distillation unit may include:

[0051] The second identification subunit is used to identify the question samples in the second question sample subset using the student model, and obtain the distribution information of the question samples in the second question sample subset;

[0052] The loss calculation subunit is used to calculate the model loss information between the reference distribution information obtained by the enhanced teacher model based on the question sample recognition and the distribution information obtained by the student model based on the question sample recognition.

[0053] The adjustment subunit is used to adjust the student model using the model loss information to obtain the target student model.

[0054] In one embodiment, the loss calculation subunit may include:

[0055] An adjustment module is used to adjust the smoothness of the reference distribution information and the distribution information to obtain adjusted reference distribution information and adjusted distribution information.

[0056] The logarithmic operation module is used to perform logarithmic operations on the adjusted distribution information to obtain logarithmically operated distribution information, and to perform logarithmic operations on the adjusted reference distribution information to obtain logarithmically operated reference distribution information;

[0057] The multiplication module is used to multiply the adjusted distribution information and the logarithmic distribution information to obtain multiplied distribution information, and to multiply the adjusted reference distribution information and the logarithmic reference distribution information to obtain multiplied reference distribution information.

[0058] The addition module is used to add the multiplied reference distribution information and the multiplied distribution information to obtain the model loss information.

[0059] In one embodiment, the adjustment subunit may include:

[0060] The acquisition module is used to acquire the tag information of the question samples;

[0061] The loss calculation module is used to calculate the label loss information between the distribution information of the question samples and the label information;

[0062] The loss fusion module is used to fuse the label loss information and the model loss information to obtain fused loss information;

[0063] The parameter adjustment module is used to adjust the model parameters of the student model based on the fused loss information to obtain the target student model.

[0064] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative embodiments described above.

[0065] Accordingly, this application also provides a storage medium storing instructions that, when executed by a processor, implement any of the model distillation methods provided in this application.

[0066] This application embodiment can obtain a question sample set and a teacher model for question recommendation. The question sample set includes a first question sample subset and a second question sample subset. Data augmentation is performed on the question samples in the first question sample subset to obtain augmented question samples. The augmented question samples are then used to augment the teacher model to obtain an augmented teacher model. The augmented teacher model is then used to identify question samples in the second question sample subset to obtain reference distribution information for the question samples in the second question sample subset. A student model is then obtained, and the student model is used to learn from the reference distribution information identified by the augmented teacher model to obtain a target student model. By using the target student model obtained in this application embodiment for question recommendation, both the real-time performance and the quality of question recommendation can be considered. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a schematic diagram of a scenario for the model distillation method provided in an embodiment of this application;

[0069] Figure 2 This is a schematic flowchart of the model distillation method provided in the embodiments of this application;

[0070] Figure 3 This is another scenario illustration of the model distillation method provided in the embodiments of this application;

[0071] Figure 4 This is a schematic diagram of a scenario illustrating the title information provided in an embodiment of this application;

[0072] Figure 5 This is another scenario illustration of the model distillation method provided in the embodiments of this application;

[0073] Figure 6 This is another schematic flowchart of the model distillation method provided in the embodiments of this application;

[0074] Figure 7 This is a schematic diagram of the model distillation apparatus provided in the embodiments of this application;

[0075] Figure 8 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0076] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. However, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0077] This application proposes a model distillation method, which can be executed by a model distillation apparatus integrated into a computer device. The computer device may include at least one of a terminal and a server. That is, the model distillation method proposed in this application can be executed by a terminal, a server, or jointly by a terminal and a server capable of communicating with each other.

[0078] The terminal may include, but is not limited to, smartphones, tablets, laptops, personal computers (PCs), smart home appliances, wearable electronic devices, VR / AR devices, in-vehicle terminals, intelligent voice interaction devices, etc.

[0079] A server can be an interconnecting server between multiple heterogeneous systems or a backend server. It can also be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms, etc.

[0080] It should be noted that the embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0081] In one embodiment, such as Figure 1The model distillation apparatus can be integrated into a computer device such as a terminal or server to implement the model distillation method proposed in this application. Specifically, the server 11 can obtain a question sample set and a teacher model for question recommendation, wherein the question sample set includes a first question sample subset and a second question sample subset; perform data augmentation on the question samples in the first question sample subset to obtain augmented question samples; use the augmented question samples to perform model augmentation on the teacher model to obtain an augmented teacher model; use the augmented teacher model to identify the question samples in the second question sample subset to obtain reference distribution information of the question samples in the second question sample subset; obtain a student model, and use the student model to learn from the reference distribution information identified by the augmented teacher model to obtain a target student model. Then, the server 11 can send the target student model to the terminal 10 so that the terminal 10 can recommend questions through the target student model.

[0082] The following will provide a detailed description of each example. It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.

[0083] This application will describe the embodiments from the perspective of a model distillation apparatus, which can be integrated into a computer device, such as a server or a terminal.

[0084] like Figure 2 The present invention provides a model distillation method, the specific process of which includes:

[0085] 101. Obtain a question sample set and a teacher model for question recommendation, wherein the question sample set includes a first question sample subset and a second question sample subset.

[0086] The question sample set includes question samples, which can be used for model training and prediction.

[0087] In one embodiment, the sample question can be a variant question. A variant question is a type of question with the characteristic of multiple solutions to one problem or one problem with multiple solutions. Variant questions are helpful in cultivating students' divergent and differentiated thinking; solving one problem is equivalent to solving many others, requiring less deep thinking but increasing efficiency.

[0088] In one embodiment, the question sample set may include a first question sample subset and a second question sample subset. The first and second question sample subsets each include a portion of the question samples from the first question sample set. For example, if the question sample set includes multiple variant questions, these variant questions can be divided into two parts, with one part stored in the first question sample subset and the other part stored in the second question sample subset. Alternatively, a portion of the question samples can be randomly selected from the first question sample subset and stored therein, and then another portion can be randomly selected from the first question sample subset and stored therein in the second question sample subset.

[0089] In one embodiment, the question samples in the first question sample subset can be data augmented to obtain augmented question samples. Then, the augmented question samples can be used to augment the teacher model to obtain an augmented teacher model.

[0090] The second sample subset can serve as the training set needed for the model distillation process when the enhanced teacher model transfers "knowledge" to the student model.

[0091] By using different question sample sets in the two stages of enhancing the teacher model to an enhanced teacher model and transferring the "knowledge" from the enhanced teacher model to the student model, the model can be exposed to more diverse data, thereby improving its generalization ability.

[0092] The teacher model is an AI model used for question recommendation. Based on the question information input by the user, the teacher model filters questions from a question database that are related to the input question and recommends these related questions to the user.

[0093] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0094] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0095] For example, the teacher model can be at least one of the following: Convolutional Neural Networks (CNN), De-Convolutional Networks (DN), Deep Neural Networks (DNN), Deep Convolutional Inverse Graphics Networks (DCIGN), Region-based Convolutional Networks (RCNN), Faster Region-based Convolutional Networks (FasterRCNN), and Bidirectional Encoder Representations from Transformers (BERT) models.

[0096] In one embodiment, the method proposed in this application can be applied to all smart education scenarios, for example, it can be applied to a homework engine. One of the core features of a homework engine is variation questions, and related product functions include intelligent question generation, practice for weak knowledge points, error notebooks, personalized manuals, etc. A homework engine based on variation questions can help cultivate students' divergent and differentiated thinking, making one task equivalent to many, requiring less deep thinking and increasing efficiency.

[0097] 102. Perform data augmentation on the question samples in the first question sample subset to obtain augmented question samples.

[0098] In one embodiment, the teacher model is an artificial intelligence model for question recommendation. To improve the quality of question recommendations, the teacher model is generally structurally complex, which makes question recommendation time-consuming and results in insufficient real-time performance. Therefore, model distillation can be performed on the teacher model, allowing the "knowledge" learned by the teacher model to be transferred to the less complex student model, thereby improving the real-time performance of question recommendations. However, while the student model can improve the real-time performance of question recommendations, the performance degradation is very significant. Therefore, this application proposes a model distillation method, such as... Figure 3 As shown, this model distillation method can enhance the performance of the teacher model to obtain an enhanced teacher model. Then, the "knowledge" learned by the enhanced teacher model is transferred to the student model, so that the student model can not only improve the efficiency of question recommendation, but also ensure the effectiveness of question recommendation. This achieves better results while keeping the model lightweight, saving resources and improving performance.

[0099] This application's embodiments can enhance the teacher model from two dimensions: a data dimension and a model dimension. The data dimension refers to performing data augmentation on the question samples in the first question sample subset to obtain augmented question samples. The model dimension refers to using the augmented question samples to enhance the teacher model, resulting in an augmented teacher model.

[0100] In one embodiment, data augmentation can be performed on the question samples based on the relationships between them. These relationships can refer to similarity. For example, if question sample A and question sample B share the same or similar knowledge points, then they are related. Similarly, if the knowledge points in question sample A and question sample B both belong to the same trigonometric function range, then they are related. Furthermore, if the knowledge points in question sample A are always combined with the knowledge points in question sample B in the questions, then they are also related.

[0101] In one embodiment, the question samples in the first subset of question samples can be augmented by leveraging the properties of the relationships between question samples to obtain augmented question samples.

[0102] Among them, the nature of the relationship between the question samples can refer to the exchange and transitivity between data.

[0103] Commutativity can be categorized into positive and negative commutativity. Positive commutativity means that if ab is similar, then ba is also similar. Negative commutativity means that if ab is not similar, then ba is also not similar.

[0104] Transitivity can include positive and negative transitivity. Positive transitivity means that if ab is similar and bc is similar, then ac is similar. Negative transitivity means that if ab is similar and bc is not similar, then ac is not similar.

[0105] In one embodiment, there are several ways to augment the question samples in the first subset of question samples by leveraging the properties of the relationships between question samples to obtain augmented question samples. For example, question samples can be augmented directly based on the positive transitivity of the question samples.

[0106] Specifically, the step "data augmentation of the question samples in the first question sample subset to obtain augmented question samples" may include:

[0107] Identify the current question sample in the first question sample subset to obtain related question samples that have a preset relationship with the current question sample in the first question sample subset;

[0108] Identify and associate target question samples with preset association relationships within the first question sample subset, wherein the target question samples include other question samples in the first question sample subset besides the current question sample;

[0109] Based on the preset association between the current question sample and related question samples, as well as the preset association between related question samples and the target question sample, the association between the current question sample and the target question sample is established, resulting in an enhanced question sample.

[0110] The current question sample can refer to the question sample that is currently undergoing correct data processing.

[0111] In one embodiment, the relationships between question samples in the question sample set can be pre-defined manually. However, manual setting may miss some relationships between question samples. Therefore, data augmentation can be used to establish relationships between samples that are related but have not yet been established.

[0112] In one embodiment, the current question sample in the first question sample subset can be identified to obtain associated question samples in the first question sample subset that have a preset association relationship with the current question sample. Then, target question samples with preset association relationships are identified and associated with question samples in the first question sample subset, wherein the target question samples include other question samples in the first question sample subset besides the current question sample.

[0113] For example, the current question sample is question sample A. By identifying the relationship between question sample A and question sample B, question sample B is considered a related question sample. Then, by identifying question sample B, it is found that question sample B is related to question samples A, C, and D. At this point, the target question sample can include question samples C and D, excluding question sample A.

[0114] In one embodiment, an enhanced question sample can be obtained by establishing an association between the current question sample and the target question sample based on a preset association between the current question sample and the associated question sample, and a preset association between the associated question sample and the target question sample.

[0115] For example, since question sample A and question sample B are related, and question sample B and question sample C are related, according to the transitivity property, question sample A and question sample C are related, and then the relationship between question sample A and question sample C can be established.

[0116] For example, since problem sample A and problem sample B are related, and problem sample B and problem sample D are related, according to the transitivity property, problem sample A and problem sample D are related. Then, the relationship between problem sample A and problem sample D can be established.

[0117] In one embodiment, if the question samples are directly enhanced based on the positive transitivity of the question samples, it is found that the enhanced question samples do not perform well. The specific reasons are as follows:

[0118] For example, if sample questions a and b have a similarity score of 0.7, and sample questions b and c also have a similarity score of 0.7, after forward propagation, the similarity score between sample questions a and c might only be 0.49, less than 0.5. In this case, sample questions a and c might be considered dissimilar. Furthermore, determining the similarity between sample questions generally relies on the labeling of the knowledge points in the sample questions. If some sample questions have labeling errors, especially those with multiple knowledge points, forward propagation can amplify these errors. For example, if sample questions a and b are dissimilar but are mistakenly labeled as similar, forward propagation will generate two incorrect samples, indirectly amplifying the error.

[0119] For example, suppose there are four sample questions Q1, Q2, Q3, and Q4, and the knowledge points corresponding to each sample question are as follows:

[0120] Q1: Trigonometric functions, definition and standard equation of an ellipse

[0121] Q2: Trigonometric function synthesis and mathematical induction

[0122] Q3: Mathematical induction, geometric sequence

[0123] Q4: Mathematical induction, the definition and standard equation of an ellipse

[0124] If we directly enhance the question samples based on the positive transitivity of the question samples, we can conclude that Q1 and Q2 are similar, Q2 and Q3 are similar, and after positive transitivity, Q1 and Q3 are similar. However, in reality, Q1 and Q3 test significantly different knowledge points and are not actually similar questions because the correlation weakens during transitivity; Q1 and Q3 do not have any overlapping knowledge points.

[0125] To overcome the above problems, this application proposes an enhanced version of positive transfer. Specifically, the transfer is performed under the premise of constraining a single knowledge point of the question sample. This is because the annotation of a single knowledge point sample is generally more accurate and more reliable, resulting in better quality of the question sample after transfer.

[0126] In one embodiment, in order to overcome the above problems, when establishing the association between the current question sample and the target question sample, it can be determined whether the current question sample and the target question sample have the same knowledge point information. Only when they do, is the association between the current question sample and the target question sample established.

[0127] Specifically, the step "based on the preset association between the current question sample and the related question sample, and the preset association between the related question sample and the target question sample, establish the association between the current question sample and the target question sample to obtain the enhanced question sample" can include:

[0128] Identify the knowledge point information of the current question sample and the knowledge point information of the target question sample;

[0129] The knowledge point information of the current question sample is compared with the knowledge point information of the target question sample to obtain the comparison results;

[0130] Based on the comparison results, the correlation between the current question sample and the target question sample is established to obtain the enhanced question sample.

[0131] The knowledge point information in the question sample can refer to which knowledge points are included in the question content. For example, the knowledge point information in the question sample may include trigonometric functions, the definition and standard equation of an ellipse. Another example is that the knowledge point information in the question sample may include trigonometric functions and mathematical induction, and so on.

[0132] In one embodiment, the knowledge point information of the current question sample and the knowledge point information of the target question sample can be identified. Then, the knowledge point information of the current question sample and the target question sample are compared to obtain a comparison result. By comparing the knowledge point information of the current question sample and the target question sample, it can be determined whether the current question sample and the target question sample have the same knowledge points. If the current question sample and the target question sample have the same knowledge points, an association relationship is established between the current question sample and the target question sample, resulting in an enhanced question sample. If the current question sample and the target question sample do not have the same knowledge points, no association relationship is established between the current question sample and the target question sample, resulting in an enhanced question sample.

[0133] The enhanced version of forward propagation proposed in this application only propagates knowledge points when the question samples have the same single knowledge point constraint, thereby improving the quality of the enhanced question samples.

[0134] For example, as described above, Q4 and Q2 are similar questions, and Q2 and Q1 are similar questions. Therefore, Q4 and Q1 also have the possibility of being similar questions. In this case, we can determine whether Q4 and Q1 share the same knowledge points. Since Q4 and Q1 share the same knowledge points—the definition and standard equation of an ellipse—we can conclude that Q4 and Q1 are also similar.

[0135] Data augmentation is applied to the question samples in the first subset of questions, establishing relationships between question samples that are related but not yet linked, thus expanding the range of related questions. When the augmented question samples are used to train the teacher model, the quality of the training process is improved, thereby enhancing the overall quality of the augmented teacher model.

[0136] 103. Use the enhanced question samples to enhance the teacher model, resulting in the enhanced teacher model.

[0137] In one embodiment, in addition to enhancing the teacher model from the perspective of data, the teacher model can also be enhanced from the perspective of the model itself. Specifically, the teacher model can be enhanced using enhanced question samples to obtain an enhanced teacher model.

[0138] In one embodiment, ensemble learning methods can be used to enhance the teacher model.

[0139] Ensemble learning is not a standalone machine learning algorithm, but rather a method that combines multiple machine learning machines (base learners) to accomplish a learning task. The basic principle of ensemble learning is that, given a training set of data, we train several individual weak learners, and through a specific combination strategy, we can ultimately form a strong learner, thus achieving the goal of leveraging the strengths of multiple learners.

[0140] Here, a strong learner is a concept relative to a weak learner. A strong learner refers to a learning algorithm that can predict results with considerable accuracy. In contrast, a weak learner typically produces significantly worse predictions. For example, in the embodiments of this application, the enhanced teacher model can be a strong learner, while the teacher model can be a weak learner.

[0141] Here, the base learner can refer to an individual learner in ensemble learning, and the base learner is usually a weak learner. For example, in the embodiments of this application, the teacher model can be a base learner.

[0142] In one embodiment, ensemble learning employs various construction algorithms. Examples include Bagging, Random Forest, AdaBoost, and Gradient Boosting Tree, among others.

[0143] In one embodiment, these ensemble learning construction algorithms can be used to augment the teacher model, resulting in an augmented teacher model.

[0144] For example, the teacher model can be augmented using algorithms such as Bagging, Random Forest, AdaBoost, or Gradient Boosting Tree to obtain an augmented teacher model.

[0145] In one embodiment, the enhanced teacher model can also be obtained by increasing the structural complexity of the teacher model, because generally, the more complex the model structure, the higher its fitting ability. Specifically, multiple teacher models can be trained and combined to form the enhanced teacher model. These multiple teacher models can be models with the same network structure and model parameters. Alternatively, the teacher models can also be models with different network structures and model parameters.

[0146] Specifically, the step "using the enhanced question samples to enhance the teacher model, resulting in an enhanced teacher model" can include:

[0147] Each teacher model is trained using the enhanced question samples to obtain the trained teacher model for each teacher model.

[0148] Based on the training results of the post-trained teacher model, a performance metric for the post-trained teacher model is generated, wherein the performance metric characterizes the quality of the post-trained teacher model in recommending questions.

[0149] Based on the performance metrics of each trained teacher model, the trained teacher models are combined into an enhanced teacher model.

[0150] In one embodiment, each teacher model can be trained using the enhanced question samples to obtain the trained teacher model corresponding to each teacher model.

[0151] For example, training each teacher model using augmented question samples can involve inputting two augmented question samples into the teacher model, which then determines whether the two augmented question samples are related and outputs the result. This result can then be compared with a pre-set correct result, and the coefficients of the teacher model can be adjusted based on the comparison.

[0152] For example, the quality of training teacher models can be improved by adjusting the augmented question samples used to train multiple teacher models. Specifically, the step "training each teacher model using the augmented question samples to obtain the trained teacher model for each teacher model" can include:

[0153] Initialize the weight coefficients for each augmented question sample, where the weight coefficients represent the value of the augmented question sample for training;

[0154] The teacher model was trained using the enhanced question samples to obtain the initial trained teacher model and the initial training results;

[0155] Based on the initial training results, the weight coefficients of the augmented question samples are updated to obtain the updated weight coefficients of the augmented question samples.

[0156] Based on the updated weight coefficients of the enhanced question samples, select the target question samples from the enhanced question samples;

[0157] The initial trained teacher model is trained using the target question samples to obtain the trained teacher model.

[0158] For example, assuming there are m augmented question samples, we can first initialize each augmented question sample with the same weight coefficient, i.e., 1 / m. The weight coefficient of each augmented question sample represents its value for training, i.e., the probability that the augmented question sample will be selected into the training set by the teacher model. Generally, the larger the weight coefficient of an augmented question sample, the greater its value for training, and the higher its probability of being selected into the training set by the teacher model.

[0159] Then, the teacher model can be trained using all the augmented question samples to obtain the initial trained teacher model and initial training results. The model typically requires multiple training iterations before its performance meets the requirements. Each training iteration that does not meet the performance requirements can be referred to as the initial trained teacher model. When the performance of the trained teacher model meets the requirements, it can be called the trained teacher model.

[0160] Then, based on the initial training results, the weight coefficients of the augmented question samples can be updated to obtain the updated weight coefficients of the augmented question samples.

[0161] For example, there are 10 augmented question samples, w0 to w9, with the weight coefficient of each augmented question sample initialized to 0.1. Then, the teacher model is trained using these 10 augmented question samples, resulting in the initial trained teacher model and initial training results. Upon reviewing the initial training results, it is found that the predictions for question samples w6, w7, and w8 are incorrect. In this case, the weight coefficients of w6, w7, and w8 can be increased, while the weight coefficients of the other augmented question samples can be decreased.

[0162] Then, based on the updated weight coefficients of the augmented question samples, target question samples can be selected from the augmented question samples. These target question samples can include the question samples selected as part of the training set. Generally, samples with higher weight coefficients have a higher probability of being selected, while those with lower weight coefficients have a lower probability of being selected.

[0163] Then, the target question samples can be used to train the initially trained teacher model to obtain the trained teacher model.

[0164] Through the embodiments of this application, the teacher model can continuously focus on those difficult-to-distinguish question samples during the training process, thereby improving the quality of the trained teacher model. For example, during training, the teacher model often mispredicts some question samples with rich knowledge points. Through the embodiments of this application, the teacher model can focus on these question samples that are consistently mispredicted, thereby continuously improving the teacher model's sensitivity and predictive ability to these question samples, and thus improving the prediction quality of the teacher model.

[0165] In one embodiment, a performance metric for the trained teacher model can be generated based on the training results of the trained teacher model. This performance metric characterizes the quality of the trained teacher model's question recommendation capabilities. In other words, this performance metric allows for comparison of the training quality among multiple trained teacher models.

[0166] For example, the performance metric could be the error rate of the model's predictions of samples. For instance, the samples that were predicted incorrectly could be included in the total training sample to obtain the performance metric.

[0167] For example, given 10 augmented question samples, w0 to w9, with an initial weight coefficient of 0.1 for each augmented question sample, and if the prediction results for question samples w6, w7, and w8 are incorrect when the initial training results are used for identification, then the performance metric can be 0.3.

[0168] In one embodiment, trained teacher models can be combined into an enhanced teacher model based on the performance metrics of each trained teacher model.

[0169] Specifically, the step "combining the trained teacher models into an enhanced teacher model based on the performance metrics of each trained teacher model" may include:

[0170] Generate the combined weight coefficients for each trained teacher model based on performance metrics.

[0171] Each trained teacher model is fused together based on the combined weight coefficients to obtain the fused teacher model;

[0172] The merged teacher model is constructed into an enhanced teacher model based on the preset constructor.

[0173] For example, performance metrics can be converted into combined weighting coefficients using the following formula:

[0174]

[0175] Where α can represent the combined weight coefficient, and ∈ can represent the performance measurement index.

[0176] Then, each trained teacher model can be multiplied by its corresponding combined weight coefficients to obtain the fused teacher model. The fused teacher model can then be used to construct the enhanced teacher model according to a predefined constructor function. This predefined constructor function can be a sign function, a sigmoid function, etc.

[0177] For example, the trained teacher model can be combined into an enhanced teacher model according to the following formula:

[0178]

[0179] Where G(x) can represent the enhanced teacher model, h i (x) can represent the i-th teacher model, α i It can represent the weight combination coefficient corresponding to the i-th teacher model, N can refer to the total number of teacher models, and x can refer to the input of the model.

[0180] By combining the trained teacher models into an enhanced teacher model according to the above formula, the teacher models with lower prediction error rates have greater influence and play a more decisive role in the final enhanced teacher model, while the teacher models with higher prediction error rates have less influence and play a smaller decisive role. In other words, higher-quality teacher models account for a larger proportion of the final enhanced teacher model, and vice versa, thereby enhancing the quality of the enhanced teacher model.

[0181] The enhanced teacher model is better than the teacher model in terms of generalization and expressive power, thus enabling the distillation process to be learned better.

[0182] 104. Use the enhanced teacher model to identify the question samples in the second question sample subset and obtain the reference distribution information of the question samples in the second question sample subset.

[0183] In one embodiment, after obtaining the enhanced teacher model, the "knowledge" learned by the enhanced teacher model can be transferred to the student model model, thereby improving the real-time performance of question recommendations through the student model and improving the quality of question recommendations for student solutions compared with the prior art.

[0184] Specifically, the enhanced teacher model can be used to identify question samples in the second question sample subset to obtain reference distribution information for the question samples in the second question sample subset. Then, the student model is obtained, and the student model is used to learn from the reference distribution information obtained by the enhanced teacher model to obtain the target student model.

[0185] In one embodiment, question information can be extracted from the question sample, and then reference distribution information of the question sample can be obtained based on the question information.

[0186] Specifically, the step "using the enhanced teacher model to identify question samples in the second question sample subset and obtain reference distribution information of question samples in the second question sample subset" may include:

[0187] Information is extracted from the question samples to obtain the question information of the question samples;

[0188] The question information is normalized to obtain normalized question information;

[0189] The enhanced teacher model is used to identify the normalized question information to obtain the reference distribution information of the question samples in the second question sample subset.

[0190] The question sample information may include a description of the question. For example, the question sample information may include the question stem, options, question type, applicable grade level, learning stage, subject, knowledge point, and chapter, etc. Figure 4 The image shown can be a schematic diagram of the question information.

[0191] In one embodiment, the problem samples may be obtained through various means, such as scanning from books, downloading from the internet, etc. Therefore, there may be inconsistencies in the expression of professional terms or formulas among the problem samples, which can affect the quality of model distillation. Therefore, after obtaining the problem information of the problem samples, the problem information can be normalized to obtain normalized problem information.

[0192] Normalizing the question information in the sample questions can refer to normalizing various types of content, such as technical terms, formulas, and formats, within the question information. The normalization methods for different types of content may differ. The specific "steps" for "normalizing the question information to obtain normalized question information" can include:

[0193] Identify at least one type of content to be normalized from the question information;

[0194] Determine the normalization method for the content to be normalized of the current type;

[0195] The content to be normalized is transformed using a normalization method to obtain the normalized question information.

[0196] For example, the problem samples might be obtained from the internet, such as through learning websites. In this case, the problem samples would contain web page symbols. For instance, the problem information might include Hyper Text Markup Language (HTML) symbols or Cascading Style Sheets (CSS) symbols, etc. In this case, these symbols can be removed to prevent them from affecting the quality of the model distillation, thereby impacting the quality of the student's model.

[0197] One method is to use regular expression matching to remove webpage symbols from the question information of the sample questions.

[0198] For example, different question samples might use different terms to express the same meaning. Some question samples might describe "Euclidean metric" as "Euclidean distance," while others might describe it as "Euclidean distance." Therefore, these technical terms can be normalized to avoid affecting the model distillation results.

[0199] This can be achieved by pre-setting a table for replacing technical terms, which records the normalized terms corresponding to different terms expressing the same meaning. The server can then normalize the technical terms based on this table.

[0200] Furthermore, different question samples may use different formulas to express the same meaning. In this case, these formulas can be normalized to avoid affecting the model distillation effect. For example, the syntax parsing tool Antlr can be used to normalize the formulas.

[0201] In one embodiment, the enhanced teacher model can be used to identify the normalized question information to obtain the reference distribution information of the question samples in the second question sample subset.

[0202] Specifically, the step "using the enhanced teacher model to identify the normalized question information and obtain the reference distribution information of the question samples in the second question sample subset" may include:

[0203] Feature extraction is performed on the normalized question information to obtain the feature information of the normalized question information;

[0204] The feature information is pooled to obtain the pooled feature information;

[0205] The pooled feature information is fitted to a distribution to obtain the distribution information of the question samples in the second question sample subset.

[0206] In one embodiment, feature extraction can be performed on the normalized question information to obtain the feature information of the normalized question information.

[0207] For example, the model structure of the enhanced teacher model can be as follows: Figure 5 As shown. (Through) Figure 5 As can be seen, the enhanced teacher model can include four parts: question normalization, L-layer Transformer, pooling operation, and classification layer.

[0208] After normalizing the sample questions, features of the normalized question information can be extracted using the Transformer in the augmented teacher model to obtain the feature information of the normalized question information. The augmented teacher model can include L layers of Transformers, each mainly consisting of self-attention mechanisms, hierarchical normalization, and forward fully connected layers. The self-attention mechanism is primarily implemented using a multi-head attention mechanism.

[0209] Then, pooling operations can be performed on the feature information to obtain pooled feature information.

[0210] Pooling is a crucial concept in convolutional neural networks, essentially a form of downsampling. There are various non-linear pooling functions, with max pooling being the most common. It divides the input information into several rectangular regions and outputs the maximum value for each sub-region. Besides max pooling, pooling layers can also use other pooling functions, such as average pooling and even L2-norm pooling.

[0211] For example, max pooling or average pooling operations can be performed on feature information, and so on.

[0212] Then, the pooled feature information can be fitted to a distribution to obtain the distribution information of the question samples in the second question sample subset. This classification layer can be composed of a three-layer neural network, and its function is to classify the sample questions, thereby recommending questions to the user based on the classification results.

[0213] 105. Obtain the student model, and use the student model to learn the reference distribution information obtained from the enhanced teacher model to obtain the target student model.

[0214] In one embodiment, after obtaining the reference distribution information of the question samples in the second question sample subset, a student model can be obtained. The student model is then used to learn the reference distribution information identified by the enhanced teacher model, thereby achieving the transfer of the "knowledge" of the enhanced teacher model to the student model.

[0215] The student model's structure can be similar to the enhanced teacher model, but the difference lies in the student model's network structure being much simpler. For example, ... Figure 5 As shown, the structure of the student model is similar to that of the augmented teacher model, the difference being that the student model has fewer Transformer layers than the augmented teacher model. For example, the student model has K Transformer layers, while the augmented teacher model has L Transformer layers, where L is less than K.

[0216] In one embodiment, the student model can be a pre-trained model. The pre-training objectives primarily include two tasks: a language task involving a full-word mask model and a task involving next-sentence prediction. By pre-training the student model, it gains a certain foundation of "knowledge." When the knowledge from the augmented teacher model is transferred to the student model, the student model can learn the augmented teacher model's "knowledge" better and faster, thereby improving the overall quality of the target student model.

[0217] In one embodiment, the student model can be trained using question samples from a second question sample subset, and then the reference distribution information identified by the augmented teacher model can be used as the "learning target" of the student model during the training process, thereby guiding the student model to learn the knowledge of the augmented teacher model.

[0218] Specifically, the step "obtaining the student model, using the student model to learn the reference distribution information obtained from the enhanced teacher model, and obtaining the target student model" can include:

[0219] The student model is used to identify the question samples in the second question sample subset to obtain the distribution information of the question samples in the second question sample subset;

[0220] The model loss information is calculated between the reference distribution information obtained by the enhanced teacher model based on the question sample recognition and the distribution information obtained by the student model based on the question sample recognition.

[0221] The student model is adjusted using the model loss information to obtain the target student model.

[0222] For example, the enhanced teacher model identifies question sample A and obtains its reference distribution information. Then, the student model is also used to identify question sample A and obtain its distribution information. Based on the reference and distribution information of question sample A, the model loss information of the student model can then be calculated. Finally, this model loss information is used to adjust the student model.

[0223] In one embodiment, the step "calculating the model loss information between the reference distribution information obtained by the enhanced teacher model based on the question sample recognition and the distribution information obtained by the student model based on the question sample recognition" may include:

[0224] Adjust the smoothness of the reference distribution information and the distribution information to obtain the adjusted reference distribution information and the adjusted distribution information;

[0225] Logarithmic operation is performed on the adjusted distribution information to obtain the logarithmically adjusted distribution information, and logarithmic operation is performed on the adjusted reference distribution information to obtain the logarithmically adjusted reference distribution information;

[0226] Multiply the adjusted distribution information and the logarithmic distribution information to obtain the multiplied distribution information, and multiply the adjusted reference distribution information and the logarithmic reference distribution information to obtain the multiplied reference distribution information.

[0227] The model loss information is obtained by adding the reference distribution information and the multiplied distribution information together.

[0228] Specifically, the model loss information of the student model can be calculated using the following formula:

[0229]

[0230] Among them, z T This can represent the distribution information of the question samples identified by the enhanced teacher model, z s This can represent the distribution information of the question samples identified by the student model. Here, τ can represent the temperature coefficient, used to smooth the conditional distribution. σ T () can represent the enhanced teacher model, σ s () can represent the student model. Where L CE () can refer to the cross-entropy function, which can be expressed as follows:

[0231]

[0232] In one embodiment, the label loss information between the distribution information obtained by the student model based on the question samples and its labels can also be calculated. Then, based on the label loss information and the model loss information, the model parameters of the student model are adjusted to obtain the target student model.

[0233] Specifically, the step of "adjusting the student model using model loss information to obtain the target student model" may include:

[0234] Obtain the tag information of the question samples;

[0235] Calculate the label loss information between the distribution information and label information of the question samples;

[0236] The label loss information and the model loss information are fused to obtain the fused loss information;

[0237] The model parameters of the student model are adjusted based on the fused loss information to obtain the target student model.

[0238] For example, the label loss information between the distribution information and label information of the question samples can be calculated based on the cross-entropy function.

[0239] For example, label loss information can be represented as follows:

[0240]

[0241] Here, f can represent the distribution information of the question samples, and f′ can represent the label information corresponding to the question samples.

[0242] Then, the label loss information and the model loss information can be fused to obtain the fused loss information. Specifically:

[0243] Total Loss = λ * l KD +(1-λ)l CE

[0244] Where λ can represent the balance factor, l KD The model loss function, l CE This represents the label loss function.

[0245] Then, the model parameters of the student model can be adjusted based on the fused loss information to obtain the target student model.

[0246] In one embodiment, the present application also tested the target student model and the teacher model, and the test results are shown in Table 1:

[0247]

[0248]

[0249] Table 1: Test Results

[0250] The test results clearly show that the target student model can effectively improve the real-time performance of question recommendations, and its performance in question recommendation is slightly better than that of the teacher model.

[0251] In this embodiment, a question sample set and a teacher model for question recommendation can be obtained. The question sample set includes a first question sample subset and a second question sample subset. Data augmentation is performed on the question samples in the first question sample subset to obtain augmented question samples. The augmented question samples are then used to augment the teacher model to obtain an augmented teacher model. The augmented teacher model is then used to identify question samples in the second question sample subset to obtain reference distribution information for the question samples in the second question sample subset. A student model is then obtained, and the student model is used to learn from the reference distribution information identified by the augmented teacher model to obtain a target student model. This application augments the teacher model to obtain an augmented teacher model, resulting in a model whose performance is significantly better than the original teacher model. When the knowledge from the augmented teacher model is "transferred" to the student model, the target student model can maintain its lightweight nature while ensuring that its question recommendation performance does not decline significantly, and may even outperform the original teacher model.

[0252] Secondly, the embodiments of this application can enhance the teacher model from two dimensions: a data dimension and a model dimension. The data dimension refers to data augmentation of the question samples in the first question sample subset, resulting in enhanced question samples. The model dimension refers to using the enhanced question samples to enhance the teacher model, resulting in an enhanced teacher model. By enhancing the teacher model from these two dimensions, the performance of the enhanced teacher model can be significantly improved. Therefore, when the knowledge from the enhanced teacher model is "transferred" to the student model, the student model can learn more knowledge, leading to a substantial improvement in the performance of the target student model.

[0253] Based on the method described in the above embodiments, the following examples will provide further detailed explanations.

[0254] This application will use the integration of a model distillation method on a server as an example to illustrate the method of this application.

[0255] In one embodiment, such as Figure 6 As shown, a model distillation method is described below:

[0256] 201. The server obtains a question sample set and a teacher model for question recommendation, wherein the question sample set includes a first question sample subset and a second question sample subset.

[0257] For example, the question sample set includes multiple question samples, where each question sample can be a variant question. A variant question is a type of question that has the characteristic of multiple solutions to one problem or one solution to multiple problems.

[0258] In one embodiment, the question sample set may include a first question sample subset and a second question sample subset. The first and second question sample subsets each include a portion of the question samples in the question sample set. For example, if the question sample set includes multiple variant questions, these variant questions can be divided into two parts, with one part stored in the first question sample subset and the other part stored in the second question sample subset.

[0259] The teacher model is an AI model used for question recommendation. Based on the question information input by the user, the teacher model filters questions from a question database that are related to the input question and recommends these related questions to the user.

[0260] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0261] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0262] For example, the teacher model can be at least one of the following: Convolutional Neural Networks (CNN), De-Convolutional Networks (DN), Deep Neural Networks (DNN), Deep Convolutional Inverse Graphics Networks (DCIGN), Region-based Convolutional Networks (RCNN), Faster Region-based Convolutional Networks (FasterRCNN), and Bidirectional Encoder Representations from Transformers (BERT) models.

[0263] In one embodiment, the teacher model is an AI model for question recommendation. To improve the quality of question recommendations, the teacher model is generally complex in structure, which makes question recommendation time-consuming and results in insufficient real-time performance. Model distillation can be performed on the teacher model to transfer the "knowledge" learned by the teacher model to the less complex student model, thereby improving the real-time performance of question recommendations. However, while the student model can improve the real-time performance of question recommendations, the performance degradation is significant. Therefore, this application proposes a model distillation method that enhances the performance of the teacher model to obtain an enhanced teacher model. Then, the "knowledge" learned by the enhanced teacher model is transferred to the student model, enabling the student model to not only improve the efficiency of question recommendation but also maintain the effectiveness of question recommendation. This achieves better results while maintaining a lightweight model, saving resources and improving performance.

[0264] 202. The server performs data augmentation on the question samples in the first question sample subset to obtain augmented question samples.

[0265] This application's embodiments can enhance the teacher model from two dimensions: a data dimension and a model dimension. The data dimension refers to performing data augmentation on the question samples in the first question sample subset to obtain augmented question samples. The model dimension refers to using the augmented question samples to enhance the teacher model, resulting in an augmented teacher model.

[0266] In one embodiment, the question samples in the first subset of question samples can be augmented by leveraging the properties of the relationships between question samples to obtain augmented question samples.

[0267] Among them, the nature of the relationship between the question samples can refer to the exchange and transitivity between data.

[0268] Commutativity can be categorized into positive and negative commutativity. Positive commutativity means that if ab is similar, then ba is also similar. Negative commutativity means that if ab is not similar, then ba is also not similar.

[0269] Transitivity can include positive and negative transitivity. Positive transitivity means that if ab is similar and bc is similar, then ac is similar. Negative transitivity means that if ab is similar and bc is not similar, then ac is not similar.

[0270] In one embodiment, if the question samples are directly enhanced based on the positive transitivity of the question samples, it is found that the enhanced question samples do not perform well. The specific reasons are as follows:

[0271] For example, if sample questions a and b have a similarity score of 0.7, and sample questions b and c also have a similarity score of 0.7, after forward propagation, the similarity score between sample questions a and c might only be 0.49, less than 0.5. In this case, sample questions a and c might be considered dissimilar. Furthermore, determining the similarity between sample questions generally relies on the labeling of the knowledge points in the sample questions. If some sample questions have labeling errors, especially those with multiple knowledge points, forward propagation can amplify these errors. For example, if sample questions a and b are dissimilar but are mistakenly labeled as similar, forward propagation will generate two incorrect samples, indirectly amplifying the error.

[0272] To overcome the above problems, this application proposes an enhanced version of positive transfer. Specifically, the transfer is performed under the premise of constraining a single knowledge point of the question sample. This is because the annotation of a single knowledge point sample is generally more accurate and more reliable, resulting in better quality of the question sample after transfer.

[0273] For example, suppose there are four sample questions Q1, Q2, Q3, and Q4, and the knowledge points corresponding to each sample question are as follows:

[0274] Q1: Trigonometric functions, definition and standard equation of an ellipse

[0275] Q2: Trigonometric function synthesis and mathematical induction

[0276] Q3: Mathematical induction, geometric sequence

[0277] Q4: Mathematical induction, the definition and standard equation of an ellipse

[0278] Based on the original principle of positive propagation, the correlation between sample questions is identified as follows: Q1 and Q2 are similar, Q2 and Q3 are similar, and after positive propagation, Q1 and Q3 are similar. However, in reality, Q1 and Q3 test significantly different knowledge points and are not actually similar questions because the correlation weakens during the propagation process; Q1 and Q3 do not have any overlapping knowledge points.

[0279] The enhanced version of forward propagation proposed in this application only propagates knowledge points when the question samples have the same single knowledge point constraint, thereby improving the quality of the enhanced question samples.

[0280] For example, if Q4 and Q2 are similar questions, and Q2 and Q1 are similar questions, then Q4 and Q1 also have the possibility of being similar questions. In this case, we can determine whether Q4 and Q1 share the same knowledge points. Since Q4 and Q1 share the same knowledge points—the definition and standard equation of an ellipse—we can conclude that Q4 and Q1 are also similar.

[0281] In this way, the number of relevant question samples can be increased while ensuring the quality of the enhanced question samples.

[0282] 203. The server uses the enhanced question samples to enhance the teacher model, resulting in the enhanced teacher model.

[0283] In one embodiment, the teacher model can be augmented using the augmented question samples to obtain an augmented teacher model.

[0284] One approach is to enhance the teacher model using ensemble learning.

[0285] Ensemble learning is not a single machine learning algorithm, but rather a process of building and combining multiple machine learning machines (base learners) to accomplish a learning task. The basic principle of ensemble learning is that, given a training set of data, we train several individual weak learners, and through a specific combination strategy, we can ultimately form a strong learner, thus achieving the goal of leveraging the strengths of multiple learners.

[0286] Here, a strong learner is a concept relative to a weak learner. A strong learner refers to a learning algorithm that can predict results with considerable accuracy. In contrast, a weak learner typically produces significantly worse predictions. For example, in the embodiments of this application, the enhanced teacher model can be a strong learner, while the teacher model can be a weak learner.

[0287] Here, the base learner can refer to an individual learner in ensemble learning, and the base learner is usually a weak learner. For example, in the embodiments of this application, the teacher model can be a base learner.

[0288] In one embodiment, ensemble learning employs various construction algorithms. Examples include Bagging, Random Forest, AdaBoost, and Gradient Boosting Tree, among others.

[0289] In one embodiment, these ensemble learning construction algorithms can be used to augment the teacher model, resulting in an augmented teacher model.

[0290] For example, the teacher model can be augmented using algorithms such as Bagging, Random Forest, AdaBoost, or Gradient Boosting Tree to obtain an augmented teacher model.

[0291] In one embodiment, a teacher model can be used as a base learner, and then in each iteration, the sample weights of the previous erroneous samples can be increased. Finally, multiple base learners are combined to form an enhanced teacher model.

[0292] For example, assuming there are m augmented question samples, we can first assign the same weight, 1 / m, to each augmented question sample. The weight of each augmented question sample represents the probability that it will be selected into the training set by the teacher model. If the augmented question sample is correctly classified by the teacher model, its weight will decrease. If it is misclassified by the teacher model, its weight will increase. Assuming the teacher model is a BERT model, augmenting it into an augmented teacher model requires T iterations.

[0293] In each iteration, if an augmented question sample has been accurately classified, its weight is reduced in the next training set; conversely, if an augmented question sample is not accurately classified, its weight is increased. Simultaneously, the discourse power corresponding to the teacher model can be calculated. Then, the weights of the augmented question samples are updated to obtain updated question samples. Next, a new training set is selected based on the updated question sample weights, and this new training set is used to train the next teacher model. This entire training process continues iteratively.

[0294] Finally, the various trained teacher models are combined into an enhanced teacher model. After the training process of each teacher model is completed, the teacher model with the lower prediction error rate has greater influence and plays a more decisive role in the final enhanced teacher model, while the teacher model with the higher prediction error rate has less influence and plays a less decisive role. In other words, teacher models with lower error rates account for a larger proportion of the final enhanced teacher model, and vice versa.

[0295] 204. The server uses the enhanced teacher model to identify the question samples in the second question sample subset and obtains the reference distribution information of the question samples in the second question sample subset.

[0296] In one embodiment, after obtaining the enhanced teacher model, the "knowledge" learned by the enhanced teacher model can be transferred to the student model model, thereby improving the real-time performance of question recommendations through the student model and improving the quality of question recommendations for student solutions compared with the prior art.

[0297] In one embodiment, the enhanced teacher model's structure can be as follows: Figure 5 As shown. (Through) Figure 5 As can be seen, the enhanced teacher model can include four parts: question normalization, L-layer Transformer, pooling operation, and classification layer.

[0298] In one embodiment, when transferring the "knowledge" learned by the enhanced teacher model to the student model in the model structure, the enhanced teacher model can normalize the question information of the question samples based on the question information in the second question sample subset. Normalizing the question information can refer to normalizing professional terminology, formulas, and formats within the question information.

[0299] For example, the problem samples might be obtained from the internet, such as through learning websites. In this case, the problem samples would contain web page symbols. For instance, the problem information might include Hyper Text Markup Language (HTML) symbols or Cascading Style Sheets (CSS) symbols, etc. In this case, these symbols can be removed to prevent them from affecting the quality of the model distillation, thereby impacting the quality of the student's model.

[0300] One method is to use regular expression matching to remove webpage symbols from the question information of the sample questions.

[0301] For example, different question samples might use different terms to express the same meaning. Some question samples might describe "Euclidean metric" as "Euclidean distance," while others might describe it as "Euclidean distance." Therefore, these technical terms can be normalized to avoid affecting the model distillation results.

[0302] This can be achieved by pre-setting a table for replacing technical terms, which records the normalized terms corresponding to different terms expressing the same meaning. The server can then normalize the technical terms based on this table.

[0303] Furthermore, different question samples may use different formulas to express the same meaning. In this case, these formulas can be normalized to avoid affecting the model distillation effect. For example, the syntax parsing tool Antlr can be used to normalize the formulas.

[0304] In one embodiment, after normalizing the sample questions, features can be extracted from the normalized question information using a Transformer to obtain the feature information of the normalized question information. The enhanced teacher model can include L layers of Transformers, each mainly consisting of self-attention mechanisms, hierarchical normalization, and forward fully connected layers. The self-attention mechanism is primarily implemented using a multi-head attention mechanism.

[0305] In one embodiment, after the L-layer Transformer outputs feature information, pooling operations can be performed on the feature information to obtain pooled feature information.

[0306] Pooling is a crucial concept in convolutional neural networks, essentially a form of downsampling. There are various non-linear pooling functions, with max pooling being the most common. It divides the input information into several rectangular regions and outputs the maximum value for each sub-region. Besides max pooling, pooling layers can also use other pooling functions, such as average pooling and even L2-norm pooling.

[0307] For example, max pooling or average pooling operations can be performed on feature information, and so on.

[0308] In one embodiment, a classification layer can be used to fit the distribution of the pooled feature information to obtain the distribution information of the sample questions. This classification layer can be composed of a three-layer neural network, and its function is to classify the sample questions, thereby recommending questions to the user based on the classification results.

[0309] 205. The server obtains the student model and uses it to learn the reference distribution information obtained from the enhanced teacher model to obtain the target student model.

[0310] In one embodiment, the student model can be used to learn the distribution information obtained from the enhanced teacher model to obtain the target student model.

[0311] The student model can be a pre-trained model, which can further improve the model quality of the target student model.

[0312] In one embodiment, such as Figure 5 As shown, the structure of the student model is similar to that of the augmented teacher model, the difference being that the student model has fewer Transformer layers than the augmented teacher model. For example, the student model has K Transformer layers, while the augmented teacher model has L Transformer layers, where L is less than K.

[0313] In one embodiment, learning the distribution information identified by the augmented teacher model using the student model can refer to calculating the loss function of the student model using the distribution information identified by the teacher model. This allows the student model to continuously adjust the parameters in the model according to the loss function (equivalent to "learning" from the augmented teacher model) to obtain the target learning model.

[0314] Specifically, for example, the enhanced teacher model identifies question sample A, obtaining the first distribution information of question sample A. Then, the student model is also used to identify question sample A, obtaining the second distribution information of question sample A. Then, the loss information between the first and second distribution information of question sample A can be calculated. The loss information between the first and second distribution information of question sample A can be calculated according to the following formula:

[0315]

[0316] Among them, z T This can represent the distribution information of the question samples identified by the enhanced teacher model, z s This can represent the distribution information of the question samples identified by the student model. Here, τ can represent the temperature coefficient, used to smooth the conditional distribution. σ T() can represent the enhanced teacher model, σ s () can represent the student model. Where L CE () can refer to the cross-entropy function, which can be expressed as follows:

[0317]

[0318] Furthermore, the distribution information of question sample A and the loss information between its labels can be calculated. Specifically, the distribution information of question sample A and the loss information between its labels can be calculated using the cross-entropy function.

[0319] For example, the loss information between the first and second distribution information of the problem sample A is calculated to be l. KD The loss information between the distribution information of sample A and its label is l. CE Then, the two sets of loss information can be integrated to obtain the integrated loss information. The parameters of the student model can then be adjusted using the integrated loss information.

[0320] For example, the integrated loss information can be represented as follows:

[0321] Total Loss = λ * l KD +(1-λ)l CE

[0322] Where λ can represent the balance factor, l KD The model loss function, l CE This represents the label loss function.

[0323] In this embodiment, the server obtains a question sample set and a teacher model for question recommendation. The question sample set includes a first question sample subset and a second question sample subset. The server performs data augmentation on the question samples in the first question sample subset to obtain augmented question samples. The server uses the augmented question samples to augment the teacher model to obtain an augmented teacher model. The server uses the augmented teacher model to identify the question samples in the second question sample subset to obtain reference distribution information of the question samples in the second question sample subset. The server obtains a student model and uses the student model to learn from the reference distribution information identified by the augmented teacher model to obtain a target student model. The target student model obtained through this embodiment can improve both the real-time performance and the quality of question recommendation.

[0324] To better implement the model distillation method provided in the embodiments of this application, one embodiment also provides a model distillation apparatus, which can be integrated into a computer device. The meanings of the terms used are the same as in the model distillation method described above, and specific implementation details can be found in the description of the method embodiments.

[0325] In one embodiment, a model distillation apparatus is provided, which can be specifically integrated into a computer device, such as... Figure 7 As shown, the model distillation apparatus includes: an acquisition unit 301, a data augmentation unit 302, a model augmentation unit 303, a recognition unit 304, and a distillation unit 305, as detailed below:

[0326] The acquisition unit 301 is used to acquire a question sample set and a teacher model for question recommendation, wherein the question sample set includes a first question sample subset and a second question sample subset;

[0327] The data augmentation unit 302 is used to augment the question samples in the first question sample subset to obtain augmented question samples;

[0328] The model enhancement unit 303 is used to enhance the teacher model using the enhanced question samples to obtain the enhanced teacher model.

[0329] The identification unit 304 is used to identify the question samples in the second question sample subset using the enhanced teacher model, and obtain the reference distribution information of the question samples in the second question sample subset;

[0330] Distillation unit 305 is used to acquire a student model and learn the reference distribution information identified by the enhanced teacher model using the student model to obtain a target student model.

[0331] In one embodiment, the data enhancement unit 302 may include:

[0332] The first relation identification subunit is used to identify the current question sample in the first question sample subset and obtain the associated question samples in the first question sample subset and the current question sample that have a preset association relationship.

[0333] The second relationship identification subunit is used to identify target question samples that have a preset relationship with the associated question samples in the first question sample subset, wherein the target question samples include other question samples in the first question sample subset other than the current question sample;

[0334] The relationship establishment subunit is used to establish the relationship between the current question sample and the target question sample based on the preset relationship between the current question sample and the associated question sample, and the preset relationship between the associated question sample and the target question sample, so as to obtain the enhanced question sample.

[0335] In one embodiment, the relationship-establishing subunit may include:

[0336] The knowledge point recognition module is used to identify the knowledge point information of the current question sample and the knowledge point information of the target question sample;

[0337] The comparison module is used to compare the knowledge point information of the current question sample with the knowledge point information of the target question sample to obtain the comparison result;

[0338] The relationship establishment module is used to establish the association between the current question sample and the target question sample based on the comparison results, so as to obtain the enhanced question sample.

[0339] In one embodiment, the model enhancement unit 303 may include:

[0340] The training subunit is used to train each teacher model using the enhanced question samples to obtain the trained teacher model corresponding to each teacher model.

[0341] The indicator generation subunit is used to generate a performance measurement indicator of the trained teacher model based on the training results of the trained teacher model, wherein the performance measurement indicator characterizes the quality of the trained teacher model in recommending questions.

[0342] The constituent sub-units are used to combine the trained teacher models into the enhanced teacher model based on the performance metrics of each trained teacher model.

[0343] In one embodiment, the training subunit may include:

[0344] An initialization module is used to initialize the weight coefficients of each augmented question sample, wherein the weight coefficients represent the value of the augmented question sample for training.

[0345] The first training module is used to train the teacher model using the enhanced question samples to obtain the initial trained teacher model and the initial training results.

[0346] An update module is used to update the weight coefficients of the enhanced question samples based on the initial training results, so as to obtain the updated weight coefficients of the enhanced question samples.

[0347] The filtering module is used to filter out target question samples from the enhanced question samples based on the updated weight coefficients of the enhanced question samples.

[0348] The second training module is used to train the initial trained teacher model using the target question samples to obtain the trained teacher model.

[0349] In one embodiment, the constituent subunit may include:

[0350] The coefficient generation module is used to generate combined weight coefficients for each trained teacher model based on the performance metric.

[0351] The model fusion module is used to fuse each trained teacher model according to the combined weight coefficients to obtain a fused teacher model.

[0352] The model construction module is used to construct the enhanced teacher model from the fused teacher model according to the preset constructor.

[0353] In one embodiment, the identification unit 304 may include:

[0354] The information extraction subunit is used to extract information from the question sample to obtain the question information of the question sample;

[0355] The normalization subunit is used to normalize the question information to obtain normalized question information.

[0356] The first identification subunit is used to identify the normalized question information using the enhanced teacher model to obtain the reference distribution information of the question samples in the second question sample subset.

[0357] In one embodiment, the normalization subunit may include:

[0358] The content recognition module is used to identify at least one type of content to be normalized in the question information;

[0359] The method determination module is used to determine the normalization method for the content to be normalized of the current type;

[0360] The content conversion module is used to convert the content to be normalized using the normalization method to obtain the normalized question information.

[0361] In one embodiment, the identification subunit may include:

[0362] The feature extraction module is used to extract features from the normalized question information to obtain the feature information of the normalized question information;

[0363] The pooling module is used to perform pooling operations on the feature information to obtain pooled feature information.

[0364] The distribution fitting module is used to perform distribution fitting on the pooled feature information to obtain the distribution information of the question samples in the second question sample subset.

[0365] In one embodiment, the distillation unit 305 may include:

[0366] The second identification subunit is used to identify the question samples in the second question sample subset using the student model, and obtain the distribution information of the question samples in the second question sample subset;

[0367] The loss calculation subunit is used to calculate the model loss information between the reference distribution information obtained by the enhanced teacher model based on the question sample recognition and the distribution information obtained by the student model based on the question sample recognition.

[0368] The adjustment subunit is used to adjust the student model using the model loss information to obtain the target student model.

[0369] In one embodiment, the loss calculation subunit may include:

[0370] An adjustment module is used to adjust the smoothness of the reference distribution information and the distribution information to obtain adjusted reference distribution information and adjusted distribution information.

[0371] The logarithmic operation module is used to perform logarithmic operations on the adjusted distribution information to obtain logarithmically operated distribution information, and to perform logarithmic operations on the adjusted reference distribution information to obtain logarithmically operated reference distribution information;

[0372] The multiplication module is used to multiply the adjusted distribution information and the logarithmic distribution information to obtain multiplied distribution information, and to multiply the adjusted reference distribution information and the logarithmic reference distribution information to obtain multiplied reference distribution information.

[0373] The addition module is used to add the multiplied reference distribution information and the multiplied distribution information to obtain the model loss information.

[0374] In one embodiment, the adjustment subunit may include:

[0375] The acquisition module is used to acquire the tag information of the question samples;

[0376] The loss calculation module is used to calculate the label loss information between the distribution information of the question samples and the label information;

[0377] The loss fusion module is used to fuse the label loss information and the model loss information to obtain fused loss information;

[0378] The parameter adjustment module is used to adjust the model parameters of the student model based on the fused loss information to obtain the target student model.

[0379] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0380] The aforementioned model distillation device can achieve both real-time performance and quality of question recommendations when using artificial intelligence models for question recommendation.

[0381] This application also provides a computer device, which may include a terminal or a server. For example, the computer device may serve as a model distillation terminal, such as a mobile phone, tablet computer, etc.; or it may serve as a server, such as a model distillation server. Figure 8 As shown, it illustrates the structural diagram of the terminal involved in the embodiments of this application, specifically:

[0382] The computer device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 8 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0383] The processor 401 is the control center of the computer device, connecting various parts of the computer device through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user page, and application programs, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0384] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0385] The computer device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0386] The computer device may also include an input unit 404, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0387] Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the computer device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0388] Obtain a question sample set and a teacher model for question recommendation, wherein the question sample set includes a first question sample subset and a second question sample subset;

[0389] Data augmentation is performed on the question samples in the first question sample subset to obtain augmented question samples;

[0390] The teacher model is enhanced using the enhanced question samples to obtain the enhanced teacher model.

[0391] The enhanced teacher model is used to identify the question samples in the second question sample subset to obtain the reference distribution information of the question samples in the second question sample subset;

[0392] A student model is obtained, and the reference distribution information identified by the enhanced teacher model is used to learn the target student model.

[0393] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0394] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations of the above embodiments.

[0395] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by a computer program, or by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0396] Therefore, embodiments of this application also provide a storage medium storing a computer program that can be loaded by a processor to execute the steps in any of the model distillation methods provided in embodiments of this application. For example, the computer program can execute the following steps:

[0397] Obtain a question sample set and a teacher model for question recommendation, wherein the question sample set includes a first question sample subset and a second question sample subset;

[0398] Data augmentation is performed on the question samples in the first question sample subset to obtain augmented question samples;

[0399] The teacher model is enhanced using the enhanced question samples to obtain the enhanced teacher model.

[0400] The enhanced teacher model is used to identify the question samples in the second question sample subset to obtain the reference distribution information of the question samples in the second question sample subset;

[0401] A student model is obtained, and the reference distribution information identified by the enhanced teacher model is used to learn the target student model.

[0402] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0403] Since the computer program stored in the storage medium can execute the steps in any of the model distillation methods provided in the embodiments of this application, the beneficial effects that any of the model distillation methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0404] The above provides a detailed description of a model distillation method, apparatus, computer equipment, and storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A model distillation method, characterized by, The method comprises the following steps: obtaining a question sample set and a teacher model for question recommendation, wherein the question sample set comprises a first question sample subset and a second question sample subset; performing data enhancement on question samples in the first question sample subset based on positive transmission according to a preset association relationship, to obtain enhanced question samples, wherein the preset association relationship is knowledge point similarity, knowledge point range and knowledge point combination of the question samples, the knowledge point similarity is that two knowledge points are the same or similar, the knowledge point range is that two knowledge points have the same knowledge range, the knowledge point combination is that two knowledge points are combined to obtain a question, the data enhancement is to associate question samples in the first question sample subset that meet the preset association relationship, and the positive transmission is transmission under the premise of constraint on a single knowledge point of a question sample; performing model enhancement on the teacher model by using the enhanced question samples, to obtain an enhanced teacher model; performing identification on question samples in the second question sample subset by using the enhanced teacher model, to obtain reference distribution information of the question samples in the second question sample subset; obtaining a student model, learning the reference distribution information obtained by the enhanced teacher model by using the student model, to obtain a target student model, wherein the student model is a pre-trained model, and the student model realizes a language task and a next sentence prediction task of a full-word coverage mask model.

2. The method of claim 1, wherein, The data enhancement on the question samples in the first question sample subset to obtain the enhanced question samples comprises the following steps: performing identification on a current question sample in the first question sample subset, to obtain associated question samples in the first question sample subset that have a preset association relationship with the current question sample; identifying question samples in the first question sample subset that have a preset association relationship with the associated question samples, wherein the question samples comprise other question samples in the first question sample subset except the current question sample; establishing an association relationship between the current question sample and the question samples based on a preset association relationship between the current question sample and the associated question samples and a preset association relationship between the associated question samples and the question samples, to obtain the enhanced question samples.

3. The method of claim 2, wherein, The establishment of the association relationship between the current question sample and the question samples based on the preset association relationship between the current question sample and the associated question samples and the preset association relationship between the associated question samples and the question samples to obtain the enhanced question samples comprises the following steps: identifying knowledge point information of the current question sample and knowledge point information of the question samples; comparing the knowledge point information of the current question sample and the knowledge point information of the question samples, to obtain a comparison result; establishing the association relationship between the current question sample and the question samples based on the comparison result, to obtain the enhanced question samples.

4. The method of claim 1, wherein, The teacher model is a plurality of teacher models; the model enhancement of the teacher model by using the enhanced question sample comprises: training each teacher model by using the enhanced question sample to obtain a corresponding trained teacher model of each teacher model; generating a performance measurement index of the trained teacher model according to a training result of the trained teacher model, wherein the performance measurement index represents a quality of question recommendation of the trained teacher model; combining the trained teacher models into the enhanced teacher model according to the performance measurement index of each trained teacher model.

5. The method of claim 4, wherein, The training of each teacher model by using the enhanced question sample to obtain a corresponding trained teacher model of each teacher model comprises: initializing a weight coefficient of each enhanced question sample, wherein the weight coefficient represents a value of the enhanced question sample for training; training the teacher model by using the enhanced question sample to obtain an initial trained teacher model and an initial training result; updating the weight coefficient of the enhanced question sample based on the initial training result to obtain an updated weight coefficient of the enhanced question sample; selecting a target usage question sample from the enhanced question sample according to the updated weight coefficient of the enhanced question sample; training the initial trained teacher model by using the target usage question sample to obtain the trained teacher model.

6. The method of claim 4, wherein, The combination of the trained teacher models into the enhanced teacher model according to the performance measurement index of each trained teacher model comprises: generating a combination weight coefficient of each trained teacher model based on the performance measurement index; fusing each trained teacher model according to the combination weight coefficient to obtain a fused teacher model; constructing the fused teacher model into the enhanced teacher model according to a preset constructor.

7. The method of claim 1, wherein, The identification of the question sample in the second question sample subset by using the enhanced teacher model to obtain reference distribution information of the question sample in the second question sample subset comprises: performing information extraction on the question sample to obtain question information of the question sample; performing normalization processing on the question information to obtain normalized question information; identifying the normalized question information by using the enhanced teacher model to obtain the reference distribution information of the question sample in the second question sample subset.

8. The method of claim 7, wherein, The normalization processing of the question information to obtain normalized question information comprises: identifying at least one type of to-be-normalized content in the question information; determining a normalization mode of the current type of to-be-normalized content; performing content conversion on the to-be-normalized content by using the normalization mode to obtain the normalized question information.

9. The method of claim 7, wherein, The identification of the normalized question information by using the enhanced teacher model to obtain the reference distribution information of the question sample in the second question sample subset comprises: performing feature extraction on the normalized question information to obtain feature information of the normalized question information; perform a pooling operation on the feature information to obtain pooled feature information; perform distribution fitting on the pooled feature information to obtain distribution information of the question samples in the second question sample subset.

10. The method of claim 1, wherein, The student model is obtained, and the student model is used to learn the reference distribution information identified by the enhanced teacher model to obtain a target student model, including: using the student model to identify the question samples in the second question sample subset to obtain the distribution information of the question samples in the second question sample subset; calculating the model loss information between the reference distribution information identified by the enhanced teacher model based on the question samples and the distribution information identified by the student model based on the question samples; adjust the student model using the model loss information to obtain the target student model.

11. The method of claim 10, wherein, The calculation of the model loss information between the reference distribution information identified by the enhanced teacher model based on the question samples and the distribution information identified by the student model based on the question samples includes: adjust the smoothness of the reference distribution information and the distribution information to obtain adjusted reference distribution information and adjusted distribution information; performing logarithmic operation on the adjusted distribution information to obtain logarithmic operation distribution information, and performing logarithmic operation on the adjusted reference distribution information to obtain logarithmic operation reference distribution information; multiply the adjusted distribution information and the logarithmic operation distribution information to obtain multiplied distribution information, and multiply the adjusted reference distribution information and the logarithmic operation reference distribution information to obtain multiplied reference distribution information; add the multiplied reference distribution information and the multiplied distribution information to obtain the model loss information.

12. The method of claim 10, wherein, The use of the model loss information to adjust the student model to obtain the target student model includes: obtain the label information of the question sample; calculate the label loss information between the distribution information of the question sample and the label information; fuse the label loss information and the model loss information to obtain fused loss information; adjust the model parameters of the student model based on the fused loss information to obtain the target student model.

13. A model distillation apparatus, characterized by, including: an acquisition unit configured to acquire a question sample set and a teacher model for question recommendation, wherein the question sample set includes a first question sample subset and a second question sample subset; The data enhancement unit is configured to perform data enhancement on the question samples in the first question sample subset based on positive transmission according to a preset correlation relationship, to obtain enhanced question samples, wherein the preset correlation relationship is a knowledge point similarity, a knowledge point range, and a knowledge point combination of the question samples, the knowledge point similarity is that two knowledge points are the same or similar, the knowledge point range is that two knowledge points have the same knowledge range, the knowledge point combination is that two knowledge points are combined to obtain a question, the data enhancement is to associate the question samples in the first question sample subset that meet the preset correlation relationship, and the positive transmission is performed under the premise of constraint on a single knowledge point of the question sample. The model enhancement unit is configured to perform model enhancement on the teacher model by using the enhanced question samples, to obtain an enhanced teacher model. The recognition unit is configured to perform recognition on the question samples in the second question sample subset by using the enhanced teacher model, to obtain reference distribution information of the question samples in the second question sample subset. The distillation unit is configured to obtain a student model, to learn the reference distribution information obtained by the enhanced teacher model by using the student model, and to obtain a target student model, wherein the student model is a pre-trained model, and the student model implements a language task and a next sentence prediction task of a full word coverage mask model.

14. A computer device, comprising: The computer readable storage medium stores a plurality of instructions, and the instructions are adapted to be loaded by the processor to perform the steps in the model distillation method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer program or instructions are executed by the processor to implement the steps in the model distillation method according to any one of claims 1 to 12.

16. A computer program product comprising computer programs or instructions, characterized in that, ​

Images