Model distillation method, apparatus, medium, apparatus and computer program product

Abstract

The invention relates to a model distillation method and device, a medium, equipment and a computer program product. The method comprises the following steps: acquiring a word list of a teacher model and a word list of a student model for performing model distillation; determining a shared word list corresponding to the teacher model and the student model based on the word list of the teacher model and the word list of the student model; on the basis of the shared word list, probability distributions output by the teacher model and the student model are converted, conversion probability distributions corresponding to the teacher model and the student model are obtained, and the conversion probability distributions are probability distributions corresponding to words in the shared word list; determining the target loss of the student model based on the conversion probability distribution corresponding to the teacher model and the conversion probability distribution corresponding to the student model; and training the student model based on the target loss.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular, to a model distillation method, apparatus, medium, device, and computer program product. Background Art

[0002] As the performance of large language models (LLMs) increases, the model size and parameters are also increasing, but the increasing model size will bring huge challenges to reasoning and application. In actual business scenarios, due to high traffic and high latency requirements, only relatively small-scale models can be used under the existing hardware resources.

[0003] Model distillation is a model compression technology that aims to transfer the knowledge of a complex, large model (usually called a teacher model) to another smaller, simpler model (usually called a student model). How to implement model distillation to obtain a lightweight model is an urgent problem to be solved in this field. Summary of the invention

[0004] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] In a first aspect, the present disclosure provides a model distillation method, the method comprising: Get the vocabulary of the teacher model and the vocabulary of the student model for model distillation; Based on the vocabulary of the teacher model and the vocabulary of the student model, determining a shared vocabulary corresponding to the teacher model and the student model, wherein the words in the shared vocabulary are words existing in both the vocabulary of the teacher model and the vocabulary of the student model; Based on the shared vocabulary, respectively convert the probability distributions output by the teacher model and the student model to obtain conversion probability distributions corresponding to the teacher model and the student model, respectively, wherein the conversion probability distributions are probability distributions corresponding to words in the shared vocabulary; Determining a target loss of the student model based on a transition probability distribution corresponding to the teacher model and a transition probability distribution corresponding to the student model; The student model is trained based on the target loss.

[0006] In a second aspect, the present disclosure provides a model distillation device, the device comprising: An acquisition module, used to acquire a vocabulary of a teacher model and a vocabulary of a student model for model distillation; A first determination module is used to determine a shared vocabulary corresponding to the teacher model and the student model based on the vocabulary of the teacher model and the vocabulary of the student model, wherein the words in the shared vocabulary are words existing in both the vocabulary of the teacher model and the vocabulary of the student model; A processing module, configured to convert the probability distributions output by the teacher model and the student model respectively based on the shared vocabulary, and obtain conversion probability distributions corresponding to the teacher model and the student model respectively, wherein the conversion probability distributions are probability distributions corresponding to words in the shared vocabulary; A second determination module is used to determine the target loss of the student model based on the transition probability distribution corresponding to the teacher model and the transition probability distribution corresponding to the student model; A training module is used to train the student model based on the target loss.

[0007] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which implements the steps of the method described in the first aspect when the computer program is executed by a processing device.

[0008] In a fourth aspect, the present disclosure provides an electronic device, including: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of the method described in the first aspect.

[0009] In a fifth aspect, the present disclosure provides a computer program product, including a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0010] In the above technical solution, for models using different vocabularies, the probability distribution of the model output can be converted through the shared vocabularies corresponding to different models, so that the knowledge of any trained model can be distilled to the student model, thereby increasing the scope of use of the model distillation method and improving the accuracy of the student model. The trained student model can be inferred at a smaller scale and maintain similar performance to the teacher model, which facilitates the deployment and application of the model and reduces the performance requirements for hardware devices. In addition, the model distillation method in the present disclosure can be trained with the prediction results of the teacher model as the goal, so as to train on unlabeled data, reduce the dependence on labeled data, and reduce the workload of data labeling.

[0011] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 is a flow chart of a model distillation method provided according to one embodiment of the present disclosure.

[0013] Figure 2 is a block diagram of a model distillation apparatus provided according to one embodiment of the present disclosure.

[0014] Figure 3 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0015] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0016] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0017] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0018] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0019] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0020] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0021] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0022] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0023] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0025] At the same time, it is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0026] Figure 1 As shown, it is a flow chart of a model distillation method provided according to an embodiment of the present disclosure, as shown Figure 1 As shown, the method may include: In step 11, the vocabulary of the teacher model and the vocabulary of the student model for model distillation are obtained.

[0027] In the process of model distillation, the original large model is recorded as the teacher model, the new small model is recorded as the student model, the label in the original training set can be recorded as the hard label, and the probability distribution output by the teacher model is recorded as the soft label. For example, the teacher model can be a large language model or a natural language processing (NLP) model.

[0028] In model distillation, the probability distribution of the output of the teacher model can be used as the target, that is, the final prediction result of the teacher model is used as the target, and the student model is trained to learn the knowledge of the teacher model to obtain prediction results that are approximate to the teacher model. By training with soft targets, the student model can learn more details and knowledge of the teacher model, such as the relative weight of the probability distribution and the relationship between categories, to improve the generalization ability of the student model, so that the student model can be inferred at a smaller scale and maintain similar performance to the teacher model.

[0029] The applicant has found that in different LLM scenarios, different open source models often have different vocabulary sizes and different vocabulary units. Therefore, the probability distributions output by models using different vocabularies cannot be directly compared, making it difficult to determine the relationship between their probability distributions. And the vocabularies of different models are different, but there are a large number of words in common between different vocabularies. Based on this, in the disclosed embodiment, the probability distribution is converted based on the vocabulary by obtaining the vocabularies of different models to achieve comparability of probability distribution. As an example, in this step, the vocabulary of the model can be directly obtained based on the configuration information of the model. Usually, the vocabulary of the teacher model is larger than that of the student model. For example, the vocabulary of the teacher model T1 contains 100,000 words, and the vocabulary of the student model S contains 60,000 words.

[0030] In step 12, based on the vocabulary of the teacher model and the vocabulary of the student model, a shared vocabulary corresponding to the teacher model and the student model is determined, wherein the words in the shared vocabulary are words that exist in both the vocabulary of the teacher model and the vocabulary of the student model.

[0031] The shared vocabulary can be obtained by taking the intersection of different vocabulary. If there is only one teacher model, the words in the vocabulary of the teacher model are taken as set J1, and the words in the vocabulary of the student model are taken as set J0, then the shared vocabulary Vc can be expressed as Vc=J1∩J0. For example, the shared vocabulary of the teacher model T1 and the student model S contains 48,000 words.

[0032] If there are multiple teacher models, such as the words in the vocabulary of teacher model T1 as set J1, the words in the vocabulary of teacher model T2 as set J2, the words in the vocabulary of teacher model T3 as set J3, and the words in the vocabulary of the student model as set J0, then the shared vocabulary Vc can be expressed as Vc=J1∩J2∩J3∩J0.

[0033] In step 13, based on the shared vocabulary, the probability distributions output by the teacher model and the student model are converted respectively to obtain conversion probability distributions corresponding to the teacher model and the student model respectively, wherein the conversion probability distributions are probability distributions corresponding to words in the shared vocabulary.

[0034] Among them, the training of LLM is essentially to do a classification task in each token dimension, where the number of categories is the size of the vocabulary. As in the example above, the teacher model outputs probability distributions for 100,000 words as categories, while the student model outputs probability distributions for 60,000 words as categories, then the categories in the probability distributions output by the teacher model and the student model are different. Therefore, in the disclosed embodiment, the probability distribution of the model output can be transformed through a shared vocabulary to convert it into the probability space corresponding to the shared vocabulary, that is, the probability distribution output by the teacher model is converted into the probability distribution of the words in the shared vocabulary as categories, and the probability distribution output by the student model is converted into the probability distribution of the words in the shared vocabulary as categories. Thus, the conversion probability distribution corresponding to the teacher model and the student model is the same type of probability distribution (that is, the number of classification categories is the same, and the meaning corresponding to each category id is the same), so that the probability distributions output by models using different vocabularies are comparable.

[0035] In step 14, based on the transition probability distribution corresponding to the teacher model and the transition probability distribution corresponding to the student model, the target loss of the student model is determined.

[0036] Among them, the target loss can be determined by calculating the distance between different transition probability distributions, such as the target loss can be calculated using KL divergence (Kullback-Leibler divergence or relative entropy) as the loss function. KL divergence is an asymmetric measure of the difference between two probability distributions, and is used to calculate the difference between one probability distribution (such as the transition probability distribution corresponding to the teacher model) and another probability distribution (such as the transition probability distribution corresponding to the student model). The smaller the KL divergence, the closer the two distributions are, and the student model can be optimized by minimizing the distance between the transition probability distribution corresponding to the teacher model and the transition probability distribution corresponding to the student model. Among them, the calculation method of KL divergence can be calculated based on the general calculation method in the field, which is not limited here.

[0037] In step 15, the student model is trained based on the target loss.

[0038] In this step, the student model can be trained with the goal of minimizing the target loss. Continuing with the previous example, the KL divergence can be minimized and the parameters of the student model can be optimized based on the target loss and by gradient descent, so as to train the student model and enable it to distill knowledge from the teacher model.

[0039] In the above technical solution, for models using different vocabularies, the probability distribution of the model output can be converted through the shared vocabularies corresponding to different models, so that the knowledge of any trained model can be distilled to the student model, thereby increasing the scope of use of the model distillation method and improving the accuracy of the student model. The trained student model can be inferred at a smaller scale and maintain similar performance to the teacher model, which facilitates the deployment and application of the model and reduces the performance requirements for hardware devices. In addition, the model distillation method in the present disclosure can be trained with the prediction results of the teacher model as the goal, so as to train on unlabeled data, reduce the dependence on labeled data, and reduce the workload of data labeling.

[0040] In a possible embodiment, based on the shared vocabulary, converting the probability distributions output by the teacher model and the student model respectively to obtain the conversion probability distributions corresponding to the teacher model and the student model respectively may include: For each model in the teacher model and the student model, a transformation matrix corresponding to the model is determined based on the vocabulary of the model and the shared vocabulary.

[0041] Among them, the number of words in the vocabulary of the teacher model is M, and the number of words in the shared vocabulary is N. Then, a conversion matrix corresponding to the teacher model can be generated based on the vocabulary of the teacher model and the shared vocabulary, and the conversion matrix can be an M×N matrix. The number of words in the vocabulary of the student model is L, and the number of words in the shared vocabulary is N. Then, a conversion matrix corresponding to the student model can be generated based on the vocabulary and the shared vocabulary, and the conversion matrix can be an L×N matrix.

[0042] As an example, determining the conversion matrix corresponding to the model based on the vocabulary of the model and the shared vocabulary may include: For each word in the model, the similarity between the vector representation of the word and the vector representation of each word in the shared vocabulary is used as the conversion vector representation corresponding to the word, wherein the order of the words in the shared vocabulary in the conversion vector representation corresponding to each word is the same.

[0043] Among them, for the word R[i] in the teacher model, its corresponding conversion vector is expressed as follows:

[0044] in, The vector representation of the i-th word in the vocabulary used to represent the model, The vector representation used to represent the first word in the shared vocabulary, The vector representation used to represent the Nth word in the shared vocabulary, and so on, will not be repeated here. That is to say and The cosine similarity of can be determined based on the calculation method of cosine similarity in the art.

[0045] Among them, the order of words in the shared vocabulary is fixed, which can be pre-set based on the actual application scenario, that is, the order 1-N of N words in the shared vocabulary is fixed, so that the words in the shared vocabulary corresponding to each column in the conversion vector representation are the same.

[0046] The vector representation of each word can be determined by a pre-trained feature extraction model, such as the output of the embedding layer in the feature extraction model can be used as the vector representation of the word. The vector representation of the word can be determined by a common word vector generation method in the art, which will not be described in detail here.

[0047] Therefore, through the above technical scheme, for each word in the vocabulary of the model, the conversion vector representation of the word can be obtained by the similarity between the word and each word in the shared vocabulary, so that the words in the vocabulary of the model can be represented by the word similarity in the shared vocabulary, thereby realizing the association between the probability space corresponding to the vocabulary of the model and the probability space corresponding to the shared vocabulary.

[0048] Afterwards, the conversion matrix is ​​generated according to the conversion vector representation corresponding to each word.

[0049] As an example, the transformation vector corresponding to each word can be represented as a row vector in the transformation matrix to obtain the transformation matrix. Then each row in the transformation matrix of the model corresponds to a word in the vocabulary of the model, and each column in the transformation matrix of the model corresponds to a word in the vocabulary of the shared model.

[0050] In actual application scenarios, the similarity between the words in the model's vocabulary and all the words in the shared vocabulary may be relatively small. Therefore, R[i] may contain some vectors whose rows are approximately all zeros. Such vectors are not conducive to model learning and convergence. Based on this, the present disclosure provides the following embodiments.

[0051] As another example, generating the conversion matrix according to the conversion vector representation corresponding to each word may include: The conversion vector representation corresponding to each word is normalized to obtain the normalized vector representation corresponding to the word.

[0052] Among them, for the conversion vector corresponding to each word, it can be normalized by the softmax function, such as , That is, it is used to represent the normalized vector representation. Among them, the softmax function is a common function in this field and is not limited here.

[0053] Afterwards, the normalized vector representation corresponding to each word is used as a row vector in the transformation matrix to obtain the transformation matrix.

[0054] Among them, the normalized vector representation corresponding to each word can be used as a row vector in the transformation matrix to obtain the transformation matrix. Then each row in the transformation matrix of the model corresponds to a word in the vocabulary of the model, and each column in the transformation matrix of the model corresponds to a word in the vocabulary of the shared model.

[0055] Therefore, through this technical solution, the conversion vector representation of the words in the vocabulary of the model can be further normalized to improve the effectiveness of the features in the vector representation determined based on the shared vocabulary, avoid the amount of information in the conversion vector representation determined when the similarity between the words in the vocabulary of the model and all the words in the shared vocabulary is relatively small, facilitate the subsequent model learning and convergence process, and improve the efficiency and accuracy of model distillation.

[0056] Furthermore, after the conversion matrix is ​​determined, the probability distribution output by the model can be converted based on the conversion matrix corresponding to the model to obtain the conversion probability distribution corresponding to the model.

[0057] Among them, the number of words in the vocabulary of the teacher model is M, then the probability distribution output by the teacher model can be expressed as a 1×M row vector, corresponding to the probability of each of the M words. The transformation matrix of the teacher model is an M×N matrix. As an example, the product of the probability distribution output by the model and the transformation matrix corresponding to the model can be used as the transformation probability distribution. Therefore, by multiplying the probability distribution of the teacher model and the transformation matrix, a 1×N matrix can be obtained, that is, the transformation probability distribution of the teacher model.

[0058] Similarly, if the number of words in the vocabulary of the student model is L, the probability distribution output by the student model can be expressed as a 1×L row vector, corresponding to the probability of each word in the L words. The transformation matrix of the student model is an L×N matrix. As an example, by multiplying the probability distribution of the student model and the transformation matrix, a 1×N matrix can be obtained, i.e., the transformation probability distribution of the student model.

[0059] Therefore, through the above technical solution, the probability distributions output by the student model and the teacher model using different vocabularies can be converted into the same space for representation, so that the conversion probability distribution of the student model and the conversion probability distribution of the teacher model are comparable, so that in the subsequent training process of the student model, the probability distribution output by the student model can be made as close as possible to the probability distribution output by the teacher model.

[0060] In a possible embodiment, there are multiple teacher models. In this embodiment, the transformation matrix corresponding to each teacher model can be determined based on the method described above, and then the probability distribution output by each teacher model can be transformed to obtain the transformation probability distributions corresponding to the multiple teacher models.

[0061] Accordingly, determining the target loss of the student model based on the transition probability distribution corresponding to the teacher model and the transition probability distribution corresponding to the student model includes: The conversion probability distributions corresponding to the multiple teacher models are fused to obtain the target fusion probability distribution.

[0062] Among them, it can be seen from the above that the conversion probability distributions corresponding to different models are comparable distributions in the same space. As an example, the mean of the conversion probability distributions of multiple teacher models can be used as the target fusion probability. As another example, the conversion probability distributions of multiple teacher models can be weighted averaged based on the prediction accuracy of each teacher model, and the weight of the teacher model is determined based on its corresponding prediction accuracy. For example: r=W T1 × T1 + W T2 × T2 +W T3 × T3 , W T1 + W T2 +W T3 =1 Among them, r is used to represent the target fusion probability distribution, W T1 Used to represent the weight of the teacher model T1, r T1 Used to represent the transition probability distribution of the teacher model T1, W T2 Used to represent the weight of the teacher model T2, r T2 Used to represent the transition probability distribution of the teacher model T2, W T3 Used to represent the weight of the teacher model T3, r T3 Used to represent the transition probability distribution of the teacher model T3.

[0063] Afterwards, the target loss of the student model is determined based on the target fusion probability distribution and the conversion probability distribution corresponding to the student model.

[0064] The target loss can be obtained by performing loss calculation based on the target fusion probability distribution and the conversion probability distribution corresponding to the student model and using the KL divergence as a loss function.

[0065] Therefore, through the above technical solution, the capabilities of multiple teacher models can be combined to achieve the fusion and complementarity of the capabilities of multiple teacher models, so as to better provide learning goals for the student model. By fusing the output predictions of multiple teacher models and distilling them into a student model, the process of knowledge distillation of a student model by multiple teacher models is realized, the learning ability of the student model is improved, and then the usage scenarios of the obtained student model are improved.

[0066] Based on the same inventive concept, the present disclosure also provides a model distillation device, such as Figure 2 As shown, the device 10 comprises: An acquisition module 100 is used to acquire a vocabulary of a teacher model and a vocabulary of a student model for model distillation; A first determination module 200 is used to determine a shared vocabulary corresponding to the teacher model and the student model based on the vocabulary of the teacher model and the vocabulary of the student model, wherein the words in the shared vocabulary are words existing in both the vocabulary of the teacher model and the vocabulary of the student model; A processing module 300 is used to convert the probability distributions output by the teacher model and the student model respectively based on the shared vocabulary to obtain conversion probability distributions corresponding to the teacher model and the student model respectively, wherein the conversion probability distributions are probability distributions corresponding to words in the shared vocabulary; A second determination module 400 is used to determine the target loss of the student model based on the transition probability distribution corresponding to the teacher model and the transition probability distribution corresponding to the student model; The training module 500 is used to train the student model based on the target loss.

[0067] Optionally, the processing module includes: A first determination submodule is used to determine, for each model in the teacher model and the student model, a transformation matrix corresponding to the model based on the vocabulary of the model and the shared vocabulary; The first processing submodule is used to transform the probability distribution output by the model based on the transformation matrix corresponding to the model to obtain the transformation probability distribution corresponding to the model.

[0068] Optionally, the first determining submodule includes: A second determination submodule is used for, for each word in the model, using the similarity between the vector representation of the word and the vector representation of each word in the shared vocabulary as the conversion vector representation corresponding to the word, wherein the order of the words in the shared vocabulary in the conversion vector representation corresponding to each word is the same; The generating submodule is used to generate the conversion matrix according to the conversion vector representation corresponding to each word.

[0069] Optionally, the generating submodule includes: The second processing submodule is used to normalize the conversion vector representation corresponding to each word to obtain the normalized vector representation corresponding to the word; The third processing submodule is used to represent the normalized vector corresponding to each word as a row vector in the transformation matrix to obtain the transformation matrix.

[0070] Optionally, the first processing submodule is further used for: The product of the probability distribution output by the model and the transformation matrix corresponding to the model is used as the transformation probability distribution.

[0071] Optionally, there are multiple teacher models; The second determining module comprises: A fusion submodule, used for fusing the conversion probability distributions corresponding to the multiple teacher models to obtain a target fusion probability distribution; The third determination submodule is used to determine the target loss of the student model based on the target fusion probability distribution and the conversion probability distribution corresponding to the student model.

[0072] Reference below Figure 3 , which shows a schematic diagram of the structure of an electronic device (such as a terminal device or a server) 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0073] like Figure 3As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0074] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0075] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0076] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0077] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0078] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0079] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a vocabulary of a teacher model and a vocabulary of a student model for model distillation; determines a shared vocabulary corresponding to the teacher model and the student model based on the vocabulary of the teacher model and the vocabulary of the student model, wherein the words in the shared vocabulary are words existing in both the vocabulary of the teacher model and the vocabulary of the student model; based on the shared vocabulary, respectively converts the probability distributions output by the teacher model and the student model to obtain conversion probability distributions corresponding to the teacher model and the student model, respectively, wherein the conversion probability distributions are probability distributions corresponding to the words in the shared vocabulary; determines a target loss for the student model based on the conversion probability distributions corresponding to the teacher model and the conversion probability distributions corresponding to the student model; and trains the student model based on the target loss.

[0080] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0081] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0082] The modules described in the embodiments of the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, an acquisition module may also be described as a "module for acquiring a vocabulary of a teacher model and a vocabulary of a student model for model distillation".

[0083] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0084] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0085] According to one or more embodiments of the present disclosure, Example 1 provides a model distillation method, wherein the method includes: Get the vocabulary of the teacher model and the vocabulary of the student model for model distillation; Based on the vocabulary of the teacher model and the vocabulary of the student model, determining a shared vocabulary corresponding to the teacher model and the student model, wherein the words in the shared vocabulary are words existing in both the vocabulary of the teacher model and the vocabulary of the student model; Based on the shared vocabulary, respectively convert the probability distributions output by the teacher model and the student model to obtain conversion probability distributions corresponding to the teacher model and the student model, respectively, wherein the conversion probability distributions are probability distributions corresponding to words in the shared vocabulary; Determining a target loss of the student model based on a transition probability distribution corresponding to the teacher model and a transition probability distribution corresponding to the student model; The student model is trained based on the target loss.

[0086] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein, based on the shared vocabulary, respectively converting the probability distributions output by the teacher model and the student model to obtain the conversion probability distributions corresponding to the teacher model and the student model, respectively, includes: For each model in the teacher model and the student model, based on the vocabulary of the model and the shared vocabulary, determine a transformation matrix corresponding to the model; The probability distribution output by the model is transformed based on the transformation matrix corresponding to the model to obtain the transformation probability distribution corresponding to the model.

[0087] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 2, wherein determining the transformation matrix corresponding to the model based on the vocabulary of the model and the shared vocabulary includes: For each word in the model, the similarity between the vector representation of the word and the vector representation of each word in the shared vocabulary is used as the conversion vector representation corresponding to the word, wherein the order of the words in the shared vocabulary in the conversion vector representation corresponding to each word is the same; The conversion matrix is ​​generated according to the conversion vector representation corresponding to each word.

[0088] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 3, wherein generating the conversion matrix according to the conversion vector representation corresponding to each word includes: Normalize the conversion vector representation corresponding to each word to obtain the normalized vector representation corresponding to the word; The normalized vector representation corresponding to each word is used as a row vector in the transformation matrix to obtain the transformation matrix.

[0089] According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 2, wherein the converting the probability distribution output by the model based on the conversion matrix corresponding to the model to obtain the conversion probability distribution corresponding to the model includes: The product of the probability distribution output by the model and the transformation matrix corresponding to the model is used as the transformation probability distribution.

[0090] According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 1, wherein the teacher model is multiple; The determining the target loss of the student model based on the transition probability distribution corresponding to the teacher model and the transition probability distribution corresponding to the student model includes: Fusing the conversion probability distributions corresponding to the multiple teacher models to obtain a target fusion probability distribution; Based on the target fusion probability distribution and the transition probability distribution corresponding to the student model, a target loss of the student model is determined.

[0091] According to one or more embodiments of the present disclosure, Example 7 provides a model distillation device, the device comprising: An acquisition module, used to acquire a vocabulary of a teacher model and a vocabulary of a student model for model distillation; A first determination module is used to determine a shared vocabulary corresponding to the teacher model and the student model based on the vocabulary of the teacher model and the vocabulary of the student model, wherein the words in the shared vocabulary are words existing in both the vocabulary of the teacher model and the vocabulary of the student model; A processing module, configured to convert the probability distributions output by the teacher model and the student model respectively based on the shared vocabulary, and obtain conversion probability distributions corresponding to the teacher model and the student model respectively, wherein the conversion probability distributions are probability distributions corresponding to words in the shared vocabulary; A second determination module is used to determine the target loss of the student model based on the transition probability distribution corresponding to the teacher model and the transition probability distribution corresponding to the student model; A training module is used to train the student model based on the target loss.

[0092] According to one or more embodiments of the present disclosure, Example 8 provides a computer-readable medium having a computer program stored thereon, which implements the steps of any of the methods described in Examples 1-6 when executed by a processing device.

[0093] According to one or more embodiments of the present disclosure, Example 9 provides an electronic device, including: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of the method described in any one of Examples 1-6.

[0094] According to one or more embodiments of the present disclosure, Example 10 provides a computer program product, including a computer program, which implements the steps of any one of the methods of Examples 1-6 when executed by a processor.

[0095] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0096] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0097] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A model distillation method, characterized in that: The method comprises: Get the vocabulary of the teacher model and the vocabulary of the student model for model distillation; Based on the vocabulary of the teacher model and the vocabulary of the student model, determining a shared vocabulary corresponding to the teacher model and the student model, wherein the words in the shared vocabulary are words existing in both the vocabulary of the teacher model and the vocabulary of the student model; Based on the shared vocabulary, respectively convert the probability distributions output by the teacher model and the student model to obtain conversion probability distributions corresponding to the teacher model and the student model, respectively, wherein the conversion probability distributions are probability distributions corresponding to words in the shared vocabulary; Determining a target loss of the student model based on a transition probability distribution corresponding to the teacher model and a transition probability distribution corresponding to the student model; The student model is trained based on the target loss.

2. The method according to claim 1, characterized in that: The converting the probability distributions output by the teacher model and the student model based on the shared vocabulary to obtain conversion probability distributions corresponding to the teacher model and the student model respectively includes: For each model in the teacher model and the student model, based on the vocabulary of the model and the shared vocabulary, determine a transformation matrix corresponding to the model; The probability distribution output by the model is transformed based on the transformation matrix corresponding to the model to obtain the transformation probability distribution corresponding to the model.

3. The method according to claim 2, characterized in that The step of determining a conversion matrix corresponding to the model based on the vocabulary of the model and the shared vocabulary includes: For each word in the model, the similarity between the vector representation of the word and the vector representation of each word in the shared vocabulary is used as the conversion vector representation corresponding to the word, wherein the order of the words in the shared vocabulary in the conversion vector representation corresponding to each word is the same; The conversion matrix is ​​generated according to the conversion vector representation corresponding to each word.

4. The method according to claim 3, characterized in that: The step of generating the conversion matrix according to the conversion vector representation corresponding to each word includes: Normalize the conversion vector representation corresponding to each word to obtain the normalized vector representation corresponding to the word; The normalized vector representation corresponding to each word is used as a row vector in the transformation matrix to obtain the transformation matrix.

5. The method according to claim 2, characterized in that: The converting the probability distribution output by the model based on the conversion matrix corresponding to the model to obtain the conversion probability distribution corresponding to the model includes: The product of the probability distribution output by the model and the transformation matrix corresponding to the model is used as the transformation probability distribution.

6. The method according to claim 1, characterized in that There are multiple teacher models; The determining the target loss of the student model based on the transition probability distribution corresponding to the teacher model and the transition probability distribution corresponding to the student model includes: Fusing the conversion probability distributions corresponding to the multiple teacher models to obtain a target fusion probability distribution; Based on the target fusion probability distribution and the transition probability distribution corresponding to the student model, a target loss of the student model is determined.

7. A model distillation apparatus, characterized in that: The device comprises: An acquisition module, used to acquire a vocabulary of a teacher model and a vocabulary of a student model for model distillation; A first determination module is used to determine a shared vocabulary corresponding to the teacher model and the student model based on the vocabulary of the teacher model and the vocabulary of the student model, wherein the words in the shared vocabulary are words existing in both the vocabulary of the teacher model and the vocabulary of the student model; A processing module, configured to convert the probability distributions output by the teacher model and the student model respectively based on the shared vocabulary, and obtain conversion probability distributions corresponding to the teacher model and the student model respectively, wherein the conversion probability distributions are probability distributions corresponding to words in the shared vocabulary; A second determination module is used to determine the target loss of the student model based on the transition probability distribution corresponding to the teacher model and the transition probability distribution corresponding to the student model; A training module is used to train the student model based on the target loss.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processing device, the steps of the method according to any one of claims 1 to 6 are implemented.

9. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images