Model distillation method and device, session method and device, electronic equipment and storage medium

Abstract

The invention provides a model distillation method and device, a session method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to a knowledge distillation technology and a large model technology. The method comprises the following steps: acquiring a pre-trained teacher model and a newly constructed student model; inputting the training sample into a teacher model and a student model, and adjusting parameters of the student model according to loss between output of a middle feature layer of the teacher model and output of a middle feature layer of the student model to obtain a first middle model; and inputting the training sample into the teacher model and the first intermediate model, and adjusting parameters of the first intermediate model according to the loss between the output of the teacher model prediction layer and the output of the first intermediate model prediction layer and the loss between the output of the first intermediate model prediction layer and the sample label to obtain a student model for executing the target task. The model distillation process is performed in multiple stages, so that the condition that the student model cannot converge due to few parameters is avoided, and the model distillation efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to knowledge distillation technology and large language model technology, and specifically to a model distillation method, a conversation method, an apparatus, an electronic device, a storage medium and a computer program product. Background Art

[0002] With the rapid development of artificial intelligence technology, algorithm models based on deep learning have been widely used in many fields such as image recognition, speech processing, and natural language understanding. These algorithm models often rely on large amounts of data and complex network structures to improve their prediction or recognition accuracy. However, in actual application scenarios, especially in edge computing and mobile devices, the algorithm model calls are large, while hardware resources are limited, so it is necessary to lightweight models with complex structures. Summary of the invention

[0003] The present disclosure provides a model distillation method, a conversation method, an apparatus, an electronic device, a storage medium, and a computer program product.

[0004] According to one aspect of the present disclosure, a model distillation method is provided, comprising:

[0005] Obtain a pre-trained teacher model for performing the target task and a newly constructed student model with fewer parameters than the teacher model;

[0006] In response to inputting the training sample into the teacher model and the student model, adjusting the parameters of the student model according to the loss between the intermediate feature layer output of the teacher model and the intermediate feature layer output of the student model to obtain a first intermediate model;

[0007] In response to inputting the training sample into the teacher model and the first intermediate model, the parameters of the first intermediate model are adjusted according to the loss between the teacher model prediction layer output and the first intermediate model prediction layer output, and the loss between the first intermediate model prediction layer output and the true label of the training sample to obtain a student model for performing the target task.

[0008] According to one aspect of the present disclosure, a conversation method is provided, comprising:

[0009] Get the conversation text entered by the user;

[0010] Verify and identify the conversation text through a lightweight text recognition model; wherein the lightweight text recognition model is obtained by distillation through the model distillation method of the embodiment of the present disclosure;

[0011] If the verification is passed, the conversation text will be sent to the large model for processing, and the processing results will be fed back to the user.

[0012] According to another aspect of the present disclosure, there is provided a model distillation apparatus, comprising:

[0013] An acquisition module, used to acquire a pre-trained teacher model for performing a target task and a newly constructed student model with fewer parameters than the teacher model;

[0014] A first-stage distillation module, configured to, in response to inputting a training sample into the teacher model and the student model, adjust a parameter of the student model according to a loss between an output of an intermediate feature layer of the teacher model and an output of an intermediate feature layer of the student model to obtain a first intermediate model;

[0015] The second stage distillation module is used to adjust the parameters of the first intermediate model in response to inputting the training sample into the teacher model and the first intermediate model according to the loss between the output of the teacher model prediction layer and the output of the first intermediate model prediction layer, and the loss between the output of the first intermediate model prediction layer and the true label of the training sample, so as to obtain a student model for performing the target task.

[0016] According to another aspect of the present disclosure, there is provided a conversation device, comprising:

[0017] A content acquisition module is used to acquire the conversation text input by the user;

[0018] A recognition module, used to verify and recognize the conversation text through a lightweight text recognition model; wherein the lightweight text recognition model is obtained by distillation through the model distillation method of the embodiment of the present disclosure;

[0019] The session processing module is used to send the session text to the large model for processing when the verification is passed, and feed back the processing results to the user.

[0020] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0021] at least one processor; and

[0022] a memory communicatively connected to at least one processor; wherein,

[0023] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to execute the model distillation method or session method of any embodiment of the present disclosure.

[0024] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the model distillation method or conversation method of any embodiment of the present disclosure.

[0025] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the model distillation method or conversation method of any embodiment of the present disclosure when executed by a processor.

[0026] According to the technology disclosed in the present invention, the model distillation training process is divided into multiple stages, and different losses are used in each stage to guide the learning process of the student model, thereby avoiding the problem that the student model cannot converge due to fewer parameters, improving the efficiency of the model distillation, and reducing the occupation of computing resources.

[0027] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0029] Figure 1 is a schematic flow chart of a model distillation method according to an embodiment of the present disclosure;

[0030] Figure 2 is a schematic flow chart of another model distillation method according to an embodiment of the present disclosure;

[0031] Figure 3 is a flow chart of a conversation method according to an embodiment of the present disclosure;

[0032] Figure 4 is a schematic structural diagram of a model distillation device according to an embodiment of the present disclosure;

[0033] Figure 5 is a structural schematic diagram of a conversation device according to an embodiment of the present disclosure;

[0034] Figure 6 It is a block diagram of an electronic device used to implement the model distillation method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0036] In the disclosed solution, when lightweight processing is performed on a model with a complex structure, a distillation training method can be used. For example, the training samples are input into a pre-trained teacher model and a newly constructed student model; the total loss is calculated based on the loss between the output of the teacher model and the output of the student model, and the loss between the output of the student model and the sample label; and then the parameters of the student model are adjusted according to the total loss to obtain the final required lightweight student model. However, this method has certain shortcomings: due to the small number of parameters in the student model, it is difficult to learn the complex features of the teacher model, which makes it difficult to converge the training and makes the efficiency of model distillation training low; in addition, due to the simplification of the student model, gradient diffusion or gradient vanishing problems are prone to occur on complex tasks, making the training process unstable and affecting the performance of the final model. Based on this, the present disclosure proposes a new model distillation method, that is, a model distillation method using multiple stages and multiple losses. The specific implementation process can be seen in the following embodiments.

[0037] Figure 1 is a schematic diagram of a model distillation method according to an embodiment of the present disclosure, which can be applied to a scenario where a model with a complex structure is lightweight. The method can be performed by a model distillation device, which is implemented in software and / or hardware and configured in an electronic device, which can be a server device.

[0038] like Figure 1 As shown, the model distillation method specifically includes the following:

[0039] S101. Obtain a pre-trained teacher model for performing a target task and a newly constructed student model with fewer parameters than the teacher model.

[0040] In the disclosed embodiments, the target task may be any one of a natural language processing task, an image processing task, and a speech processing task; wherein the natural language processing task may be a text classification task, a text generation task, a text translation task, etc.; the image processing task may be an image classification task, an image segmentation task, an image generation task, etc.; the speech processing task may include a speech recognition task, a speech synthesis task, etc. In this way, the disclosed solution can perform lightweight processing on models that perform tasks in different scenarios, enriching the applicable scenarios of the model distillation method.

[0041] The teacher model is a large, high-performance model that is pre-trained to perform the target task. For example, if the target task is a text classification task, the teacher model used to perform the target task is a text classification model, and the training samples used in the model distillation process are pre-labeled text data; similarly, if the target task is an image classification task, the teacher model used to perform the target task is an image classification model, and the training samples used in the model distillation process are pre-labeled image data; if the target task is a speech recognition task, the teacher model used to perform the target task is a speech recognition model, and the training samples used in the model distillation process are pre-labeled speech data.

[0042] The student model is a newly constructed model with a simpler structure and fewer parameters than the teacher model. The purpose of distillation training for the student model is to learn as much knowledge and capabilities of the teacher model as possible while keeping the model size small by imitating the behavior and knowledge representation of the teacher model, so as to achieve better performance on specific tasks.

[0043] In the disclosed embodiment, after obtaining the teacher model and the student model, the student model is trained by staged distillation training so that the student model after distillation training can perform the target task. Specifically, the first distillation training stage is performed according to step S102, and the second distillation training stage is performed according to step S103 on the basis of the completion of the first distillation training stage.

[0044] S102. In response to inputting the training samples into the teacher model and the student model, adjusting the parameters of the student model according to the loss between the intermediate feature layer output of the teacher model and the intermediate feature layer output of the student model to obtain a first intermediate model.

[0045] In the disclosed embodiment, the training sample is determined by the target task performed by the teacher model. For example, if the target task is a text classification task, the training sample is text data with pre-labeled types. The structures of the teacher model and the student model both include an intermediate feature layer for extracting sample features. For example, if the teacher model is a trained text classification model and the student model is a text classification model to be distilled and trained, the intermediate feature layers of the student model and the teacher model may include an embedding layer, an attention layer, a fully connected layer, etc. It can be understood that since the teacher model is a trained large-scale high-performance model and the student model is a model with a simple structure and fewer parameters, the number of intermediate feature layers of the teacher model is much larger than the number of intermediate feature layers of the student model. Therefore, to perform the first stage of model distillation training, it is necessary to first determine the matching relationship between the intermediate feature layer of the teacher model and the intermediate feature layer of the student model; the matching relationship can be determined by analyzing the architecture of the teacher model and the student model, and then determining the matching feature layer between the two based on factors such as network depth and semantic hierarchy. On this basis, the training samples are simultaneously input into the teacher model and the student model, and the loss between the output of the intermediate feature layer of the teacher model and the output of the intermediate feature layer of the student model is determined according to the matching relationship; that is, according to the matching relationship, the feature loss between each pair of matching feature layers is calculated, and the total feature loss is determined by weighted summation according to the feature loss weights corresponding to each pair of matching feature layers; according to the total feature loss, the parameters of the student model are adjusted to obtain the first intermediate model. Specifically, the gradient of the student model parameters can be calculated using the back propagation algorithm based on the calculated total feature loss; the parameters of the student model are updated according to the gradient using an optimization algorithm (such as the stochastic gradient descent algorithm) to obtain the first intermediate model.

[0046] It can be understood that determining the matching relationship between the intermediate feature layer of the teacher model and the intermediate feature layer of the student model can ensure the accuracy of the loss calculated between the outputs of the intermediate feature layers of the two models; after implementing the first stage of distillation training according to step S102, the first intermediate model obtained can quickly learn from the teacher model how to extract accurate features from the data.

[0047] S103. In response to inputting the training sample into the teacher model and the first intermediate model, according to the loss between the prediction layer output of the teacher model and the prediction layer output of the first intermediate model, and the loss between the prediction layer output of the first intermediate model and the true label of the training sample, the parameters of the first intermediate model are adjusted to obtain a student model for performing the target task.

[0048] In the disclosed embodiment, since the intermediate feature layer of the first intermediate model has completed distillation training in the first stage, the second stage mainly continues distillation training based on the output of the teacher model and the student model prediction layer. The second stage can be further divided into the first sub-stage and the second sub-stage.

[0049] In a specific implementation, the process of the first sub-stage includes: in response to inputting the training sample into the teacher model and the first intermediate model, determining the first soft target loss between the teacher model prediction layer output (e.g., probability distribution) and the first intermediate model prediction layer output (e.g., probability distribution), and the first hard target loss between the first intermediate model prediction layer output and the true label of the training sample; wherein the first soft target loss is the cross entropy loss between the teacher model prediction layer output and the first intermediate model prediction layer output calculated according to the cross entropy formula; the first hard target loss is the cross entropy loss between the first intermediate model prediction layer output and the true label of the training sample calculated according to the cross entropy formula. In addition, in the first sub-stage, the loss weight corresponding to the first soft target loss is greater than the loss weight corresponding to the first hard target loss; it can be understood that the loss weight corresponding to the first soft target loss is greater than the loss weight corresponding to the first hard target loss, which can ensure that the distillation training of the first sub-stage is mainly based on the prediction layer output loss of the first intermediate model and the teacher model, so that the first intermediate model can learn the discrimination ability of the teacher model as soon as possible. Further, according to the first soft target loss, the first hard target loss and the corresponding loss weights, the parameters of the first intermediate model are adjusted to obtain the second intermediate model. In specific implementation, the total prediction loss can be determined by weighted summation based on the first soft target loss, the first hard target loss and their corresponding loss weights; then, based on the total prediction loss, the gradient of the first intermediate model parameters is calculated using the back propagation algorithm; and the optimization algorithm (such as the stochastic gradient descent algorithm) is used to update the parameters of the first intermediate model according to the gradient to obtain the second intermediate model.

[0050] The process of the second sub-stage includes: in response to inputting the training sample into the teacher model and the second intermediate model, determining the second soft target loss between the output of the teacher model prediction layer and the output of the second intermediate model prediction layer, and the second hard target loss between the output of the second intermediate model prediction layer and the true label of the training sample; the second soft target loss is the cross entropy loss between the output of the teacher model prediction layer and the output of the second intermediate model prediction layer calculated according to the cross entropy formula; the second hard target loss is the cross entropy loss between the output of the second intermediate model prediction layer and the true label of the training sample calculated according to the cross entropy formula. In addition, in the second sub-stage, the loss weight corresponding to the second soft target loss is less than the loss weight corresponding to the second hard target loss. It can be understood that the loss weight corresponding to the second soft target loss is less than the loss weight corresponding to the second hard target loss, which can ensure that the distillation training of the second sub-stage is mainly based on the loss between the output of the second intermediate model prediction layer and the sample label, so that the second intermediate model can further improve the discrimination ability of the second intermediate model according to its own prediction results and true results. Further, according to the second soft target loss, the second hard target loss and the corresponding loss weights, the parameters of the second intermediate model are adjusted to obtain a student model for performing the target task. In specific implementation, the total prediction loss can be determined by weighted summation based on the second soft target loss, the second hard target loss and the corresponding loss weights; then, based on the total prediction loss, the back propagation algorithm is used to calculate the gradient of the second intermediate model parameters; using an optimization algorithm (such as a stochastic gradient descent algorithm), the parameters of the second intermediate model are updated according to the gradient to obtain the student model used to perform the target task. It can be understood that by subdividing the second stage of distillation training into two sub-stages, each sub-stage is mainly trained with different prediction losses, so that the student model can quickly and accurately learn the discrimination ability of the teacher model.

[0051] In the disclosed embodiment, the model distillation training process is divided into multiple stages, and each stage uses different losses to guide the learning process of the student model, so that the student model converges quickly, which can effectively improve the efficiency of model distillation and reduce the long-term occupation of computing resources by the student model due to the failure to converge during training.

[0052] Figure 2 FIG. 1 is a flow chart of another model distillation method according to an embodiment of the present disclosure. Figure 2 As shown, the model distillation method specifically includes the following steps:

[0053] S201. Obtain a pre-trained teacher model for performing a target task and a newly constructed student model with fewer parameters than the teacher model.

[0054] In the disclosed embodiment, the number of intermediate feature layers of the teacher model is an integer multiple of the number of intermediate feature layers of the student model; thus, the matching relationship between the intermediate feature layers of the teacher model and the intermediate feature layers of the student model can be quickly determined based on the multiple value, thereby ensuring the efficiency of subsequent model distillation training.

[0055] S202. Determine a matching relationship between the intermediate feature layers of the teacher model and the intermediate feature layers of the student model according to a multiple value between the number of intermediate feature layers of the teacher model and the number of intermediate feature layers of the student model.

[0056] For example, the teacher model has 6 intermediate feature layers and the student model has 3 intermediate feature layers, then the 2nd, 4th, and 6th layers of the teacher model can be matched with the 1st, 2nd, and 3rd layers of the student model respectively according to the multiplier value 2. It can be understood that this one-to-one matching relationship helps to directly calculate the loss between each pair of matching feature layers.

[0057] S203. In response to inputting the training samples into the teacher model and the student model, the loss between the intermediate feature layer output of the teacher model and the matched intermediate feature layer output of the student model is determined according to the matching relationship; according to the loss, the parameters of the student model are adjusted to obtain a first intermediate model.

[0058] Exemplarily, after the training samples are simultaneously input into the teacher model and the student model, the first loss between the output of the second layer of the teacher model and the output of the first layer of the student model is calculated according to a preset loss function (e.g., mean square error function); at the same time, the second loss between the output of the fourth layer of the teacher model and the output of the second layer of the student model is calculated; the third loss between the output of the sixth layer of the teacher model and the output of the third layer of the student model is calculated. Then, according to the first loss, the second loss, the third loss and the loss weight corresponding to each layer of loss, the total feature loss is calculated by weighted summation; then, according to the total feature loss, the parameters of the student model are adjusted to obtain the first intermediate model.

[0059] S204. In response to inputting the training sample into the teacher model and the first intermediate model, according to the loss between the prediction layer output of the teacher model and the prediction layer output of the first intermediate model, and the loss between the prediction layer output of the first intermediate model and the true label of the training sample, the parameters of the first intermediate model are adjusted to obtain a student model for performing the target task.

[0060] The specific implementation process of step 204 can refer to the description of the above embodiment, which will not be repeated here.

[0061] In the disclosed embodiment, because the number of intermediate feature layers of the teacher model is an integer multiple of the number of intermediate feature layers of the student model, the matching relationship between the intermediate feature layers of the teacher model and the intermediate feature layers of the student model can be quickly and accurately determined based on the multiple relationship, and the loss between each pair of matching feature layers can be subsequently quickly calculated based on the matching relationship, so as to complete the first stage of distillation training of the student model based on the loss, which can effectively improve the efficiency of the first stage of distillation training.

[0062] Figure 3 The present invention is a flowchart of a conversation method according to an embodiment of the present invention. The present invention is applicable to a scenario in which a question-and-answer conversation is implemented by combining a lightweight text recognition model and a large language model. The method can be executed by a conversation reply device, which is implemented in software and / or hardware and configured in an electronic device, which can be a mobile terminal device.

[0063] like Figure 3 As shown, the session method specifically includes the following steps:

[0064] S301: Acquire the conversation text input by the user.

[0065] When using a large model for a conversation, the conversation text input by the user must first be obtained so that the large model can provide feedback based on the conversation text. The conversation text can be the text content directly entered by the user in the text input box. In addition, if the user inputs by voice, the user's input voice is converted into the corresponding conversation text through a lightweight speech recognition model. The lightweight speech recognition model is a model for performing speech recognition tasks obtained by staged distillation training using the model distillation method disclosed in the present invention. The specific model distillation process can be found in the above embodiment.

[0066] S302: Verify and recognize the conversation text using a lightweight text recognition model.

[0067] Among them, the lightweight text recognition model is a model for performing text classification tasks obtained by staged distillation training using the model distillation method disclosed in the present invention. The specific model distillation training process can be found in the above embodiment and will not be repeated here. The lightweight text recognition model is used to verify and identify the conversation text, mainly to detect whether the conversation text includes sensitive words (for example, words involving vulgarity, violence, etc. that violate regulations or public order and good morals). If sensitive words are included, the verification fails, and the user can be informed in the form of a pop-up window that the conversation text input does not meet the specifications, and it is prohibited to send the conversation text to the large model. If the verification and identification pass, step S303 is executed.

[0068] S303. If the verification is passed, the conversation text is sent to the large model for processing, and the processing result is fed back to the user.

[0069] In the disclosed embodiment, after the conversation text is sent to the big model, the big model understands the conversation text (e.g., lexical analysis or semantic analysis), and retrieves relevant knowledge based on the conversation text; then generates a reply text based on the understanding results and the retrieval results; and finally feeds back the reply text to the user in text form or by converting it into voice form.

[0070] In the disclosed embodiment, the lightweight text recognition model can quickly perform preliminary verification and recognition on the conversation text before sending it to the large model, so as to avoid the large model processing low-quality or non-compliant inputs, thereby saving the processing time of the large model and improving the efficiency of the entire conversation system; and the lightweight text recognition model is obtained through multi-stage distillation training and will not take up too many computing resources.

[0071] Figure 4 Schematic diagram of the structure of the model distillation device according to the embodiment of the present disclosure. The present embodiment can be applied to the scene of lightweight processing of models with complex structures. The device can implement the model distillation method of any embodiment of the present disclosure. Figure 4 As shown, the device 400 specifically includes:

[0072] An acquisition module 401 is used to acquire a pre-trained teacher model for performing a target task and a newly constructed student model having fewer parameters than the teacher model;

[0073] The first stage distillation module 402 is used for adjusting the parameters of the student model according to the loss between the intermediate feature layer output of the teacher model and the intermediate feature layer output of the student model in response to inputting the training sample into the teacher model and the student model to obtain a first intermediate model;

[0074] The second-stage distillation module 403 is used to adjust the parameters of the first intermediate model in response to inputting the training sample into the teacher model and the first intermediate model according to the loss between the teacher model prediction layer output and the first intermediate model prediction layer output, and the loss between the first intermediate model prediction layer output and the true label of the training sample, so as to obtain a student model for performing the target task.

[0075] In some embodiments, the first stage distillation module comprises:

[0076] A matching relationship determination unit, used to determine the matching relationship between the intermediate feature layer of the teacher model and the intermediate feature layer of the student model;

[0077] A loss calculation unit, configured to determine, in response to inputting the training sample into the teacher model and the student model, a loss between an intermediate feature layer output of the teacher model and a matched intermediate feature layer output of the student model according to a matching relationship;

[0078] The distillation unit is used to adjust the parameters of the student model according to the loss to obtain the first intermediate model.

[0079] In some embodiments, the number of intermediate feature layers of the teacher model is an integer multiple of the number of intermediate feature layers of the student model;

[0080] The matching relationship determination unit is specifically used for:

[0081] According to the multiple value between the number of intermediate feature layers of the teacher model and the number of intermediate feature layers of the student model, the matching relationship between the intermediate feature layers of the teacher model and the intermediate feature layers of the student model is determined.

[0082] In some embodiments, the second stage distillation module is specifically used to:

[0083] In response to inputting the training sample into the teacher model and the first intermediate model, determining a first soft target loss between the teacher model prediction layer output and the first intermediate model prediction layer output, and a first hard target loss between the first intermediate model prediction layer output and the true label of the training sample; wherein the loss weight corresponding to the first soft target loss is greater than the loss weight corresponding to the first hard target loss;

[0084] According to the first soft target loss, the first hard target loss and the corresponding loss weights, the parameters of the first intermediate model are adjusted to obtain the second intermediate model;

[0085] In response to inputting the training sample into the teacher model and the second intermediate model, determining a second soft target loss between the output of the teacher model prediction layer and the output of the second intermediate model prediction layer, and a second hard target loss between the output of the second intermediate model prediction layer and the true label of the training sample; wherein the loss weight corresponding to the second soft target loss is less than the loss weight corresponding to the second hard target loss;

[0086] According to the second soft target loss, the second hard target loss and their corresponding loss weights, the parameters of the second intermediate model are adjusted to obtain a student model for performing the target task.

[0087] In some embodiments, the target task is any one of a natural language processing task, an image processing task, and a speech processing task; the training sample is any one of text data, image data, and speech data.

[0088] The above-mentioned product can execute the method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0089] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0090] Figure 5 Schematic diagram of the structure of the conversation device according to the embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the scenario of combining a lightweight text recognition model and a large language model to realize a question-and-answer conversation. The device can realize the conversation method of any embodiment of the present disclosure. Figure 5 As shown, the device 500 specifically includes:

[0091] Content acquisition module 501, used to acquire the conversation text input by the user;

[0092] The recognition module 502 is used to verify and recognize the conversation text through a lightweight text recognition model; wherein the lightweight text recognition model is obtained by distillation training through the model distillation method disclosed in the present invention;

[0093] The conversation processing module 503 is used to send the conversation text to the large model for processing when the verification is passed, and feed back the processing result to the user.

[0094] The above-mentioned product can execute the method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0095] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0096] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0097] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0098] like Figure 6As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0099] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0100] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the model distillation method. For example, in some embodiments, the model distillation method may be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the model distillation method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the model distillation method in any other appropriate manner (e.g., by means of firmware).

[0101] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0102] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0103] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0104] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0105] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0106] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs that are executed on the corresponding computers and have a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services. The server may also be a server of a distributed system, or a server combined with a blockchain.

[0107] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning technology, big data processing technology, knowledge graph technology, and other major directions.

[0108] Cloud computing refers to a technology system that uses network access to elastically scalable shared physical or virtual resource pools. Resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for technical applications such as artificial intelligence and blockchain, as well as model training.

[0109] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions provided by this disclosure can be achieved, and this document does not limit this.

[0110] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A model distillation method, comprising: Obtain a pre-trained teacher model for performing a target task and a newly constructed student model having fewer parameters than the teacher model; In response to inputting a training sample into the teacher model and the student model, adjusting a parameter of the student model according to a loss between an intermediate feature layer output of the teacher model and an intermediate feature layer output of the student model to obtain a first intermediate model; In response to inputting the training sample into the teacher model and the first intermediate model, the parameters of the first intermediate model are adjusted according to the loss between the prediction layer output of the teacher model and the prediction layer output of the first intermediate model, and the loss between the prediction layer output of the first intermediate model and the true label of the training sample to obtain a student model for performing the target task.

2. The method according to claim 1, wherein: In response to inputting the training sample into the teacher model and the student model, adjusting the parameters of the student model according to the loss between the intermediate feature layer output of the teacher model and the intermediate feature layer output of the student model to obtain a first intermediate model, including: Determine a matching relationship between the intermediate feature layer of the teacher model and the intermediate feature layer of the student model; In response to inputting a training sample into the teacher model and the student model, determining a loss between an intermediate feature layer output of the teacher model and a matched intermediate feature layer output of the student model according to the matching relationship; According to the loss, the parameters of the student model are adjusted to obtain a first intermediate model.

3. The method according to claim 2, wherein: The number of intermediate feature layers of the teacher model is an integer multiple of the number of intermediate feature layers of the student model; The determining of the matching relationship between the intermediate feature layer of the teacher model and the intermediate feature layer of the student model includes: According to the multiple value between the number of intermediate feature layers of the teacher model and the number of intermediate feature layers of the student model, the matching relationship between the intermediate feature layers of the teacher model and the intermediate feature layers of the student model is determined.

4. The method according to claim 1, wherein: In response to inputting the training sample into the teacher model and the first intermediate model, adjusting the parameters of the first intermediate model according to the loss between the prediction layer output of the teacher model and the prediction layer output of the first intermediate model, and the loss between the prediction layer output of the first intermediate model and the true label of the training sample, to obtain a student model for performing the target task, including: In response to inputting the training sample into the teacher model and the first intermediate model, determining a first soft target loss between the teacher model prediction layer output and the first intermediate model prediction layer output, and a first hard target loss between the first intermediate model prediction layer output and the true label of the training sample; wherein the loss weight corresponding to the first soft target loss is greater than the loss weight corresponding to the first hard target loss; According to the first soft target loss, the first hard target loss and their corresponding loss weights, adjusting the parameters of the first intermediate model to obtain a second intermediate model; In response to inputting the training sample into the teacher model and the second intermediate model, determining a second soft target loss between the teacher model prediction layer output and the second intermediate model prediction layer output, and a second hard target loss between the second intermediate model prediction layer output and the true label of the training sample; wherein the loss weight corresponding to the second soft target loss is less than the loss weight corresponding to the second hard target loss; According to the second soft target loss, the second hard target loss and their corresponding loss weights, the parameters of the second intermediate model are adjusted to obtain a student model for performing the target task.

5. The method according to claim 1 or 4, wherein: The target task is any one of a natural language processing task, an image processing task, and a speech processing task; the training sample is any one of text data, image data, and speech data.

6. A conversation method, comprising: Get the conversation text entered by the user; Verifying and identifying the conversation text by means of a lightweight text recognition model; wherein the lightweight text recognition model is obtained by distillation of the method according to any one of claims 1 to 6; If the verification is passed, the conversation text is sent to the large model for processing, and the processing result is fed back to the user.

7. A model distillation apparatus comprising: An acquisition module, used to acquire a pre-trained teacher model for performing a target task and a newly constructed student model having fewer parameters than the teacher model; A first-stage distillation module, configured to, in response to inputting a training sample into the teacher model and the student model, adjust a parameter of the student model according to a loss between an output of an intermediate feature layer of the teacher model and an output of an intermediate feature layer of the student model to obtain a first intermediate model; The second-stage distillation module is used for adjusting the parameters of the first intermediate model in response to inputting the training sample into the teacher model and the first intermediate model according to the loss between the prediction layer output of the teacher model and the prediction layer output of the first intermediate model, and the loss between the prediction layer output of the first intermediate model and the true label of the training sample, so as to obtain a student model for performing the target task.

8. The device according to claim 7, wherein: The first stage distillation module comprises: A matching relationship determination unit, used to determine the matching relationship between the intermediate feature layer of the teacher model and the intermediate feature layer of the student model; a loss calculation unit, configured to determine, in response to inputting a training sample into the teacher model and the student model, a loss between an intermediate feature layer output of the teacher model and a matched intermediate feature layer output of the student model according to the matching relationship; The distillation unit is used to adjust the parameters of the student model according to the loss to obtain a first intermediate model.

9. The device according to claim 8, wherein: The number of intermediate feature layers of the teacher model is an integer multiple of the number of intermediate feature layers of the student model; The matching relationship determination unit is specifically used for: According to the multiple value between the number of intermediate feature layers of the teacher model and the number of intermediate feature layers of the student model, the matching relationship between the intermediate feature layers of the teacher model and the intermediate feature layers of the student model is determined.

10. The device according to claim 7, wherein: The second stage distillation module is specifically used for: In response to inputting the training sample into the teacher model and the first intermediate model, determining a first soft target loss between the teacher model prediction layer output and the first intermediate model prediction layer output, and a first hard target loss between the first intermediate model prediction layer output and the true label of the training sample; wherein the loss weight corresponding to the first soft target loss is greater than the loss weight corresponding to the first hard target loss; According to the first soft target loss, the first hard target loss and their corresponding loss weights, adjusting the parameters of the first intermediate model to obtain a second intermediate model; In response to inputting the training sample into the teacher model and the second intermediate model, determining a second soft target loss between the teacher model prediction layer output and the second intermediate model prediction layer output, and a second hard target loss between the second intermediate model prediction layer output and the true label of the training sample; wherein the loss weight corresponding to the second soft target loss is less than the loss weight corresponding to the second hard target loss; According to the second soft target loss, the second hard target loss and their corresponding loss weights, the parameters of the second intermediate model are adjusted to obtain a student model for performing the target task.

11. The device according to claim 7 or 10, wherein: The target task is any one of a natural language processing task, an image processing task, and a speech processing task; the training sample is any one of text data, image data, and speech data.

12. A conversation device, comprising: A content acquisition module is used to acquire the conversation text input by the user; A recognition module, used to verify and recognize the conversation text through a lightweight text recognition model; wherein the lightweight text recognition model is obtained by distillation through the method described in any one of claims 1 to 6; The conversation processing module is used to send the conversation text to the large model for processing when the verification is passed, and feed back the processing result to the user.

13. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-6.

15. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images