Model distillation method, apparatus, and storage medium

Abstract

The application relates to model distillation in the field of artificial intelligence, and specifically discloses a model distillation method, a model distillation device and a storage medium. The method comprises the following steps: obtaining a text sample, the text sample comprising at least one pair of entity labels, each pair of entity labels in the at least one pair of entity labels being used for representing an entity type; inputting the text sample into a teacher model and a student model respectively, and determining a loss set; and adjusting model parameters of the student model according to the loss set, so as to train the student model. The embodiment of the application is beneficial to improving the accuracy of text intent recognition of the student model after model distillation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a model distillation method, apparatus, and storage medium. Background Technology

[0002] Natural language understanding (NLU) tasks mainly consist of intent recognition and slot filling. For example, when a voice assistant executes a user's command, the machine needs to determine the intent expressed in the user's speech and then translate that intent into a command, i.e., slot filling. The accuracy of intent recognition and slot filling has a significant impact on the proper execution of user commands and the improvement of the quality of the dialogue system.

[0003] BERT-based models are currently the mainstream architecture for NLU, significantly improving the recognition accuracy of existing models. However, they suffer from excessively large model parameters and long inference latency. In practical applications, knowledge distillation techniques are typically used to miniaturize the model, resulting in a student model, which is then used for spoken language comprehension.

[0004] For large models (teacher models), the identification of intent and slot filling are mainly done through sentence structure and keywords. However, in actual business scenarios, a sentence may contain multiple intents, which makes it difficult for the teacher model to accurately determine which intent it corresponds to. This results in a high misclassification rate for teacher models in such sentence structures.

[0005] Therefore, the student model derived from the teacher model is also unable to distinguish the accurate intent of this type of sentence, resulting in low accuracy of the student model in spoken language comprehension and text recognition, and a poor user experience. Summary of the Invention

[0006] This application provides a model distillation method, apparatus, and storage medium. Distilling the teacher model's ability to recognize text intent through entity labels enables the student model to also recognize text intent through entity labels, thereby improving the accuracy of text intent recognition.

[0007] In a first aspect, embodiments of this application provide a model distillation method, including:

[0008] Obtain a text sample, wherein the text sample includes at least one pair of entity labels, and each pair of entity labels is used to represent an entity type;

[0009] The text samples are input into a teacher model and a student model respectively to determine a loss set, which includes a first loss, a second loss, and a third loss. The first loss is determined based on the difference between a first intent and a second intent, which are obtained by the teacher model and the student model, respectively, through intent recognition of the text samples based on at least one pair of entity labels. The second loss is determined based on the difference between a first feature vector and a second feature vector corresponding to each entity label in the at least one pair of entity labels, which are obtained by encoding each entity label by the teacher model and the student model respectively. The third loss is determined based on the difference between a first slot filling result and a second slot filling result corresponding to each word in the text sample, wherein the first slot filling result and the second slot filling result corresponding to each word are obtained by the teacher model and the student model, respectively, through slot filling of each word based on at least one pair of entity labels.

[0010] The model parameters of the student model are adjusted according to the loss set in order to train the student model.

[0011] It should be understood that the teacher model is a trained model that possesses the ability to recognize text intent based on the entity types represented by entity labels. Therefore, in this embodiment, during the distillation of the teacher model, it is also necessary to align the teacher model and the student model at the entity label level, so that the student model also has the ability to recognize text intent based on entity labels. This allows for subsequent inference using the student model, where intent can be recognized based on the entity types represented by entity labels in the text to be identified. This enables model inference based on prior knowledge, thereby improving the accuracy of text intent recognition.

[0012] In some possible implementations, one entity label in each pair of entity labels is used to indicate the starting position of the first entity word in the text sample, and the other entity label is used to indicate the ending position of the first entity word in the text sample; and the entity type represented by each entity label is the same as or different from the entity type corresponding to the first entity word.

[0013] As can be seen, in this embodiment, by using a pair of entity labels to indicate the position of entities in the text sample, it is possible to accurately determine which words in the text sample are entities. Thus, in the slot filling stage, slot filling results corresponding to the type of the entity can be filled for these words, improving the accuracy of the slot filling results, thereby making the recognition accuracy of the trained student model higher.

[0014] In some possible implementations, the loss set further includes a fourth loss, which is determined based on the difference between the third and fourth word vectors corresponding to each word in the text sample, the third and fourth word vectors being obtained by encoding each word by the teacher model and the student model, respectively.

[0015] As can be seen in this implementation, since the teacher model has better performance and more accurate word encoding, the student model and the teacher model need to be aligned at the word encoding level so that the student model can learn the teacher model's ability to encode words, thereby improving the accuracy of word encoding.

[0016] In some possible implementations, the loss set further includes a fifth loss, which is determined based on the difference between the first target word vector and the second target word vector corresponding to each word in the text sample. The first target word vector and the second target word vector corresponding to each word in the text sample are obtained by attention weighting the third word vector and the fourth vector corresponding to each word through the attention mechanism in the teacher model and the student model, respectively.

[0017] As can be seen in this implementation, since the teacher model has better performance and the attention fusion pair is more accurate, the student model and the teacher model need to be aligned at the word encoding level so that the student model can learn the teacher model's ability to fuse word vectors based on the attention mechanism, thereby improving the accuracy of attention fusion.

[0018] In some possible implementations, adjusting the model parameters of the student model based on the loss set includes:

[0019] The first loss, the second loss, the third loss, the fourth loss, and the fifth loss are weighted to obtain the target loss; the model parameters of the student model are adjusted according to the target loss.

[0020] As can be seen, in this embodiment, since the teacher model has good text intent recognition performance and the student model is aligned with the teacher model in multiple dimensions, the student model can learn the performance of the teacher model in multiple dimensions. Thus, even with fewer parameters in the student model, it can still use good text intent recognition performance for intent recognition.

[0021] In some possible implementations, the first text sequence is obtained;

[0022] The first text sequence is augmented to obtain at least one second text sequence, wherein each of the at least one second text sequence has the same intent as the first text sequence;

[0023] Replace the second entity word in each of the at least one second text sequence to obtain at least one third text sequence corresponding to each of the at least one second text sequence;

[0024] At least one pair of entity labels are added to each of the at least one third text sequence corresponding to each second text sequence to obtain the text sample.

[0025] As can be seen, in this embodiment, text enhancement and entity word replacement are performed on the first text sequence, resulting in a richer corpus for the third text sequence. Using such corpus for model distillation can make the generalization ability of the distilled student model stronger.

[0026] In some possible implementations, the text enhancement of the first text sequence to obtain at least one second text sequence includes:

[0027] Determine at least one second word corresponding to the first word, wherein the first word is any word in the first text sequence other than stop words, entity words, and vertical category words;

[0028] The first word in the first text sequence is replaced by each of the at least one second word to obtain the at least one second text sequence.

[0029] As can be seen, in this embodiment, by replacing the first word, the resulting second text sequence has the same intent as the first text sequence, thereby increasing the training corpus with the same intent but different expressions. By using such training corpus to distill the model, it is possible to identify different expressions corresponding to the same intent, thus improving the model's generalization ability.

[0030] In some possible implementations, determining at least one second word corresponding to the first word includes:

[0031] The first word in the first text sequence is masked to obtain the fourth text sequence;

[0032] Based on the fourth text sequence, word prediction is performed to obtain at least one second word corresponding to the first word.

[0033] As can be seen, in this embodiment, the randomness of the generation of the second word is ensured by prediction, thereby simulating the randomness of each expression in different expressions of the same intention in daily life.

[0034] In some possible implementations, replacing the second entity word in each of the at least one second text sequence to obtain at least one third text sequence corresponding to each of the at least one second text sequence includes:

[0035] Obtain at least one third entity word from a pre-constructed entity dictionary that corresponds to a second entity word in each of the at least one second text sequences, wherein each third entity word belongs to the same domain as the second entity word;

[0036] Each third entity word in the at least one third entity word is used to replace the second entity word in each second text sequence to obtain at least one third text sequence corresponding to each second text sequence.

[0037] As can be seen, in this embodiment, entity words are replaced to increase the richness of the training corpus in a domain. The resulting network model can identify the intent corresponding to each entity in the domain, thereby improving the generalization ability of the network model.

[0038] In some possible implementations, adding at least one pair of entity tags to each of the at least one third text sequence corresponding to each second text sequence to obtain the text sample includes:

[0039] In the case that each third text sequence contains vertical words, at least one pair of entity labels is added to the third entity words in each third text sequence to obtain the text sample, wherein the entity type represented by each pair of entity labels is different from the entity type corresponding to the third entity words in the third text sequence;

[0040] In the case that each third text sequence does not contain vertical words, a third entity word in each third text sequence is determined, and at least one entity type corresponding to the third entity word is determined. At least one target entity type is selected from the at least one entity type, and at least one pair of entity labels is added to each third text sequence to represent the at least one target entity type, thereby obtaining the text sample. Each pair of entity labels is used to represent a target entity type.

[0041] As can be seen in this embodiment, when vertical category words are included, different entity labels are added to the third entity word to simulate a scenario where the text sequence contains noise. Using such text samples, the model is distilled, allowing the student model to learn to prioritize intent recognition and slot filling based on vertical category words when the text sequence contains them, while ignoring the role of entity labels. This ensures correct inference even when the text sequence contains noise in the subsequent inference stage (the stage of applying the model). When vertical category words are not included, entity labels are added to the third text sequence to represent the entity type corresponding to the third entity, enabling the distilled student model to learn intent recognition based on entity labels, thereby achieving intent recognition based on prior knowledge and improving the accuracy of intent recognition.

[0042] Secondly, embodiments of this application provide a model distillation apparatus, comprising:

[0043] A transceiver unit is used to acquire text samples, wherein the text samples include at least one pair of entity tags, and each pair of entity tags is used to characterize an entity type.

[0044] A processing unit is configured to input the text samples into a teacher model and a student model respectively, and determine a loss set, the loss set including a first loss, a second loss, and a third loss. The first loss is determined based on the difference between a first intent and a second intent, which are obtained by the teacher model and the student model respectively, and by performing intent recognition on the text samples according to at least one pair of entity labels. The second loss is determined based on the difference between a first feature vector and a second feature vector corresponding to each entity label in the at least one pair of entity labels, which are obtained by encoding each entity label by the teacher model and the student model respectively. The third loss is determined based on the difference between a first slot filling result and a second slot filling result corresponding to each word in the text sample, wherein the first slot filling result and the second slot filling result corresponding to each word are obtained by the teacher model and the student model respectively, and by performing slot filling on each word according to at least one pair of entity labels.

[0045] The processing unit is further configured to adjust the model parameters of the student model according to the loss set in order to train the student model.

[0046] In some possible implementations, one entity label in each pair of entity labels is used to indicate the starting position of the first entity word in the text sample, and the other entity label is used to indicate the ending position of the first entity word in the text sample; and the entity type represented by each entity label is the same as or different from the entity type corresponding to the first entity word.

[0047] In some possible implementations, the loss set further includes a fourth loss, which is determined based on the difference between the third and fourth word vectors corresponding to each word in the text sample, the third and fourth word vectors being obtained by encoding each word by the teacher model and the student model, respectively.

[0048] In some possible implementations, the loss set further includes a fifth loss, which is determined based on the difference between the first target word vector and the second target word vector corresponding to each word in the text sample. The first target word vector and the second target word vector corresponding to each word in the text sample are obtained by attention weighting the third word vector and the fourth vector corresponding to each word through the attention mechanism in the teacher model and the student model, respectively.

[0049] In some possible implementations, the processing unit is specifically used for adjusting the model parameters of the student model based on the loss set:

[0050] The first loss, the second loss, the third loss, the fourth loss, and the fifth loss are weighted to obtain the target loss;

[0051] The model parameters of the student model are adjusted based on the target loss.

[0052] In some possible implementations, before the transceiver unit acquires the text sample, the transceiver unit is further configured to acquire a first text sequence;

[0053] The processing unit is further configured to: have each second text sequence in the text sequence have the same intent as the first text sequence; replace the second entity word in each of the at least one second text sequence to obtain at least one third text sequence corresponding to each of the second text sequences; and add at least one pair of entity tags to each of the at least one third text sequence corresponding to each of the second text sequences to obtain the text sample.

[0054] In some possible implementations, in terms of text enhancement of the first text sequence to obtain at least one second text sequence, the processing unit is specifically configured to:

[0055] Determine at least one second word corresponding to the first word, wherein the first word is any word in the first text sequence other than stop words, entity words, and vertical category words;

[0056] The first word in the first text sequence is replaced by each of the at least one second word to obtain the at least one second text sequence.

[0057] In some possible implementations, the processing unit is specifically configured to: determine at least one second word corresponding to the first word;

[0058] The first word in the first text sequence is masked to obtain the fourth text sequence;

[0059] Based on the fourth text sequence, word prediction is performed to obtain at least one second word corresponding to the first word.

[0060] In some possible implementations, the processing unit is specifically configured to: replace the second entity word of each of the at least one second text sequence to obtain at least one third text sequence corresponding to each of the at least one second text sequence;

[0061] Obtain at least one third entity word from a pre-constructed entity dictionary that corresponds to a second entity word in each of the at least one second text sequences, wherein each third entity word belongs to the same domain as the second entity word;

[0062] Each third entity word in the at least one third entity word is used to replace the second entity word in each second text sequence to obtain at least one third text sequence corresponding to each second text sequence.

[0063] In some possible implementations, the processing unit is specifically used to obtain the text sample by adding at least one pair of entity tags to each of the at least one third text sequence corresponding to each second text sequence.

[0064] In the case that each third text sequence contains vertical words, at least one pair of entity labels is added to the third entity words in each third text sequence to obtain the text sample, wherein the entity type represented by each pair of entity labels is different from the entity type corresponding to the third entity words in the third text sequence;

[0065] In the case that each third text sequence does not contain vertical words, a third entity word in each third text sequence is determined, and at least one entity type corresponding to the third entity word is determined. At least one target entity type is selected from the at least one entity type, and at least one pair of entity labels is added to each third text sequence to represent the at least one target entity type, thereby obtaining the text sample. Each pair of entity labels is used to represent a target entity type.

[0066] Thirdly, embodiments of this application provide a model distillation apparatus. The text intent recognition apparatus includes a processor connected to a memory for storing computer programs. The processor executes the computer programs stored in the memory to cause the text intent recognition apparatus to perform the method as described in any one of the first aspects.

[0067] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run, implements the methods performed by the model distillation apparatus in the above aspects.

[0068] Fifthly, a computer program product is provided, the computer program product comprising: computer program code, which, when the computer program code is run, causes the methods executed by the model distillation apparatus in the above aspects to be performed. Attached Figure Description

[0069] Figure 1 This application provides a schematic diagram of slot filling;

[0070] Figure 2 This application provides a schematic diagram of a model framework for model distillation;

[0071] Figure 3 This application provides a flowchart illustrating a method for constructing text samples;

[0072] Figure 4 This application provides a schematic diagram of a generalized first text sequence;

[0073] Figure 5 This application provides a schematic flowchart of a model distillation method;

[0074] Figure 6 This application provides a schematic diagram of a network model;

[0075] Figure 7 This application provides a schematic diagram of the structure of a dialogue system;

[0076] Figure 8This application provides a schematic diagram of a model distillation apparatus;

[0077] Figure 9 This application provides a schematic diagram of another model distillation apparatus. Detailed Implementation

[0078] To facilitate understanding of this application, relevant technical knowledge involved in the embodiments of this application will be introduced first.

[0079] Currently, models are used for Non-Logical Understanding (NLU), such as for understanding spoken language. To ensure the model's accuracy and efficiency, a teacher model is first trained. Generally, teacher models have more parameters and a more complex structure, making them more accurate for NLU but relatively slower. Therefore, the teacher model can be distilled to obtain a student model with fewer parameters and a simpler structure; then, the student model is used for NLU.

[0080] For example, labeled training data can be input into teacher and student models for training, and the likelihood estimates of each teacher and student model can be output. The likelihood estimates of the teacher models can be pooled to output the pooled likelihood estimates. The difference between the pooled likelihood estimates of the teacher models and the student models can be measured. The parameters of the student models can be updated to obtain the student model whose likelihood estimate is closest to the pooled likelihood estimate of the teacher models.

[0081] It can be seen that by making the output of the student model close to that of the teacher model, the same recognition effect as the teacher model can be achieved when the student model has fewer parameters, thus ensuring recognition accuracy and efficiency.

[0082] However, for teacher models to perform NLU, it is mainly achieved through intent recognition and slot filling. Moreover, currently, intent recognition and slot filling are mainly based on sentence structure and keywords. In actual business scenarios, a sentence structure and keywords may contain multiple intents. As a result, teacher models cannot accurately determine which intent it corresponds to, which leads to a high misclassification rate of teacher models in such sentence structures.

[0083] like Figure 1As shown, in the multimedia vertical domain, the intent to play music, video, and audio all contain the phrase "play XXX." The model cannot distinguish the intent from the identification slot based solely on the phrase and keywords. For example, "Please play 'Seven Mile Fragrance' for me." Since "Seven Mile Fragrance" could be a music entity, a video entity, or another entity, if the current user's intent is to play the music "Seven Mile Fragrance," but the teacher model cannot recognize that "Seven Mile Fragrance" is a music entity, it will be unable to accurately identify the user's intent, leading to a misinterpretation of the user's intent.

[0084] Therefore, the student model derived from the teacher model is also unable to distinguish the accurate intent of this type of sentence, resulting in low accuracy of the student model in spoken language comprehension and text recognition, and a poor user experience.

[0085] Therefore, how to develop a student model with high accuracy in spoken language comprehension is an urgent problem to be solved.

[0086] To facilitate understanding of this application, the relevant terms used in this application are introduced herein.

[0087] Vertical category terms: Terms related to a vertical category, used to indicate the intent type of the statement containing that vertical category term.

[0088] Stop words: Stop words, also known as function words, have little actual meaning compared to other words. The most common stop words are determiners such as "the," "a," "an," "that," and "those," which help describe nouns and express concepts such as location or quantity in the text.

[0089] Entity words, including nouns and pronouns, are used to indicate objectively existing entities.

[0090] See Figure 2 , Figure 2 This is a schematic diagram of a model framework for model distillation provided in an embodiment of this application. The model framework includes a text sample construction module and a knowledge distillation module.

[0091] like Figure 2 As shown, the text sample construction module first enhances the original text, that is, enhances the first text sequence to obtain at least one second text sequence; then, it performs dynamic dictionary enhancement on each second text sequence, that is, it replaces the second entity words in each second text sequence to obtain at least one third text sequence; finally, it enhances the entity tags, that is, it adds at least one pair of entity tags to each third text sequence, and each text sample with at least one entity tag is considered as a text sample.

[0092] The knowledge distillation module inputs the constructed text samples into the teacher model and student model respectively. Then, it aligns the hidden layer representations of the teacher model and student model, as well as the entity representations of the teacher model and student model. The alignment of the hidden layer representations includes the word vectors of each word in the text samples output by the teacher model and student model, the intent of the text to be identified, and the slot filling results of each word. The alignment of the entity representations includes the feature vectors of each entity label output by the teacher model and student model.

[0093] See Figure 3 , Figure 3 This is a flowchart illustrating a method for constructing text samples according to an embodiment of this application. The text samples are used in subsequent model distillation processes. The method includes the following steps:

[0094] 301: Get the first text sequence.

[0095] 302: Perform text enhancement on the first text sequence to obtain at least one second text sequence, wherein each of the at least one second text sequence has the same intent as the first text sequence.

[0096] For example, a first word in the first text sequence and at least one second word corresponding to the first word are determined, wherein the first word is any word in the first text sequence other than stop words, entity words, and vertical category words; each of the at least one second word is used to replace the first word in the first text sequence to obtain at least one second text sequence. The first word in the first text sequence can be determined by dictionary matching or keyword recognition, etc. This application does not limit the method of identifying the first word.

[0097] It can be seen that the enhancement of the first text sequence is mainly to expand the corpus and simulate the scenario where people can express an intention in multiple ways in spoken language. Stop words are replaced because they do not help with intention recognition and therefore do not need to be replaced; however, replacing entity words and category words would change the intention of the first text sequence. Therefore, replacing words other than stop words, entity words, and category words in the first text sequence will not change the intention of the first text sequence, but will increase the richness of the training samples. Using such training samples to train the model results in a model with strong generalization ability.

[0098] It should be noted that the “training samples” and “text samples” mentioned in this application are essentially the same.

[0099] For example, the first word can be masked to obtain a fourth text sequence; then, word prediction can be performed based on the fourth text sequence to obtain at least one second word corresponding to the first word. Word prediction can be performed using a pre-trained model, such as the BERT model.

[0100] For example, the first text sequence is "I want to hear the confession balloon". The first word "want" can be masked to obtain the fourth text sequence "I [MASK] hear the confession balloon", where [MASK] indicates that the word at this position is masked. Then, the fourth text sequence is input into the BERT model to predict the word at the masked position, resulting in the second words "desire" and "want". Therefore, the second words can be used to replace the first word "want", resulting in the second text sequences "I want to hear the confession balloon" and "I want to hear the confession balloon". This expands a single text sequence into multiple text sequences with the same intent without changing the text intent, simulating different ways of expressing the same intent and increasing the richness of the text corpus.

[0101] 303: Replace the second entity word of each of the at least one second text sequence to obtain at least one third text sequence corresponding to each of the second text sequences.

[0102] For example, a second entity word is determined in each second text sequence, and the second entity word in each second text sequence is replaced to obtain at least one third text sequence corresponding to each second text sequence. Here, the entity word is the entity in the second text sequence, but it corresponds to different entity types. For example, the entity word "confession balloon" is an entity, and the entity types corresponding to this entity include music entities, video entities, audio entities, etc. Therefore, the entity words and entities mentioned in this application are essentially the same. The determination of the second entity word in each second text sequence can be achieved through a pre-trained model, such as a recurrent neural network, a convolutional neural network, a BERT model, etc.; or by searching for the second entity word contained in the second text sequence through a pre-constructed entity dictionary.

[0103] It can be seen that if the entity words are not replaced, the text sequences corresponding to the entity words in a certain field may be relatively few. After training the model with such training samples, the intention of the text sequences containing certain entity words in this field can be recognized, but the intention of the text sequences containing other entity words in this field cannot be recognized. For example, most of the entity words in the original training samples are "Qi Li Xiang". If the entity words are not replaced, the model trained subsequently may be able to recognize the intention of the text sequences containing "Qi Li Xiang", but cannot recognize the intention of the text sequences containing "Confession Balloon" and "Dong Feng Po". Therefore, during the process of constructing training samples, replacing the entity words can increase the number of text samples corresponding to each type of entity word. This can prevent the model from overfitting and improve the generalization ability of the model.

[0104] Therefore, at least one third entity word corresponding to the second entity word (in the same field) can be queried from the pre-constructed entity dictionary; then, each second entity word in each second text sequence is replaced with each of the at least one third entity word to obtain at least one third text sequence corresponding to each second text sequence.

[0105] For example, the second text sequences are respectively "I want to listen to Qi Li Xiang" and "I would like to listen to Qi Li Xiang", and the second entity word in each second text sequence is recognized as "Qi Li Xiang". The third entity words corresponding to "Qi Li Xiang" queried from the entity dictionary library are respectively "Confession Balloon" and "Dong Feng Po". Then each second entity word can be replaced with each third entity word to obtain the third text sequences as "I want to listen to Confession Balloon", "I want to listen to Confession Balloon", "I would like to listen to Confession Balloon", "I would like to listen to Confession Balloon", thereby constructing multiple training samples in the music field. After training the model with these training samples, "Confession Balloon" will also be regarded as a music entity during the recognition process, thus improving the generalization ability for the music field.

[0106] 304: Add at least one pair of entity tags to each of the at least one third text sequence corresponding to each second text sequence to obtain text samples.

[0107] Optionally, each entity word can correspond to one or more entity types. For example, the entity types corresponding to the entity word "Qi Li Xiang" can include music entity, video entity, novel entity, and so on. Therefore, adding at least one pair of entity tags for each third text sequence means adding at least one pair of entity tags for the third entity words in each third text sequence. Each pair of entity tags is used to represent an entity type corresponding to the third entity word, and two pairs of entity tags are used to represent two different entity types among the multiple entity types corresponding to the third entity word. In this application, an example is given of adding a pair of entity tags for each third entity word in each third text sequence. The method of adding more entity tags is similar and will not be described further.

[0108] In addition, one entity tag in each pair of entity tags is used to indicate the starting position of the third entity word in the third text sequence, and the other entity tag is used to indicate the ending position of the third entity word in the third text sequence. That is to say, one entity tag in the pair of entity tags is adjacent to the first word in the third entity word, and the other entity tag is adjacent to the last word in the third entity word. Moreover, the specific compositions of the two entity tags are different so that in the recognition process, these two entity tags with different compositions can be recognized, and all the content between the two entity tags is regarded as the entity word. In this way, during the slot filling process, the accuracy of slot filling can be improved.

[0109] Exemplarily, when the third text sequence contains vertical category words, at least one pair of entity tags is added to the third entity word in the third text sequence, and the entity types represented by each pair of entity tags are different from the entity types corresponding to the third entity word. And a third text sequence with entity tags added is used as a text sample.

[0110] It can be seen that adding entity tags representing different entity types for the third entity word is used to simulate the scenario where the text sequence contains noise. Training the model with such text samples enables the model to learn to preferentially perform intent recognition and slot filling through vertical category words and ignore the role of entity tags when the text sequence contains vertical category words. In this way, during the subsequent inference stage (the stage of applying the model), correct inference can still be achieved when the text sequence contains noise.

[0111] For example, the third text sequence is "I want to listen to the music 'Seven Mile Fragrance'". Since it contains the vertical category word "music", the entity word "Seven Mile Fragrance" can be identified as a music entity. A pair of entity tags that do not represent a music entity can be added to the entity word "Seven Mile Fragrance". For example, a pair of entity tags representing a video entity, [video name-B] and [video name-E], can be added. Then the text sample is "I want to listen to the music [video name-B] Seven Mile Fragrance [video name-E]". Of course, multiple pairs of entity tags that do not represent a music entity can also be added to this entity word. For example, a pair of entity tags representing a video entity, [video name-B] and [video name-E], and a pair of entity tags representing a novel entity, [novel name-B] and [novel name-E], can be added. Then the text sample is "I want to listen to the music [novel name-B][video name-B] Seven Mile Fragrance [video name-E][novel name-E]". By training the model with such text samples, it can learn to automatically ignore entity labels and prioritize slot filling and intent recognition using vertical keywords when vertical keywords are included.

[0112] It should be understood that in this application, the entity type represented by "song name" is music, "video name" is video, "voice name" is audio, "novel name" is a novel, and "media-name" represents an entity in the media domain. "B," "E," and "I" are used to indicate word position information; for example, "B" indicates the starting position, "E" indicates the ending position, and "I" indicates a word in the middle. Subsequent uses of terms similar to those explained here will not be further explained.

[0113] If the third text sequence does not contain vertical words, determine the third entity word in the third text sequence, obtain at least one entity type corresponding to the third entity word from the entity dictionary, and select at least one target entity type from the at least one entity type; then, add at least one pair of entity labels to the third text sequence to represent the at least one target entity type, and take a third text sequence with added entity labels as a text sample, wherein each pair of entity labels is used to represent a target entity type.

[0114] Since an entity word can correspond to one or more entity types, multiple text sequences containing the same entity word may have different intentions due to the different entity types they correspond to. Therefore, to simulate scenarios in people's daily lives where the same entity word expresses different intentions, at least one target entity type can be selected from at least one entity type corresponding to the third entity word. Then, an entity label can be added to each target entity type to generate multiple text samples.

[0115] In addition, in order to ensure the randomness of the selected target entity type, that is, to simulate the randomness of the occurrence of different entity types, at least one target entity type can be selected from the at least one entity type according to a preset sampling rate; or, the at least one target entity type can be randomly selected from the at least one entity type.

[0116] For example, the third text sequence is "I want to listen to 'Seven Mile Fragrance'". Since this third text sequence does not contain vertical category words, at least one entity type corresponding to the entity word "Seven Mile Fragrance" can be queried from the entity dictionary. For example, the entity type corresponding to the third entity word can be a music entity, a video entity, a novel entity, etc. Then, the selected target entity types are music entity and video entity, so different entity tags can be added to "Seven Mile Fragrance", resulting in two text samples: "I want to listen to [song name-B] Seven Mile Fragrance [song name-E]" and "I want to listen to [video name-B] Seven Mile Fragrance [video name-E]". This constructs two text samples containing the same entity word but with different intentions, and adds corresponding entity tags to each intention. In this way, the different entity tags indicate that the intentions corresponding to these two text samples are to play music and to play video, respectively. The distilled student model learns to identify the intention of text based on the entity tags of entity words even without vertical category words, thereby enabling the distilled student model to accurately identify the user's intention based on the entity tags corresponding to entity words under the same sentence structure and keywords.

[0117] Therefore, after performing the above processing on the first text sequence, it can be generalized into multiple text samples. For example, as shown... Figure 4 As shown, the first text sequence is "I want to hear the confession balloon". First, replace the first word "hear" to obtain m second words. Therefore, we can obtain m second text sequences. Figure 4 Only one second text sequence is shown; then, by replacing the second entity words in each second text sequence, n third entity words are obtained, which in turn yields n*m third text sequences. Figure 4Only one third text sequence is shown; then, k pairs of entity tags are added to each third text sequence, and finally one first text sequence is generalized into m * n * k text samples. Figure 4 Only one text sample is shown.

[0118] See Figure 5 , Figure 5 which is a schematic flowchart of a model distillation method provided by an embodiment of this application. The content identical to that in the Figure 3 shown embodiment will not be described herein again. The method of this embodiment includes the following steps. This method is applied to a model distillation device. The method includes the following steps:

[0119] 501: Obtain a text sample, where the text sample includes at least one pair of entity tags, and each pair of entity tags in the at least one pair of entity tags is used to characterize an entity type.

[0120] Wherein, the first entity word is any entity word in the text sample.

[0121] Wherein, this text sample is constructed by the method in Figure 2 and will not be elaborated here.

[0122] Wherein, each entity word can correspond to one or more entity types. For example, the entity word "Jasminum Sambac" can correspond to a music entity or a video entity.

[0123] Exemplarily, the entity type represented by each pair of entity words can be an entity type corresponding to the first entity word in this text sample, or not an entity type corresponding to the first entity word. From the method of constructing the text sample described above Figure 2 , it can be seen that if the text sample contains vertical category words, the entity type represented by each pair of entity tags is not the entity type corresponding to the first entity word, and if the text sample does not contain vertical category words, the entity type represented by each pair of entity tags is an entity type corresponding to the first entity word.

[0124] In addition, one entity tag in each pair of entity tags is used to indicate the starting position of the first entity word in this text sample, and the other entity tag is used to indicate the ending position of the first entity word in this text sample. That is to say, one entity tag in this pair of entity tags is adjacent to the first word in the first entity word, and the other entity tag is adjacent to the last word in the first entity word. And, the two entity tags in each pair of entity tags correspond to the same entity. However, the specific compositions of the two entity tags are different, so that in the recognition process, these two entity tags can be recognized, and the content between these two entity words can be used as an entity word, and in the process of slot filling, the accuracy of slot filling can be improved.

[0125] For example, the text sample is "I want to listen to 'Seven Mile Fragrance'", and the entity tags are used to represent music entities. The text sample after adding entity tags becomes "I want to listen to [song Name-B] Seven Mile Fragrance [song Name-E]". The entity tags are then [song Name-B] and [song Name-E], and the content between [song Name-B] and [song Name-E] consists entirely of entity words. Here, "song Name" indicates that the entity type represented by this pair of entity tags is a music entity, and "B" and "E" represent the first and second entity tags in this pair, respectively.

[0126] 502: Input the text samples into the teacher model and student model respectively to determine the loss set.

[0127] The teacher model is the trained model. For example, it is trained using the text samples described above, thus possessing the ability to recognize text intent through entity labels. For instance, the constructed text samples can be input into the teacher model to obtain predictions for slot filling for each word and the intent of the text sample. Then, based on the predictions and actual results (i.e., pre-labeled slot filling results) for each word, a sixth loss is obtained; based on the predictions and actual results (i.e., pre-labeled intent) for the intent of the text sample, a seventh loss is obtained; finally, the sixth and seventh losses are weighted to obtain an eighth loss. The model parameters are then adjusted using this eighth loss and gradient descent until the teacher model converges, completing the training of the teacher model.

[0128] It should be understood that after training the teacher model using such text samples, during the process of intent recognition of text to be recognized with added entity labels, if the text to be recognized contains vertical category words, then the intent of the text to be recognized is ignored, and the intent of the text to be recognized is recognized by the vertical category words. If the text to be recognized does not contain vertical category words, then the intent entity of the text to be recognized is determined according to the entity type represented by the entity label in the text to be recognized. It should be understood that if multiple pairs of entity labels are added, multiple intents corresponding to the multiple entity types represented by the multiple entity labels will be output, thereby achieving the output of all intents corresponding to all possible entity types of the entity when the intent of the entity is unclear, that is, the corresponding entity type is uncertain.

[0129] For example, the loss set includes a first loss, a second loss, and a third loss.

[0130] The first loss is determined based on the difference between the first intent and the second intent, which are obtained by the teacher model and the student model respectively performing intent recognition on the text sample.

[0131] For example, such as Figure 6 As shown, text samples [x1, x2, x3, ...] are input into the teacher model and student model respectively, and start symbols [CLS] and end symbols [SEP] are inserted into the text samples as start and end signals for recognition. In subsequent processing, the start and end symbols are treated as words in the text sample. The slot filling result [y] of the slot filling layer is processed by the intent classification layer of the teacher model. l ,y s 1,y s 2,y s 3,……,y s n Intent mapping (i.e., post-processing of slot filling results) is performed to obtain the first intent. This involves mapping the slot filling results to the corresponding strings based on the mapping relationship between the slot filling results and the strings. For example, if the character corresponding to the starting word of a music entity is pre-set to the number 1, then the slot filling result for that word can be mapped to 1. Therefore, after mapping the slot filling result of each word to its corresponding character, the first intent (string) corresponding to the text sample can be obtained. The slot filling results [k] of the slot filling layer are then processed by the graph classification layer of the student model. l ,k s 1,k s 2,k s 3, ..., k s n Intent mapping (i.e., post-processing of slot filling results) is performed to obtain the second intent; finally, the first intent is used as the supervision label of the second intent, and the first loss is determined based on the difference between the first intent and the second intent.

[0132] The second loss is determined based on the difference between the first word vector and the second word vector corresponding to each pair of entity labels in the at least one pair of entity labels. The first word vector and the second word vector corresponding to each pair of entity labels are obtained by encoding each pair of entity labels by the teacher model and the student model, respectively.

[0133] Specifically, similar to encoding text or characters, each entity label in at least one pair of entity labels can be encoded through the encoding layer of the teacher model to obtain a first feature vector corresponding to each entity label, and each entity label can be encoded through the student model to obtain a second feature vector corresponding to each entity label. Furthermore, during the encoding process, the semantic information of the entity labels is encoded into the word vector of each word. For example, this encoding layer can be a BERT-based encoding layer. Therefore, during the encoding of each word, the first word vector of each word and the first feature vector corresponding to each entity label can be fused based on the self-attention mechanism of the BERT model, resulting in a third word vector for each word, i.e., [CLS, h1, h2, h3, ..., SEP], which contains the semantic information of each word in the text sample. Figure 6 As shown, a pair of entity labels LA and LB are added to the entity word x3. Then, the entity labels LA and LB are encoded through the encoding layer of the teacher model to obtain the first feature vector corresponding to LA and the first feature vector corresponding to LB. The entity labels LA and LB are encoded through the student model to obtain the first feature vector corresponding to LA and the second feature vector corresponding to LB. Then, the first feature vector of each entity label is used as the supervision label of the second feature vector corresponding to that entity label, and the difference between the two (e.g., Euclidean distance) is calculated to obtain multiple differences. Finally, the average of the multiple differences is used as the second loss.

[0134] The third loss is determined based on the difference between the first slot filling result and the second slot filling result for each word in the text sample. The first slot filling result and the second slot filling result for each word are obtained by filling the slots for the word by the teacher model and the student model, respectively.

[0135] Specifically, the slot filling layer of the teacher model fills the slots for each word using the output of the attention layer, obtaining the first slot filling result for each word. For example, the target feature vector of each word can be input into a fully connected layer and subjected to softmax normalization to obtain a normalized feature vector for each word. The slot corresponding to the largest dimension in the normalized feature vector is then mapped to the slot filling result for that word. For instance, if the normalized feature vector of a word is [0.11, 0.21, 0.47, 0.11, 0.10], the maximum probability is determined to be 0.47, and the slot filling result corresponding to this dimension is "B-song name". Therefore, the slot filling result for that word is determined to be "B-song". The name is used to fill the slots of each word by applying the output of the attention layer to the slot filling layer of the student model, resulting in the second slot filling result for each word. Then, the first slot filling result for each word is used as the supervision label for the second slot filling result corresponding to that word, and the difference between the first and second slot filling results for each word (e.g., Euclidean distance) is determined to obtain multiple differences. Finally, the average of these multiple differences is used as the second loss.

[0136] For example, such as Figure 6 As shown, the output of the attention layer can be obtained through the slot-filling layer of the teacher model, i.e., the first target word vector of each word [c]. l ,c 1 ,c 2 ,c 3 ,……,c n For each word in the text sample, slot filling is performed to obtain the first slot filling result for each word in the text sample. l ,y s 1,y s 2,y s 3,……y s n ]; The output of the attention layer through the slot-filling layer of the student model, i.e., the target feature vector of each word [t l ,t 1 ,t 2 ,t 3 ,……,t n ], fill the slots for each word in the text sample to obtain the second slot filling result [k] for each word in the text sample. l ,k s 1,k s 2,k s 3,……k s n ]. Among them, y l kl Fill the first slot corresponding to the start symbol [CLS] and end symbol [SEP] in the text sample with the result, y s n and k s n The second slot filling results corresponding to the start symbol [CLS] and end symbol [SEP] in the text sample are then determined. Finally, the difference between the first and second slot filling results for each word is determined, resulting in multiple differences. The average of these multiple differences is used as the second loss.

[0137] 503: Adjust the model parameters of the student model according to the loss set to train the student model.

[0138] For example, the first loss, the second loss, and the third loss are weighted to obtain the first target loss; the model parameters of the student model are adjusted according to the first target loss and the gradient descent method until the student model converges, thus completing the training of the student model.

[0139] As can be seen in this embodiment, during the model distillation process, the recognition results of the teacher model on entity labels are aligned with those of the student model. Since the entity label represents the type of entity, the student model is able to perform intent recognition and slot filling by combining entity labels, thereby improving the accuracy of the student model in intent recognition and slot filling, and accurately identifying the intent corresponding to texts with the same sentence structure and keywords. In addition, the student model can accurately identify the position of the entity based on the entity label, which is beneficial to the rapid convergence of the student model. Finally, the training sample corpus has high richness, and there are a large number of training samples in each domain, which can improve the generalization ability of the model.

[0140] In some possible implementations, the loss set also includes a fourth loss, which is determined based on the difference between the third and fourth word vectors corresponding to each word in the text sample, which are obtained by word embedding of each word by the teacher model and the student model, respectively.

[0141] Specifically, the encoding layer of the teacher model can encode each word and entity label in the text sample to obtain the third word vector for each word. For example, the encoding layer of the teacher model can encode each word to obtain the first word vector for each word, and encode each entity label to obtain the first feature vector for each entity label. Then, through the self-attention mechanism in this encoding layer, the first word vector of each word and the first feature vector of each entity label are fused to obtain the third word vector for each word. Similarly, the encoding layer of the student model can encode each word in the text sample to obtain the fourth word vector for each word. Each word is encoded to obtain its second word vector. Each entity label is encoded to obtain its second feature vector. Then, through the self-attention mechanism in this encoding layer, the first word vector of each word and the second feature vector of each entity label are fused to obtain the fourth word vector of each word. The encoding of each word can be performed using the BERT model, which will not be elaborated further. Finally, the third word vector of each word is used as the supervision label for the fourth word vector of that word, and the difference between the third word vector and the fourth word vector of each word (e.g., Euclidean distance) is calculated to obtain multiple differences. The average of these multiple differences is used as the fourth loss.

[0142] For example, such as Figure 6 As shown, text samples [x1,x2,x3,...] are input into the teacher model and student model, respectively, where x1, x2, and x3 are words in the text samples. Each word [CLS,x1,x2,x3,...,SEP] in the text sample is encoded through the encoding layer of the teacher model, resulting in the third word vector [CLS,h1,h2,h3,...,SEP] corresponding to each word. Similarly, each word [x1,x2,x3,...] in the text sample is encoded through the encoding layer of the student model, resulting in the fourth word vector [CLS,z1,z2,z3,...,SEP]. The differences between the third and fourth word vectors corresponding to each word in the text sample are determined, resulting in multiple differences, specifically calculated between x1 and z1, x2 and z2, x3 and z3,... Finally, the average of these multiple differences is used as the second loss.

[0143] Therefore, the model parameters of the student model can be adjusted according to the first loss, second loss, third loss and fourth loss to train the student model. That is, the first loss, second loss, third loss and fourth loss are weighted to obtain the second target loss. The model parameters of the student model are adjusted according to the second target loss until the student model converges, thus completing the training of the student model.

[0144] In some possible implementations, the loss set also includes a fifth loss, which is determined based on the difference between the first target word vector and the second target word vector corresponding to each word in the text sample. The first target word vector and the second target word vector corresponding to each word in the text sample are obtained by performing attention processing on the third word vector and the fourth vector corresponding to each word through the attention layers in the teacher model and the student model, respectively.

[0145] like Figure 6 As shown, the third word vector of each word output by the encoding layer of the teacher model is weighted by attention through the attention layer to obtain the first target word vector for each word. Similarly, the fourth word vector of each word output by the encoding layer of the student model is weighted by attention through the attention layer to obtain the second target word vector for each word. The self-attention weighting of the word vectors through the attention layer is a current technique and will not be elaborated further; for example, a self-attention mechanism can be used. Finally, the first target word vector of each word is used as the supervision label for the corresponding second target word vector, and the difference (e.g., Euclidean distance) between the first target word vector and the corresponding second target word vector is calculated, resulting in multiple differences. The average of these multiple differences is used as the fifth loss.

[0146] Therefore, the model parameters of the student model can be adjusted based on the first, second, third, fourth, and fifth losses to train the student model. Specifically, the first, second, third, fourth, and fifth losses are weighted to obtain the third target loss; the model parameters of the student model are then adjusted based on this third target loss until the student model converges, completing the training of the student model.

[0147] It should be understood that although both the teacher and student models have encoding layers, attention layers, slot-filling layers, and intent classification layers, the number of network layers and / or model parameters in the teacher and student models can be the same or different. This application does not impose any restrictions on this. For example, the encoding layer in the teacher model may consist of x sub-network layers (such as convolutional layers), while the encoding layer in the student model may consist of y sub-network layers, where x is greater than y. Generally, the student model has fewer network layers and fewer network parameters than the teacher model.

[0148] See Figure 7 , Figure 7 This is a schematic diagram of the structure of a dialogue system provided in an embodiment of this application. The dialogue system 700 includes an ASR (Acoustic Recognition) module 710, a NLU (Natural Language Understanding) module 720, a DM (Dialogue Management) module 730, and a TTS (Text-to-Speech) module 740, wherein:

[0149] The speech recognition module 710 is used to receive speech information and convert the speech information into text to be recognized;

[0150] The language understanding module 720 is used to fill the slots in the text to be recognized using the student model trained as described above, and to obtain the slot filling result for each word in the text to be recognized.

[0151] The dialogue management module 730 is used to perform post-processing based on the slot filling results of each word to obtain the intent of the text to be recognized. The post-processing of the slot filling results of each word is existing technology and will not be described further.

[0152] The text-to-speech module 740 is used to convert the intent of the text to be recognized into corresponding audio information and output the audio information.

[0153] See Figure 8 , Figure 8 This is a schematic diagram of a model distillation apparatus provided in an embodiment of this application. The model distillation apparatus 800 includes a transceiver unit 801 and a processing unit 802, wherein:

[0154] The transceiver unit 801 is used to acquire text samples, wherein the text samples include at least one pair of entity tags, and each pair of entity tags is used to characterize an entity type.

[0155] Processing unit 802 is configured to input the text samples into a teacher model and a student model respectively, and determine a loss set, the loss set including a first loss, a second loss, and a third loss. The first loss is determined based on the difference between a first intent and a second intent, which are obtained by the teacher model and the student model, respectively, through intent recognition of the text samples based on at least one pair of entity labels. The second loss is determined based on the difference between a first feature vector and a second feature vector corresponding to each entity label in the at least one pair of entity labels, which are obtained by encoding each entity label by the teacher model and the student model respectively. The third loss is determined based on the difference between a first slot filling result and a second slot filling result corresponding to each word in the text sample, wherein the first slot filling result and the second slot filling result corresponding to each word are obtained by the teacher model and the student model, respectively, through slot filling of each word based on at least one pair of entity labels.

[0156] The processing unit 802 is further configured to adjust the model parameters of the student model according to the loss set in order to train the student model.

[0157] In some possible implementations, one entity label in each pair of entity labels is used to indicate the starting position of the first entity word in the text sample, and the other entity label is used to indicate the ending position of the first entity word in the text sample; and the entity type represented by each entity label is the same as or different from the entity type corresponding to the first entity word.

[0158] In some possible implementations, the loss set further includes a fourth loss, which is determined based on the difference between the third and fourth word vectors corresponding to each word in the text sample, the third and fourth word vectors being obtained by encoding each word by the teacher model and the student model, respectively.

[0159] In some possible implementations, the loss set further includes a fifth loss, which is determined based on the difference between the first target word vector and the second target word vector corresponding to each word in the text sample. The first target word vector and the second target word vector corresponding to each word in the text sample are obtained by attention weighting the third word vector and the fourth vector corresponding to each word through the attention mechanism in the teacher model and the student model, respectively.

[0160] In some possible implementations, the processing unit 802 is specifically configured to: adjust the model parameters of the student model based on the loss set.

[0161] The first loss, the second loss, the third loss, the fourth loss, and the fifth loss are weighted to obtain the target loss;

[0162] The model parameters of the student model are adjusted based on the target loss.

[0163] In some possible implementations, before the transceiver unit acquires the text sample, the transceiver unit 801 is also used to acquire the first text sequence;

[0164] The processing unit 802 is further configured to: have each second text sequence in the text sequence have the same intent as the first text sequence; replace the second entity word of each second text sequence in the at least one second text sequence to obtain at least one third text sequence corresponding to each second text sequence; and add at least one pair of entity tags to each third text sequence in the at least one third text sequence corresponding to each second text sequence to obtain the text sample.

[0165] In some possible implementations, in order to perform text enhancement on the first text sequence to obtain at least one second text sequence, the processing unit 802 is specifically configured to:

[0166] Determine at least one second word corresponding to the first word, wherein the first word is any word in the first text sequence other than stop words, entity words, and vertical category words;

[0167] The first word in the first text sequence is replaced by each of the at least one second word to obtain the at least one second text sequence.

[0168] In some possible implementations, in determining at least one second word corresponding to the first word, the processing unit 802 is specifically configured to:

[0169] The first word in the first text sequence is masked to obtain the fourth text sequence;

[0170] Based on the fourth text sequence, word prediction is performed to obtain at least one second word corresponding to the first word.

[0171] In some possible implementations, in replacing the second entity word of each of the at least one second text sequence to obtain at least one third text sequence corresponding to each second text sequence, the processing unit 802 is specifically configured to:

[0172] Obtain at least one third entity word from a pre-constructed entity dictionary that corresponds to a second entity word in each of the at least one second text sequences, wherein each third entity word belongs to the same domain as the second entity word;

[0173] Each third entity word in the at least one third entity word is used to replace the second entity word in each second text sequence to obtain at least one third text sequence corresponding to each second text sequence.

[0174] In some possible implementations, in obtaining the text sample by adding at least one pair of entity tags to each of the at least one third text sequence corresponding to each second text sequence, the processing unit 802 is specifically used for:

[0175] In the case that each third text sequence contains vertical words, at least one pair of entity labels is added to the third entity words in each third text sequence to obtain the text sample, wherein the entity type represented by each pair of entity labels is different from the entity type corresponding to the third entity words in the third text sequence;

[0176] In the case that each third text sequence does not contain vertical words, a third entity word in each third text sequence is determined, and at least one entity type corresponding to the third entity word is determined. At least one target entity type is selected from the at least one entity type, and at least one pair of entity labels is added to each third text sequence to represent the at least one target entity type, thereby obtaining the text sample. Each pair of entity labels is used to represent a target entity type.

[0177] See Figure 9 , Figure 9 This is a schematic diagram of a model distillation apparatus provided in an embodiment of this application. The model distillation apparatus 900 includes a memory 901, a processor 902, and a transceiver 903. They are connected via a bus 904. The memory 901 is used to store relevant instructions and data, and can transmit the stored data to the processor 902.

[0178] Processor 902 is used to read relevant instructions from memory 901 and perform the following operations:

[0179] The transceiver 903 is controlled to acquire a text sample, the text sample including at least one pair of entity tags, each pair of entity tags being used to characterize an entity type;

[0180] The text samples are input into a teacher model and a student model respectively to determine a loss set, which includes a first loss, a second loss, and a third loss. The first loss is determined based on the difference between a first intent and a second intent, which are obtained by the teacher model and the student model, respectively, through intent recognition of the text samples based on at least one pair of entity labels. The second loss is determined based on the difference between a first feature vector and a second feature vector corresponding to each entity label in the at least one pair of entity labels, which are obtained by encoding each entity label by the teacher model and the student model respectively. The third loss is determined based on the difference between a first slot filling result and a second slot filling result corresponding to each word in the text sample, wherein the first slot filling result and the second slot filling result corresponding to each word are obtained by the teacher model and the student model, respectively, through slot filling of each word based on at least one pair of entity labels.

[0181] The model parameters of the student model are adjusted according to the loss set in order to train the student model.

[0182] Specifically, the aforementioned processor 902 can be Figure 8 The transceiver 903 in the processing unit 802 of the model distillation apparatus 800 of the illustrated embodiment may be... Figure 8 The transceiver unit 801 of the model distillation apparatus 800 in the embodiment described above.

Claims

1. A model distillation method, characterized in that, include: Obtain a text sample, wherein the text sample includes at least one pair of entity labels, and each pair of entity labels is used to represent an entity type; The text samples are input into a teacher model and a student model respectively to determine a loss set, which includes a first loss, a second loss, and a third loss. The first loss is determined based on the difference between a first intent and a second intent, which are obtained by the teacher model and the student model, respectively, through intent recognition of the text samples based on at least one pair of entity labels. The second loss is determined based on the difference between a first feature vector and a second feature vector corresponding to each entity label in the at least one pair of entity labels, which are obtained by encoding each entity label by the teacher model and the student model respectively. The third loss is determined based on the difference between a first slot filling result and a second slot filling result corresponding to each word in the text sample, wherein the first slot filling result and the second slot filling result corresponding to each word are obtained by the teacher model and the student model, respectively, through slot filling of each word based on at least one pair of entity labels. The model parameters of the student model are adjusted according to the loss set in order to train the student model.

2. The method according to claim 1, characterized in that, In each pair of entity tags, one entity tag indicates the starting position of the first entity word in the text sample, and the other entity tag indicates the ending position of the first entity word in the text sample; and the entity type represented by each entity tag is the same as or different from the entity type corresponding to the first entity word.

3. The method according to claim 1 or 2, characterized in that, The loss set also includes a fourth loss, which is determined based on the difference between the third word vector and the fourth word vector corresponding to each word in the text sample. The third word vector and the fourth word vector corresponding to each word in the text sample are obtained by encoding each word by the teacher model and the student model, respectively.

4. The method according to claim 3, characterized in that, The loss set also includes a fifth loss, which is determined based on the difference between the first target word vector and the second target word vector corresponding to each word in the text sample. The first target word vector and the second target word vector corresponding to each word in the text sample are obtained by attention weighting the third word vector and the fourth vector corresponding to each word through the attention mechanism in the teacher model and the student model, respectively.

5. The method according to claim 4, characterized in that, The step of adjusting the model parameters of the student model based on the loss set includes: The first loss, the second loss, the third loss, the fourth loss, and the fifth loss are weighted to obtain the target loss; The model parameters of the student model are adjusted based on the target loss.

6. The method according to any one of claims 1-5, characterized in that, Before obtaining text samples, the method further includes: Obtain the first text sequence; The first text sequence is augmented to obtain at least one second text sequence, wherein each of the at least one second text sequence has the same intent as the first text sequence; Replace the second entity word in each of the at least one second text sequence to obtain at least one third text sequence corresponding to each of the at least one second text sequence; At least one pair of entity labels are added to each of the at least one third text sequence corresponding to each second text sequence to obtain the text sample.

7. The method according to claim 6, characterized in that, The step of enhancing the first text sequence to obtain at least one second text sequence includes: Determine at least one second word corresponding to the first word, wherein the first word is any word in the first text sequence other than stop words, entity words, and vertical category words; The first word in the first text sequence is replaced by each of the at least one second word to obtain the at least one second text sequence.

8. The method according to claim 6 or 7, characterized in that, The determination of at least one second word corresponding to the first word includes: The first word in the first text sequence is masked to obtain the fourth text sequence; Based on the fourth text sequence, word prediction is performed to obtain at least one second word corresponding to the first word.

9. The method according to any one of claims 6-8, characterized in that, The step of replacing the second entity word in each of the at least one second text sequence to obtain at least one third text sequence corresponding to each of the at least one second text sequence includes: Obtain at least one third entity word from a pre-constructed entity dictionary that corresponds to a second entity word in each of the at least one second text sequences, wherein each third entity word belongs to the same domain as the second entity word; Each third entity word in the at least one third entity word is used to replace the second entity word in each second text sequence to obtain at least one third text sequence corresponding to each second text sequence.

10. The method according to any one of claims 6-9, characterized in that, The step of adding at least one pair of entity labels to each of the at least one third text sequence corresponding to each second text sequence to obtain the text sample includes: In the case that each third text sequence contains vertical words, at least one pair of entity labels is added to the third entity words in each third text sequence to obtain the text sample, wherein the entity type represented by each pair of entity labels is different from the entity type corresponding to the third entity words in the third text sequence; In the case that each third text sequence does not contain vertical words, a third entity word in each third text sequence is determined, and at least one entity type corresponding to the third entity word is determined. At least one target entity type is selected from the at least one entity type, and at least one pair of entity labels is added to each third text sequence to represent the at least one target entity type, thereby obtaining the text sample. Each pair of entity labels is used to represent a target entity type.

11. A model distillation apparatus, characterized in that, include: A transceiver unit is used to acquire text samples, wherein the text samples include at least one pair of entity tags, and each pair of entity tags is used to characterize an entity type. A processing unit is configured to input the text samples into a teacher model and a student model respectively, and determine a loss set, the loss set including a first loss, a second loss, and a third loss. The first loss is determined based on the difference between a first intent and a second intent, which are obtained by the teacher model and the student model respectively, and by performing intent recognition on the text samples according to at least one pair of entity labels. The second loss is determined based on the difference between a first feature vector and a second feature vector corresponding to each entity label in the at least one pair of entity labels, which are obtained by encoding each entity label by the teacher model and the student model respectively. The third loss is determined based on the difference between a first slot filling result and a second slot filling result corresponding to each word in the text sample, wherein the first slot filling result and the second slot filling result corresponding to each word are obtained by the teacher model and the student model respectively, and by performing slot filling on each word according to at least one pair of entity labels. The processing unit is further configured to adjust the model parameters of the student model according to the loss set in order to train the student model.

12. The apparatus according to claim 11, characterized in that, In each pair of entity tags, one entity tag indicates the starting position of the first entity word in the text sample, and the other entity tag indicates the ending position of the first entity word in the text sample; and the entity type represented by each entity tag is the same as or different from the entity type corresponding to the first entity word.

13. The apparatus according to claim 11 or 12, characterized in that, The loss set also includes a fourth loss, which is determined based on the difference between the third word vector and the fourth word vector corresponding to each word in the text sample. The third word vector and the fourth word vector corresponding to each word in the text sample are obtained by encoding each word by the teacher model and the student model, respectively.

14. The apparatus according to claim 13, characterized in that, The loss set also includes a fifth loss, which is determined based on the difference between the first target word vector and the second target word vector corresponding to each word in the text sample. The first target word vector and the second target word vector corresponding to each word in the text sample are obtained by attention weighting the third word vector and the fourth vector corresponding to each word through the attention mechanism in the teacher model and the student model, respectively.

15. The apparatus according to claim 14, characterized in that, In adjusting the model parameters of the student model based on the loss set, the processing unit is specifically used for: The first loss, the second loss, the third loss, the fourth loss, and the fifth loss are weighted to obtain the target loss; The model parameters of the student model are adjusted based on the target loss.

16. The apparatus according to any one of claims 11-15, characterized in that, Before the transceiver unit acquires the text sample, the transceiver unit is also used to acquire a first text sequence; The processing unit is further configured to: have each second text sequence in the text sequence have the same intent as the first text sequence; replace the second entity word in each of the at least one second text sequence to obtain at least one third text sequence corresponding to each of the second text sequences; and add at least one pair of entity tags to each of the at least one third text sequence corresponding to each of the second text sequences to obtain the text sample.

17. The apparatus according to claim 16, characterized in that, In performing text enhancement on the first text sequence to obtain at least one second text sequence, the processing unit is specifically configured to: Determine at least one second word corresponding to the first word, wherein the first word is any word in the first text sequence other than stop words, entity words, and vertical category words; The first word in the first text sequence is replaced by each of the at least one second word to obtain the at least one second text sequence.

18. The apparatus according to claim 16 or 17, characterized in that, In determining at least one second word corresponding to the first word, the processing unit is specifically configured to: The first word in the first text sequence is masked to obtain the fourth text sequence; Based on the fourth text sequence, word prediction is performed to obtain at least one second word corresponding to the first word.

19. The apparatus according to any one of claims 16-18, characterized in that, In replacing the second entity word of each of the at least one second text sequence to obtain at least one third text sequence corresponding to each second text sequence, the processing unit is specifically configured to: Obtain at least one third entity word from a pre-constructed entity dictionary that corresponds to a second entity word in each of the at least one second text sequences, wherein each third entity word belongs to the same domain as the second entity word; Each third entity word in the at least one third entity word is used to replace the second entity word in each second text sequence to obtain at least one third text sequence corresponding to each second text sequence.

20. The apparatus according to any one of claims 16-19, characterized in that, In obtaining the text sample by adding at least one pair of entity labels to each of at least one third text sequence corresponding to each second text sequence, the processing unit is specifically used for: In the case that each third text sequence contains vertical words, at least one pair of entity labels is added to the third entity words in each third text sequence to obtain the text sample, wherein the entity type represented by each pair of entity labels is different from the entity type corresponding to the third entity words in the third text sequence; In the case that each third text sequence does not contain vertical words, a third entity word in each third text sequence is determined, and at least one entity type corresponding to the third entity word is determined. At least one target entity type is selected from the at least one entity type, and at least one pair of entity labels is added to each third text sequence to represent the at least one target entity type, thereby obtaining the text sample. Each pair of entity labels is used to represent a target entity type.

21. A model distillation apparatus, characterized in that, The device includes a processor connected to a memory for storing computer programs, and the processor is configured to execute the computer programs stored in the memory to cause the terminal device to perform the method as described in any one of claims 1 to 10.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method as described in any one of claims 1 to 10.

Images