Intention recognition model training method, intention recognition method and intention recognition device

Abstract

The invention provides an intention recognition model training method and device, and the method comprises the steps: obtaining first training data, inputting the first training data into a first model and a second model, taking the first model as a teacher model, and taking the second model as a student model, performing distillation calculation on the first target layer structures of the first model and the second model by utilizing a knowledge distillation algorithm, so that the second model is used for determining an intention classification label corresponding to the text data; the first target layer structure comprises an embedded layer, a multi-head attention layer, an encoder layer and a dense layer. Therefore, distillation and mapping can be carried out at the beginning stage of the algorithm, namely, an embedded layer, the middle part of the algorithm, namely, a multi-head attention layer and an encoder layer, and the ending part of the algorithm, namely, a dense layer, so that the distillation of the algorithm is more thorough, and the student model and the teacher model are closer to each other; the intention recognition accuracy of the second model is higher, and the model recognition speed is higher.

Description

Technical Field

[0001] The present application relates to the field of model training technology, and in particular to an intent recognition model training method, an intent recognition method and a device. Background Art

[0002] An intelligent outbound call system is a platform used to perform calling tasks on customers. It generally integrates automatic speech recognition (ASR) technology, text to speech (TTS) technology, and natural language understanding (NLU) technology. In actual applications, an intelligent outbound call system can automatically dial a user's phone, play pre-recorded voice to the user through a computer, and conduct human-computer dialogue with the customer. Afterwards, the intelligent outbound call system can also perform text recognition and other processing on the audio generated by the call to analyze the customer's intentions.

[0003] Current intent recognition methods are mainly based on native Bert-related algorithms or algorithms such as TextCNN. Although native Bert-related algorithms have impressive effects, they face efficiency issues when deployed in complex systems, especially when the hardware conditions of the deployment equipment are limited. For example, when the hardware conditions are low and the concurrency is high, the algorithm's reasoning efficiency drops significantly. For algorithms such as TextCNN, although they are highly efficient and can support high concurrency, their recognition effect is significantly lower than that of Bert, especially when faced with complex intent situations. Summary of the invention

[0004] The embodiments of the present application provide an intent recognition model training method, an intent recognition method and a device to solve the problem of poor recognition efficiency and accuracy of traditional methods.

[0005] In a first aspect, an embodiment of the present application provides an intent recognition model training method, comprising: obtaining first training data, the first training data comprising multiple text data with intent classification labels; inputting the first training data into a first model and a second model, using the first model as a teacher model and the second model as a student model, and using a knowledge distillation algorithm to perform distillation calculations on the first target layer structures of the first model and the second model, so that the second model is used to determine the intent classification label corresponding to the text data; wherein the model parameter amount of the first model is greater than that of the second model, and the first model has been trained with the first training data, and the first target layer structure is an embedding layer, a multi-head attention layer, an encoder layer, and a dense layer.

[0006] In one feasible manner, before the step of inputting the first training data into the first model and the second model, taking the first model as the teacher model and the second model as the student model, and performing distillation calculation on the first target layer structure of the first model and the second model using the knowledge distillation algorithm, it also includes: obtaining the second training data, the second training data including a plurality of text data without the intent classification label; inputting the second training data into the first model and the second model, taking the first model as the teacher model and the second model as the student model, and performing distillation calculation on the second target layer structure of the first model and the second model using the knowledge distillation algorithm, so that the second model is used to understand the semantics of the text data, and the second target layer structure is a multi-head attention layer and an encoder layer.

[0007] In one achievable method, the first training data and the second training data also include coding ID information corresponding to the text data; before the second training data is input into the first model and the second model, the first model is used as the teacher model and the second model is used as the student model, and the step of using the knowledge distillation algorithm to perform distillation calculations on the second target layer structure of the first model and the second model also includes: for each text data of the first training data and the second training data, determining whether the text length of the text data exceeds a first preset length; if the text length of the text data exceeds the first preset length, truncating the text data so that the text length of the text data is equal to the first preset length; querying the coding mapping table to determine the coding ID information corresponding to each coding unit in the text data to form a first coding sequence.

[0008] In one achievable method, after the step of determining whether the character length of the text data exceeds a first preset length, the method further includes: if the character length of the text data does not exceed the first preset length, querying a coding mapping table to determine the coding ID information corresponding to each coding unit in the text data to form a subsequence; adding a placeholder value at the end of the subsequence to form a second coding sequence, and the character length of the second coding sequence is equal to the character length of the first coding sequence.

[0009] In one implementable manner, the first training data and the second training data also include mask ID information and symbol type coding information corresponding to the text data; wherein the mask ID information is equal to a first default value; after the step of forming a first coding sequence, it also includes: based on each bit of coding ID information in the first coding sequence, determining the symbol type coding information; wherein the symbol type coding information corresponding to the coding ID information is a second default value; after the step of forming a second coding sequence, it also includes: based on each bit of coding ID information and a placeholder value in the second coding sequence, determining the symbol type coding information; wherein the symbol type coding information corresponding to the placeholder value is a third default value.

[0010] In an achievable manner, the model parameters include at least the dimension Embedding-size of the embedding layer and the related parameters of the encoder layer, and the related parameters of the encoder layer include at least the dimension hidden_size of the hidden layer, the number of attention heads num_attention_heads, the number of hidden layers num_hidden_layers, and the dimension intermediate_size of the intermediate layer of the feedforward neural network; wherein, the dimension Embedding-size of the embedding layer of the first model is 768, and the dimension Embedding-size of the embedding layer of the second model is 192; the number of attention heads num_attention of the first model is _heads is 768, and the number of attention heads num_attention_heads of the second model is 192; the number of hidden layers num_hidden_layers of the first model is 12, and the number of hidden layers num_hidden_layers of the second model is 8; the number of hidden layers num_hidden_layers of the first model is 12, and the number of hidden layers num_hidden_layers of the second model is 2, the intermediate layer dimension intermediate_size of the feedforward neural network of the first model is 3072, and the intermediate layer dimension intermediate_size of the feedforward neural network of the second model is 768.

[0011] In one implementable manner, the knowledge distillation algorithm includes at least multiple loss functions; the step of using the knowledge distillation algorithm to perform distillation calculations on the first target layer structures of the first model and the second model includes: using the target loss function to calculate the output difference between the first model and each corresponding first target layer structure of the second model; wherein the target loss function is one of the multiple loss functions, and the target loss function corresponds to the type of the first target layer structure; performing weighted summation on the output differences to obtain the distillation loss; and using the distillation loss to modify the target parameters of the second model, wherein the target parameters are the weights and biases of the second model.

[0012] In the second aspect, an embodiment of the present application provides an intent recognition method, including: obtaining a speech segment to be recognized; converting the speech segment into first text data; inputting the first text data into a second model, determining the intent classification label corresponding to the speech segment, so as to determine the intent corresponding to the speech segment; wherein the second model is pre-trained by the intent recognition model training method in the first aspect and its various implementation methods.

[0013] In a third aspect, an embodiment of the present application provides an intent recognition model training device, comprising: a first acquisition module, used to acquire first training data, the first training data comprising multiple text data with intent classification labels; a strong distillation module, used to input the first training data into a first model and a second model, with the first model as a teacher model and the second model as a student model, and using a knowledge distillation algorithm to perform distillation calculations on the first target layer structures of the first model and the second model, so that the second model is used to determine the intent classification label corresponding to the text data; wherein the model parameter amount of the first model is greater than that of the second model, and the first model has been trained with the first training data, and the first target layer structure is an embedding layer, a multi-head attention layer, an encoder layer and a dense layer.

[0014] In a fourth aspect, an embodiment of the present application provides an intent recognition device, comprising: a second acquisition module, used to acquire a speech segment to be recognized; a conversion module, used to convert the speech segment into first text data; an intent recognition module, used to input the first text data into a second model, determine the intent classification label corresponding to the speech segment, so as to determine the intent corresponding to the speech segment; wherein the second model is pre-trained by the intent recognition model training method in the first aspect and its various implementation methods.

[0015] From the above content, it can be seen that the embodiment of the present application provides an intent recognition model training method, an intent recognition method and a device, the intent recognition model training method includes obtaining first training data, the first training data includes multiple text data with intent classification labels; inputting the first training data into the first model and the second model, taking the first model as the teacher model and the second model as the student model, and using the knowledge distillation algorithm to perform distillation calculation on the first target layer structure of the first model and the second model, so that the second model is used to determine the intent classification label corresponding to the text data; wherein the model parameter amount of the first model is greater than that of the second model, and the first model has been trained with the first training data, and the first target layer structure is an embedding layer, a multi-head attention layer, an encoder layer and a dense layer. In this way, distillation and mapping can be performed at the beginning stage of the algorithm (embedding layer), the middle part of the algorithm (multi-head attention layer and encoder layer) and the end part of the algorithm (dense layer), thereby making the distillation of the algorithm more thorough, ensuring that the student model and the teacher model are closer, and making the intent recognition accuracy of the second model higher and the model recognition speed faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a scenario applicable to an embodiment of the present application;

[0017] Figure 2 An overall flow chart of the intent recognition model training method and the intent recognition method provided in the embodiments of the present application;

[0018] Figure 3 A flowchart of the intent recognition model training method provided in an embodiment of the present application;

[0019] Figure 4 A text ID mapping table provided for an embodiment of the present application;

[0020] Figure 5 A schematic diagram of fine-tuning the first model provided in an embodiment of the present application;

[0021] Figure 6 A schematic diagram of pre-distillation provided in an embodiment of the present application;

[0022] Figure 7 A schematic diagram of strong distillation provided in an embodiment of the present application;

[0023] Figure 8 A flowchart of the intent recognition method provided in an embodiment of the present application;

[0024] Fig. 9 A schematic diagram of the structure of the intention recognition model training device provided in an embodiment of the present application;

[0025] Fig.10 A schematic diagram of the structure of an intention recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0027] With the development of deep learning, models have become more and more complex and large, such as large-scale pre-trained models such as BERT and GPT. Although these large models perform well on many tasks, their computational cost and storage requirements are also very high. This makes them difficult to apply directly in resource-constrained environments (such as mobile devices, real-time applications, etc.). Therefore, knowledge distillation technology came into being to solve this problem.

[0028] Knowledge Distillation: Knowledge distillation is a model compression technique that aims to transfer the knowledge learned from a large and complex model (called the teacher model) to a smaller and more efficient model (called the student model). The goal of this method is to make the student model as close to the teacher model as possible in performance, but with significantly lower computing resource requirements, reducing the model's computing resource requirements while maintaining performance.

[0029] Specifically, the teacher model is a complex and large model that has been trained. The teacher model can serve as a knowledge source to provide soft labels to the student model, that is, the output probability distribution of the model, rather than just hard labels (such as category numbers). Due to its complexity and size, the teacher model usually has high accuracy and strong expressiveness, and accordingly, the teacher model requires more computing resources and storage space.

[0030] The student model is a small, simple, lightweight model that can be trained by learning the output of the teacher model to improve its own performance, so as to achieve performance as close to the teacher model as possible while reducing computing resource requirements. The student model is usually more efficient than the teacher model and can run in a resource-constrained environment.

[0031] Figure 1 A schematic diagram of a scenario applicable to an embodiment of the present application.

[0032] like Figure 1 As shown, the scenario includes a call center 10, a server 20, an object storage service (OSS) platform 30, and a called user 40. The call center 10 is one of the modules of the intelligent call system, which can execute call tasks for the called user 40. Regardless of whether the call is connected, the call center 10 can upload the channel data generated by the call to the OSS platform, wherein the OSS platform can provide low-cost and reliable cloud storage services, and storage based on the OSS platform 30 is a convenient cloud storage method. It can be understood that the OSS platform 30 here is only an exemplary introduction to a cloud storage technology. In actual applications, other means or cloud platforms can also be used to store the channel data generated by the call, and the embodiments of the present application do not specifically limit this.

[0033] Furthermore, the server 20 can access the OSS platform 30 to obtain channel data and perform a series of processing on the channel data, such as call intention recognition, etc. Afterwards, the server 20 can feed back the recognition result to the call center 10 so that the call center 10 can further grasp the actual situation of the called user 40.

[0034] Figure 2 An overall flow chart of the intent recognition model training method and intent recognition method provided in the embodiments of the present application.

[0035] Figure 3 A flowchart of the intent recognition model training method provided in an embodiment of the present application.

[0036] like Figure 2 and 3 As shown, the intention recognition model training method corresponds to the training stage of the model, and the intention recognition method corresponds to the reasoning stage of the model. The intention recognition model training method provided in the embodiment of the present application may include the following steps:

[0037] S100: Acquire first training data.

[0038] The first training data includes a plurality of text data with intent classification labels. The first training data may be determined based on an actual outbound call scenario.

[0039] Specifically, when the call center 10 makes an outbound call, the call center 10 can upload the channel data generated by the call to the OSS platform, and the channel data records the real conversation between the call center 10 and the called user 40. The embodiment of the present application can obtain channel data from the OSS platform, and convert the channel data into text data based on technologies such as automatic speech recognition (ASR). Furthermore, the text data can be divided according to the conversation rounds, and each round of conversation can be regarded as a sub-data unit, and then the intent classification label is determined for the sub-data by manual classification. It is also possible to use machine learning or natural language processing technology to automatically classify the text data of each conversation round. After that, combined with manual proofreading and optimization, the accuracy of the classification results is ensured.

[0040] Exemplarily, the intent classification label may include no response, request for deferred repayment, and repayment confirmation, etc. The specific intent classification label can be determined based on the actual application scenario, and the embodiments of the present application do not specifically limit this. The text data corresponding to the conversation between the call center 10 and the called user 40 is as follows: "Call center 10: Hello, Mr. Li, the loan you borrowed on this platform has reached the repayment date. May I ask if you can repay it on time. Mr. Li: I can repay it." Or, "Call center 10: Hello, Mr. Li, the loan you borrowed on this platform has reached the repayment date. May I ask if you can repay it on time. Mr. Li: Please help me apply for an extension." At this time, it can be determined that the intent classification label of the text data is repayment confirmation or request for deferred repayment.

[0041] The embodiment of the present application does not specifically limit the amount of the first training data, which can be determined based on actual training needs.

[0042] S200: Acquire second training data.

[0043] Among them, the second training data includes multiple text data without intent classification labels. In an embodiment of the present application, a large amount of text data can be obtained from public network data, and the large amount of data may not be associated with a specific task (such as an intent recognition task). In some implementations, the amount of the large amount of data can be at least 76GB, which can be adjusted based on actual needs, and the embodiment of the present application does not specifically limit this.

[0044] In the embodiment of the present application, the step of further processing the first training data and the second training data is also included, and the processing steps are described in detail below.

[0045] In the embodiment of the present application, the first training data and the second training data also include the coding ID information (text input_id) corresponding to the text data. The coding ID information can be determined by the following steps S301-S305.

[0046] S301: for each piece of text data in the first training data and the second training data, determining whether the character length of the text data exceeds a first preset length.

[0047] The length of a word can be measured by the number of Chinese characters, numbers, letters, punctuation marks, spaces, special symbols, etc. in the text data, or by the number of characters corresponding to the encoding that can be understood by the computer language. For example, for the text data "Can I use a mobile phone with an iOS system to repay a loan?", when measured by the number of words, each Chinese character, English letter, number, punctuation mark, space, special symbol, etc. in the text data can be calculated as a unit, and the length of the word in this case is 15. When measured by the number of characters corresponding to the encoding, when the encoding method is UTF-8, the length of the word in the text data is 10*3 (Chinese characters) + 5*1 (other characters) = 35.

[0048] In the embodiment of the present application, the first preset length can be set to 15 or 20, and different first preset lengths can be set based on different measurement standards, which is not specifically limited in the embodiment of the present application.

[0049] S302: If the character length of the text data exceeds a first preset length, the text data is truncated to make the character length of the text data equal to the first preset length.

[0050] For example, when the text data is "Can an iOS phone be used to repay the loan? I don't have an Android phone", the text data can be intercepted and only "Can an iOS phone be used to repay the loan" can be retained so that the character length of the text data is equal to 15. In this way, the efficiency of subsequent text data processing can be improved.

[0051] S303: Query the encoding mapping table to determine the encoding ID information corresponding to each encoding unit in the text data to form a first encoding sequence.

[0052] In Natural Language Processing (NLP) and deep learning, the input for computer processing is usually text data, but the computer cannot directly understand and process the entire text. Therefore, in order to further analyze and process the text, it is necessary to convert the text into smaller, more structured units so that the computer can better understand and process the text data. This step is usually called "tokenization" and these units are called "tokens". Further, "tokenization encoding" refers to the process of converting text or tokens into digital representations. This digital representation is usually in a form that the model can understand and process, so that the computer can perform various analyses and modeling on the text.

[0053] In the embodiments of the present application, text can be directly tokenized and encoded. Specifically, in the embodiments of the present application, encoding ID information can be assigned to each Chinese character, digit, letter, symbol, etc. Each Chinese character, digit, letter, and symbol is a coding unit, realizing the tokenization and encoding process of a text.

[0054] Figure 4 This is the character ID mapping table provided by the embodiments of the present application.

[0055] As Figure 4 shown, the character ID mapping table can include rows and columns. Among them, each character occupies a separate row, and the row number of this row represents the unique encoding ID information of the Chinese character. When it is necessary to convert the text into the corresponding encoding ID information, only the row number of each character (coding unit) in the mapping table needs to be found. In this way, text data can be conveniently converted into digital representation. For example, by looking up the mapping table, the encoding ID information of the Chinese character "戈" can be obtained as 2763. This method not only simplifies the text processing process but also improves the processing efficiency and accuracy.

[0056] It can be understood that Figure 4 only exemplary shows the encoding ID information corresponding to some Chinese characters. The character ID mapping table provided by the embodiments of the present application can include various characters such as Chinese characters, letters, digits, and symbols. The embodiments of the present application do not make specific limitations on this.

[0057] In the embodiments of the present application, the number of digits of the encoding ID information can be fixed. For example, the encoding ID information corresponding to the character in the first row of the character ID mapping table can be 0001. In some implementation manners, the number of digits of the encoding ID information can also be unfixed, and can be specifically determined by the specific position of the character in the character ID mapping table. For example, the encoding ID information corresponding to the character in the second row of the character ID mapping table can be 2.

[0058] S304: If the character length of the text data does not exceed the first preset length, query the encoding mapping table to determine the encoding ID information corresponding to each coding unit in the text data, and form a subsequence.

[0059] Exemplarily, for the text data "我是中国人" (I am Chinese), when the first preset length is 10 characters, the character length of this text data is 5 characters, which does not exceed the first preset length. Based on the encoding mapping table, the subsequence formed by its corresponding encoding ID information can be "12345".

[0060] It is understandable that the coding sequence formed by the coding ID information is composed of numbers. In the embodiment of the present application, the second preset length can be used to measure the character length of the coding sequence formed by the coding ID information. The second preset length is, for example, 128Byte or 512Byte. The specific value of the second preset length depends on the first preset length and the number of bits of the coding ID information. For example, when the first preset length is 10 characters and the number of bits of the coding ID information is determined by the specific position of the text in the text ID mapping table, the second preset length is also not fixed. When the first preset length is 32 characters and the number of bits of the coding ID information is a fixed value of 4, the second preset length is 128Byte. When the first preset length is 128 characters and the number of bits of the coding ID information is a fixed value of 4, the second preset length is 512Byte.

[0061] S305: Add a placeholder value at the end of the subsequence to form a second encoding sequence, and the character length of the second encoding sequence is equal to the character length of the first encoding sequence.

[0062] In the embodiment of the present application, the placeholder value can be 1 or 0. The character length of the subsequence "12345" corresponding to the text data "I am Chinese" is 5 bytes. When the second preset length is 10 bytes, 5 placeholder values ​​0 can be added to the end of the subsequence to form the following second encoding sequence [1234500000].

[0063] In an embodiment of the present application, the first training data and the second training data also include mask ID information (text type_id) corresponding to the text data. The mask ID information is used to represent different speakers corresponding to the text data. The mask ID information can be equal to a first default value, and the first default value can be 0, indicating the first speaker. Correspondingly, the mask ID information can be equal to a fourth default value, and the fourth default value can be equal to 1, indicating the second speaker, and can also be equal to a fifth default value, and the fifth default value can be equal to 2, indicating the third speaker. Since the embodiment of the present application focuses on identifying the intention of the called user 40 to reply, only the reply content of the called user 40 can be intercepted for training. Then, the text data corresponding to the first training data are all sent by the same speaker, and the text data corresponding to the second training data are also sent by the same speaker, so the mask ID information can be equal to the first default value.

[0064] In some implementations, when the called user 40 replies, two speakers may speak simultaneously or alternately. Therefore, the mask ID information may also be equal to the fourth default value or the fifth default value, which is not specifically limited in the embodiments of the present application.

[0065] Further, in the embodiment of the present application, the character length of the mask ID information may be equal to the first coding sequence and the second coding sequence. Exemplarily, when the second coding sequence is [1234500000], the mask ID information may be [0000000000].

[0066] In the embodiment of the present application, the first training data and the second training data also include symbol type coding information (text attention_id) corresponding to the text data, wherein the symbol type coding information is used to indicate whether the first coding sequence and the second coding sequence include valid information. The following step S306 may also be included after step S303.

[0067] S306: Determine symbol type coding information based on each bit of coding ID information in the first coding sequence; wherein the symbol type coding information corresponding to the coding ID information is a second default value.

[0068] The first coding sequence does not include a placeholder value, and each bit of coding ID information is determined by text data. Therefore, the symbol type coding information can be determined based on each bit of coding ID information, and the symbol type coding information is equal to the second default value, and the character length of the symbol type coding information can be equal to the first coding sequence. The second default value can be equal to 1. In this way, it can be indicated that each bit in the first coding sequence corresponding to the text data is valid information determined by normal text, rather than no information.

[0069] Step S305 may be followed by the following step S307.

[0070] S307: Determine the symbol type coding information based on each bit coding ID information and the placeholder value in the second coding sequence; wherein the symbol type coding information corresponding to the placeholder value is a third default value.

[0071] In addition to the coding ID information, the second coding sequence also includes a placeholder value. The symbol type coding information can be determined based on each bit of coding ID information and the placeholder value, and the character length of the symbol type coding information can be equal to the second coding sequence. The third default value can be equal to 0. In this way, it can be indicated that the second coding sequence corresponding to the text data includes valid information determined by normal text, and also includes a placeholder without information. Exemplarily, when the second coding sequence is [1234500000], the symbol type coding information can be [1111100000].

[0072] Continue to see Figure 2 and Figure 3 ,After the training data preparation is completed, the model can be further trained.

[0073] Specifically, the embodiment of the present application also includes the following step S400.

[0074] S400: Inputting the encoding ID information, mask ID information and symbol type encoding information corresponding to the text data in the first training data into the first model to train the first model.

[0075] The first model can be a Bert model with a large number of model parameters, which can be used as a teacher model to prepare for the subsequent distillation step. The first model can include an embedding layer (Bert-embedding), a multi-head attention layer (Muti-HeadAttention), an encoder layer (Bert-encoder) and a dense layer (dense).

[0076] Figure 5 A schematic diagram of fine-tuning the first model provided in an embodiment of the present application.

[0077] like Figure 5 As shown in the figure, during fine-tuning, [input_id, type_id, attention_id] can be input into Bert-embedding, where input_id represents the encoding ID information corresponding to the text data, type_id represents the mask ID information corresponding to the text data, and attention_id represents the symbol type encoding information corresponding to the text data. The output of Bert-embedding can be input into Bert-encoder. Further, the output of Bert-encoder can be subjected to dropout processing, where dropout is a commonly used regularization technique in deep learning, which can reduce the overfitting phenomenon of neural networks. Based on the dropout technology, the output values ​​of some neurons can be randomly set to 0 during the training process, that is, these neurons can be "discarded", thereby reducing the dependency between neurons and preventing the model from over-relying on specific neurons. After that, the output data of dropout is input into the dense layer to obtain the recognition result and achieve fine-tuning.

[0078] Step S400 can also be called the downstream task fine-tuning stage. This stage mainly uses the relevant downstream task data to fine-tune the corresponding pre-trained model of Bert (such as Bert-base) and outputs the corresponding fine-tuned downstream task model. The fine-tuned model obtained in this stage can already be used for intent recognition related tasks and can be used as a teacher model for distillation calculations.

[0079] Specifically, fine-tuning is a supervised learning process, and the specific steps are as follows:

[0080] (1) Loading a pre-trained model: This process may refer to obtaining a pre-trained Bert-base model.

[0081] (2) Preparing downstream task data: In the embodiment of the present application, the downstream task refers to the intent recognition task, and preparing downstream task data refers to obtaining a specific annotated data set. Each sample in this data set contains input text and its corresponding label (such as category, emotion, etc.). In the embodiment of the present application, preparing downstream task data specifically refers to obtaining the data determined in steps S100, S200 and steps S301-S307, such as [input_id, type_id, attention_id].

[0082] (3) Training the model: Use the labeled dataset of the downstream task to further train the pre-trained model. This process is called "fine-tuning". Fine-tuning allows the model to learn the characteristics and rules of a specific task while retaining its general language representation capabilities.

[0083] Furthermore, the second model can adopt the basic architecture of the TinyBert model. Compared with the Bert model, the TinyBert model has been simplified in terms of the number of layers, hidden layer parameters, etc. The second model can include an embedding layer (TinyBert-embedding), a multi-head attention layer (Muti-Head Attention), an encoder layer (TinyBert-encoder) and a dense layer (dense).

[0084] In the embodiment of the present application, the model parameters of the first model and the second model include at least the dimension Embedding-size of the embedding layer and the relevant parameters of the encoder layer, and the relevant parameters of the encoder layer include at least the hidden layer dimension hidden_size, the number of attention heads num_attention_heads, the number of hidden layers num_hidden_layers, and the intermediate layer dimension intermediate_size of the feedforward neural network; wherein, the dimension Embedding-size of the embedding layer of the first model is 768, and the dimension Embedding-size of the embedding layer of the second model is 192; the number of attention heads num_attention_heads of the first model is num_hidden_layers, and the number of hidden layers num_hidden_layers is num_hidden_layers. tion_heads is 768, and the number of attention heads of the second model is 192; the number of hidden layers of the first model is 12, and the number of hidden layers of the second model is 8; the number of hidden layers of the first model is 12, and the number of hidden layers of the second model is 2, the intermediate_size of the intermediate layer of the feedforward neural network of the first model is 3072, and the intermediate_size of the intermediate layer of the feedforward neural network of the second model is 768. Table 1 compares the model parameters of the first model and the second model in a tabular form.

[0085] Table 1 Model distillation information comparison table

[0086]

[0087] Furthermore, before the distillation calculation, the parameters of the second model can be randomly initialized.

[0088] Randomly initializing the model means assigning random initial values ​​to the parameters of the model (such as weights and biases). The specific method of random initialization can be uniform distribution initialization, normal distribution initialization, Xavier (Glorot) initialization or He initialization, which is not specifically limited in the embodiments of the present application.

[0089] Furthermore, the step S400 may further include the following step S500.

[0090] S500: Input the second training data into the first model and the second model, use the first model as the teacher model and the second model as the student model, and use the knowledge distillation algorithm to perform distillation calculations on the second target layer structures of the first model and the second model, so that the second model can be used to understand the semantics of the text data. The second target layer structure is a multi-head attention layer and an encoder layer.

[0091] The step of inputting the second training data into the first model and the second model may include inputting encoding ID information, mask ID information and symbol type encoding information corresponding to the second training data, as well as an intent recognition label.

[0092] In the field of machine learning, the knowledge of the teacher model can be transferred to the student model through distillation calculation. For example, through the use of soft targets, the output probability distribution of the teacher model is passed as a supervision signal to the student model. In this way, the student model can retain the decision-making process and generalization ability of the teacher model as much as possible while reducing the number of parameters and computational complexity. Therefore, when training the student model, a loss function is usually defined.

[0093] Figure 6 A schematic diagram of pre-distillation provided in an embodiment of the present application.

[0094] like Figure 6 As shown, the embodiment of the present application can distill the second target layer structure (multi-head attention layer and encoder layer). Specifically, the output (Attention) of the multi-head attention layer of the first model and the second model can be distilled, and then the output (Sequence output) of the encoder layer can be distilled to finally obtain the distilled second model (TinyBert distillation model output).

[0095] An example distillation code is as follows:

[0096] tmp_loss=loss_mse(student_rep,teacher_rep);

[0097] rep_loss+=tmp_loss;

[0098] Among them, loss_mse is the mean square error loss function, student_rep represents the output representation of the student model, and teacher_rep represents the output representation of the teacher model. This code is used to calculate the mean square error loss between the output representation of the student model and the output representation of the teacher model, and accumulate the mean square error loss to the rep_loss variable.

[0099] It should be noted that, in the embodiment of the present application, step S500 is the first distillation of the second model (student model), which can also be called pre-distillation. This stage mainly uses a large amount of data to achieve mapping between pre-trained models, thereby obtaining a pre-trained student model for strong distillation.

[0100] The mapping here refers to the hierarchical mapping or correspondence from the teacher model to the student model. Specifically, it involves mapping the hierarchical structure in the teacher model to the hierarchical structure in the student model for model distillation. In an embodiment of the present application, the mapping may include several strategies: taking the first 8 layers of the teacher model as part of the structure of the student model. Alternatively, taking the last 8 layers of the teacher model as part of the structure of the student model. Alternatively, selecting the layers of the teacher model at intervals to cover multiple depth levels of the teacher model, so that the student model can obtain the diversity and knowledge of the teacher model as much as possible.

[0101] It should be noted that the number of Attention heads in the embodiment of the present application is 8, which makes up for the disadvantage of reduced features compared to the model with 6 Attention heads, and at the same time makes the model parameter amount smaller than TinyBert-3L. That is, the model provided by the embodiment of the present application has advantages in terms of model size and parameter amount, and can ensure the model effect.

[0102] Furthermore, the step S500 also includes the following step S600.

[0103] S600: Input the first training data into the first model and the second model, use the first model as the teacher model and the second model as the student model, and use the knowledge distillation algorithm to perform distillation calculations on the first target layer structures of the first model and the second model, so that the second model is used to determine the intent classification label corresponding to the text data; wherein the model parameter amount of the first model is greater than that of the second model, and the first model has been trained with the first training data, and the first target layer structure is an embedding layer, a multi-head attention layer, an encoder layer and a dense layer.

[0104] Among them, the step of inputting the first training data into the first model and the second model specifically inputs the encoding ID information, mask ID information and symbol type encoding information corresponding to the first training data, as well as the intent recognition label.

[0105] It is worth noting that in the embodiment of the present application, step S600 is the second distillation of the second model (student model), which can also be called strong distillation. In this stage, by introducing the embedding layer and the logits layer, it is ensured that distillation and mapping can be performed at the beginning of the algorithm (embedding layer), the middle part of the algorithm (multi-head attention layer and encoder layer), and the end part of the algorithm (dense layer), thereby making the distillation of the algorithm more thorough and more powerful, ensuring that the student model and the teacher model are closer and learn more completely, and further strengthening the distillation training in the corresponding downstream tasks.

[0106] Figure 7 A schematic diagram of the strong distillation provided in the embodiments of the present application.

[0107] like Figure 7 As shown, the embodiment of the present application can distill the first target layer structure (embedding layer, multi-head attention layer, encoder layer and dense layer). Specifically, the output (embedding) of the first model and the second model embedding layer can be distilled, and then the output (Attention) of the multi-head attention layer can be distilled, and then the output (Sequence output) of the encoder layer can be distilled, and then the output (logits) of the dense layer can be distilled to finally obtain the distilled second model (distillation model output).

[0108] In the embodiment of the present application, the knowledge distillation algorithm includes at least multiple loss functions; step S600 may include the following steps S601-S603.

[0109] S601: Calculate the output difference between the first target layer structures corresponding to each of the first model and the second model using the target loss function; wherein the target loss function is one of a plurality of loss functions, and the target loss function corresponds to the type of the first target layer structure.

[0110] For example, the loss function of the embedding layer distillation can be L embd =MSE(E S W e ,E T ).

[0111] Among them, E S and E T Represent the embedding layer outputs of the student and teacher models, respectively, and W e It is the linear transformation matrix used to map the output of the embedding layer of the student model to the embedding space of the teacher model, and MSE represents the mean square error loss function.

[0112] Exemplarily, the distillation of the embedding layer can be implemented based on the following code:

[0113] emb_loss=loss_mse(student_embedding,teacher_embedding);

[0114] Among them, loss_mse is the mean square error loss function, student_embedding represents the embedding layer output of the student model, and teacher_embedding represents the embedding layer output of the teacher model.

[0115] By comparing the MSE loss between the outputs of the embedding layers, we can ensure that the representation of the student model in the embedding space is similar to that of the teacher model.

[0116] Exemplarily, the distillation of the output (logits) of the dense layer can be implemented based on the following code.

[0117] cls_loss=soft_cross_entropy(student_logits / self.args["temperature"];

[0118] teacher_logits / self.args["temperature"]);

[0119] tr_cls_loss+=cls_loss.item();

[0120] Among them, soft_cross_entropy is the cross entropy loss function, student_logits represents the dense layer output of the student model, teacher_logits represents the dense layer output of the teacher model, and self.args["temperature"] represents the temperature parameter. This code is used to calculate the cross entropy loss cls_loss between the dense layer output of the student model and the dense layer output of the teacher model, and accumulate the cross entropy loss cls_loss to the tr_cls_loss variable.

[0121] S602: Perform weighted summation on the output differences to obtain the distillation loss.

[0122] Specifically, the output differences corresponding to each first target layer structure (embedding layer, multi-head attention layer, encoder layer, dense layer) are weighted summed to obtain the distillation loss.

[0123] S603: Modify the target parameters of the second model using the distillation loss, where the target parameters are the weights and biases of the second model.

[0124] Specifically, based on the calculated distillation loss, an optimization algorithm (such as gradient descent, Adam, RMSprop, etc.) is used to update the parameters of the second model (student model). By iterating this process, the second model will gradually approach the performance of the first model (teacher model) and learn the feature representation and intent recognition capabilities of the teacher model. Among them. Weights and biases are the basic parameters of each neuron in the neural network, which are used to calculate the output of the neuron. The weight determines the degree of influence of the input data on the output of the neuron, and the bias is a fixed value used to adjust the output of the neuron. The weights and biases can be updated through the back-propagation algorithm to minimize the prediction error of the model.

[0125] From the above content, it can be seen that the embodiment of the present application provides an intent recognition model training method, which can encode text data into an ID form (encoding ID information), and output mask encoding and symbol type encoding at the same time; it can also train and generate a teacher model (first model) of the target intent for use in the distillation stage, and adopt targeted downstream task finetune to further improve the effect of the first model in the target intent. Afterwards, by adopting a large batch of data, the mapping between pre-trained models can be achieved, thereby obtaining a pre-trained student model for strong distillation. Afterwards, by designing a distillation scheme from the front, middle and back positions of the model, the model can be made more thorough during distillation, so that the student model (second model) is closer to the teacher model (first model). In other words, in the embodiment of the present application, the distillation is strengthened by enhancing the model stage of distillation. And because the structure of the second model after distillation is relatively simple, its efficiency is also significantly improved.

[0126] The following describes the inference phase of the model.

[0127] Figure 8 A flowchart of the intent recognition method provided in an embodiment of the present application.

[0128] Combination Figure 2 and Figure 8 The intention recognition method provided in the embodiment of the present application may include the following steps S701-S703.

[0129] S701: Obtain a speech segment to be recognized.

[0130] The voice segment may be generated in a specific intent recognition scenario. For example, in a human-computer dialogue scenario, what a person says may be recorded to form a voice segment, and then the person's speaking intention may be recognized.

[0131] S702: Convert the speech segment into first text data.

[0132] In this step, the speech segment can be converted into text data, namely, first text data, by speech recognition technology. The first text data is the data to be recognized.

[0133] S703: Input the first text data into the second model to determine the intent classification label corresponding to the speech segment to determine the intent corresponding to the speech segment; wherein the second model is pre-trained by the aforementioned intent recognition model training method.

[0134] When the first text data is input into the second model, the first text data can be further processed based on steps S301-S307 to obtain coding ID information, mask ID information and symbol type coding information, and then the above information can be input into the second model for intention recognition, so that the speaker's speaking intention can be determined. The recognition time is short and the accuracy is high.

[0135] Fig. 9 A schematic diagram of the structure of the intent recognition model training device provided in an embodiment of the present application.

[0136] like Fig. 9 As shown, the intention recognition model training device provided in the embodiment of the present application may include:

[0137] A first acquisition module 1001 is used to acquire first training data, where the first training data includes a plurality of text data with intent classification labels;

[0138] The strong distillation module 1002 is used to input the first training data into the first model and the second model, use the first model as the teacher model and the second model as the student model, and use the knowledge distillation algorithm to perform distillation calculations on the first target layer structures of the first model and the second model, so that the second model is used to determine the intent classification label corresponding to the text data; wherein the model parameter amount of the first model is greater than that of the second model, and the first model has been trained with the first training data, and the first target layer structure is an embedding layer, a multi-head attention layer, an encoder layer and a dense layer.

[0139] In some implementations, the first acquisition module 1001 is further used to acquire second training data, where the second training data includes a plurality of text data without intent classification labels;

[0140] It also includes a pre-trained student model module, which is used to input the second training data into the first model and the second model, using the first model as the teacher model and the second model as the student model, and using the knowledge distillation algorithm to perform distillation calculations on the second target layer structures of the first model and the second model, so that the second model can be used to understand the semantics of the text data, and the second target layer structure is a multi-head attention layer and an encoder layer.

[0141] In some implementations, the first training data and the second training data further include coding ID information corresponding to the text data; and further include a text data coding module, which is used to determine, for each text data of the first training data and the second training data, whether the text length of the text data exceeds a first preset length; if the text length of the text data exceeds the first preset length, intercept the text data so that the text length of the text data is equal to the first preset length; query the coding mapping table to determine the coding ID information corresponding to each coding unit in the text data to form a first coding sequence. If the text length of the text data does not exceed the first preset length, query the coding mapping table to determine the coding ID information corresponding to each coding unit in the text data to form a subsequence; add a placeholder value at the end of the subsequence to form a second coding sequence, and the character length of the second coding sequence is equal to the character length of the first coding sequence.

[0142] After the step of forming the first coding sequence, the text data coding module is further used to determine the symbol type coding information based on each bit of coding ID information in the first coding sequence; wherein the symbol type coding information corresponding to the coding ID information is a second default value;

[0143] After the step of forming the second coding sequence, the text data coding module is further used to determine the symbol type coding information based on each bit coding ID information and placeholder value in the second coding sequence; wherein the symbol type coding information corresponding to the placeholder value is the third default value.

[0144] In some implementations, the model parameter quantity includes at least the dimension Embedding-size of the embedding layer and the related parameters of the encoder layer, and the related parameters of the encoder layer include at least the dimension hidden_size of the hidden layer, the number of attention heads num_attention_heads, the number of hidden layers num_hidden_layers, and the dimension intermediate_size of the intermediate layer of the feedforward neural network; wherein, the dimension Embedding-size of the embedding layer of the first model is 768, and the dimension Embedding-size of the embedding layer of the second model is 192; the number of attention heads num_attention_ heads is 768, and the number of attention heads num_attention_heads of the second model is 192; the number of hidden layers num_hidden_layers of the first model is 12, and the number of hidden layers num_hidden_layers of the second model is 8; the number of hidden layers num_hidden_layers of the first model is 12, and the number of hidden layers num_hidden_layers of the second model is 2, the intermediate layer dimension intermediate_size of the feedforward neural network of the first model is 3072, and the intermediate layer dimension intermediate_size of the feedforward neural network of the second model is 768.

[0145] In some implementations, the knowledge distillation algorithm includes at least multiple loss functions; the strong distillation module is also used to: use the target loss function to calculate the output difference between the first model and each corresponding first target layer structure of the second model; wherein the target loss function is one of the multiple loss functions, and the target loss function corresponds to the type of the first target layer structure; perform weighted summation on the output differences to obtain the distillation loss; use the distillation loss to modify the target parameters of the second model, and the target parameters are the weights and biases of the second model.

[0146] Fig.10 A schematic diagram of the structure of an intention recognition device provided in an embodiment of the present application.

[0147] like Fig.10 As shown, the intention recognition device provided in the embodiment of the present application may include:

[0148] The second acquisition module 2001 is used to acquire the speech segment to be recognized;

[0149] A conversion module 2002, used for converting the speech segment into first text data;

[0150] The intention recognition module 2003 is used to input the first text data into the second model, determine the intention classification label corresponding to the speech segment, and determine the intention corresponding to the speech segment; wherein the second model is pre-trained by the aforementioned intention recognition model training method.

[0151] In a specific implementation, the present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the storage medium may include some or all of the steps in each embodiment of the intent recognition model training method and the intent recognition method provided by the present invention, and may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0152] It is easy to understand that those skilled in the art can combine, split, reorganize, etc. the embodiments of the present application to obtain other embodiments based on the several embodiments provided in the present application, and these embodiments do not exceed the protection scope of the present application.

[0153] The above specific implementation methods further explain in detail the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above are only specific implementation methods of the embodiments of the present application and are not used to limit the protection scope of the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A method for training an intent recognition model, characterized in that: include: Acquire first training data, where the first training data includes a plurality of text data with intent classification labels; The first training data is input into the first model and the second model, the first model is used as the teacher model and the second model is used as the student model, and the first target layer structure of the first model and the second model is distilled using a knowledge distillation algorithm, so that the second model is used to determine the intent classification label corresponding to the text data; wherein the model parameter amount of the first model is greater than that of the second model, and the first model has been trained with the first training data, and the first target layer structure is an embedding layer, a multi-head attention layer, an encoder layer and a dense layer.

2. The intention recognition model training method according to claim 1, characterized in that: Before the step of inputting the first training data into the first model and the second model, taking the first model as the teacher model and the second model as the student model, and performing distillation calculation on the first target layer structure of the first model and the second model using the knowledge distillation algorithm, the method further includes: Acquire second training data, where the second training data includes a plurality of text data without intent classification labels; The second training data is input into the first model and the second model, with the first model as the teacher model and the second model as the student model, and the knowledge distillation algorithm is used to perform distillation calculation on the second target layer structures of the first model and the second model, so that the second model can be used to understand the semantics of the text data, and the second target layer structure is the multi-head attention layer and the encoder layer.

3. The intention recognition model training method according to claim 2, characterized in that: The first training data and the second training data also include encoding ID information corresponding to the text data; Before the step of inputting the second training data into the first model and the second model, taking the first model as a teacher model and the second model as a student model, and performing distillation calculation on the second target layer structure of the first model and the second model using a knowledge distillation algorithm, the method further includes: For each piece of text data of the first training data and the second training data, determining whether the length of the text data exceeds a first preset length; If the character length of the text data exceeds a first preset length, truncating the text data so that the character length of the text data is equal to the first preset length; The coding mapping table is queried to determine the coding ID information corresponding to each coding unit in the text data to form a first coding sequence.

4. The intention recognition model training method according to claim 3, characterized in that: After the step of determining whether the length of the text data exceeds a first preset length, the method further includes: If the length of the text data does not exceed the first preset length, query the encoding mapping table to determine the encoding ID information corresponding to each encoding unit in the text data to form a subsequence; A placeholder value is added at the end of the subsequence to form a second encoding sequence, and the character length of the second encoding sequence is equal to the character length of the first encoding sequence.

5. The intention recognition model training method according to claim 4, characterized in that: The first training data and the second training data further include mask ID information and symbol type encoding information corresponding to the text data; wherein the mask ID information is equal to a first default value; After the step of forming the first coding sequence, the method further comprises: Determine the symbol type encoding information based on each bit of the encoding ID information in the first encoding sequence; wherein the symbol type encoding information corresponding to the encoding ID information is a second default value; After the step of forming the second coding sequence, the method further comprises: The symbol type coding information is determined based on the coding ID information and the placeholder value of each bit in the second coding sequence; wherein the symbol type coding information corresponding to the placeholder value is a third default value.

6. The intention recognition model training method according to claim 1, characterized in that: The model parameters include at least the dimension Embedding-size of the embedding layer and the relevant parameters of the encoder layer, and the relevant parameters of the encoder layer include at least the hidden layer dimension hidden_size, the number of attention heads num_attention_heads, the number of hidden layers num_hidden_layers and the intermediate layer dimension intermediate_size of the feedforward neural network; wherein, the dimension Embedding-size of the embedding layer of the first model is 768, and the dimension Embedding-size of the embedding layer of the second model is 192; the number of attention heads num_attention_heads of the first model is num_hidden_layers, and the intermediate layer dimension intermediate_size of the feedforward neural network is num_attention_heads. ads is 768, and the number of attention heads num_attention_heads of the second model is 192; the number of hidden layers num_hidden_layers of the first model is 12, and the number of hidden layers num_hidden_layers of the second model is 8; the number of hidden layers num_hidden_layers of the first model is 12, and the number of hidden layers num_hidden_layers of the second model is 2, the intermediate layer dimension intermediate_size of the feedforward neural network of the first model is 3072, and the intermediate layer dimension intermediate_size of the feedforward neural network of the second model is 768.

7. The intention recognition model training method according to claim 1, characterized in that: The knowledge distillation algorithm includes at least a plurality of loss functions; The step of performing distillation calculation on the first target layer structure of the first model and the second model by using a knowledge distillation algorithm includes: Calculating the output difference between the first target layer structure corresponding to each of the first model and the second model using a target loss function; wherein the target loss function is one of the multiple loss functions, and the target loss function corresponds to the type of the first target layer structure; Performing a weighted summation on the output differences to obtain a distillation loss; The distillation loss is used to modify target parameters of the second model, where the target parameters are weights and biases of the second model.

8. A method for intention recognition, characterized in that: include: Obtain the speech segment to be recognized; Converting the speech segment into first text data; The first text data is input into the second model to determine the intent classification label corresponding to the speech segment to determine the intent corresponding to the speech segment; wherein the second model is pre-trained by the intent recognition model training method described in any one of claims 1-7.

9. An intention recognition model training device, characterized in that: include: A first acquisition module, used to acquire first training data, wherein the first training data includes a plurality of text data with intent classification labels; A strong distillation module is used to input the first training data into the first model and the second model, use the first model as the teacher model and the second model as the student model, and use the knowledge distillation algorithm to perform distillation calculations on the first target layer structures of the first model and the second model, so that the second model is used to determine the intent classification label corresponding to the text data; wherein the model parameter amount of the first model is greater than that of the second model, and the first model has been trained with the first training data, and the first target layer structure is an embedding layer, a multi-head attention layer, an encoder layer and a dense layer.

10. An intention recognition device, characterized in that: include: A second acquisition module, used to acquire a speech segment to be recognized; A conversion module, used for converting the speech segment into first text data; An intention recognition module is used to input the first text data into a second model to determine the intention classification label corresponding to the speech segment to determine the intention corresponding to the speech segment; wherein the second model is pre-trained by the intention recognition model training method described in any one of claims 1-7.

Images