Intention recognition method based on teacher-student framework

Abstract

One or more embodiments of the present specification provide a teacher-student framework-based intention recognition method. The method is applied to an intention recognition system comprising a teacher model and a student model, the student model is obtained through semi-supervised learning based on the teacher model, and the method comprises the following steps: receiving an intention recognition request initiated by a user; inputting the request into a teacher model and obtaining a first identification result output by the teacher model; inputting the request into a student model and obtaining a second identification result output by the student model; and carrying out fusion processing on the first identification result and the second identification result, and outputting a comprehensive identification result obtained by processing to a user. According to the method, the final comprehensive recognition result is generated through the fusion strategy, the advantages of the teacher model and the student model can be comprehensively utilized, the problems of performance bottleneck, low generalization ability and the like caused by tight weight coupling of the teacher model and the student model in a traditional teacher-student framework are solved, and the accuracy of the comprehensive recognition result is improved.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the field of artificial intelligence, and in particular to an intent recognition method based on a teacher-student framework. BACKGROUND

[0002] Traditional intent recognition schemes rely on rule systems or supervised learning models, which not only require a large amount of labeled data, but also have weak generalization ability due to the limitation of labeling methods. With the development of semi-supervised learning (SSL) technology, it is possible to generate pseudo-labels using unlabeled data.

[0003] As a typical application of SSL technology, the traditional teacher-student framework is a knowledge distillation method that combines a small amount of labeled data and a large amount of unlabeled data to improve model performance. The core idea is to use a teacher model to generate pseudo-labels for unlabeled data, and then use a student model to learn these pseudo-labels and labeled data, thereby realizing knowledge transfer and performance optimization of the student model.

[0004] In related technologies, the student model is usually used to handle real intent recognition requests. However, in the semi-supervised learning process, the student model learns the model parameters of the teacher model highly, so the parameters of the trained student model are often highly coupled with the teacher model, and the prediction accuracy of both models is usually close and the accuracy of each model is low. This results in the intent recognition result obtained by using the student model to reason often having low accuracy and the model having poor generalization performance, which needs to be improved. SUMMARY

[0005] Therefore, one or more embodiments of the present specification provide technical solutions as follows:

[0006] According to a first aspect of one or more embodiments of the present specification, an intent recognition method based on a teacher-student framework is provided, applied to an intent recognition system including a teacher model and a student model, the student model being obtained based on semi-supervised learning of the teacher model, and the method comprising:

[0007] receiving an intent recognition request initiated by a user;

[0008] inputting the intent recognition request into the teacher model and obtaining a first recognition result output by the teacher model; and inputting the intent recognition request into the student model and obtaining a second recognition result output by the student model;

[0009] The first recognition result and the second recognition result are fused, and a comprehensive recognition result obtained through the fusion is output to the user.

[0010] According to a second aspect of one or more embodiments of the present specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to the first aspect by running the executable instructions.

[0011] According to a third aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, which stores computer instructions that, when executed by a processor, implement the steps of the method according to the first aspect.

[0012] According to a fourth aspect of one or more embodiments of the present specification, a computer program product is provided, comprising computer program / instructions that, when executed by a processor, implement the steps of the method according to the first aspect.

[0013] As can be seen from the above embodiments, the student model is obtained by the teacher model and semi-supervised learning training, and on this basis, the teacher model and the student model are used simultaneously to infer the intent recognition request initiated by the user, and the intent recognition results output by the two models are fused, and finally the comprehensive recognition result obtained through the fusion is output to the user as the final response to the request.

[0014] It can be understood that the present scheme simultaneously uses the teacher model and the student model to undertake the intent recognition request initiated by the user, and obtains the comprehensive recognition result through the comprehensive processing of the intent recognition results of the two models, so as to ensure that the final response output to the user is made according to the inference logic of the teacher model and the student model. Since the inference error of the model usually conforms to the normal distribution or the Poisson distribution, according to the error joint distribution law in the fusion processing (such as weighted operation), for the case that the model accuracy is close due to the high coupling of model parameters, no matter how the accuracy of the two models is, the present scheme can improve the accuracy of the comprehensive recognition result to a certain extent by simultaneously using the two models and post-fusing the output results of the two models. In addition, simultaneously using the two models for inference also avoids the performance bottleneck that may exist when using a single student model to undertake user requests, and improves the generalization performance of the system to a certain extent (the generalization of the system as a whole is higher than that of a single model).

[0015] To illustrate the accuracy improvement, assume that the error distributions of the two models are both normal distributions, and the prediction accuracy of the teacher model is 90% and that of the student model is 80%. According to the error joint distribution law corresponding to the normal distribution, the accuracy of the result obtained after fusion (such as weighted average) is about 1-(10%*20%) = 98%. Obviously, the prediction accuracy of the intent recognition system (which is composed of the teacher model and the student model) as a whole is higher than that of the student model and higher than that of the teacher model. Obviously, compared with the related technical solution of using only the student model to undertake the user's intent recognition request, the intent recognition system of the present solution has a higher prediction accuracy as a whole. In fact, the lower the single accuracy of the teacher model and the student model, the more significant the accuracy improvement effect of the intent recognition system of the present solution. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 FIG. 1 is a schematic diagram of an architecture of an intent recognition system provided by an example embodiment.

[0017] Figure 2 FIG. 2 is a flowchart of an intent recognition method based on a teacher-student framework provided by an example embodiment.

[0018] Figure 3 FIG. 3 is a schematic diagram of a semi-supervised learning and request response process provided by an example embodiment.

[0019] Figure 4 FIG. 4 is a schematic diagram of a structure of a device provided by an example embodiment.

[0020] Figure 5 FIG. 5 is a block diagram of an intent recognition apparatus based on a teacher-student framework provided by an example embodiment. DETAILED DESCRIPTION

[0021] Traditional intent recognition schemes rely on rule systems or supervised learning models, and such schemes not only require a large amount of labeled data, but also have weak generalization ability due to the limitation of labeling methods. With the development of semi-supervised learning technology, it is possible to generate pseudo labels using unlabeled data.

[0022] As a typical application of SSL technology, the traditional teacher-student framework is a method for improving model performance by combining a small amount of labeled data and a large amount of unlabeled data. The core idea is to use the teacher model to generate pseudo labels for the unlabeled data, and then use the student model to learn these pseudo labels and labeled data, so as to realize the knowledge transfer and performance optimization of the student model.

[0023] In the related art, only the student model is usually used to undertake real intent recognition requests. However, because the student model highly learns the model parameters of the teacher model in the semi-supervised learning process, the parameters of the trained student model are often highly coupled with the teacher model, which leads to low accuracy and poor generalization performance of the intent recognition result obtained by using only the student model for reasoning. In other words, the student model may focus too much on imitating the specific parameter values (or weight values) of the teacher model rather than learning the abstract features in the sample data, which may lead to the student model performing well on the training set but poorly on completely new data, such as poor generalization performance of the response to various types of intent recognition requests in real scenarios.

[0024] To solve the foregoing technical problems in the related art, the present specification proposes a brand-new intent recognition scheme (specifically, a brand-new POST-TEACHER framework), that is, after semi-supervised learning is completed, the teacher model and the student model are simultaneously used to respond to the intent recognition requests initiated by the user, so as to improve the generalization performance of the system in response to the intent recognition requests, and improve the accuracy of the comprehensive recognition result fed back to the user through post-fusion. The scheme will be described below in combination with the accompanying drawings and related embodiments.

[0025] Figure 1 FIG. 1 is an exemplary embodiment of an architecture schematic diagram of an intent recognition system. As shown in FIG. 1, the system can include a server 11 and a plurality of electronic devices, such as a mobile phone 14, a mobile phone 15, a mobile phone 16, and the like. Figure 1

[0026] In the running process, the server 11 can run the server-side program of an application to realize the related functions of the application, such as when the server 11 runs the program of an intent recognition service, it can realize the service end (hereinafter referred to as the intent recognition service end) of the service, such as the service end of a dialogue platform. Exemplarily, the dialogue platform can be a task-oriented dialogue platform, such as an intelligent customer service, a voice assistant, and the like.

[0027] ​In addition to the server 11 and the electronic devices, the intent recognition system can also interface or include servers that implement specific functions, such as the server 12 and / or the server 13, etc., where the server 12 can run a program of a model training service to implement a model training server for semi-supervised learning of student models according to the methods described in the embodiments of the present specification. The server 13 can run a program of a model deployment service to implement a model running server, such as locally deploying and running the trained teacher model and / or student model, and providing the corresponding intent recognition service (i.e., responding to the intent recognition request to infer the corresponding intent recognition result) through the callable interface of the model to the intent recognition server. Of course, the aforementioned teacher model and student model can also be run locally on the server 11 (in which case the intent recognition system can not contain the aforementioned server 13) so that the intent recognition server can quickly call the model, and will not be described again.

[0028] Any of the aforementioned server 11, server 12 and server 13 can be a physical server containing a standalone host, or the server 11 can be a virtual server carried by a host cluster.

[0029] Any of the electronic devices can run a program of a client side of an application to implement the related functions of the application. For example, when the electronic device runs a program of a model training service, it can implement a model training client of the service (such as the mobile phone 14 as a model training client used by technical personnel, etc.), and the user of the electronic device (i.e., the aforementioned technical personnel) can control the model training process through the client (such as configuring the model architecture and interface, specifying the loss function and parameter adjustment condition, adjusting the model parameters, etc.). Similarly, when the electronic device runs a program of an intent recognition service, it can implement an intent recognition client of the service (such as the mobile phone 15 and the mobile phone 16 as intent recognition clients used by ordinary users, etc.), and the ordinary user can initiate an intent recognition request through the client.

[0030] The application program of any of the above clients can be started and run on the corresponding electronic device. For example, the program can be a native application installed on the electronic device, or can be an applet, a fast application, or other similar forms. When using web page technologies such as HTML5 or the like, the relevant functions can be implemented through a page displayed by a browser, which can be a standalone browser application or a browser module embedded in some application. Of course, the intent recognition client can also be integrated into the client of other applications (such as online shopping applications, social applications, travel applications, logistics applications, etc.) as a functional plug-in of the client, so as to recommend objects to the user in the process of providing corresponding services by the above-mentioned applications. Further description is omitted. In addition, a mobile phone is only one type of electronic device that can be used by a user. In fact, the user can also use other types of electronic devices such as a PC (Personal Computer), a tablet device, a notebook computer, a PDA (Personal Digital Assistant), a wearable device (such as smart glasses, a smart watch, etc.), and the like, and the embodiments of the present disclosure are not limited in this regard.

[0031] It can be understood that from the hardware point of view, the intent recognition system includes terminal devices (i.e., the aforementioned electronic devices such as mobile phones 14-16, etc.) and servers (such as the aforementioned servers 11-13, etc.). From the software point of view, the system includes an intent recognition client (such as the client-side program of the intent recognition service running in mobile phones 15 and 16) and an intent recognition server (such as the server-side program of the intent recognition service running in server 11); more specifically, the system includes a trained teacher model and a student model.

[0032] The user described in the present disclosure can be a legitimate user who has completed registration in the intent recognition system or the aforementioned dialogue platform, or can be a temporary user (or a "guest" of the system) who has not registered, and the embodiments of the present disclosure are not limited in this regard. Any user can initiate an intent recognition request through the intent recognition client used by the user (such as inputting a question text or a voice in the dialogue interface provided by the aforementioned dialogue platform); correspondingly, the intent recognition server can call the deployed teacher model and student model to perform reasoning and obtain corresponding first and second recognition results, respectively, and then fuse the two results to obtain a comprehensive recognition result and return it to the intent recognition client for the user to view.

[0033] The network 10 for interaction between the electronic device such as the mobile phone 14 and the server 11 can be implemented by a wired or wireless network based on the communication mode supported by the corresponding electronic device, and the present specification does not limit this. For example, the mobile phone 14 usually only supports wireless communication, and thus a wireless network can be used to implement the communication. The PC can communicate with the server 11 in a wired or wireless manner, and details are not described herein.

[0034] Figure 2 FIG. 1 is a flowchart of an exemplary embodiment of a method for intent recognition based on a teacher-student framework, which is applied to an intent recognition system including a teacher model and a student model. As shown in FIG. 1, the method includes the following steps 102-106. Figure 2

[0035] Step 102, receiving an intent recognition request initiated by a user.

[0036] The user can initiate the intent recognition request through an intent recognition client used by the user. For example, the user can participate in a dialogue on a dialogue platform built by a related party, and propose a question in the form of text or voice. Correspondingly, the platform can initiate a corresponding intent recognition request to the intent recognition system for the question, so that the system responds to the request and obtains a corresponding comprehensive recognition result according to the method described in the following embodiments, and finally outputs the result to the user for viewing.

[0037] Exemplarily, the dialogue platform can be a task-oriented dialogue platform such as an intelligent customer service or a voice assistant. In this case, the intent recognition system can be used as a customer service robot or a voice assistant to dialogue with the user, thereby solving various problems encountered by the user in production and life, and having high practical value.

[0038] Step 104, inputting the intent recognition request into the teacher model, and obtaining a first recognition result output by the teacher model; and inputting the intent recognition request into the student model, and obtaining a second recognition result output by the student model.

[0039] As described above, the intent recognition system includes a teacher model and a student model, wherein the teacher model is a pre-trained model, and the student model and the teacher model constitute a teacher-student framework. The student model is obtained based on the teacher model and semi-supervised learning, that is, the student model is obtained by training a basic model through the teacher model and semi-supervised learning.

[0040] ​In an embodiment, the student model can be configured to have a smaller scale than the teacher model. For example, the student model can have a relatively smaller number of parameters (e.g., weights, biases, etc.), a relatively lower complexity (e.g., fewer layers of Transformers, fewer dimensions of each hidden layer, fewer functional components, etc.), a relatively smaller storage space for a model file (e.g., a.pt file for PyTorch, a.pb file for TensorFlow, etc.), a relatively lower computational complexity (e.g., a smaller number of floating-point operations, etc.), a smaller scale of training data, etc. It can be appreciated that a student model with a relatively smaller scale not only has a faster training speed, but also generally requires fewer resources for running, as it is easier to deploy on an end device (e.g., a mobile phone, an embedded device, etc.), more suitable for edge computing or a memory-limited environment (e.g., as a mobile AI assistant, etc.), has a shorter delay and a faster response in a real-time interactive scenario (e.g., a dialogue robot, etc.), can provide real-time / near real-time translation, has a lower energy consumption, and can be applied to a privacy protection scenario that requires a localized deployment, with a more flexible and secure deployment and running.

[0041] In an embodiment, the teacher model and the student model can employ the same or different network architectures. In addition, the data types that the two models can recognize (i.e., the types of intent recognition requests that the models respond to) can be one or more, and in the case of recognizing multiple types of data, the model is a multi-modal model. The data types can be any one of text, speech, image, video, etc. For example, both models can be rule-based models or conventional deep learning models (BERT, Bidirectional Encoder Representations from Transformers).

[0042] In an embodiment, the teacher model and the student model can also be LLMs (Large Language Models), i.e., the two models are trained based on an LLM. By introducing an LLM in the teacher-student framework, both the teacher model and the student model can better capture complex intent distributions, helping to improve the accuracy of the intent recognition system in intent recognition. Compared with traditional machine learning models, LLMs can break through the context length limit by expanding the attention mechanism (such as rotating position encoding, local sensitive hashing) and a larger model size, and can model long-range dependencies of thousands to millions of tokens, thus having stronger text understanding capabilities. Based on this, LLMs can not only generate fluent and natural text (such as articles, code, stories, etc.), support multi-turn dialogue and context memory, but also can achieve logical reasoning, mathematical calculation, common sense inference, and cross-domain association through a Chain of Thought (CoT), which helps the intent recognition system to more accurately complete intent recognition.

[0043] In an embodiment, the teacher model can be pre-trained based on a large amount of sample data, such as being trained based on a newly built large model network architecture.

[0044] Alternatively, since there are benchmark models (or base models) available in the industry at present, the teacher model can also be obtained by fine-tuning (Fine-Tuning, or supervised fine-tuning) a benchmark LLM. For example, in the case where the intent recognition system described in the present solution needs to be connected to a task-oriented dialogue platform (at this time, the user will initiate the aforementioned intent recognition request through the platform), the sample data associated with the task-oriented dialogue platform can be obtained first, and the benchmark LLM is fine-tuned based on the sample data to obtain the teacher model. The sample data associated with the task-oriented dialogue platform can be filtered from historical dialogues that occur on the platform, such as using high-quality questions initiated by historical users as samples, and using the correct answers (which can be human customer service or robot replies) given by the platform as the labels of the samples, and the sample data is composed of the samples and their labels. It can be understood that because the sample data is associated with the task-oriented dialogue platform, these sample data can fully reflect the preferences and habits of the platform users in initiating intent recognition requests, which helps to realize a special-purpose question and answer platform in a specific field. Of course, the sample data can also be constructed without using data from a specific platform to ensure that the teacher model has high universality and wide adaptability to various requests.

[0045] In the fine-tuning process, all or most of the parameters of the teacher model can be adjusted to achieve comprehensive training of the model as much as possible. Alternatively, to reduce the number of parameter adjustments to speed up the fine-tuning efficiency, the benchmark model can also be fine-tuned based on the LoRA (Low-Rank Adaptation) algorithm.

[0046] The core idea of the LoRA algorithm is to introduce two low-rank matrices to decompose the update matrix of the parameters. For example, the update matrix of the benchmark LLM can be first decomposed into two low-rank matrices, and the rank r of the two low-rank matrices satisfies r << m and r << n, where m and n are the number of rows and columns of the parameter matrix of the benchmark matrix; then, after inputting the sample data into the benchmark LLM, the element values of the low-rank matrices can be adjusted according to the deviation between the sample prediction results output by the benchmark LLM and the sample labels.

[0047] Illustratively, assuming that the parameter matrix of a certain layer of the benchmark LLM is W (shape m x n), and the corresponding update matrix is ΔW, ΔW is decomposed into two low-rank matrices B and A based on the LoRA algorithm. Next, the input data is first passed through the original pre-trained model to obtain the output result, and then the loss between the prediction result and the real label is calculated according to the task loss function (such as cross-entropy loss), and then the gradients of the loss with respect to the low-rank parameters A and B are calculated by the back propagation algorithm, and finally the low-rank parameters A and B are updated using the optimizer according to the calculated gradients. The adjusted parameter matrix is as follows:

[0048] W' = W + ΔW = W + BA

[0049] In LoRA fine-tuning, only parameters need to be added to specific layers of the benchmark LLM, which significantly reduces the number of parameters that need to be updated, thereby saving GPU resources and storage space. In this way, while keeping most of the parameters of the benchmark LLM frozen, only a limited number of parameters can be fine-tuned to achieve an effect close to full fine-tuning, thereby significantly reducing the amount of calculation in the fine-tuning process and improving the training speed. In addition, the introduction of the efficient fine-tuning method of LoRA based on the powerful language understanding ability of LLM makes the MM1 more accurate when generating pseudo labels, which helps to reduce the introduction of noise.

[0050] In the process of fine-tuning in the foregoing manner, fine-tuning can be stopped when the stop condition is met. The stop condition can be that the number of training times reaches a preset number (such as the second historical interaction record of the user is used up), or the convergence degree of the teacher model meets the requirement, the model error is less than a preset threshold, etc., which will not be repeated here.

[0051] After the fine-tuning is completed in the above manner, the parameters obtained through the fine-tuning can be saved, so that the parameters are directly loaded into the benchmark LLM in a subsequent model calling stage, and the teacher model is obtained.

[0052] In an embodiment, after the pre-trained teacher model or the teacher model obtained through temporary training is acquired, when the student model is obtained through semi-supervised learning based on the teacher model, a first sample request and a first sample benchmark label thereof can be acquired first, and the student model to be trained is called to predict a first sample prediction label of the first sample request; then a first deviation between the first sample prediction label and the first sample benchmark label is determined, and the model parameters of the student model are adjusted according to the first deviation. As can be known from the calculation manner of the first deviation, the deviation is used to represent the error of the student model to be trained itself in the process of text understanding and generation, and therefore, the model parameters of the student model are adjusted based on the first deviation, so that the above error of the model can be reduced as much as possible in the training process, and the accuracy of the intent recognition (i.e., text understanding and generation) of the student model is improved.

[0053] Further, a second sample request can also be acquired, and the student model to be trained is called to predict a second sample prediction label of the second sample request, and the teacher model is called to predict a second sample benchmark label of the second sample request; and a second deviation between the second sample prediction label and the second sample benchmark label is determined. Based on this, when the model parameters of the student model are adjusted according to the first deviation, the comprehensive deviation of the student model can be determined based on the first deviation and the second deviation first, and then the model parameters of the student model are adjusted according to the comprehensive deviation. As can be known from the calculation manner of the second deviation, the deviation is used to represent the inter-model deviation between the student model and the teacher model, i.e., the gap between the intent recognition ability (i.e., the text understanding and generation ability) of the student model and the intent recognition ability of the teacher model. Therefore, the comprehensive deviation of the student model determined based on the first deviation and the second deviation together can reflect the error of the model itself and the inter-model deviation between the model and the teacher model (i.e., the learning object of the teacher model), and therefore, the model parameters are adjusted based on the comprehensive deviation, so that the above error / deviation of the model can be reduced, and the intent recognition ability of the student model itself is improved while the above ability of the model gradually approaches the teacher model.

[0054] In addition, the parameters of the student model can be updated by using a gradient back propagation algorithm, which calculates the gradient of each parameter with respect to a loss function to guide the update of the model parameters. The specific update manner can be referred to the description in the related art, which will not be described herein.

[0055] Figure 3Fig. 1 is a schematic diagram of a semi-supervised learning and request-response process provided by an example embodiment. As shown in Fig. 1, a training process (i.e., a semi-supervised learning process) indicated by black arrows can use a small amount of labeled data (including a first sample request xi and a first sample reference label Yi) and a large amount of unlabeled data (only including a second sample request x2). Figure 3

[0056] Based on the above data, on the one hand, xi can be input into a student model to obtain a first sample predicted label yi output by the student model after inference, and then a first deviation, i.e., lossi(yi, Yi), can be calculated based on the first sample predicted label yi and the first sample reference label Yi. On the other hand, x2 can be input into a teacher model to obtain a second sample reference label Y2 output by the teacher model after inference (the label can be regarded as a pseudo label generated by the teacher model for the unlabeled data x2), and x2 can be input into the student model to obtain a second sample predicted label y2 output by the student model after inference; then a second deviation, i.e., loss2(y2, Y2), can be calculated based on the second sample predicted label y2 and the second sample reference label Y2. Then, a comprehensive deviation Loss = lossi + loss2 can be obtained.

[0057] Wherein, any one of the labels yi, Yi, y2 and Y2 can be a hard label (Hard Labels, such as directly selecting the category with the highest probability as a pseudo label, i.e., retaining when the probability > threshold, otherwise discarding), or a soft label (Soft Labels, i.e., taking the probability distribution / probability value output by the model as a label), which is not limited by the embodiments of the present specification. In addition, the first deviation can be a cross-entropy loss (when a hard label is used) or a KL divergence (when a soft label is used) between yi and Yi; the second deviation can be a mean square error between y2 and Y2. Of course, other suitable algorithms can also be selected, and the specific calculation formulas of the two can be referred to the description of related technologies, which will not be described here.

[0058] In an embodiment, in addition to keeping the parameters of the teacher model unchanged as a fixed learning target, the teacher model can also be synchronously trained in the process of querying the student model, so as to enable the teacher model to adapt to the training process of the student model and improve the distillation effect, thereby improving the overall intent recognition capability and intelligent level of the intent recognition system (the teacher model can also learn new knowledge and / or rules from unlabeled data). For example, in the process of training the student model, the parameters of the teacher model can also be jointly adjusted according to the adjusted parameters of the student model, thereby realizing joint optimization of the two models.

[0059] ​Specifically, after the student parameter is updated, the model parameter of the teacher model can be updated using an Exponential Moving Average (EMA) algorithm:

[0060]

[0061] wherein, is the updated teacher model parameter, is the teacher model parameter before being updated, is the updated student model parameter. The EMA coefficient a is small (e.g., 0.9), the teacher model is updated faster, and the student parameter changes more sensitive, which is more suitable for the stage of rapid iteration of the student model. The EMA coefficient a is large (e.g., 0.999), the teacher model is smoother, which is suitable for the later convergence stage, and can avoid overfitting to the local noise of the student. Therefore, the value of a can be dynamically adjusted during the joint optimization process, such as using a smaller a in the early stage of training, and gradually increasing the value of a in the later stage of training.

[0062] Through the EMA algorithm, the parameter update of the student model can be fused, the teacher model is gradually evolved, and the soft label more suitable for the current ability of the student is output, avoiding the too complex supervision signal in the early training. Moreover, EMA has low-pass filtering characteristics, which can filter the high-frequency noise in the student model parameter update, and generate more stable teacher model parameters. In addition, this method is suitable for online distillation, so that the teacher model can absorb the optimization experience of the student model in real time, form a benign interaction between models, and accelerate the improvement of the overall intent recognition ability of the intent recognition system.

[0063] In step 206, the first recognition result and the second recognition result are fused and processed, and the comprehensive recognition result obtained by the processing is output to the user.

[0064] With the teacher model and the student model trained by the foregoing method, the intent recognition system can input the intent recognition request initiated by the user into the teacher model to obtain the first recognition result output by the model after understanding and reasoning, and input the request into the student model to obtain the second recognition result output by the model after understanding and reasoning. Any of the above recognition results is a result text (such as an answer to the user text) generated by the corresponding model based on its powerful text generation capability, which matches the user intent (such as what problem the user wants to ask, what language style the user is used to, etc.) represented by the intent recognition request after fully understanding the text features and analyzing the text semantics of the intent recognition request.

[0065] It can be understood that the intention recognition capabilities of the teacher model and the student model are not completely the same, and therefore the recognition results (i.e., the first recognition result and the second recognition result) given respectively for the same intention recognition request are usually also not the same. At this time, the comprehensive recognition result can be generated by fusing the two results, which can ensure that the result can cover the overall meaning of the first recognition result and the second recognition result, can reflect the overall intention recognition capability of the intention recognition system, and thus improve the accuracy of the result.

[0066] In an embodiment, when the first recognition result and the second recognition result are fused, the weight values of the teacher model and the student model can be determined first, and then the first recognition result and the second recognition result are weighted according to the weight values to generate the comprehensive recognition result. By determining appropriate weight values and weighted average operation, it can be ensured that the comprehensive recognition result can fully reflect the overall meaning of the first recognition result and the second recognition result.

[0067] As shown in Figure 3 Unlike the conventional scheme, the reasoning process shown by the red arrow involves not only the student model but also the student model, i.e., the present scheme uses two models to process the intention recognition request at the same time. Among them, the comprehensive recognition result S can be calculated according to the first recognition result s1 and the second recognition result s2 according to the following formula:

[0068] S = k1 * s1 + k2 * s2

[0069] Among them, the coefficient k1 of the teacher model and the coefficient k2 of the student model can be flexibly set according to the actual scene, such as k2 = 1 - k1.

[0070] In an embodiment, the weight value matched to the request type of the intention recognition request can be determined according to the request type. For example, different weight values can be set for different request types in advance according to the request scenarios that the teacher model and the student model are good at, such as k1 = 0.7, k2 = 0.3 for shopping-related problems, k1 = 0.8, k2 = 0.2 for tourism-related problems, k1 = 0.5, k2 = 0.5 for medical-related problems, and the like, which will not be repeated. Then when the intention recognition request is received, the corresponding values of k1 and k2 can be determined according to the request type to which the request belongs, so as to calculate a relatively accurate comprehensive recognition result. In this way, the teacher model and the student model are allowed to be good at coping with different request scenarios respectively, so as to realize a professional model for a specific field, which helps to improve the overall intention recognition capability of the intention recognition system.

[0071] The above describes particular embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in an order other than that in the embodiments and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or possible.

[0072] As can be seen from the above embodiments, the scheme simultaneously uses the teacher model and the student model to undertake the intent recognition request initiated by the user, and the comprehensive recognition result is obtained after the intent recognition results of the two are comprehensively processed, so that the final response output to the user can be ensured to be made according to the inference logic of the teacher model and the student model. Since the inference error of the model usually conforms to the normal distribution or the Poisson distribution, according to the error joint distribution law in the fusion processing (such as weighted operation), for the case that the model accuracy is close due to the high coupling of the model parameters, even if the accuracy of the two models is low, the scheme still can improve the accuracy of the comprehensive recognition result to a certain extent by using the two models and the output results of the two models are fused, and avoids the performance bottleneck that may exist when using a single student model to undertake the user request, thereby improving the generalization of the system.

[0073] Further explanation: it is assumed that the error distribution of the two models conforms to the normal distribution, if the prediction accuracy of the teacher model is 90%, and the prediction accuracy of the student model is 80%, according to the error joint distribution law corresponding to the normal distribution, the accuracy of the result obtained after fusion (such as weighted average) is about 1-(10%*20%) = 98%. Obviously, the prediction accuracy of the intent recognition system (which is composed of the teacher model and the student model) is not only higher than that of the student model, but also higher than that of the teacher model. Obviously, compared with the related technical scheme of using only the student model to undertake the intent recognition request of the user, the intent recognition system of the present scheme has higher prediction accuracy. Even, the lower the single accuracy of the teacher model and the student model is, the more significant the accuracy improvement effect of the intent recognition system described in the present scheme is.

[0074] In an embodiment, in the case that the intent recognition system is connected to the session platform for receiving real user-initiated intent recognition requests, the system can use historical intent recognition requests that have been processed to build samples and use the samples to periodically or irregularly iteratively train the student model (or even the teacher model). For example, the student model can be iteratively trained according to a preset period (such as daily, weekly, monthly, etc.) or a preset request processing amount (such as processing 100 or 1000 million requests, etc.), so as to further improve the overall recognition capability of the intent recognition system based on real traffic.

[0075] Figure 4 is a schematic structural diagram of an apparatus provided by an exemplary embodiment. Please refer to Figure 4 At the hardware level, the apparatus includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410, and can also include other hardware required by functions. One or more embodiments of the present specification can be implemented in a software manner, such as reading a corresponding computer program from the non-volatile memory 410 into the memory 408 by the processor 402 and then running. Of course, in addition to the software implementation, one or more embodiments of the present specification do not exclude other implementation manners, such as a logic device or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device.

[0076] Please refer to Figure 5 The intent recognition apparatus based on the teacher-student framework can be applied to an apparatus as shown in Figure 4 to implement the technical solutions of the present specification. Wherein, the apparatus is applied to an intent recognition system including a teacher model and a student model, and the student model is obtained by semi-supervised learning based on the teacher model. Wherein, the apparatus includes:

[0077] The request receiving unit 501 is configured to receive an intent recognition request initiated by a user;

[0078] The request recognition unit 502 is configured to input the intent recognition request into the teacher model and obtain a first recognition result output by the teacher model; and input the intent recognition request into the student model and obtain a second recognition result output by the student model;

[0079] The result fusion unit 503 is configured to perform fusion processing on the first recognition result and the second recognition result, and output a comprehensive recognition result obtained by processing to the user.

[0080] Optionally, the semi-supervised learning unit 504 is further configured to:

[0081] obtain a first sample request and a first sample reference label thereof, and call the student model to be trained to predict a first sample prediction label of the first sample request;

[0082] determine a first deviation between the first sample prediction label and the first sample reference label, and adjust a model parameter of the student model according to the first deviation.

[0083] Optionally,

[0084] The second deviation determination unit 505 is further configured to: obtain a second sample request, call the student model to be trained to predict a second sample prediction label of the second sample request, and call the teacher model to predict a second sample reference label of the second sample request; and determine a second deviation between the second sample prediction label and the second sample prediction label.

[0085] The semi-supervised learning unit 504 is specifically configured to: determine a comprehensive deviation of the student model based on the first deviation and the second deviation, and adjust the model parameter of the student model according to the comprehensive deviation.

[0086] Optionally, the system further comprises:

[0087] The joint optimization unit 506 is configured to jointly adjust a parameter of the teacher model according to the adjusted parameter of the student model.

[0088] Optionally, the teacher model and the student model are both large language models (LLM), and a size of the teacher model is greater than a size of the student model.

[0089] Optionally, the user initiates the intent recognition request through a task-oriented dialogue platform, and the fine-tuning unit 507 is further configured to:

[0090] obtain sample data associated with the task-oriented dialogue platform, and fine-tune a reference LLM based on the sample data to obtain the teacher model.

[0091] Optionally, the fine-tuning unit 507 is specifically configured to:

[0092] decompose an update matrix of the reference LLM into two low-rank matrices, a rank r of the two low-rank matrices satisfying r << m and r << n, where m and n are a number of rows and a number of columns of a parameter matrix of the reference matrix;

[0093] After inputting the sample data into the reference LLM, adjust an element value of the low-rank matrix according to a deviation between a sample prediction result output by the reference LLM and the sample label.

[0094] Optionally, the result fusion unit 503 is specifically configured to:

[0095] determine weight values of the teacher model and the student model, and perform weighted operation on the first recognition result and the second recognition result according to the weight values to generate the comprehensive recognition result.

[0096] Optionally, the result fusion unit 503 is specifically configured to:

[0097] determine a weight value matched to the request type according to the request type of the intent recognition request.

[0098] Based on the same idea as the above method, the specification also provides an electronic device, comprising: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of the method as described in any of the above embodiments by running the executable instructions.

[0099] Based on the same idea as the above method, the specification also provides a computer readable storage medium having stored thereon computer instructions, which, when executed by a processor, implement the steps of the method as described in any of the above embodiments.

[0100] Based on the same idea as the above method, the specification also provides a computer program product, comprising computer program / instructions, which, when executed by a processor, implement the steps of the method as described in any of the above embodiments.

Claims

1. An intent recognition method based on a teacher-student framework, characterized in that, Applied to an intention recognition system including a teacher model and a student model, where the student model is obtained through semi-supervised learning based on the teacher model, the method includes: Receiving an intention recognition request initiated by a user; Inputting the intention recognition request into the teacher model and obtaining a first recognition result output by the teacher model; and, inputting the intention recognition request into the student model and obtaining a second recognition result output by the student model; Performing fusion processing on the first recognition result and the second recognition result, and outputting a comprehensive recognition result obtained through processing to the user.

2. The method according to claim 1, characterized in that, Obtaining the student model through semi-supervised learning based on the teacher model includes: Obtaining a first sample request and its first sample reference label, and calling the student model to be trained to predict a first sample prediction label of the first sample request; Determining a first deviation between the first sample prediction label and the first sample reference label, and adjusting model parameters of the student model according to the first deviation.

3. The method according to claim 2, wherein It further includes: obtaining a second sample request, calling the student model to be trained to predict a second sample prediction label of the second sample request, and calling the teacher model to predict a second sample reference label of the second sample request; and, determining a second deviation between the second sample prediction label and the second sample prediction label; Adjusting the model parameters of the student model according to the first deviation between the first sample prediction label and the first sample reference label includes: determining a comprehensive deviation of the student model based on the first deviation and the second deviation, and adjusting the model parameters of the student model according to the comprehensive deviation.

4. The method according to claim 2 or 3, characterized in that, It further includes: Jointly adjusting the parameters of the teacher model according to the adjusted parameters of the student model.

5. The method according to claim 1, characterized in that, Both the teacher model and the student model are large language models LLM, and the scale of the teacher model is larger than the scale of the student model.

6. The method according to claim 5, characterized in that, The user initiates the intention recognition request through a task-oriented dialogue platform, and obtaining the teacher model includes: Obtaining sample data associated with the task-oriented dialogue platform, and fine-tuning a benchmark LLM based on the sample data to obtain the teacher model.

7. The method according to claim 6, characterized in that, The fine-tuning the benchmark LLM based on the sample data includes: Decomposing an update matrix of the benchmark LLM into two low-rank matrices, where the rank r of the two low-rank matrices satisfies r << m and r << n, where m and n are the number of rows and columns of the parameter matrix of the benchmark matrix; After inputting the sample data into the benchmark LLM, adjusting the element values of the low-rank matrix according to the deviation between the sample prediction result output by the benchmark LLM and the sample label.

8. The method according to claim 1, characterized in that, The performing fusion processing on the first recognition result and the second recognition result includes: Determining weight values of the teacher model and the student model, and performing weighted operations on the first recognition result and the second recognition result according to the weight values to generate the comprehensive recognition result.

9. The method according to claim 8, characterized in that, Determining the weight values of the teacher model and the student model includes: Based on the intent to identify the request type, a weight value matching the request type is determined.

10. An electronic device, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-9 by executing the executable instructions.

11. A computer-readable storage medium, characterized in that, It stores computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-9.

12. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-9.

Images