Quality inspection method based on knowledge distillation and multi-modal combination
By employing a quality inspection method based on knowledge distillation and multimodal integration, and using the squeezeBERT model and cross-entropy loss function to train a student model, and fusing audio and text feature vectors for sentiment prediction, this approach solves the problems of low efficiency and high cost in traditional quality inspection, achieving high-precision quality inspection results and standardization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SUNYARD SYST ENG CO LTD
- Filing Date
- 2021-10-26
- Publication Date
- 2026-07-21
AI Technical Summary
Traditional manual quality inspection is inefficient and costly in call center and human customer service operations. The quality inspection standards are not uniform, it is difficult to identify the emotions of customers and customer service representatives, and the lack of labeled data makes machine quality inspection difficult.
A quality control method based on knowledge distillation and multimodal integration is adopted. The squeezeBERT model is used as the teacher model and trained on audio and text data. The complexity of the student model is improved by knowledge distillation and data augmentation. The student model is trained using the cross-entropy loss function and the audio and text feature vectors are fused for sentiment prediction.
It improved the recognition accuracy of the quality inspection model, achieved standardization and fairness in emotion recognition, reduced operating costs, reduced staff turnover, and improved quality inspection efficiency.
Smart Images
Figure CN113919369B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of natural language processing and speech recognition technology, specifically to a quality inspection method based on knowledge distillation and multimodal combination. Background Technology
[0002] Traditional call centers and human customer service are burdensome, and the massive volume of calls increases the associated quality inspection workload. Quality inspectors often have to listen to eight hours of recordings daily, and even five minutes per recording is insufficient to complete their daily tasks. Not only is the industry's quality inspection coverage low, but manual sampling is also inefficient. Furthermore, companies face difficulties in recruiting quality inspectors, with lengthy training periods. The repetitive workload leads to burnout among quality inspectors, resulting in high turnover and further increasing company costs. Quality inspection standards are heavily influenced by the inspectors' perceptions and subjectivity, making it difficult to standardize standards and ensure fairness and impartiality. Manual quality inspection can easily overlook the value and risks inherent in calls, failing to identify and resolve problems promptly.
[0003] With the development of big data, cloud computing, and AI technologies, machine quality inspection can effectively solve this problem. Machine quality inspection can perform speech recognition and conversion on received call data, transforming the voice files into accurate text files. Then, using natural language processing technology, it can identify and process the text data according to preset quality inspection rules, ultimately outputting fair and standardized quality inspection results.
[0004] Emotion recognition for customers and customer service representatives is a crucial part of quality inspection. It primarily involves analyzing call service quality through customer and customer service emotion analysis to promptly identify problems during quality inspection. Voice recognition technology converts audio files into text files, and the text data undergoes keyword and text similarity checks to uncover hidden customer and customer service emotions within sensitive words. Furthermore, it analyzes for significant emotional fluctuations during calls by detecting interruptions, abnormal hang-ups, and energy level checks. However, accurately labeling customer and customer service emotions is costly and time-consuming, requiring a high level of expertise from the labelers. Consequently, machine quality inspection often lacks sufficient label data and suffers from unevenly distributed label data, making it difficult for machines to identify customer and customer service emotions. Summary of the Invention
[0005] This invention provides a quality inspection method based on knowledge distillation and multimodal integration, employing the following technical solution:
[0006] A quality inspection method based on knowledge distillation and multimodal integration includes:
[0007] Build and train a quality inspection audio model;
[0008] Build and train a quality inspection text model;
[0009] The audio data is input into the trained quality inspection audio model to learn and obtain audio feature vectors;
[0010] Text data is input into a trained quality inspection text model to learn and obtain text feature vectors;
[0011] The audio feature vector and the text feature vector are fused to form a new fused feature vector;
[0012] Sentiment prediction is performed based on the fused feature vectors.
[0013] Furthermore, the quality inspection audio model is a first-teacher-student knowledge distillation model.
[0014] Furthermore, in the first teacher-student knowledge distillation model, the teacher model T is a squeezeBERT and the student model S is a CNN model.
[0015] Furthermore, the method for training the first teacher-student knowledge distillation model is as follows:
[0016] The teacher model T was fine-tuned using the original audio data M to obtain a teacher model T1 suitable for the quality inspection emotion classification task;
[0017] The teacher model T1 predicts new audio data M1 to obtain hard labels, which are then used to pre-train the student model S.
[0018] The parameters obtained from the pre-training guidance are shared to obtain the student model S1;
[0019] The teacher model T1 is used to predict the original audio data M and the new audio data M1 to obtain a set of soft labels and a set of hard labels;
[0020] The student model S1 is used to predict the original audio data M and the new audio data M1 to obtain a set of soft labels and a set of hard labels;
[0021] The student model S1 was trained by backpropagation using cross-entropy as the total loss to obtain the audio model S2.
[0022] Furthermore, the original audio data M is processed into a Mel spectrum and used as the input to the model.
[0023] Furthermore, the quality inspection text model is a second teacher-student knowledge distillation model.
[0024] Furthermore, in the second teacher-student knowledge distillation model, the teacher model R is squeezeBERT, and the student model P is the textCNN model.
[0025] Furthermore, the method for training the second teacher-student knowledge distillation model is as follows:
[0026] The teacher model R was fine-tuned using the original text data Q to obtain teacher model R1, which is suitable for the quality inspection emotion classification task.
[0027] The teacher model R1 predicts new text data Q1 to obtain hard labels, which are then used to pre-train the student model P.
[0028] The parameters obtained from the pre-training guidance are shared to obtain the student model P1;
[0029] Two sets of soft and hard labels were obtained by using the teacher model R1 and the student model P1 to predict the original text data Q and the new text data Q1, respectively.
[0030] The student model P1 was trained by backpropagation using cross-entropy as the total loss to obtain the text model P2.
[0031] Furthermore, the original text data Q mentioned above is processed into word vectors and then used as the input to the model.
[0032] Furthermore, the specific method for sentiment prediction based on the fused feature vector is as follows:
[0033] The fused feature vectors are input into a fully connected layer for sentiment prediction.
[0034] The advantage of this invention lies in the quality inspection method based on knowledge distillation and multimodal combination. For audio and text data in quality inspection, the squeezeBERT model is used as the teacher model, and data augmentation is combined to pre-train the student model. Network parameter sharing is used to increase the complexity of the student model. Finally, the cross-entropy loss function is used to train the student model with soft and hard labels to obtain audio and text models respectively. Then, the features of the two models are fused to predict the final quality inspection emotion classification result, which effectively improves the model recognition accuracy. Attached Figure Description
[0035] Figure 1 This is a schematic diagram of the quality inspection method based on knowledge distillation and multimodal combination of the present invention. Detailed Implementation
[0036] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.
[0037] like Figure 1The following describes a quality inspection method based on knowledge distillation and multimodal combination, comprising the following steps: S1: Building and training a quality inspection audio model. S2: Building and training a quality inspection text model. S3: Inputting audio data into the trained quality inspection audio model to learn audio feature vectors. S4: Inputting text data into the trained quality inspection text model to learn text feature vectors. S5: Fusing the audio feature vectors and text feature vectors to form a new fused feature vector. S6: Performing sentiment prediction based on the fused feature vector. This method fuses the features of the two models to predict the final quality inspection sentiment classification result, effectively improving the model's recognition accuracy. The method is described in detail below.
[0038] For step S1: Build and train the quality inspection audio model.
[0039] Preferably, in this embodiment, the quality inspection audio model is a first teacher-student knowledge distillation model. Knowledge distillation, as a model compression method, uses a "teacher-student" framework. After performing softmax classification on the teacher model, the probability distribution between 0 and 1 is used as soft labels, and the real labels are used as hard labels. The soft and hard labels are combined to train the student model so that it can obtain more knowledge from the teacher model.
[0040] In this embodiment, the teacher model T of the first teacher-student knowledge distillation model is a squeezeBERT, and the student model S is a CNN model.
[0041] As a preferred implementation method, the method for training the first teacher-student knowledge distillation model is as follows:
[0042] The teacher model T is fine-tuned using the original audio data M to obtain a teacher model T1 suitable for the quality inspection emotion classification task. SqueezeBERT is trained on a general corpus and needs to be fine-tuned using domain-specific corpora. Based on the pre-trained model squeezeBERT, only the parameters of the later layers need to be updated. According to the pre-trained model, the data is processed into the format required by the model and then fed into the model for training to obtain the fine-tuned model.
[0043] The teacher model T1 predicts new audio data M1 to obtain hard labels, which are then used to pre-train the student model S. The new audio data M1 is obtained by labeling new external quality-checked audio data.
[0044] The parameters obtained from the pre-training guidance are shared to obtain the student model S1.
[0045] The teacher model T1 is used to predict the original audio data M and the new audio data M1, resulting in a set of soft labels and a set of hard labels. The student model S1 is used to predict the original audio data M and the new audio data M1, resulting in a set of soft labels and a set of hard labels.
[0046] After the teacher model T1 produces its predicted output (hard labels), a Softmax (multi-class classification) or Sigmoid (binary classification) transformation is applied to obtain a softened probability distribution (soft labels), with values between 0 and 1, exhibiting a more moderate distribution. The hard labels represent the true annotations of the samples. The total loss is designed as a weighted average of the cross-entropy of the soft and hard labels (CE loss). A larger weighting coefficient for the soft label cross-entropy indicates a greater reliance on the teacher model T1 for transfer learning, helping the student model S1 more easily identify simple samples. However, in the later stages of training, the weight of soft targets needs to be appropriately reduced to allow the true annotations to help identify difficult samples. Furthermore, the inference performance of the teacher model T1 is generally superior to that of the student model S1.
[0047] Total_loss=λ×CE(ti,si)+(1-λ)×CE(yi,si)
[0048] Where λ is the balancing parameter, CE is the cross-entropy, si is the current output of student model S1, ti is the current soft-label output of teacher model T1, and yi is the original hard label. The cross-entropy loss of the combined soft and hard labels is used as the total loss for backpropagation to train student model S1, which becomes the final audio model S2.
[0049] Preferably, the original audio data M is processed into a Mel spectrum and used as the input to the model.
[0050] For step S2: Build a quality inspection text model and train it.
[0051] Similar to step S1, the quality inspection text model is the second teacher-student knowledge distillation model.
[0052] In a preferred implementation, the teacher model R of the second teacher-student knowledge distillation model is squeezeBERT, and the student model P is textCNN.
[0053] As a preferred implementation method, the method for training the second teacher-student knowledge distillation model is as follows:
[0054] The teacher model R was fine-tuned using the original text data Q to obtain teacher model R1, which is suitable for the quality inspection emotion classification task.
[0055] The teacher model R1 predicts new text data Q1 to obtain hard labels, which are then used to pre-train the student model P.
[0056] The parameters obtained from the pre-training guidance are shared to obtain the student model P1.
[0057] Two sets of soft and hard labels were obtained by using the teacher model R1 and the student model P1 to predict the original text data Q and the new text data Q1, respectively.
[0058] The student model P1 was trained by backpropagation using cross-entropy as the total loss to obtain the text model P2.
[0059] As a preferred implementation, the original text data Q described above is processed by word vectors and then used as the input to the model.
[0060] For step S3: Input the audio data into the trained quality inspection audio model to learn and obtain the audio feature vector.
[0061] Specifically, the audio data to be analyzed is input into a trained quality inspection audio model, and audio feature vectors are obtained through the convolutional layers of the quality inspection audio model.
[0062] Preferably, the audio data to be analyzed is preprocessed before being input into the trained quality inspection audio model.
[0063] For step S4: Input the text data into the trained quality inspection text model to learn and obtain the text feature vector.
[0064] Similarly, the text data to be analyzed is input into the trained quality inspection text model, and the text feature vector is obtained through the convolutional layer of the quality inspection text model.
[0065] Preferably, the text data to be analyzed is preprocessed before being input into the trained audio quality inspection model.
[0066] For step S5: Fuse the audio feature vector and the text feature vector to form a new fused feature vector.
[0067] The audio feature vectors and text feature vectors learned from the two models above are concatenated and fused to obtain a new fused feature vector.
[0068] For step S6: perform sentiment prediction based on the fused feature vector.
[0069] The fused feature vector obtained in step S5 is fed into the fully connected layer for prediction, and the prediction result is obtained.
[0070] Specifically, the final result of sentiment label prediction is the average of N segments. The input audio is cut into N segments of fixed length, and the audio features are also divided into N segments. After feature fusion, they are also fused into n segments. Therefore, the final sentiment label output of the model is obtained by averaging the N segments.
[0071] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.
Claims
1. A quality inspection method based on knowledge distillation and multimodal combination, characterized in that, Include: Build and train a quality inspection audio model; Build and train a quality inspection text model; The audio data is input into the trained quality inspection audio model to learn and obtain audio feature vectors; Text data is input into the trained quality inspection text model to learn and obtain text feature vectors; The audio feature vector and the text feature vector are fused to form a new fused feature vector; Sentiment prediction is performed based on the fused feature vector; The quality inspection audio model is a first teacher-student knowledge distillation model, where the teacher model T of the first teacher-student knowledge distillation model is squeezeBERT and the student model S is a CNN model. The method for training the first teacher-student knowledge distillation model is as follows: The teacher model T is fine-tuned using the original audio data M to obtain a teacher model T1 suitable for the quality inspection emotion classification task. The fine-tuning is to update the parameters of the later layers based on the pre-trained model squeezeBERT. The teacher model T1 predicts new audio data M1 to obtain hard labels, which are then used to pre-train the student model S. The new audio data M1 is obtained by labeling new external quality inspection audio data. The parameters obtained from the pre-training guidance are shared to obtain the student model S1; The teacher model T1 is used to predict the original audio data M and the new audio data M1 to obtain a set of soft labels and a set of hard labels; The student model S1 is used to predict the original audio data M and the new audio data M1 to obtain a set of soft labels and a set of hard labels; The student model S1 is trained by backpropagation using the cross-entropy loss of the soft and hard labels as the total loss, and becomes the final audio model S2. The quality inspection text model is the second teacher-student knowledge distillation model, where the teacher model R of the second teacher-student knowledge distillation model is squeezeBERT and the student model P is the textCNN model. The method for training the second teacher-student knowledge distillation model is as follows: The teacher model R was fine-tuned using the original text data Q to obtain teacher model R1, which is suitable for the quality inspection emotion classification task. The teacher model R1 predicts new text data Q1 to obtain hard labels, which are then used to pre-train the student model P. The parameters obtained from the pre-training guidance are shared to obtain the student model P1; Two sets of soft and hard labels were obtained by using the teacher model R1 and the student model P1 to predict the original text data Q and the new text data Q1, respectively. The student model P1 was trained by backpropagation using cross-entropy as the total loss to obtain the text model P2. Furthermore, the input audio is cut into N segments of fixed length, and the audio features are also divided into N segments. After feature fusion, they are also fused into n segments. Therefore, the final sentiment label output of the model is obtained by averaging the N segments.
2. The quality inspection method based on knowledge distillation and multimodal combination according to claim 1, characterized in that, The original audio data M is processed into a Mel spectrum and used as the input to the model.
3. The quality inspection method based on knowledge distillation and multimodal combination according to claim 2, characterized in that, The original text data Q mentioned above is processed into word vectors and then used as the input to the model.
4. The quality inspection method based on knowledge distillation and multimodal combination according to claim 3, characterized in that, The specific method for emotion prediction based on the fused feature vector is as follows: The fused feature vector is input into a fully connected layer for sentiment prediction.