Customer service quality evaluation method

Abstract

The embodiment of the invention provides a customer service quality evaluation method, and the method comprises the steps: obtaining call information in a call process between a user and a customer service, the call information comprises call text information, call voice information and call feature information, and the call feature information is determined based on the call text information and the call voice information; processing the call text information, the call voice information and the call feature information through a first model in a service quality evaluation model to obtain a call content vector, a voice emotion vector and a call feature vector; and processing the call content vector, the voice emotion vector and the call feature vector through a second model in the service quality evaluation model to generate user evaluation information. According to the embodiment of the invention, the accuracy of evaluating the service quality of the customer service staff is improved.

Description

Technical Field

[0001] The present application belongs to the technical field of customer service quality assessment, and in particular, relates to a customer service quality assessment method. Background Art

[0002] With the continuous development of the service industry, various telecom operators, banks, e-commerce after-sales, etc. have telephone customer service. Telephone customer service communicates directly with users to provide users with consulting, problem solving and other services, which can directly affect the user's service experience and the corporate image in the user's mind. Therefore, it is crucial to accurately evaluate the quality of customer service.

[0003] However, in real life, customer service quality is generally evaluated based on the evaluation information provided by users after a call, such as the ratings entered by users on a keyboard, feedback from users on evaluation text messages received by electronic devices, etc. Since the evaluation information provided by users is entirely dependent on individual subjective consciousness, some user feedback cannot truly reflect the customer service quality, resulting in a low accuracy in evaluating the customer service quality. Summary of the invention

[0004] The embodiment of the present application provides a customer service quality evaluation method, which improves the accuracy of evaluating customer service quality.

[0005] In a first aspect, an embodiment of the present application provides a customer service quality evaluation method, the method comprising:

[0006] Obtaining call information during a call between a user and customer service, including call text information, call voice information, and call feature information, where the call feature information is determined based on the call text information and call voice information;

[0007] The call text information, the call voice information and the call feature information are processed by the first model in the service quality evaluation model to obtain a call content vector, a voice emotion vector and a call feature vector;

[0008] The call content vector, the voice emotion vector and the call feature vector are processed by the second model in the service quality evaluation model to generate user evaluation information;

[0009] Among them, the call content vector is used to reflect the user's satisfaction with the customer service of the customer service from the aspect of call content; the voice emotion vector is used to reflect the user's satisfaction with the customer service of the customer service from the aspect of user emotion; the call feature vector is used to reflect the user's satisfaction with the customer service of the customer service from the aspect of call characteristics; the user evaluation information is used to evaluate the user's satisfaction with the customer service quality of the customer service during the call.

[0010] In an optional implementation of the first aspect, the first model includes a first sub-model, a second sub-model, and a third sub-model;

[0011] The call text information, call voice information and call feature information are processed by the first model in the service quality evaluation model to obtain a call content vector, a voice emotion vector and a call feature vector, including:

[0012] Input the call text information into the first sub-model, the first sub-model includes J neural network layers, obtain the output information of I neural network layers in the first sub-model respectively, perform maximum pooling processing on the I output information, and obtain the call content vector, wherein I is less than or equal to J, the I neural network layers include the Jth neural network layer, the J-1th neural network layer, ... the J-K+1th neural network layer, J=I+K, and I, J, and K are all positive integers;

[0013] Inputting the call voice information into the second sub-model, extracting at least one acoustic feature from the call voice information through the second sub-model, and processing the at least one acoustic feature using an attention mechanism to obtain a voice emotion vector;

[0014] The call feature information is input into the third sub-model, which includes multiple decision trees, each decision tree includes multiple leaf nodes, and the call feature vector is obtained by mapping the multiple features included in the call feature information to the multiple leaf nodes of the multiple decision trees.

[0015] In an optional implementation of the first aspect, the call content vector, the voice emotion vector, and the call feature vector are processed by the second model in the service quality evaluation model to generate user evaluation information, including:

[0016] The call content vector, the voice emotion vector and the call feature vector are processed by the second model in the service quality evaluation model to obtain call content embedding information, voice emotion embedding information and call feature embedding information;

[0017] The cross-attention mechanism is used to fuse the call content embedding information, voice emotion embedding information, and call feature embedding information to obtain the total fusion vector.

[0018] User evaluation information is generated based on the total fusion vector, voice emotion vector and call feature vector.

[0019] In an optional implementation of the first aspect, the call content embedding information, the voice emotion embedding information, and the call feature embedding information are fused using a cross attention mechanism to obtain a total fusion vector, including:

[0020] Fusing the voice emotion embedding information and the call feature embedding information to obtain first fused embedding information, where the number of dimensions of the first fused embedding information is the same as the number of dimensions of the call content embedding information;

[0021] Take the call content embedding information as the key vector, the call content embedding vector as the value vector, and the first fused embedding vector as the query vector. Input the key vector, value vector and query vector into the cross-attention mechanism function. Process the key vector, value vector and query vector through the cross-attention mechanism function to obtain the total fused vector.

[0022] In an optional implementation of the first aspect, before processing the call text information, the call voice information, and the call feature information by the first model in the service quality evaluation model to obtain the call content vector, the voice emotion vector, and the call feature vector, the method further includes:

[0023] Obtain a training sample set, the training sample set includes multiple training samples and sample labels corresponding to each training sample, each training sample includes a call information sample during a call between a user and customer service, the call information sample includes a text information sample, a voice information sample, and a feature information sample, and the feature information sample is determined based on the text information sample and the voice information sample;

[0024] For each training sample, perform the following operations:

[0025] The text information sample, the voice information sample and the feature information sample corresponding to the training sample are processed by the first preset model in the preset service quality evaluation model to obtain the call content vector, the voice emotion vector and the call feature vector corresponding to the training sample;

[0026] The call content vector, the voice emotion vector and the call feature vector corresponding to the training sample are processed by a second preset model in the preset service quality evaluation model to obtain reference user evaluation information;

[0027] Based on the first loss function, the reference user evaluation information corresponding to the target training sample and the sample label corresponding to the target training sample are processed to obtain a first loss function value, where the target training sample is any one of the multiple training samples;

[0028] Processing the call content vector, the voice emotion vector, and the call feature vector corresponding to the target training sample based on the second loss function to obtain a second loss function value;

[0029] Based on the target loss function value of the preset service quality evaluation model, the target loss function value is equal to the sum of the first loss function value and the second loss function value, and the preset service quality evaluation model is trained using multiple training samples to obtain a trained service quality evaluation model.

[0030] In an optional implementation of the first aspect, before processing the call content vector, the voice emotion vector, and the call feature vector corresponding to the target training sample based on the second loss function to obtain the second loss function value, the method further includes:

[0031] Obtain a first sub-loss function, a second sub-loss function, and a third sub-loss function, where the dimensional information of the sample includes call content dimensional information, voice emotion dimensional information, and call feature dimensional information, the first sub-loss function is used to learn the contrast loss between the same dimensional information of different samples, the second sub-loss function is used to learn the contrast loss between different dimensional information of different samples, and the third sub-loss function is used to learn the difference loss of different dimensional information of the same sample;

[0032] Based on the first sub-loss function, the second sub-loss function, the third sub-loss function, and the preset loss function weights, a second loss function is constructed.

[0033] In an optional implementation of the first aspect, dimension information of the first sample is obtained, where the dimension information of the first sample includes call content dimension information, voice emotion dimension information, and call feature dimension information;

[0034] For the first dimension information of the first sample, obtain the first dimension information of N first positive samples and the first dimension information of M first negative samples, where the first dimension information is any one of call content dimension information, voice emotion dimension information, and call feature dimension information;

[0035] Constructing a first sub-loss function based on the first dimension information of the N first positive samples and the first dimension information of the M first negative samples;

[0036] For the first dimensional information of the first sample, obtain the second dimensional information of P second positive samples and the second dimensional information of Q second negative samples, where the second dimensional information is the dimensional information of the call content dimensional information, the voice emotion dimensional information and the call feature dimensional information except the first dimensional information;

[0037] Constructing a second sub-loss function based on the second dimension information of the P second positive samples and the second dimension information of the Q second negative samples;

[0038] For the first dimension information of the first sample, the second dimension information is used as a positive sample to construct a third sub-loss function;

[0039] Among them, the sample labels of the N first positive samples and the sample labels of the P second positive samples are the same as the sample label of the first sample, the sample labels of the M first negative samples and the sample labels of the Q second negative samples are different from the sample label of the first sample, and N, M, P, and Q are all positive integers.

[0040] In an optional implementation of the first aspect, obtaining a training sample set includes:

[0041] Obtaining a plurality of training samples to be processed and a sample label of each of the training samples to be processed, wherein the sample label of each training sample to be processed can be used to characterize the sample emotion of the training sample to be processed;

[0042] For each of the plurality of training samples to be processed, matching the text content of the training sample to be processed by using the text matching information to obtain a matching result, wherein the text matching information includes a plurality of text matching elements and an element emotion label of each text matching element, wherein the element emotion reflected by the element emotion label of each text matching element is an emotion of a different emotion type from the sample emotion;

[0043] A plurality of training samples whose matching results meet preset conditions are determined from the plurality of training samples to be processed, and a sample label of each training sample is determined based on the sample label of each training sample to be processed.

[0044] In an optional implementation of the first aspect, the text matching information includes word information and / or sentence information;

[0045] The word information includes multiple words to be matched and the word emotion label of each word to be matched, the sentence information includes multiple sentences to be matched and the sentence emotion label of each sentence to be matched, the word emotion of each word to be matched and the sentence emotion of each sentence to be matched are emotions of different emotion types from the sample emotion, the word emotion of each word to be matched is determined based on the word emotion label of the word to be matched, and the sentence emotion of each sentence to be matched is determined based on the sentence emotion label of the sentence to be matched.

[0046] In an optional implementation of the first aspect, the emotion type includes a first emotion type, a second emotion type, and a third emotion type;

[0047] When the sample emotion of the training sample to be processed is of the first emotion type, the emotion type of the word emotion of each word to be matched is of the second emotion type or the third emotion type, and / or the emotion type of the sentence emotion of each sentence to be matched is of the second emotion type or the third emotion type;

[0048] When the sample emotion of the training sample to be processed is of the second emotion type, the emotion type of the word emotion of each word to be matched is of the first emotion type or the third emotion type, and / or the emotion type of the sentence emotion of each sentence to be matched is of the first emotion type or the third emotion type;

[0049] When the sample emotion of the training sample to be processed is the third emotion type, the emotion type of the word emotion of each word to be matched is the first emotion type or the second emotion type, and / or the emotion type of the sentence emotion of each sentence to be matched is the first emotion type or the second emotion type.

[0050] In a second aspect, an embodiment of the present application provides a customer service quality evaluation device, the device comprising:

[0051] An acquisition module is used to acquire call information during a call between a user and a customer service representative. The call information includes call text information, call voice information, and call feature information. The call feature information is determined based on the call text information and call voice information.

[0052] A processing module, used to process the call text information, call voice information and call feature information through the first model in the service quality evaluation model to obtain a call content vector, a voice emotion vector and a call feature vector;

[0053] The processing module is further used to process the call content vector, the voice emotion vector and the call feature vector through the second model in the service quality evaluation model to generate user evaluation information;

[0054] Among them, the call content vector is used to reflect the user's satisfaction with the customer service of the customer service from the aspect of call content; the voice emotion vector is used to reflect the user's satisfaction with the customer service of the customer service from the aspect of user emotion; the call feature vector is used to reflect the user's satisfaction with the customer service of the customer service from the aspect of call characteristics; the user evaluation information is used to evaluate the user's satisfaction with the customer service quality of the customer service during the call.

[0055] In a third aspect, an electronic device is provided, comprising: a memory for storing computer program instructions; and a processor for reading and running the computer program instructions stored in the memory to execute the customer service quality assessment method provided by any optional implementation method in the first aspect.

[0056] In a fourth aspect, a computer storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, a customer service quality assessment method provided by any optional implementation method in the first aspect is implemented.

[0057] In a fifth aspect, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the customer service quality evaluation method provided by any optional implementation method in the first aspect is implemented.

[0058] In an embodiment of the present application, call information during a call between a user and customer service can be obtained, and the call information includes call text information, call voice information, and call feature information. The call feature information is determined based on the call text information and the call voice information, and the acquired call text information, call voice information, and call feature information are processed accordingly through a service quality evaluation model to analyze the user's satisfaction with the customer service of the customer service from aspects such as call content, user emotions, and call characteristics, and generate user evaluation information. The user evaluation information is used to evaluate the user's customer service quality of the customer service during the call, avoiding the situation in the prior art where users randomly score and the scoring is relatively subjective, thereby improving the accuracy of the customer service quality evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0060] Figure 1 This is one of the flow charts of a customer service quality evaluation method provided in an embodiment of the present application;

[0061] Figure 2 This is the second flow chart of a customer service quality evaluation method provided in an embodiment of the present application;

[0062] Figure 3 This is a flowchart of a customer service quality evaluation method provided in an embodiment of the present application;

[0063] Figure 4 It is a flowchart of a training method for a service quality assessment model provided in an embodiment of the present application;

[0064] Figure 5 This is a fourth flow chart of a customer service quality evaluation method provided in an embodiment of the present application;

[0065] Figure 6 This is a flowchart of a customer service quality evaluation method provided in an embodiment of the present application;

[0066] Figure 7 It is a structural schematic diagram of a customer service quality evaluation device provided in an embodiment of the present application;

[0067] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0069] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0070] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0071] With the continuous development of the service industry, various telecom operators, banks, e-commerce after-sales, etc. have telephone customer service. Telephone customer service communicates directly with users to provide users with consulting, problem solving and other services. Whether users are satisfied with the service of telephone customer service has an important impact on the corporate image in the minds of users, user return rate and user stickiness. Therefore, how to accurately evaluate the quality of customer service is an urgent problem to be solved.

[0072] However, in real life, customer service quality is generally evaluated by the evaluation information fed back by the user after the call, such as the rating entered by the user through the keyboard, the user's feedback on the evaluation text message received by the electronic device, etc. Because the evaluation information fed back by the user depends entirely on the individual's subjective consciousness, there may be some user feedback that cannot truly reflect the customer service quality, which may lead to a low accuracy in evaluating the customer service quality.

[0073] In addition, since the process of users providing feedback on customer service quality is relatively cumbersome, most users are not inclined to spend time providing feedback after the call with customer service. This may result in less user evaluation information on customer service quality, making it difficult to accurately evaluate customer service quality.

[0074] Based on this, in order to solve the problem of low accuracy in evaluating customer service quality in the prior art, an embodiment of the present application provides a customer service quality evaluation method, which can obtain call information during a call between a user and customer service, the call information including call text information, call voice information and call feature information, the call feature information being determined based on the call text information and the call voice information, and the acquired call text information, call voice information and call feature information being processed accordingly through a service quality evaluation model, so as to analyze the user's satisfaction with the customer service of the customer service from aspects such as call content, user emotions and call characteristics, and generate user evaluation information, which is used to evaluate the user's customer service quality of the customer service during the call, thereby avoiding the situation in the prior art where users randomly score and the scoring is relatively subjective, thereby improving the accuracy of the customer service quality evaluation.

[0075] It should be noted that the execution subject of the customer service quality evaluation method provided in the embodiment of the present application can be a customer service quality evaluation device, or a control module in the customer service quality evaluation device for executing the customer service quality evaluation method. In the embodiment of the present application, the customer service quality evaluation method provided in the embodiment of the present application is explained by taking the customer service quality evaluation device executing the customer service quality evaluation method as an example.

[0076] The customer service quality evaluation method provided in the embodiment of the present application is described in detail below through specific embodiments in conjunction with the accompanying drawings.

[0077] Figure 1 It is a flow chart of a customer service quality evaluation method provided in an embodiment of the present application.

[0078] like Figure 1 As shown, the execution subject of the method may be a customer service quality evaluation device. Based on this, the method may specifically include the following steps:

[0079] S110, obtaining call information during the call between the user and the customer service.

[0080] In some embodiments, the call information may include call text information, call voice information, and call feature information, wherein the call feature information may be determined based on the call text information and the call voice information.

[0081] For ease of understanding, it should be noted that call text information may include call content in text form, which is the communication content between the user and customer service during the call. Correspondingly, call voice information may include call content in voice form.

[0082] Specifically, during a conversation between a user and a customer service representative, the customer service quality evaluation device can directly obtain the conversation voice information during the conversation between the user and the customer service representative, and then obtain the conversation text information by converting the conversation voice information from a voice form into a text form, and then obtain the conversation feature information by mining and analyzing the conversation voice information and the conversation text information. In this way, the customer service quality evaluation device can obtain the conversation text information, the conversation voice information, and the conversation feature information, that is, it can obtain the conversation information during the conversation between the user and the customer service representative.

[0083] S120, processing the call text information, call voice information and call feature information through the first model in the service quality evaluation model to obtain a call content vector, a voice emotion vector and a call feature vector.

[0084] The service quality evaluation model is a model obtained through pre-training and used to evaluate the service quality of customer service. The first model is a sub-model of the service quality model, and the first model is used to process the call text information, call voice information and call feature information included in the call information, so as to reflect the user's satisfaction with the service quality of the customer service from aspects such as call content, user emotions and call features.

[0085] In some embodiments, the call content vector is used to reflect the user's satisfaction with the customer service service from the perspective of call content, the voice emotion vector is used to reflect the user's satisfaction with the customer service service from the perspective of user emotions, and the call feature vector is used to reflect the user's satisfaction with the customer service service from the perspective of call characteristics.

[0086] Specifically, after obtaining the call information during the call between the user and the customer service, the customer service quality evaluation device can input the call text information, call voice information and call feature information included in the call information into the first model in the service quality evaluation model, so as to process the call text information through the first model of the service quality evaluation model to obtain a call content vector corresponding to the call text information, and process the call voice information through the first model of the service quality evaluation model to obtain a voice emotion vector corresponding to the call voice information, and process the call feature information through the first model of the service quality evaluation model to obtain a call feature vector corresponding to the call feature information.

[0087] S130, processing the call content vector, the voice emotion vector and the call feature vector through the second model in the service quality evaluation model to generate user evaluation information.

[0088] Among them, the second model is a sub-model in the service quality evaluation model, and the second model is used to fuse the call content vector, voice emotion vector and call feature vector output by the first model to obtain user evaluation information.

[0089] The user evaluation information is used to evaluate the quality of customer service provided by the user to the customer service during the call. For example, the user evaluation information may include evaluation scores, evaluation levels, and other relevant information that can be used to evaluate the quality of customer service provided by the user to the customer service during the call.

[0090] Specifically, after the customer service quality evaluation device obtains the call content vector, voice emotion vector and call feature vector in the first model in the service quality evaluation model, the call content vector, voice emotion vector and user feature vector can be input into the second model of the service quality evaluation model, and the call content vector, voice emotion vector and user feature vector can be processed by the second model of the service quality evaluation model to obtain user evaluation information.

[0091] In the embodiment of the present application, the call information during the call between the user and the customer service can be obtained, and the call information includes call text information, call voice information and call feature information, and the call feature information is determined based on the call text information and call voice information, and the acquired call text information, call voice information and call feature information are processed accordingly through the service quality evaluation model to analyze the user's satisfaction with the customer service of the customer service from aspects such as the call content, user emotions and call features, and generate user evaluation information, which is used to evaluate the customer service quality of the customer service during the call. In this way, the real user evaluation information can be extracted from the call information during the call between the user and the customer service, avoiding the situation where the accuracy of evaluating the customer service quality is low due to the user's arbitrary scoring, thereby improving the accuracy of evaluating the customer service quality.

[0092] In S120, the first model in the service quality evaluation model processes the call text information, the call voice information and the call feature information respectively. In order to accurately describe the specific processing process of the first model on the call text information, the call voice information and the call feature information, so as to obtain accurate call content vectors, voice emotion vectors and call feature vectors. In some embodiments, the first model may include a first sub-model, a second sub-model and a third sub-model, such as Figure 2 As shown, based on this, the above-mentioned S120 may specifically include the following steps:

[0093] S210, input the call text information into the first sub-model, obtain the output information output by I neural network layers in the first sub-model respectively, perform maximum pooling processing on the I output information, and obtain the call content vector.

[0094] The first sub-model may be a related model for identifying call text information, which is not specifically limited here. In addition, the output information may include text features captured by the corresponding neural network layer from the call text information, which is not limited here.

[0095] In some embodiments, the first sub-model may include J neural network layers, where I is less than or equal to J. The I neural network layers may include the Jth neural network layer, the J-1th neural network layer, ... the J-K+1th neural network layer, where J=I+K, and I, J, and K are all positive integers.

[0096] Specifically, after obtaining the call text information, the customer service quality evaluation device can input the call text information into the first sub-model. Since the first sub-model can include J neural network layers, the customer service quality evaluation device can obtain the output information output by I neural network layers in the first sub-model respectively, and then perform maximum pooling processing on the I output information to obtain the call content vector.

[0097] In one example, the first sub-model can be a Transformer-based bidirectional encoding representation (BERT, Bidirectional Encoder Representations from Transformers) model. Assuming that the BERT model can include J neural network layers, considering that after the call text information is input into the BERT model, each neural network layer in the BERT model can capture different text features from the call text information, so that the customer service quality evaluation method provided in the embodiment of the present application can obtain the output information of each of the I neural network layers in the J neural network layers included in the BERT model, and perform maximum pooling processing on the I output information to obtain the call content vector. In this way, the method of integrating the text features of multiple layers of output avoids the problem of low accuracy of obtaining the call content vector in the prior art that only the output information of the last layer of the model is used for processing, thereby improving the accuracy of obtaining the call content vector.

[0098] It should also be noted that, since the representation ability learned by the bottom layer of BERT is relatively general, the representation ability learned by the top layer is more task-specific, that is, more suitable for the fine-tuned classification target task. Therefore, when training the BERT model, the customer service quality evaluation method provided in the embodiment of the present application can set different learning rates for different layers of the BERT model, and the learning rate decreases layer by layer from the top layer to the bottom layer, thereby improving the speed and performance of model convergence.

[0099] S220, input the call voice information into the second sub-model, extract at least one acoustic feature from the call voice information through the second sub-model, and process the at least one acoustic feature using the attention mechanism to obtain a voice emotion vector.

[0100] Among them, the second sub-model is used to process call voice information and is used to identify relevant models of user emotions, which are not specifically limited here.

[0101] In addition, the at least one acoustic feature may include at least one of Mel-Frequency Cepstral Coefficients (MFCC), spectrogram, and Wav2vec2 Embedding, etc. Wav2vec2Embedding is the output of the wav2vec2 model, and the wav2vec2 model is an unsupervised speech pre-training model.

[0102] Considering that the user's tone and intonation will reflect the user's emotions, such as happy emotions, angry emotions, calm emotions, etc., and the user's emotions can reflect the user's satisfaction with the customer service to a certain extent, for example, if the user's emotion is happy, it means that the user is relatively satisfied with the customer service, and if the user's emotion is angry, it means that the user is dissatisfied with the customer service. Therefore, after obtaining the call voice information, the customer service quality evaluation device can input the call voice information into the second sub-model, extract at least one acoustic feature from the call voice information through the second sub-model, and use the attention mechanism to process the at least one acoustic feature to obtain a voice emotion vector.

[0103] More specifically, after the call voice information is input into the second sub-model, at least one acoustic feature is extracted from the call voice information through the second sub-model, and the at least one acoustic feature is input into the corresponding feature encoder in the second sub-model to obtain an acoustic feature vector corresponding to each acoustic feature, and then the at least one acoustic feature vector can be processed using the attention mechanism to obtain a voice emotion vector.

[0104] S230, inputting the call feature information into the third sub-model, and obtaining a call feature vector by mapping a plurality of features included in the call feature information to a plurality of leaf nodes of a plurality of decision trees.

[0105] In some embodiments, the third sub-model may include multiple decision trees, each of which may include multiple leaf nodes. In addition, the call feature information may include multiple features.

[0106] Specifically, after acquiring the call feature information, the customer service quality evaluation device can input the call feature information into the third sub-model. Since the third sub-model can include multiple decision trees and each decision tree can include multiple leaf nodes, the call feature vector can be obtained by mapping the multiple features included in the call feature information to the multiple leaf nodes of the multiple decision trees.

[0107] In one example, the third sub-model may be a gradient boosting decision tree (GBDT) model. If the GBDT model may include 100 decision trees, each decision tree may include 12 leaf nodes. If a feature falls on a leaf node, the element at the leaf node is 1, otherwise it is 0. In this way, after obtaining the call feature information, the customer service quality evaluation device may map the multiple features included in the call feature information to the multiple leaf nodes of the multiple decision trees included in the third model. In this way, a call feature vector in which multiple elements each take the value of 0 or 1 can be obtained. The vector length of the call feature vector is the total number of all leaf nodes contained in all trees in the GBDT. That is, the customer service quality evaluation device can obtain a call feature vector containing 100×12 elements.

[0108] It should also be noted that the call feature information may include multiple features, which may be specifically as follows:

[0109] (1) Call length feature: Specifically, the number of calls between the customer and the customer service representative is used as the call length feature.

[0110] (2) The number of words and / or sentences that can reflect negative emotions contained in the text content of the call information corresponding to the call feature information.

[0111] (3) The number of words that can reflect positive emotions and / or the number of sentences that can reflect positive emotions contained in the text content of the call information corresponding to the call feature information.

[0112] (4) Number of speakers. Considering that in real life, due to problems such as signals and omissions, only the user or the customer service staff speaks unilaterally while the other party does not respond after the user and the customer service are connected, which may lead to low scores from the user, the number of speakers feature is used as a feature in the call feature information. The number of speakers feature can be the number of speakers during the call. If only the user or the customer service staff is speaking during the call, the number of speakers feature is 1. If both the user and the customer service staff are speaking during the call, the number of speakers feature can be 2. It should be noted that the number of speakers feature is determined by the number of specific speakers during the call, and is limited to the two cases where the number of speakers feature is 1 or 2, which will not be elaborated on here.

[0113] (5) Customer service history rating feature. Specifically, the average rating of multiple user ratings of the corresponding customer service in the historical time period can be obtained. For example, 10 user ratings of the customer service in the historical time period can be obtained to calculate the corresponding average rating. It should be noted that if there is no historical user rating for the customer service, its customer service history rating feature can be set to a preset value, which can be based on actual experience or circumstances. For example, the preset value can be 5, which is not specifically limited here.

[0114] (6) Standardized ending features. By observing the data features, it is found that the endings of call messages with high user ratings are generally standardized, usually including "I wish you a happy life, goodbye.", "Looking forward to your very good comments, goodbye.", "I wish you a happy holiday, goodbye.", "I won't bother you, goodbye.", etc. At the end of call messages with low user ratings, there may be abnormal call interruptions or unpleasant calls that cause customers to hang up directly. Based on this, the customer service quality evaluation device can pre-define a list of standardized ending calls, calculate and store the text vector of each line of speech in the list. For a new call message, obtain the text vector of the customer service's last speech, and calculate the similarity with the text vector of each line of speech in the ending speech list, and use the maximum similarity as the standardized ending feature of speech.

[0115] From the specific contents of the multiple features included in the call feature information, it can be seen that the multiple features included in the call feature information are features of different dimensions. In order to make the multiple features comparable so that a more accurate call feature vector can be obtained later, before the call feature information is input into the third model, it is necessary to normalize the multiple features included in the call feature information respectively to obtain the normalized call feature information, that is, to normalize each feature included in the call feature information to a continuous value between the interval [0, 1].

[0116] In this embodiment, the first sub-model can be used to process the call text information, and the output information of multiple layers in the first sub-model can be integrated to obtain a more accurate call content vector. The second sub-model can be used to process the call voice information, and at least one acoustic feature can be extracted from the call voice information. The at least one acoustic feature is processed using the attention mechanism to obtain a voice emotion vector. The third sub-model can also be used to process the multiple features included in the call feature information to obtain a call feature vector. In this way, the call content vector, voice emotion vector and call feature vector can be accurately obtained, which facilitates the subsequent accurate acquisition of user evaluation information and improves the accuracy of evaluating customer service quality.

[0117] In addition, considering that the input information of the first sub-model may have a length limit, that is, the number of characters included in the input information of the first sub-model needs to be less than the preset number of characters, and the preset number of characters can be pre-set based on actual experience or circumstances. For example, the preset number of characters can be set to 512. The number of characters included in the call text information may exceed the preset number of characters. If only the part of the call text information that exceeds the preset number of characters is truncated, information loss may occur, which in turn affects the accuracy of the call content vector output by the first sub-model. Therefore, in one embodiment, before S210, the customer service quality evaluation method may also include the following steps:

[0118] The call text information is processed using a summary generation tool to obtain a text summary corresponding to the call text information.

[0119] Based on this, the above-mentioned S210 may specifically include the following steps:

[0120] The text summary corresponding to the call text information is input into the first sub-model, the output information output by I neural network layers in the first sub-model is obtained, and the I output information is subjected to maximum pooling processing to obtain a call content vector.

[0121] The summary generation tool may be a tool for extracting key information from call text information and generating a summary. For example, the summary generation tool may be the Chinese language model Llama.

[0122] Specifically, the customer service quality evaluation device needs to process the call text information using a summary generation tool before inputting the call text information into the first sub-model to obtain a text summary corresponding to the call text information, and then use the text summary instead of the call text information as the input information of the first sub-model, input the text summary corresponding to the call text information into the first sub-model, and then obtain the output information output by I neural network layers in the first sub-model, perform maximum pooling processing on the I output information, and obtain a call content vector.

[0123] In this embodiment, a text summary can be extracted from the call text information, and the text summary is used to replace the entire call text information as the input information of the first sub-model, and the text summary is input into the first sub-model to obtain the call content vector. In this way, not only the situation that the first sub-model cannot process the call text information due to its length is avoided, but also the accuracy of the output call content vector is guaranteed as much as possible, and the situation of information loss is avoided.

[0124] In order to accurately describe the specific process of processing the call content vector, the voice emotion vector and the call feature vector by the second model in the service quality evaluation model to generate user evaluation information, in one embodiment, Figure 3 As shown, the above-mentioned S130 may specifically include the following steps:

[0125] S310, processing the call content vector, the voice emotion vector and the call feature vector through the second model in the service quality evaluation model to obtain call content embedding information, voice emotion embedding information and call feature embedding information.

[0126] Among them, call content embedding information, voice emotion embedding information and call feature embedding information can all be related information in vector form.

[0127] Considering that the call content vector, voice emotion vector and call feature vector output by the first model are different types of vector data, in order to facilitate the subsequent fusion of the call content vector, voice emotion vector and call feature vector, it is necessary to convert the call content vector, voice emotion vector and call feature vector into data in the same vector space, and the modal embedding technology is a technology that converts different modal data (corresponding to call content vectors, voice emotion vectors and call feature vectors) into a universal vector representation.

[0128] Therefore, after obtaining the call content vector, voice emotion vector and call feature vector, the customer service quality evaluation device can process the call content vector, voice emotion vector and call feature vector to obtain call content embedding information, voice emotion embedding information and call feature embedding information, thereby realizing the conversion of call content vector, voice emotion vector and call feature vector from different types of data into common vectors, so as to facilitate the subsequent fusion of the call content embedding information, voice emotion embedding information and call feature embedding information.

[0129] S320, using a cross-attention mechanism to fuse the call content embedding information, the voice emotion embedding information, and the call feature embedding information to obtain a total fusion vector.

[0130] Specifically, after obtaining the call content embedding information, the voice emotion embedding vector and the call feature embedding information, the customer service quality evaluation device can use the cross-attention mechanism to fuse the acquired call content embedding information, the voice emotion embedding information and the call feature embedding information, so as to capture the dependency and correlation between the call content embedding information, the voice emotion embedding information and the call feature embedding information, and thus obtain a more accurate total fusion vector.

[0131] S330, generating user evaluation information based on the total fusion vector, the voice emotion embedding information and the call feature embedding information.

[0132] Specifically, after obtaining the total fusion vector, the customer service quality evaluation device can generate user evaluation information based on the total fusion vector, the voice emotion vector and the call feature vector.

[0133] In an example, if the call content vector is represented by m1, the voice emotion vector is represented by m2, and the call feature vector is represented by m3, based on this, the call content vector corresponding to the call content embedding information can be represented as The speech emotion embedding information corresponding to the speech emotion vector can be expressed as The call feature embedding information corresponding to the call feature vector can be expressed as

[0134] Based on this, information is embedded based on the call content. Voice emotion embedding information And call feature embedding information The total fusion vector obtained by fusion Afterwards, the total fusion vector and speech emotion embedding information And call feature embedding information After concatenation, the data are fed into the classifier f (fully connected layer and softmax function) to obtain user evaluation information. The specific process satisfies the following formula (1):

[0135]

[0136] In this embodiment, it is possible to take into account that the call content vector, voice emotion vector and call feature vector are different types of vector data, and convert the call content vector, voice emotion vector and call feature into data in the same vector space, that is, to obtain call content embedding information, voice emotion embedding information and call feature embedding information, and to utilize the cross-attention mechanism to capture the dependency and correlation between the call content embedding information, voice emotion embedding information and call feature embedding information, and accurately obtain the total fusion vector, and then obtain accurate and true user evaluation information, thereby improving the accuracy of evaluating customer service quality.

[0137] Considering that the amount of information included in the call content embedding information is greater than the amount of information included in the voice emotion embedding information and the amount of information included in the call feature embedding information, it is not conducive to the fusion of the call content embedding information, the voice emotion embedding information and the call feature embedding information. In order to effectively fuse the call content embedding information, the voice emotion embedding information and the call feature embedding information to accurately obtain the total fusion vector, in one embodiment, the above-mentioned S320 may specifically include the following steps:

[0138] Fusing the voice emotion embedding information and the call feature embedding information to obtain first fused embedding information, where the number of dimensions of the first fused embedding information is the same as the number of dimensions of the call content embedding information;

[0139] Take the call content embedding information as the key vector, the call content embedding vector as the value vector, and the first fused embedding vector as the query vector. Input the key vector, value vector and query vector into the cross-attention mechanism function. Process the key vector, value vector and query vector through the cross-attention mechanism function to obtain the total fused vector.

[0140] Specifically, after obtaining the call content embedding information, the voice emotion embedding information and the call feature embedding information, the customer service quality evaluation device can first preliminarily fuse the voice emotion embedding information and the call feature embedding information to obtain the first fused embedding information with the same number of dimensions as the call content embedding information, that is, the amount of information included in the call content embedding information is the same as the amount of information included in the first fused embedding information. Then, the call content embedding information can be used as the key vector, the call content embedding vector can be used as the value vector, and the first fused embedding vector can be used as the query vector. The key vector, value vector and query vector can be input into the cross-attention mechanism function, and the key vector, value vector and query vector can be processed by the cross-attention mechanism function to obtain the total fused vector.

[0141] In one example, the specific process of fusing the speech emotion embedding information and the call feature embedding information to obtain the first fused embedding information may satisfy the following formula (2):

[0142]

[0143] As shown in formula (2), speech emotion embedding information Call feature embedding information The concatenation vector Need to go through an explicit conversion layer f att , converted into information embedded with the call content The first fusion embedding information of the same dimension e att .

[0144] The cross attention mechanism function satisfies the following formula (3):

[0145]

[0146] Among them, W Q , W K , W V is the parameter in the cross attention mechanism function.

[0147] In this way, the call content is embedded in the message Used as the key K (corresponding to the key vector) and value V (corresponding to the value vector) in the cross-attention mechanism to spell e att Used as query Q (corresponding to the query vector), the key vector, value vector and query vector are input into the cross attention mechanism function shown in the following formula (3) to obtain the total fusion vector

[0148] In this embodiment, the voice emotion embedding information and the call feature embedding information can be preliminarily fused to obtain the first fused embedding information with the same number of dimensions as the call content embedding information. Then, the call content embedding information can be used as the key vector, the call content embedding vector can be used as the value vector, and the first fused embedding vector can be used as the query vector. The key vector, the value vector and the query vector can be input into the cross-attention mechanism function to capture the dependency and correlation between the call content embedding information, the voice emotion embedding information and the call feature embedding information. The call content embedding information, the voice emotion embedding information and the call feature embedding information can be fully utilized, and the total fusion vector can be accurately obtained.

[0149] It should be noted that the customer service quality evaluation method provided in the embodiment of the present application needs to use a pre-trained service quality evaluation model to process the call information during the call between the user and the customer service. Therefore, before S120, the service quality evaluation model needs to be trained first. Therefore, the specific implementation method of the service quality evaluation model training method provided in the embodiment of the present application is described below in conjunction with the accompanying drawings.

[0150] Figure 4 It is a flowchart of a training method for a service quality assessment model provided in an embodiment of the present application.

[0151] like Figure 4 As shown, the execution subject of the method may be a customer service quality evaluation device, based on which, the method may include the following steps:

[0152] S410, obtaining a training sample set.

[0153] Before introducing this step, it should be noted that the service quality evaluation model needs to be iterated multiple times to adjust its loss function value until the loss function value meets the training stop condition to obtain the trained service quality evaluation model. However, in each iterative training, if only one training sample is input, the sample size is too small, which is not conducive to the training adjustment of the service quality evaluation model. That is, in the process of training the service quality evaluation model, multiple training samples are required to iteratively process the service quality evaluation model.

[0154] Therefore, the training sample set may include multiple training samples and sample labels for each training sample. Each training sample may include a call information sample during a call between a user and customer service. The call information sample may include a text information sample, a voice information sample, and a feature information sample. The feature information sample may be determined based on the text information sample and the voice information sample.

[0155] In addition, the sample label of each sample to be trained can be determined based on the user evaluation information fed back by the user on the call information sample in the training sample. For example, if the user evaluation information can be a score between the interval [0, 10], the sample label of the training sample with a score in the interval [0, 3] can be set to 0, the sample label of the training sample with a score in the interval [4, 7] can be set to 1, and the sample label of the training sample with a score in the interval [8, 10] can be set to 2, without specific limitation here.

[0156] S420, for each training sample, the text information sample, voice information sample and feature information sample corresponding to the training sample are processed by the first preset model in the preset service quality evaluation model to obtain the call content vector, voice emotion vector and call feature vector corresponding to the training sample.

[0157] The preset service quality evaluation model may be a model pre-set based on experience or circumstances and used to evaluate customer service quality. The first preset model may be a sub-model of the service quality model, and the first preset model is used to process the text information sample, the voice information sample and the feature information sample included in the call information respectively.

[0158] Specifically, after obtaining the training sample set, the customer service quality evaluation device can, for each training sample among the multiple training samples included in the training sample set, input the text information sample, voice information sample and feature information sample corresponding to the training sample into the first preset model of the preset service quality evaluation model, and process the text information sample, voice information sample and call information sample corresponding to the training sample through the first preset model of the preset service quality evaluation model to obtain the call content vector, voice emotion vector and call feature vector corresponding to the training sample.

[0159] In some embodiments, the call content vector can be used to reflect the user's satisfaction with the customer service service of the customer service from the aspect of call content, the voice emotion vector can be used to reflect the user's satisfaction with the customer service service of the customer service from the aspect of user emotion, and the call feature vector can be used to reflect the user's satisfaction with the customer service service of the customer service from the aspect of call characteristics.

[0160] S430, processing the call content vector, the voice emotion vector and the call feature vector corresponding to the training sample through the second preset model in the preset service quality evaluation model to obtain reference user evaluation information.

[0161] Among them, the second preset model can be a sub-model in the service quality evaluation model, and the second preset model is used to fuse the call content vector, voice emotion vector and call feature vector corresponding to the training sample output by the first model to obtain reference user evaluation information.

[0162] The reference user evaluation information can be used to evaluate the quality of customer service provided by the user to the customer service during the call. Based on this, the reference user evaluation information can include evaluation scores, evaluation levels and other relevant information that can be used to evaluate the quality of customer service provided by the user to the customer service during the call.

[0163] Specifically, after the customer service quality evaluation device obtains the call content vector, voice emotion vector and call feature vector corresponding to the training sample in the first preset model in the preset service quality evaluation model, the call content vector, voice emotion vector and user feature vector corresponding to the training sample are input into the second preset model of the preset service quality evaluation model, and the call content vector, voice emotion vector and user feature vector corresponding to the training sample are processed by the second preset model of the preset service quality evaluation model to obtain reference user evaluation information.

[0164] S440: Process the reference user evaluation information corresponding to the target training sample and the sample label corresponding to the target training sample based on the first loss function to obtain a first loss function value.

[0165] The target training sample is any one of the multiple training samples.

[0166] Specifically, the customer service quality evaluation device can process the reference user evaluation information corresponding to the target training sample and the sample label corresponding to the target training sample based on the first loss function to obtain the first loss function value.

[0167] In one example, the first loss function may be a cross entropy loss function L cross-entropyBased on this, the reference user evaluation information corresponding to the target training sample and the sample label corresponding to the target training sample are processed based on the first loss function, and the first loss function value satisfies the following formula (4):

[0168] L c = L cross-entropy (yy ′ ) (4)

[0169] Among them, L c is the first loss function value, y is the reference user evaluation information, y ′ is the sample label.

[0170] S450, processing the call content vector, the voice emotion vector, and the call feature vector corresponding to the target training sample based on the second loss function to obtain a second loss function value.

[0171] Specifically, the customer service quality evaluation device can process the call content vector, voice emotion vector and call feature vector corresponding to the target training sample based on the second loss function, that is, perform loss calculation on the intermediate information generated in the process of generating reference user evaluation information to obtain the second loss function value.

[0172] S460, based on the target loss function value of the preset service quality evaluation model, use multiple training samples to train the preset service quality evaluation model to obtain a trained service quality evaluation model.

[0173] In some embodiments, the target loss function value may be equal to the sum of the first loss function value and the second loss function value.

[0174] Specifically, after obtaining the first loss function value and the second loss function value corresponding to each training sample, the customer service quality evaluation device can obtain the target loss function value of each training sample by calculating the sum of the first loss function value and the second loss function value corresponding to each training sample. Based on this, in order to obtain a better trained service quality evaluation model, when the target loss function value does not meet the training stop condition, the model parameters of the preset service quality evaluation model are adjusted, and the service quality evaluation model with adjusted parameters is continuously trained using multiple training samples until the target loss function value meets the training stop condition, and the trained service quality evaluation model is obtained. Among them, the training stop condition can be based on actual experience and is not specifically limited here.

[0175] It should also be noted that the specific processing process of the first preset model and the second preset model for data is different Figure 2 , Figure 3 The data processing process shown is similar and will not be described in detail here.

[0176] In this embodiment, multiple training samples and sample labels corresponding to each training sample can be obtained. For each training sample, the text information sample, voice information sample and feature information sample corresponding to the training sample can be processed by a preset service quality evaluation model to obtain reference user evaluation information. After that, not only can the final output value corresponding to the target training sample and the sample label corresponding to the target training sample be processed using the first loss function to obtain the first loss function value, but the loss of the intermediate information generated in the process of generating the reference user evaluation information can also be calculated based on the second loss function to obtain the second loss function value. Then, the target loss function value corresponding to the target training sample can be determined, and the preset service quality evaluation model can be trained using multiple training samples to obtain a more accurate image processing model.

[0177] In order to describe in detail the customer service quality evaluation method provided by the embodiment of the present application, in one embodiment, before S450, as Figure 5 As shown, the customer service quality evaluation method mentioned above may also include the following steps:

[0178] S510, obtaining a first sub-loss function, a second sub-loss function, and a third sub-loss function.

[0179] It should be noted that the dimensional information of the sample may include call content dimensional information, voice emotion dimensional information and call feature dimensional information. The sample may be a training sample or other acquired sample, which is not specifically limited here.

[0180] Based on this, the first sub-loss function is used to learn the contrast loss between the same dimensional information of different samples, the second sub-loss function is used to learn the contrast loss between different dimensional information of different samples, and the third sub-loss function is used to learn the difference loss of different dimensional information of the same sample.

[0181] S520, construct a second loss function based on the first sub-loss function, the second sub-loss function, the third sub-loss function, and the preset loss function weights.

[0182] Among them, the preset loss function weight can be the weight coefficient of each sub-loss function pre-set based on actual experience or situation, and is not specifically limited here.

[0183] Specifically, the customer service quality evaluation device can obtain a first sub-loss function, a second sub-loss function and a third sub-loss function, and after obtaining the three sub-loss functions, it can construct a second loss function based on the obtained first sub-loss function, the second sub-loss function and the third loss function, and the preset loss function weights.

[0184] In one example, the above-mentioned S520 satisfies the following formula (5):

[0185] L d =σ1L intra_modal +σ2L inter_modal +σ3L self (5)

[0186] Among them, σ1, σ2, σ3 are the preset loss function weights, L intra_modal is the first sub-loss function, L inter_modal is the second loss function, L self is the third loss function, L d is the second loss function.

[0187] In this embodiment, the first sub-loss function, the second sub-loss function and the third sub-loss function can be obtained, and based on the first sub-loss function, the second sub-loss function, the third sub-loss function, and the preset loss function weight, the second loss function is constructed, because the first sub-loss function is used to learn the contrast loss between the same dimensional information of different samples, the second sub-loss function is used to learn the contrast loss between the different dimensional information of different samples, and the third sub-loss function is used to learn the difference loss of the different dimensional information of the same sample. In this way, the intermediate loss in the generation process of the reference user evaluation information can be learned by constructing the second loss function, so that an accurate target loss function can be obtained later, and then a more accurate model can be obtained.

[0188] Considering that in the task of multimodal feature fusion for classification, there are many works dedicated to exploring the interaction within or between modalities, the relationship between training samples or prediction categories is often ignored. For service satisfaction evaluation scenarios where the classification boundaries are relatively fuzzy, some samples have the problem of easy classification confusion. It is necessary to fully learn the relationship between samples and classes so that the representations of samples of different categories have a more obvious distinction. Therefore, the customer service quality evaluation method provided in the embodiment of the present application proposes multimodal representation learning based on contrastive learning, and uses the contrastive learning loss function within and across modalities to comprehensively learn the feature interactions of samples within and between modalities.

[0189] Based on this, in order to accurately obtain the first sub-loss function, the second sub-loss function and the third sub-loss function, in one embodiment, the above-mentioned S510 may specifically include the following steps:

[0190] Acquire dimensional information of the first sample, where the dimensional information of the first sample includes call content dimensional information, voice emotion dimensional information, and call feature dimensional information;

[0191] For the first dimension information of the first sample, obtain the first dimension information of N first positive samples and the first dimension information of M first negative samples, where the first dimension information is any one of call content dimension information, voice emotion dimension information, and call feature dimension information;

[0192] Constructing a first sub-loss function based on the first dimension information of the N first positive samples and the first dimension information of the M first negative samples;

[0193] For the first dimensional information of the first sample, obtain the second dimensional information of P second positive samples and the second dimensional information of Q second negative samples, where the second dimensional information is the dimensional information of the call content dimensional information, the voice emotion dimensional information and the call feature dimensional information except the first dimensional information;

[0194] Constructing a second sub-loss function based on the second dimension information of the P second positive samples and the second dimension information of the Q second negative samples;

[0195] For the first dimension information of the first sample, the second dimension information is used as a positive sample to construct a third sub-loss function;

[0196] Among them, the sample labels of the N first positive samples and the sample labels of the P second positive samples are the same as the sample label of the first sample, the sample labels of the M first negative samples and the sample labels of the Q second negative samples are different from the sample label of the first sample, and N, M, P, and Q are all positive integers.

[0197] It should also be noted that the call content dimension information may include call content embedding information, the voice emotion dimension information may include voice emotion embedding vectors, and the call feature information may include call feature embedding information.

[0198] Specifically, the customer service quality evaluation device can obtain dimensional information of the first sample, and the dimensional information of the first sample includes call content dimensional information, voice emotion dimensional information and call feature dimensional information, and then obtain the first dimensional information of N first positive samples and the first dimensional information of M first negative samples for the first dimensional information of the first sample, so as to construct a first sub-loss function based on the first dimensional information of the N first positive samples and the first dimensional information of the M first negative samples.

[0199] Next, the customer service quality evaluation device can obtain the second dimensional information of P second positive samples and the second dimensional information of Q second negative samples for the first dimensional information of the first sample. The second dimensional information is the dimensional information among multiple dimensional information except the first dimensional information. Then, the second sub-loss function can be constructed based on the second dimensional information of the P second positive samples and the second dimensional information of the Q second negative samples.

[0200] In addition, the customer service quality evaluation device can construct a third sub-loss function for the first dimensional information of the first sample, taking the second dimensional information as a positive sample, where the second dimensional information is dimensional information among multiple dimensional information except the first dimensional information.

[0201] In an example, if the call content vector is represented as m1, the voice emotion vector is represented as m2, and the call feature vector is represented as m3, m1, m2, and m3 can all become modalities. In addition, the dimensional information (i.e., embedded information) corresponding to each of the three can be represented as e m ,m∈{m1,m2,m3}.

[0202] Since contrastive learning first requires constructing positive and negative samples, the purpose of learning is to bring the features of positive samples closer and distance the features of negative samples farther in the feature space. Within each modality, we explore contrastive learning within the modality. Positive and negative samples come from the same modality, that is, positive and negative samples target the same dimensional information. For the representation e of the current sample in the m modality, m , obtain N positive samples and M negative samples, the N positive samples can be expressed as M negative samples can be expressed as It should be noted that the positive samples come from the representation of other samples of the same category as the current sample under the m-mode, and the negative samples come from the representation of samples of different categories from the current sample under the m-mode. Then the loss objective function (first sub-loss function) of intra-modal contrastive learning can be expressed as the following formula (6):

[0203]

[0204] In this way, through contrastive learning within the sample modality, the embeddings of samples of the same category within the same modality can have greater similarity, while the embeddings of samples of different categories have lower similarity.

[0205] On different samples between the three modalities, we explore cross-modal contrastive learning to narrow the distance between the representations of samples of the same category in different modalities. m , obtain 2N (P = 2N) positive samples and 2M (Q = 2M) negative samples, the 2N positive samples can be expressed as {p1, ..., p 2N}, the 2M negative samples can be expressed as {n1,...,n 2M}, it should be noted that the positive samples come from the representation of other samples of the same category as the current sample on the two modalities other than the m-modality, and the negative samples come from the representation of other samples of different categories from the current sample on the two modalities other than the m-modality (since both positive and negative samples include samples from the other two modalities, the number of positive and negative samples can be set to twice that of contrastive learning within a single modality). The contrastive learning loss objective function between modalities (the second loss function) can be shown as follows:

[0206]

[0207] For the same sample in three modalities, we explore the semi-contrastive learning of unsupervised category information between modalities and narrow the representation gap between modalities. m , which is represented in the other two modes As a positive example, the semi-contrastive loss function (the third loss function) can be expressed as follows:

[0208]

[0209] Among them, α is the margin hyperparameter between different modes. Since the representations between different modes carry the specific information of different modes, the differences between modes cannot be completely eliminated. The margin difference α is used to allow a certain degree of representation difference between different modes.

[0210] In addition, it should be noted that, considering that the leaf node vector output by the GBDT model is affected by the number of trees and the number of leaf nodes, the call feature vector will be relatively sparse and have the problem of excessive dimension. Therefore, the customer service quality evaluation method provided in the embodiment of the present application can input the call feature vector into a shallow neural network model to obtain a lower-dimensional, dense hidden vector.

[0211] In this embodiment, a first sub-loss function for learning the contrast loss between the same dimensional information of different samples, a second sub-loss function for learning the contrast loss between different dimensional information of different samples, and a third sub-loss function for learning the difference loss of different dimensional information of the same sample can be constructed, thereby facilitating the subsequent construction of the second loss function to accurately learn the intermediate loss in the process of generating reference user evaluation information.

[0212] Considering that in real rating scenarios, users generally rate based on their own subjective consciousness, which may lead to mis-rating and random rating. For example, calls containing multiple insulting words are rated high, while calls containing expressions such as "Customer service: Can you give my service a satisfactory rating later? Thank you. Customer: No problem." are rated low. Therefore, there will be unreasonable data in the acquired training sample set. Based on this, in order to avoid the adverse effect of unreasonable data on the training of the service quality evaluation model, which will greatly reduce the performance of the service quality evaluation model, it is necessary to remove unreasonable data.

[0213] Based on this, in one embodiment, Figure 6 As shown, S410 may include the following steps:

[0214] S610: Obtain a plurality of training samples to be processed and a sample label of each training sample to be processed.

[0215] The sample label of each training sample to be processed can be used to characterize the sample emotion of the training sample to be processed.

[0216] S620: For each of the plurality of training samples to be processed, match the text content of the training sample using the text matching information to obtain a matching result.

[0217] In some embodiments, the text matching information may include multiple text matching elements and an element emotion tag of each text matching element, and the element emotion reflected by the element emotion tag of each text matching element is an emotion of a different emotion type from the sample emotion.

[0218] S630, determining a plurality of training samples whose matching results meet a preset condition from the plurality of training samples to be processed, and determining a sample label of each training sample based on the sample label of each training sample to be processed.

[0219] The preset condition may be preset based on actual experience or circumstances, for example, the preset condition may be that the matching result is empty. If the matching result includes the number of successfully matched text matching elements, the preset condition may be that the number of successfully matched text matching elements is greater than a preset number threshold, which may be preset based on actual experience or circumstances and is not specifically limited here.

[0220] Specifically, the customer service quality evaluation device can obtain multiple training samples to be processed and the sample label of each training sample to be processed, and the sample label can characterize the sample label of the training sample to be processed, and then for each training sample to be processed, the multiple text matching elements included in the text matching information can be used to match the text content of the training sample to be processed to obtain a matching result, and then multiple training samples whose matching results meet preset conditions can be determined from the multiple training samples to be processed, and the sample label of each training sample can be determined based on the sample label of each training sample to be processed.

[0221] In this embodiment, the text content of the training sample to be processed can be matched using the text matching information. The element emotion of each text matching element included in the text matching information is of a different emotion type from the sample emotion of the sample to be processed. In this way, samples whose sample labels do not match the samples can be effectively eliminated, that is, data with unreasonable scores can be effectively eliminated, thereby avoiding the adverse effects of unreasonable data on the training of the service quality evaluation model.

[0222] In some embodiments, the text matching information may include word information and / or sentence information. The word information may include multiple words to be matched and the word emotion tag of each word to be matched. The sentence information may include multiple sentences to be matched and the sentence emotion tag of each sentence to be matched, and the word emotion of each word to be matched and the sentence emotion of each sentence to be matched are emotions of different emotion types from the sample emotion. Among them, the word emotion of each word to be matched is determined based on the word emotion tag of the word to be matched, and the sentence emotion tag of each sentence to be matched is determined based on the sentence emotion tag of the sentence to be matched.

[0223] It should be noted that the word information and sentence information can be preset based on experience or circumstances, and are not specifically limited here.

[0224] In this embodiment, the text matching information for matching the training sample to be processed may include word information and sentence information. In this way, the text content of the training sample to be processed can be matched from the aspects of words, sentences, etc., ensuring the accuracy of the matching result.

[0225] In some embodiments, the emotion type may include a first emotion type, a second emotion type, and a third emotion type. The emotion corresponding to the first emotion type may be a positive emotion, such as a happy emotion, the emotion corresponding to the second emotion type may be a neutral emotion, such as a calm emotion, and the emotion corresponding to the third emotion type may be a negative emotion, such as an angry emotion, which is not specifically limited here.

[0226] Based on this, if the sample emotion of the training sample to be processed is the first emotion type, the emotion type of the word emotion of each word to be matched can be the second emotion type or the third emotion type. And / or, the emotion type of the sentence emotion of each sentence to be matched is the second emotion type or the third emotion type.

[0227] If the sample emotion of the training sample to be processed is the second emotion type, the emotion type of the word emotion of each word to be matched can be the first emotion type or the third emotion type, and / or the emotion type of the sentence emotion of each sentence to be matched is the first emotion type or the third emotion type.

[0228] If the sample emotion of the training sample to be processed is the third emotion type, then the emotion type of the word emotion of each word to be matched can be the first emotion type or the second emotion type, and / or, the emotion type of the sentence emotion of each sentence to be matched can be the first emotion type or the third emotion type.

[0229] In this embodiment, according to the sample emotion of the training sample to be processed, word information and / or sentence information different from the emotion type of the sample emotion can be selected, so that the training samples to be processed whose sample emotions do not match the text content can be screened out from multiple training samples to be processed subsequently, and then the unreasonable scored data in the training samples to be processed can be accurately eliminated to obtain accurate training samples.

[0230] In order to more accurately describe the customer service quality evaluation method provided in the embodiment of the present application, in one embodiment, before using the text matching information to match the text content of the training sample to be processed to obtain the matching result, the customer service quality evaluation method involved may also include the following steps:

[0231] Get multiple words to be matched;

[0232] For each of the multiple words to be matched, using a preset tool to obtain a sentiment score of the word to be matched;

[0233] Based on the relationship between the word sentiment score of the word to be matched and the score threshold, the word sentiment label of the word to be matched is determined.

[0234] Among them, the word sentiment score of each to-be-matched word can represent the emotion reflected by the to-be-matched word. Specifically, the higher the word sentiment score, the more positive the emotion reflected by the to-be-matched word. Conversely, the higher the word sentiment score, the more negative the emotion reflected by the to-be-matched word.

[0235] In addition, the scoring threshold may be preset based on actual experience or circumstances, which is not specifically limited here. The preset tool may be a tool preset based on actual experience or circumstances for scoring the emotions reflected by the words, for example, the preset tool may be an open source text sentiment analysis tool, which is not specifically limited here.

[0236] Specifically, the customer service quality evaluation device can obtain multiple words to be matched, and for each word to be matched in the multiple words to be matched, use a preset tool to obtain the word sentiment score of the word to be matched, and then determine the word sentiment label of the word to be matched based on the relationship between the word sentiment score of the word to be matched and the score threshold.

[0237] In this embodiment, multiple words to be matched can be obtained, and the word sentiment score of each word to be matched can be obtained by using a preset tool, and then the word sentiment label of each word to be matched can be accurately determined based on the relative relationship between the word sentiment score and the score threshold, so that the text content of the training text to be processed can be accurately matched later, so as to effectively remove unreasonable data in multiple training samples to be processed, thereby improving the effectiveness of the training sample set.

[0238] In order to more accurately describe the customer service quality evaluation method provided in the embodiment of the present application, in one embodiment, the above-mentioned step of obtaining multiple words to be matched may specifically include the following steps:

[0239] Performing word segmentation processing on the text content of each of the multiple training samples to be processed to obtain multiple first words, and obtaining the word frequency of each first word;

[0240] It is determined that among the multiple first words, multiple first words whose word frequencies are greater than a preset word frequency threshold are multiple words to be matched.

[0241] The preset word frequency threshold may be preset based on actual experience or circumstances. For example, the preset word frequency threshold may be set to 10, which is not specifically limited herein.

[0242] Specifically, the customer service quality evaluation device can perform word segmentation processing on the text content of each of the multiple training samples to be processed to obtain multiple first words, and can obtain the word frequency of each first word, and then determine that the multiple first words whose word frequency is greater than the preset word frequency threshold among the multiple first words are the words to be matched.

[0243] In this embodiment, multiple words to be matched can be obtained from multiple training samples to be processed for subsequent matching of text content of the training samples to be processed. In this way, the words to be matched can be obtained in a targeted manner, avoiding the situation of obtaining redundant words to be matched, thereby reducing costs.

[0244] It should also be noted that, in one embodiment, before performing word segmentation processing on the text content of each of the multiple training samples to be processed to obtain multiple first words and obtaining the word frequency of each first word, the customer service quality evaluation method involved above may also include the following steps:

[0245] For each of the multiple training samples to be processed, the text content of the training sample to be processed is preprocessed based on a preset preprocessing rule to obtain the preprocessed text content of the training sample to be processed.

[0246] In some embodiments, the preset preprocessing rules may include: deleting sentences in the text content of the training sample to be processed whose number of words is less than a preset number of words; and / or deleting redundant characters in the text content of the training sample to be processed, and the redundant characters may include repeated words and stop words.

[0247] The preset number of words may be a number of words pre-set based on actual experience or circumstances, and is not specifically limited here. Stop words may be, for example, auxiliary words, modal particles, and other meaningless words.

[0248] Specifically, before performing word segmentation processing on the text content of each training sample to be processed to obtain multiple first words and obtaining the word frequency of each first word, the customer service quality evaluation device can preprocess the text content of each training sample to be processed among the multiple training samples to be processed based on preset preprocessing rules to obtain the preprocessed text content of the training sample to be processed.

[0249] In this embodiment, the text content of each training sample to be processed can be preprocessed to delete short sentences with fewer words, repeated words, stop words, etc. in the text content of the training sample to be processed, thereby improving the sample quality of the training sample to be processed.

[0250] In addition, in order to accurately determine the word emotion label of the word to be matched, in some embodiments, the emotion type may include a first emotion type, a second emotion type and a third emotion type. Based on this, the word emotion label may include a first word emotion label representing the emotion type of the word emotion as a first emotion type, a second word emotion label representing the emotion type of the word emotion as a second emotion type, and a third word emotion type representing the emotion type of the word emotion as a third emotion type.

[0251] Furthermore, the scoring threshold may include a first scoring threshold and a second scoring threshold, the first scoring threshold being less than the second scoring threshold. The first scoring threshold and the second scoring threshold may be preset based on actual experience or circumstances, and are not specifically limited herein.

[0252] Based on this, based on the relationship between the word sentiment score of the word to be matched and the score threshold, the step of determining the word sentiment label of the word to be matched may specifically include the following steps:

[0253] When the word emotion score of the to-be-matched word is less than the first score threshold, determining the word emotion label of the to-be-matched word to be the first word emotion label;

[0254] When the word emotion score of the to-be-matched word is greater than or equal to the first score threshold and less than or equal to the second score threshold, determining the word emotion label of the to-be-matched word to be the second word emotion label;

[0255] When the word emotion score of the to-be-matched word is greater than the second score threshold, the word emotion label of the to-be-matched word is determined to be the third word emotion label.

[0256] Specifically, after obtaining the word emotion score of the word to be matched, the customer service quality evaluation device can compare the word emotion score with the first scoring threshold and the second scoring threshold in turn. If the word emotion score of the word to be matched is less than the first scoring threshold, it can be determined that the word emotion label of the word to be matched is the first word emotion label. If the word emotion score of the word to be matched is greater than or equal to the first scoring threshold and less than or equal to the second scoring threshold, it can be determined that the word emotion label of the word to be matched is the second word emotion label. If the word emotion score of the word to be matched is greater than the second scoring threshold, it can be determined that the word emotion label of the word to be matched is the third word emotion label.

[0257] In this embodiment, after obtaining the word emotion score of the word to be matched, the word emotion score of the word to be matched can be compared with the first scoring threshold and the second scoring threshold, so that the words to be matched can be accurately classified, and then the word emotion label of each word to be matched can be accurately determined. In this way, the accuracy of determining the word emotion label of the word to be matched is improved, which facilitates the subsequent accurate matching of the text content of the training text to be processed, so as to effectively remove unreasonable data in multiple training samples to be processed, and improve the effectiveness of the training sample set.

[0258] Based on this, in order to accurately obtain the first scoring threshold and the second scoring threshold to accurately classify the words to be matched, in one embodiment, before determining the word emotion label of the word to be matched for each of the multiple words to be matched based on the relationship between the word emotion score of the word to be matched and the scoring threshold, the method further includes:

[0259] Obtaining a plurality of second words and a preset word emotion tag and a word emotion score of each second word;

[0260] Taking the first preset value as the starting point and the second preset value as the end point, dividing according to the preset intervals to obtain a plurality of pending thresholds;

[0261] For each of the multiple pending thresholds, perform the following steps:

[0262] Divide the plurality of second words according to a threshold to be determined and a word sentiment score of each second word to obtain a first target group and a second target group, and determine a group sentiment label of the first target group and a group sentiment label of the second target group;

[0263] Based on the group emotion label of the first target group and the group emotion label of the second target group, determining the number of words in the first target group and the second target group whose preset word emotion label of the second word matches the group emotion label of the group where the second word is located;

[0264] From the word quantities of multiple pending thresholds, the pending threshold with the largest word quantity is determined as the scoring threshold.

[0265] The preset word emotion label may be a word emotion label manually preset based on actual experience or situation, and is not specifically limited here. The first preset value, the second preset value, and the preset interval are all preset based on actual experience or situation.

[0266] Additionally, in some embodiments, the first target group includes multiple second words whose word sentiment scores are less than a pending threshold, and the second target group includes multiple second words whose word sentiment scores are greater than or equal to the pending threshold.

[0267] Specifically, the customer service quality evaluation device can obtain multiple second words and the preset word emotion label and word emotion score of each second word, and take the first preset value as the starting point and the second preset value as the end point, and divide according to the preset interval to obtain multiple pending thresholds. Then, for each pending threshold in the multiple pending thresholds, the multiple second words can be divided according to the pending threshold and the word emotion score of each second word to obtain a first target group and a second target group, and the group emotion label of the first target group and the group emotion label of the second target group are determined. The first target group includes multiple second words whose word emotion scores are less than the pending threshold, and the second target group includes multiple second words whose word emotion scores are greater than or equal to the pending threshold. Then, based on the group emotion label of the first target group and the group emotion label of the second target group, the number of words in the first target group and the second target group that match the preset word emotion label of the second words and the group emotion label of the group to which the second words are located is determined, and the pending threshold with the largest number of words is determined as the scoring threshold from the number of words in the multiple pending thresholds.

[0268] In one example, since the customer service quality evaluation method provided by this embodiment can classify multiple words to be matched into three categories according to the emotions corresponding to the words, that is, words representing positive emotions, words representing negative emotions, and words representing neutral emotions, based on this, in the process of determining the first scoring threshold, the first scoring threshold is used to divide words representing negative emotions and words representing neutral emotions. If the word emotion score interval is [0, 1], in the process of determining the first scoring threshold, the first preset value can be set to 0, the second preset value can be set to 1, and the preset interval can be set to 0.01. In this way, multiple pending thresholds such as 0.01 and 0.02 can be determined. For each pending threshold, the threshold x can be taken as a value every 0.01 between 0-1, and the values ​​of the multiple second words obtained can be calculated. Word emotion scoring, multiple second words are grouped to obtain a first target group and a second target group. According to the first scoring threshold, the first target group should include words representing negative emotions, and the second target group should include words representing positive emotions and words representing neutral emotions. The actual first target group may contain words representing positive emotions or words representing neutral emotions. Therefore, the customer service quality evaluation device needs to determine, for each pending threshold, based on the group emotion label of the first target group and the group emotion label of the second target group, the number of words whose preset word emotion labels of the second words in the first target group and the second target group are consistent with the group emotion label of the group where the second words are located, and then the pending threshold with the largest number of words can be determined as the first scoring threshold from the number of words of multiple pending thresholds.

[0269] It should be noted that the process of determining the second scoring threshold is similar to the process of determining the first scoring threshold, and will not be elaborated here. However, since the second scoring threshold is greater than the first scoring threshold, in the process of determining the second scoring threshold, its corresponding first preset value should be greater than the first scoring threshold.

[0270] In this embodiment, in the process of determining the scoring threshold, the three-category threshold determination problem can be transformed into a two-category threshold determination problem, which can not only improve the accuracy of determining the scoring threshold, but also improve the efficiency of determining the scoring threshold.

[0271] Based on the same inventive concept, the embodiment of the present application also provides a customer service quality evaluation device. Figure 7 The customer service quality evaluation device provided in the embodiment of the present application is described in detail.

[0272] Figure 7 It is a structural schematic diagram of a customer service quality evaluation device provided in an embodiment of the present application.

[0273] like Figure 7As shown, the customer service quality evaluation device 700 may include: an acquisition module 710 and a processing module 720 .

[0274] The acquisition module 710 is used to acquire the call information during the call between the user and the customer service, where the call information includes call text information, call voice information and call feature information, where the call feature information is determined based on the call text information and call voice information;

[0275] The processing module 720 is used to process the call text information, the call voice information and the call feature information through the first model in the service quality evaluation model to obtain a call content vector, a voice emotion vector and a call feature vector;

[0276] The processing module 720 is further used to process the call content vector, the voice emotion vector and the call feature vector through the second model in the service quality evaluation model to generate user evaluation information;

[0277] Among them, the call content vector is used to reflect the user's satisfaction with the customer service of the customer service from the aspect of call content; the voice emotion vector is used to reflect the user's satisfaction with the customer service of the customer service from the aspect of user emotion; the call feature vector is used to reflect the user's satisfaction with the customer service of the customer service from the aspect of call characteristics; the user evaluation information is used to evaluate the user's satisfaction with the customer service quality of the customer service during the call.

[0278] In some embodiments, the first model includes a first sub-model, a second sub-model and a third sub-model; based on this, the processing module is specifically used to:

[0279] Input the call text information into the first sub-model, the first sub-model includes J neural network layers, obtain the output information of I neural network layers in the first sub-model respectively, perform maximum pooling processing on the I output information, and obtain the call content vector, wherein I is less than or equal to J, the I neural network layers include the Jth neural network layer, the J-1th neural network layer, ... the J-K+1th neural network layer, J=I+K, and I, J, and K are all positive integers;

[0280] Inputting the call voice information into the second sub-model, extracting at least one acoustic feature from the call voice information through the second sub-model, and processing the at least one acoustic feature using an attention mechanism to obtain a voice emotion vector;

[0281] The call feature information is input into the third sub-model, which includes multiple decision trees, each decision tree includes multiple leaf nodes, and the call feature vector is obtained by mapping the multiple features included in the call feature information to the multiple leaf nodes of the multiple decision trees.

[0282] In one embodiment, the processing module mentioned above is specifically used for:

[0283] The call content vector, the voice emotion vector and the call feature vector are processed by the second model in the service quality evaluation model to obtain call content embedding information, voice emotion embedding information and call feature embedding information;

[0284] The cross-attention mechanism is used to fuse the call content embedding information, voice emotion embedding information, and call feature embedding information to obtain the total fusion vector.

[0285] User evaluation information is generated based on the total fusion vector, voice emotion embedding information and call feature embedding information.

[0286] In one embodiment, the processing module is specifically used to:

[0287] Fusing the voice emotion embedding information and the call feature embedding information to obtain first fused embedding information, where the number of dimensions of the first fused embedding information is the same as the number of dimensions of the call content embedding information;

[0288] Take the call content embedding information as the key vector, the call content embedding vector as the value vector, and the first fused embedding vector as the query vector. Input the key vector, value vector and query vector into the cross-attention mechanism function. Process the key vector, value vector and query vector through the cross-attention mechanism function to obtain the total fused vector.

[0289] In one embodiment, the customer service quality assessment device mentioned above may further include a training module.

[0290] An acquisition module is used to acquire a training sample set, wherein the training sample set includes multiple training samples and sample labels corresponding to each training sample, wherein each training sample includes a call information sample during a call between a user and a customer service representative, wherein the call information sample includes a text information sample, a voice information sample, and a feature information sample, wherein the feature information sample is determined based on the text information sample and the voice information sample;

[0291] The processing module is further used to process, for each training sample, a text information sample, a voice information sample, and a feature information sample corresponding to the training sample through a first preset model in a preset service quality evaluation model to obtain a call content vector, a voice emotion vector, and a call feature vector corresponding to the training sample;

[0292] The processing module is further used to process the call content vector, the voice emotion vector and the call feature vector corresponding to the training sample through a second preset model in the preset service quality evaluation model to obtain reference user evaluation information;

[0293] The processing module is further used to process the reference user evaluation information corresponding to the target training sample and the sample label corresponding to the target training sample based on the first loss function to obtain a first loss function value, where the target training sample is any one of the multiple training samples;

[0294] The processing module is further used to process the call content vector, the voice emotion vector and the call feature vector corresponding to the target training sample based on the second loss function to obtain a second loss function value;

[0295] The training module is also used to train the preset service quality evaluation model based on the target loss function value of the preset service quality evaluation model, where the target loss function value is equal to the sum of the first loss function value and the second loss function value, using multiple training samples to obtain the trained service quality evaluation model.

[0296] In one embodiment, the customer service quality evaluation device mentioned above may further include a construction module.

[0297] The acquisition module is further used to acquire a first sub-loss function, a second sub-loss function and a third sub-loss function, the dimensional information of the sample includes call content dimensional information, voice emotion dimensional information and call feature dimensional information, the first sub-loss function is used to learn the contrast loss between the same dimensional information of different samples, the second sub-loss function is used to learn the contrast loss between different dimensional information of different samples, and the third sub-loss function is used to learn the difference loss of different dimensional information of the same sample;

[0298] A construction module is used to construct a second loss function based on the first sub-loss function, the second sub-loss function, the third sub-loss function, and a preset loss function weight.

[0299] In one embodiment, the acquisition module is specifically used to:

[0300] Acquire dimensional information of the first sample, where the dimensional information of the first sample includes call content dimensional information, voice emotion dimensional information, and call feature dimensional information;

[0301] For the first dimension information of the first sample, obtain the first dimension information of N first positive samples and the first dimension information of M first negative samples, where the first dimension information is any one of call content dimension information, voice emotion dimension information, and call feature dimension information;

[0302] Constructing a first sub-loss function based on the first dimension information of the N first positive samples and the first dimension information of the M first negative samples;

[0303] For the first dimensional information of the first sample, obtain the second dimensional information of P second positive samples and the second dimensional information of Q second negative samples, where the second dimensional information is the dimensional information of the call content dimensional information, the voice emotion dimensional information and the call feature dimensional information except the first dimensional information;

[0304] Constructing a second sub-loss function based on the second dimension information of the P second positive samples and the second dimension information of the Q second negative samples;

[0305] For the first dimension information of the first sample, the second dimension information is used as a positive sample to construct a third sub-loss function;

[0306] Among them, the sample labels of the N first positive samples and the sample labels of the P second positive samples are the same as the sample label of the first sample, the sample labels of the M first negative samples and the sample labels of the Q second negative samples are different from the sample label of the first sample, and N, M, P, and Q are all positive integers.

[0307] In one embodiment, the acquisition module is specifically used to:

[0308] Obtaining a plurality of training samples to be processed and a sample label of each of the training samples to be processed, wherein the sample label of each training sample to be processed can be used to characterize the sample emotion of the training sample to be processed;

[0309] For each of the plurality of training samples to be processed, matching the text content of the training sample to be processed by using the text matching information to obtain a matching result, wherein the text matching information includes a plurality of text matching elements and an element emotion label of each text matching element, wherein the element emotion reflected by the element emotion label of each text matching element is an emotion of a different emotion type from the sample emotion;

[0310] A plurality of training samples whose matching results meet preset conditions are determined from the plurality of training samples to be processed, and a sample label of each training sample is determined based on the sample label of each training sample to be processed.

[0311] In some embodiments, the text matching information includes word information and / or sentence information;

[0312] The word information includes multiple words to be matched and the word emotion label of each word to be matched, the sentence information includes multiple sentences to be matched and the sentence emotion label of each sentence to be matched, the word emotion of each word to be matched and the sentence emotion of each sentence to be matched are emotions of different emotion types from the sample emotion, the word emotion of each word to be matched is determined based on the word emotion label of the word to be matched, and the sentence emotion of each sentence to be matched is determined based on the sentence emotion label of the sentence to be matched.

[0313] In some embodiments, the emotion type includes a first emotion type, a second emotion type, and a third emotion type;

[0314] When the sample emotion of the training sample to be processed is of the first emotion type, the emotion type of the word emotion of each word to be matched is of the second emotion type or the third emotion type, and / or the emotion type of the sentence emotion of each sentence to be matched is of the second emotion type or the third emotion type;

[0315] When the sample emotion of the training sample to be processed is of the second emotion type, the emotion type of the word emotion of each word to be matched is of the first emotion type or the third emotion type, and / or the emotion type of the sentence emotion of each sentence to be matched is of the first emotion type or the third emotion type;

[0316] In the case where the sample emotion of the training sample to be processed is the third emotion type, the emotion type of the word emotion of each word to be matched is the first emotion type or the second emotion type, and / or the emotion type of the sentence emotion of each sentence to be matched is the first emotion type or the second emotion type.

[0317] In an embodiment of the present application, call information during a call between a user and customer service can be obtained, and the call information includes call text information, call voice information, and call feature information. The call feature information is determined based on the call text information and the call voice information, and the acquired call text information, call voice information, and call feature information are processed accordingly through a service quality evaluation model to analyze the user's satisfaction with the customer service of the customer service from aspects such as call content, user emotions, and call characteristics, and generate user evaluation information. The user evaluation information is used to evaluate the user's customer service quality of the customer service during the call, avoiding the situation in the prior art where users randomly score and the scoring is relatively subjective, thereby improving the accuracy of the customer service quality evaluation.

[0318] Each module in the customer service quality evaluation device provided in the embodiment of the present application can be implemented Figures 1 to 6 The method steps of any of the embodiments shown in the figure can achieve the corresponding technical effects, which will not be repeated here for the sake of brevity.

[0319] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0320] The electronic device may include a processor 801 and a memory 802 storing computer program instructions.

[0321] Specifically, the processor 801 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0322] The memory 802 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 802 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 802 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 802 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 802 is a non-volatile solid-state memory.

[0323] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0324] The processor 801 implements any one of the customer service quality evaluation methods in the above embodiments by reading and executing computer program instructions stored in the memory 802 .

[0325] In one example, the electronic device may further include a communication interface 803 and a bus 810. Figure 8 As shown, the processor 801, the memory 802, and the communication interface 803 are connected via a bus 810 and communicate with each other.

[0326] The communication interface 803 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0327] Bus 810 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 810 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0328] In addition, in combination with the customer service quality evaluation method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, the customer service quality evaluation method provided in the embodiment of the present application is implemented.

[0329] An embodiment of the present application also provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the customer service quality assessment method provided in the embodiment of the present application.

[0330] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0331] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0332] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0333] Aspects of the present disclosure are described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable customer service quality assessment device to produce a machine so that these instructions executed by the processor of a computer or other programmable customer service quality assessment device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0334] The above are only specific implementation methods of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the described systems, modules and units can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A customer service quality evaluation method, characterized in that: The method comprises: Acquire call information during a call between a user and a customer service representative, wherein the call information includes call text information, call voice information, and call feature information, wherein the call feature information is determined based on the call text information and the call voice information; The call text information, the call voice information and the call feature information are processed by the first model in the service quality evaluation model to obtain a call content vector, a voice emotion vector and a call feature vector; Processing the call content vector, the voice emotion vector, and the call feature vector through a second model in the service quality evaluation model to generate user evaluation information; Among them, the call content vector is used to reflect the user's satisfaction with the customer service of the customer service from the aspect of call content; the voice emotion vector is used to reflect the user's satisfaction with the customer service of the customer service from the aspect of user emotion; the call feature vector is used to reflect the user's satisfaction with the customer service of the customer service from the aspect of call characteristics; the user evaluation information is used to evaluate the user's customer service quality of the customer service during the call.

2. The method according to claim 1, characterized in that The first model includes a first sub-model, a second sub-model and a third sub-model; The process of processing the call text information, the call voice information and the call feature information by the first model in the service quality evaluation model to obtain a call content vector, a voice emotion vector and a call feature vector includes: Input the call text information into the first sub-model, the first sub-model includes J neural network layers, obtain output information respectively output by I neural network layers in the first sub-model, perform maximum pooling processing on the I output information, and obtain the call content vector, wherein I is less than or equal to J, the I neural network layers include the Jth neural network layer, the J-1th neural network layer, ... the J-K+1th neural network layer, J=I+K, and I, J, and K are all positive integers; Inputting the call voice information into the second sub-model, extracting at least one acoustic feature from the call voice information through the second sub-model, and processing the at least one acoustic feature using an attention mechanism to obtain the voice emotion vector; The call feature information is input into the third sub-model, the third sub-model includes multiple decision trees, each of the decision trees includes multiple leaf nodes, and the call feature vector is obtained by mapping the multiple features included in the call feature information to the multiple leaf nodes of the multiple decision trees.

3. The method according to claim 1, characterized in that The process of processing the call content vector, the voice emotion vector, and the call feature vector by the second model in the service quality evaluation model to generate user evaluation information includes: Processing the call content vector, the voice emotion vector and the call feature vector through the second model in the service quality evaluation model to obtain call content embedding information, voice emotion embedding information and call feature embedding information; Using a cross attention mechanism, the call content embedding information, the voice emotion embedding information, and the call feature embedding information are fused to obtain a total fusion vector; User evaluation information is generated based on the total fusion vector, the voice emotion embedding information and the call feature embedding information.

4. The method according to claim 3, characterized in that The cross-attention mechanism is used to fuse the call content embedding information, the voice emotion embedding information, and the call feature embedding information to obtain a total fusion vector, including: Fusion the speech emotion embedding information and the call feature embedding information to obtain first fused embedding information, wherein the number of dimensions of the first fused embedding information is the same as the number of dimensions of the call content embedding information; Taking the call content embedding information as the key vector, the call content embedding vector as the value vector, and the first fused embedding vector as the query vector, the key vector, the value vector and the query vector are input into a cross-attention mechanism function, and the key vector, the value vector and the query vector are processed by the cross-attention mechanism function to obtain a total fused vector.

5. The method according to claim 1, characterized in that Before the call text information, the call voice information and the call feature information are processed by the first model in the service quality evaluation model to obtain the call content vector, the voice emotion vector and the call feature vector, the method further includes: Obtain a training sample set, the training sample set including multiple training samples and sample labels corresponding to each of the training samples, each training sample including a call information sample during a call between a user and a customer service representative, the call information sample including a text information sample, a voice information sample, and a feature information sample, the feature information sample being determined based on the text information sample and the voice information sample; For each training sample, perform the following operations: The text information sample, the voice information sample and the feature information sample corresponding to the training sample are processed by a first preset model in the preset service quality evaluation model to obtain a call content vector, a voice emotion vector and a call feature vector corresponding to the training sample; The call content vector, the voice emotion vector and the call feature vector corresponding to the training sample are processed by a second preset model in the preset service quality evaluation model to obtain reference user evaluation information; Processing reference user evaluation information corresponding to a target training sample and a sample label corresponding to the target training sample based on a first loss function to obtain a first loss function value, wherein the target training sample is any one of a plurality of training samples; Processing the call content vector, the voice emotion vector, and the call feature vector corresponding to the target training sample based on the second loss function to obtain a second loss function value; Based on the target loss function value of the preset service quality evaluation model, the target loss function value is equal to the sum of the first loss function value and the second loss function value, and the preset service quality evaluation model is trained using multiple training samples to obtain a trained service quality evaluation model.

6. The method according to claim 5, characterized in that Before processing the call content vector, the voice emotion vector, and the call feature vector corresponding to the target training sample based on the second loss function to obtain the second loss function value, the method further includes: Obtain a first sub-loss function, a second sub-loss function, and a third sub-loss function, where the dimensional information of the sample includes call content dimensional information, voice emotion dimensional information, and call feature dimensional information, the first sub-loss function is used to learn the contrast loss between the same dimensional information of different samples, the second sub-loss function is used to learn the contrast loss between different dimensional information of different samples, and the third sub-loss function is used to learn the difference loss of different dimensional information of the same sample; Based on the first sub-loss function, the second sub-loss function, the third sub-loss function, and the preset loss function weight, the second loss function is constructed.

7. The method according to claim 6, characterized in that The obtaining of the first sub-loss function, the second sub-loss function and the third sub-loss function includes: Acquire dimensional information of the first sample, where the dimensional information of the first sample includes call content dimensional information, voice emotion dimensional information, and call feature dimensional information; For the first dimension information of the first sample, obtain the first dimension information of N first positive samples and the first dimension information of M first negative samples, where the first dimension information is any one of the call content dimension information, the voice emotion dimension information, and the call feature dimension information; Constructing a first sub-loss function based on the first dimension information of the N first positive samples and the first dimension information of the M first negative samples; For the first dimensional information of the first sample, obtain second dimensional information of P second positive samples and second dimensional information of Q second negative samples, where the second dimensional information is dimensional information of the call content dimensional information, the voice emotion dimensional information, and the call feature dimensional information except the first dimensional information; Constructing a second sub-loss function based on the second dimensional information of the P second positive samples and the second dimensional information of the Q second negative samples; For the first dimension information of the first sample, taking the second dimension information as a positive sample, constructing a third sub-loss function; Among them, the sample labels of the N first positive samples and the sample labels of the P second positive samples are the same as the sample label of the first sample, the sample labels of the M first negative samples and the sample labels of the Q second negative samples are different from the sample label of the first sample, and N, M, P, and Q are all positive integers.

8. The method according to claim 5, characterized in that The step of obtaining a training sample set includes: Acquire a plurality of training samples to be processed and a sample label of each of the training samples to be processed, wherein the sample label of each of the training samples to be processed can be used to characterize the sample emotion of the training sample to be processed; For each of the plurality of training samples to be processed, matching the text content of the training sample to be processed using text matching information to obtain a matching result, wherein the text matching information includes a plurality of text matching elements and an element emotion label of each of the text matching elements, and the element emotion reflected by the element emotion label of each of the text matching elements is an emotion of a different emotion type from the sample emotion; A plurality of training samples whose matching results meet a preset condition are determined from the plurality of training samples to be processed, and a sample label of each of the training samples is determined based on the sample label of each of the training samples to be processed.

9. The method according to claim 8, characterized in that The text matching information includes word information and / or sentence information; The word information includes multiple words to be matched and word emotion tags of each word to be matched, and the sentence information includes multiple sentences to be matched and sentence emotion tags of each sentence to be matched. The word emotion of each word to be matched and the sentence emotion of each sentence to be matched are emotions of different emotion types from the sample emotion. The word emotion of each word to be matched is determined based on the word emotion tag of the word to be matched, and the sentence emotion of each sentence to be matched is determined based on the sentence emotion tag of the sentence to be matched.

10. The method according to claim 9, characterized in that The emotion types include a first emotion type, a second emotion type and a third emotion type; In the case where the sample emotion of the training sample to be processed is of the first emotion type, the emotion type of the word emotion of each of the words to be matched is of the second emotion type or the third emotion type, and / or the emotion type of the sentence emotion of each of the sentences to be matched is of the second emotion type or the third emotion type; In the case where the sample emotion of the training sample to be processed is of the second emotion type, the emotion type of the word emotion of each of the words to be matched is of the first emotion type or the third emotion type, and / or the emotion type of the sentence emotion of each of the sentences to be matched is of the first emotion type or the third emotion type; When the sample emotion of the training sample to be processed is the third emotion type, the emotion type of the word emotion of each of the words to be matched is the first emotion type or the second emotion type, and / or the emotion type of the sentence emotion of each of the sentences to be matched is the first emotion type or the second emotion type.

Images