Evaluation method and device for customer service quality, storage medium and equipment

Abstract

The application discloses a customer service quality evaluation method and device, a storage medium and equipment. The method comprises the following steps: obtaining customer service monitoring information, wherein the customer service monitoring information is audio information and video information generated by a target object during service execution; adopting a first neural network model to calculate an audio feature classification evaluation result based on the audio information; adopting a second neural network model and a third neural network model to calculate a video feature classification evaluation result based on the video information, wherein the audio feature classification evaluation result and the video feature classification evaluation result both comprise at least one emotion feature evaluation result; and determining a customer service quality evaluation result corresponding to the target object according to the audio feature classification evaluation result and the video feature classification evaluation result. The application solves the technical problem of low recognition efficiency and poor evaluation effect caused by the fact that the prior art cannot accurately identify and evaluate customer service quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of financial technology, in particular to the field of intelligent algorithm and intelligent identification technology, and specifically to a customer service quality evaluation method and device, a storage medium and equipment. BACKGROUND

[0002] With the continuous development of network informatization, remote video services have been widely used. The real-time nature of video customer service processes, and the lack of effective supervision of most video customer service, have resulted in an increasing number of violations. On the one hand, existing service evaluation systems are based on customer feedback messages, and customers select satisfaction levels on the interactive interface for post-service evaluation. There are few more intelligent real-time evaluation methods. On the other hand, existing video analysis tasks are usually based on deep network models, and the training and inference of the models have brought huge computational pressure. At the same time, due to the immediacy of the video customer service process and the complexity of the audio and video data, there is currently no effective method to effectively monitor the violations of the video customer service.

[0003] To address the above problems, no effective solutions have been proposed so far. SUMMARY

[0004] The embodiments of the present application provide a customer service quality evaluation method and device, a storage medium and equipment to at least solve the technical problems of low recognition efficiency and poor evaluation effect caused by the inability of the prior art to accurately identify and evaluate customer service quality.

[0005] According to an aspect of an embodiment of the present application, a customer service quality evaluation method is provided, comprising: obtaining customer service monitoring information, wherein the customer service monitoring information is audio information and video information generated by a target object during a service process; using a first neural network model to calculate an audio feature classification evaluation result based on the audio information; using a second neural network model to calculate a video feature classification evaluation result based on the video information, wherein the audio feature classification evaluation result and the video feature classification evaluation result each include at least one emotion feature evaluation result; and determining a customer service quality evaluation result corresponding to the target object based on the audio feature classification evaluation result and the video feature classification evaluation result.

[0006] Optionally, the first neural network model is used to calculate the audio feature classification evaluation result based on the audio information, including: obtaining a speech sequence in the audio information; performing frame processing on the speech sequence to extract speech frame features of the speech sequence; performing segmentation processing on the speech frame features to obtain speech segment features of the speech sequence; fitting the speech segment features through a target function to obtain a sentiment cognitive window feature of the speech sequence; and calculating the audio feature classification evaluation result through the first neural network model based on the speech frame features, the speech segment features, and the sentiment cognitive window feature.

[0007] Optionally, the second neural network model includes a third neural network model and a fourth neural network model, and the second neural network model is used to calculate the video feature classification evaluation result based on the video information, including: performing frame processing on the video information to obtain a plurality of video images; extracting a facial image feature set of the target object from each frame of the video image; and inputting the facial image feature set into the third neural network model and the fourth neural network model respectively to obtain the video feature classification evaluation result.

[0008] Optionally, the facial image feature set of the target object is extracted from each frame of the video image, including: performing segmentation processing on each frame of the video image through a target classifier to obtain a segmented video image; performing extraction processing on the segmented video image through a fifth neural network model to obtain a facial image feature map of the target object; performing target monitoring and accurate positioning processing on the facial image feature map through the third neural network model to obtain a candidate frame region corresponding to the facial image feature map; and performing maximum pooling processing on the candidate frame region through the fourth neural network model to obtain the facial image feature set.

[0009] Optionally, the customer service quality evaluation result corresponding to the target object is determined based on the audio feature classification evaluation result and the video feature classification evaluation result, including: calculating a first evaluation score corresponding to an evaluation result of each emotional feature; summing a plurality of the first evaluation scores to obtain a second evaluation score; and determining the customer service quality evaluation result according to the second evaluation score.

[0010] Optionally, the method further includes: obtaining a first violation frequency corresponding to a first violation behavior of the target object in the audio information; obtaining a second violation frequency corresponding to a second violation behavior of the target object in the video information; and summing the first violation frequency and the second violation frequency to obtain a total violation frequency of the target object.

[0011] Optionally, the method further comprises: determining whether an evaluation index of the customer service quality of the target object reaches an alarm threshold, wherein the evaluation index comprises at least one of the following: the audio feature classification evaluation result, the video feature classification evaluation result, the customer service quality evaluation result, and the total number of violations; and issuing an alarm instruction if any one of the evaluation indexes reaches the alarm threshold.

[0012] Optionally, the obtaining the second number of violations of the second violation behavior of the target object in the video information comprises: performing frame processing on the video information to obtain a plurality of video images; determining whether the second violation behavior existing in adjacent two video images is the same violation behavior; and recording the same violation behavior as one time of the second violation behavior if the determination result is yes.

[0013] Optionally, the method further comprises: training the first neural network model based on the audio information to obtain a trained first neural network model; training the second neural network model based on the video information to obtain a trained second neural network model; and updating the first neural network model and the second neural network model respectively according to the trained first neural network model and the trained second neural network model.

[0014] Optionally, before the obtaining the speech sequence in the audio information, the method further comprises: performing segmentation processing on the audio information according to a preset time length to obtain a plurality of audio samples; determining a matching degree of a voiceprint feature of a first audio sample and a pre-stored voiceprint feature; obtaining the speech sequence if the matching degree is greater than or equal to a matching degree threshold; and issuing an alarm instruction if the matching degree is less than the matching degree threshold.

[0015] Optionally, the method further comprises: detecting a number of faces in each video image by a face recognition model; determining whether the number of faces is one; extracting the face image feature set from each video image if the determination result is yes; and issuing an alarm instruction if the determination result is no.

[0016] According to another aspect of the embodiments of the present application, there is also provided an evaluation device for customer service quality, comprising: an obtaining module, configured to obtain customer service monitoring information, wherein the customer service monitoring information is audio information and video information generated by a target object during service execution; a first calculation module, configured to calculate an audio feature classification evaluation result based on the audio information by using a first neural network model; a second calculation module, configured to calculate a video feature classification evaluation result based on the video information by using a second neural network model and a third neural network model, wherein the audio feature classification evaluation result and the video feature classification evaluation result each include an evaluation result of at least one emotional feature; and a determination module, configured to determine a customer service quality evaluation result corresponding to the target object according to the audio feature classification evaluation result and the video feature classification evaluation result.

[0017] According to another aspect of the embodiments of the present application, there is also provided an evaluation system for customer service quality, comprising: a video customer service client, configured to obtain customer service monitoring information, wherein the customer service monitoring information is audio information and video information generated by a target object during service execution; an edge computing platform server, connected to the video customer service client, configured to calculate an audio feature classification evaluation result based on the audio information by using a first neural network model; a cloud platform server, connected to the edge computing platform server, configured to calculate a video feature classification evaluation result based on the video information by using a second neural network model and a third neural network model, and determine a customer service quality evaluation result corresponding to the target object according to the audio feature classification evaluation result and the video feature classification evaluation result, wherein the audio feature classification evaluation result and the video feature classification evaluation result each include an evaluation result of at least one emotional feature.

[0018] According to another aspect of the embodiments of the present application, there is also provided a non-volatile storage medium, which stores a plurality of instructions, and the instructions are adapted to be loaded and executed by a processor to perform any of the evaluation methods for customer service quality.

[0019] According to another aspect of the embodiments of the present application, there is also provided an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform any of the evaluation methods for customer service quality.

[0020] In the embodiment of the present application, the customer service quality evaluation method is adopted, the customer service monitoring information is obtained, the customer service monitoring information is audio information and video information generated by the target object in the service process; the first neural network model is adopted, the audio feature classification evaluation result is calculated based on the audio information; the second neural network model and the third neural network model are adopted, the video feature classification evaluation result is calculated based on the video information, wherein the audio feature classification evaluation result and the video feature classification evaluation result both include the evaluation result of at least one emotional feature; the customer service quality evaluation result corresponding to the target object is determined according to the audio feature classification evaluation result and the video feature classification evaluation result, which achieves the purpose of accurately identifying and evaluating the customer service quality according to the audio and video information of the target user, thereby realizing the technical effect of improving the audio and video identification efficiency and evaluation quality, and further solving the technical problems of low identification efficiency and poor evaluation effect caused by the inability of the prior art to accurately identify and evaluate the customer service quality. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which are included to provide a further understanding of the present application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and serve to explain the present application. In the drawings:

[0022] Figure 1 It is a flowchart of a customer service quality evaluation method according to an embodiment of the present application;

[0023] Figure 2 It is a flowchart of an optional customer service quality evaluation method according to an embodiment of the present application;

[0024] Figure 3 It is a flowchart of another optional customer service quality evaluation method according to an embodiment of the present application;

[0025] Figure 4 It is a flowchart of another optional customer service quality evaluation method according to an embodiment of the present application;

[0026] Figure 5 It is a flowchart of another optional customer service quality evaluation method according to an embodiment of the present application;

[0027] Figure 6 It is a schematic structural diagram of a system for implementing the above-mentioned customer service quality evaluation method according to an embodiment of the present application;

[0028] Figure 7 It is a schematic structural diagram of a customer service quality evaluation device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the person skilled in the art better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor should belong to the protection scope of the present application.

[0030] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to include only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0031] First, for the convenience of understanding the embodiments of the present application, the following will explain some terms or nouns involved in the present application:

[0032] Edge Computing (EC) refers to an open platform integrating network, computing, storage and application core capabilities, which provides near-end services on the spot near the object or data source. The Edge Computing Point (ECP) is responsible for real-time interaction between the collected and calculated front-end data results and the cloud platform server.

[0033] Fast R-CNN introduces a RoI pooling layer based on RCNN, avoiding the R-CNN algorithm from extracting features from the same area multiple times.

[0034] The Faster R-CNN algorithm further increases the RPN candidate frame generation algorithm based on the Fast R-CNN, greatly improving the target detection speed. In fact, two networks, RPN and Fast R-CNN, are trained in Faster R-CNN. If the two networks are trained separately, the parameters of the shared convolutional layer will change, so the two networks need to share the convolutional layer, rather than being trained separately. RPN is a candidate frame recommendation algorithm, and Fast R-CNN makes detailed calculations on the position of the candidate frame and the category of the object in the frame.

[0035] Embodiment 1

[0036] With the acceleration of the commercialization process of the new generation of wireless mobile communication technology 5G, the continuous popularity of mobile intelligent terminals and the multimedia of electronic service content, various service enterprises, such as banking institutions, have researched and innovated driven to realize the transformation to smart banks and provide customers with all-round intelligent services. At present, the transaction volume of online banks, mobile banks and other electronic channels has far exceeded the transaction volume of traditional physical outlets. Remote video banks, as another extension of electronic channels, can realize the integration of various service channels and extend the external service time of banks. Video customer service is not limited by space, can intuitively display goods, and can meet the regulatory policy requirements of online transactions, greatly simulate offline business handling scenarios, give customers more thoughtful "immersive experience" services, and take into account customer emotions.

[0037] At the same time, the real-time nature of the video customer service process, and the lack of effective supervision of most video customer service, there have been more and more violations. On the one hand, the existing service evaluation system is through customer reply SMS scoring, customer selection of satisfaction on the interactive interface to conduct post-service evaluation, and there are few more intelligent real-time evaluation means; on the other hand, the existing video analysis task is usually based on a deep network model, and the training and reasoning of the model brings huge computing pressure, and at the same time, due to the immediacy of the video customer service process and the complexity of the audio and video data, there is no effective method to effectively monitor the violations of the video customer service.

[0038] Based on the above problems, the embodiment of the present application provides a method for evaluating the quality of customer service. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0039] Figure 1 The flowchart of a method for evaluating the quality of customer service according to the embodiment of the present application is shown in FIG. 1, which comprises the following steps: Figure 1 As shown in FIG. 1, the method comprises the following steps:

[0040] Step S102, obtaining customer service monitoring information, wherein the customer service monitoring information is audio information and video information generated by a target object during a service process;

[0041] Step S104, using a first neural network model to calculate an audio feature classification evaluation result based on the audio information;

[0042] In step S106, a second neural network model is used to calculate a video feature classification evaluation result based on the video information, wherein the audio feature classification evaluation result and the video feature classification evaluation result each include at least one emotion feature evaluation result.

[0043] In step S108, a customer service quality evaluation result corresponding to the target object is determined according to the audio feature classification evaluation result and the video feature classification evaluation result.

[0044] Optionally, the audio information of the target object is acquired through a video acquisition device, and the video information of the target object is acquired through an audio acquisition device.

[0045] Optionally, the first neural network model can be, but is not limited to, a cognitive mechanism-based feedback neural network model CIRNN, and the second neural network model can be, but is not limited to, a Faster R-CNN model.

[0046] Optionally, the audio information and the video information are acquired by: acquiring initial audio information and initial video information of the target object; and performing cleaning processing on the initial audio information and the initial video information to obtain the audio information and the video information.

[0047] In the embodiment of the present application, a customer service quality evaluation method is used to acquire customer service monitoring information, wherein the customer service monitoring information is audio information and video information generated by a target object during service execution; a first neural network model is used to calculate an audio feature classification evaluation result based on the audio information; a second neural network model and a third neural network model are used to calculate a video feature classification evaluation result based on the video information, wherein the audio feature classification evaluation result and the video feature classification evaluation result each include at least one emotion feature evaluation result; and a customer service quality evaluation result corresponding to the target object is determined according to the audio feature classification evaluation result and the video feature classification evaluation result, so as to accurately identify and evaluate the customer service quality according to the audio and video information of the target user, thereby achieving the technical effect of improving the audio and video identification efficiency and the evaluation quality, and further solving the technical problems of low identification efficiency and poor evaluation effect caused by the inability of the prior art to accurately identify and evaluate the customer service quality.

[0048] As an optional embodiment, Figure 2 is a flowchart of an optional customer service quality evaluation method according to the embodiment of the present application, as shown in Figure 2 The first neural network model is used to calculate an audio feature classification evaluation result based on the audio information, including:

[0049] Step S202: Obtain the speech sequence from the above audio information;

[0050] Step S204: Perform frame segmentation on the above speech sequence and extract the speech frame features of the above speech sequence.

[0051] Step S206: The above speech frame features are segmented to obtain the speech segment features of the above speech sequence.

[0052] Step S208: Fit the above speech segment features to the objective function to obtain the emotional cognitive window features of the above speech sequence;

[0053] Step S210: Based on the above-mentioned speech frame features, speech segment features, and emotion cognition window features, the above-mentioned first neural network model is used to calculate the above-mentioned audio feature classification evaluation results.

[0054] Optionally, the aforementioned speech frame features may include, but are not limited to, prosodic features, nonlinear features, phonological features, and spectral features, etc.

[0055] Optionally, the objective function may be, but is not limited to, a Gaussian function; the first neural network model may be, but is not limited to, a cognitive mechanism-based feedback neural network model (CIRNN).

[0056] It should be noted that, since the physical movement of the vocal organs has a close influence on speech formation, and this movement is relatively stable in the short term, short-time analysis of the speech signal can be performed, including framing and windowing, to obtain a speech sequence that is both time-discrete and amplitude-discrete. A 25ms Hamming window can be used, with a frame shift of 10ms, to frame the speech signal. Then, on a frame-by-frame basis, the short-time features (i.e., speech frame features) and their corresponding first-order differences are extracted from each frame. These short-time features can include prosodic features, nonlinear features, phonological features, and spectral features, and these features are then normalized. For example, let the speech frame feature vector be f(t), where... Let f(t) be an audio feature parameter. Assume that there are m types of feature parameters in the sample f(t), and each audio feature parameter is collected at time t, where t = 1, 2, ..., n:

[0057]

[0058] Optionally, the sample can be divided into segments of 100 frames each, and segment feature extraction can be performed on the speech signal to obtain the speech segment features of the target object. By extracting features on a segment-by-segment basis, the statistical features based on the segment length can remove textual relevance without weakening the expression of important prosodic characteristics. These features are extracted using the fundamental frequency and the contours of the first three formants. This feature is then fused with short-time frame features and global statistical features to improve the emotion recognition rate. For example, let the speech segment feature vector be k.(t) ,in Let k be an audio feature parameter. (t) The audio data contains m characteristic parameters, each of which is collected at time t, where t = 1, 2, ..., n.

[0059]

[0060] It should be noted that, in order to more accurately reflect the impact of the dynamic process of human emotional expression on emotion recognition, an objective function (such as a Gaussian function) is used to fit the above-mentioned speech segment features to simulate the process of human emotional expression. Based on the speech segment features, an emotion recognition window is extracted. That is, a Gaussian function is loaded onto multiple speech segment features to weight the emotions in the middle while weakening the emotions at the two ends of the speech. For example, let the speech segment features be K. N Convolution with a Gaussian function G(x) yields the emotion perception window feature: E = G(w i )*K N Where N is the total number of audio sample segments, w i It is the position of the Gaussian function corresponding to the i-th segment.

[0061] Optionally, a cognitive-mechanism-based feedback neural network model (CIRNN, i.e., the first neural network model) can be constructed. Based on the cognitive patterns of the human brain in dealing with emotions, it not only analyzes the input features of auditory information but also compares and attempts with previous experience models and the probabilistic system of the prefrontal cortex to improve the processing, orderliness, and accuracy of information. Therefore, based on feedback neural networks (RNNs), a cognitive-mechanism-based feedback neural network (CIRNN) that integrates multi-granular features is adopted, so that features from different time units participate in network training. This highlights the temporal nature of emotions, emphasizes the influence of context on emotions, and also retains the role of global characteristics in emotion recognition. The CIRNN network includes an input layer, hidden layers, a memory layer, and an output layer. The input layer takes in frame features and segment features, respectively. The memory layer is a collection of neurons fed back from the hidden layer to record the content of the hidden layer at the previous time step. The activation function of the neurons is the Sigmoid function. For example, let t be the current time step of the network, f(t) represent the frame features, k(t) represent the segment features within the time step t, E(t) represent the cognitive window features, and W... 1 W is the weight matrix from the input layer frame features f(t), segment features k(t), to the hidden layer x(t). 2 W is the weight matrix from hidden layer x(t) to hidden layer z(t). 3 Let W be the weight matrix from the hidden layer z(t) to the output layer y(t). 4 For memory layer x c The weight matrix from (t) to the hidden layer x(t). 5The input layer cognitive window feature E(t) is input to the weight matrix of the hidden layer z(t). x(t) and z(t) represent the outputs of the two hidden layers, and the specific formula is: the first hidden layer x(t) = f(W 1 (f(t) + k(t) + W 4 x c (t))) where f takes the Sigmoid function: The second hidden layer z(t) = f(W 2 x(t) + W 5 E(t)); and the output layer y(t) = f(W 3 z(t)).

[0062] As an optional embodiment, Figure 3 is a flowchart of another optional evaluation method of customer quality according to an embodiment of the present application, as Figure 3 shown, the second neural network model includes a third neural network model and a fourth neural network model, and the video feature classification evaluation result is calculated based on the video information by using the second neural network model, which includes:

[0063] In step S302, the video information is frame-processed to obtain a plurality of video images;

[0064] In step S304, the facial image feature set of the target object is extracted from each frame of the video image;

[0065] In step S306, the facial image feature set is input into the third neural network model and the fourth neural network model respectively to obtain the video feature classification evaluation result.

[0066] Optionally, the second neural network model can be but is not limited to a Faster R-CNN model; the third neural network model can be but is not limited to an RPN (Regional Proposal Network) network model; and the fourth neural network model can be but is not limited to a Fast R-CNN network model.

[0067] It should be noted that, in order to ensure the consistency of audio feature extraction and video feature extraction, the plurality of video images and the speech frame features have the same preset time length division, such as one minute per segment, video data output at 25 frames per second, etc.

[0068] Optionally, the video feature classification evaluation result can be but is not limited to: calm, happy, surprised, sad, frightened, angry, disgusted, etc.

[0069] Optionally, the output form of the video feature classification evaluation result can be but is not limited to a matrix form.

[0070] Optionally, a video stream V = {V1, V2…V q} is defined, wherein the video stream is collected by a monitoring camera installed on a video customer service client device, the monitoring camera faces the face of a video customer service personnel (i.e. a target object), and the video data (i.e. video images) corresponding to the video stream is output at 25 frames per second; wherein each frame of video image V q is represented by a matrix of l*w, wherein l is the number of rows of the video matrix, and w is the number of columns of the video matrix; and the q-th video sample is marked, q = 1, 2…n, indicating the sample number.

[0071] In an optional embodiment, the detection process of the Faster R-CNN model (the second neural network model) can be, but is not limited to: inputting a video image; using a RDN (Residual Dilated Network) residual dilated network to perform feature extraction on the image through a convolution layer in a deep network to obtain feature maps of the eyes, eyebrows and mouth of each frame in the video stream; performing target detection and accurate positioning on the feature maps through an RPN (Regional Proposal Network) network to obtain a candidate frame region; performing RoI pooling operation on the obtained candidate frame region: that is, through the method of coordinate projection, the feature region corresponding to the candidate frame region in the input image is obtained on the feature map, and the maximum pooling is performed on the region, so that the feature of the candidate frame region is obtained, and the feature size is unified. The output of the RoI pooling layer (i.e. the feature of the candidate frame region after maximum pooling of the feature map) is used as the feature vector of each candidate frame region; the feature vector of the candidate frame region is connected with a fully connected layer, and a multi-task loss function is defined, which is connected with a softmax classifier and a box bounding regressor (bounding box regression) respectively, to obtain the class and coordinate bounding box of the current region of interest respectively; performing non-maximum suppression on all obtained bounding boxes to obtain the final classification detection result (i.e. the video feature classification evaluation result).

[0072] Optionally, before detection using the Faster R-CNN model described above, the Faster R-CNN model is trained, and the training process can be but is not limited to: 1) initialize the RPN network with the convolutional layers of the pre-trained model, then train the RPN network separately, and after training, the feature parameters of the model and the RPN are updated; 2) initialize the detection network Fast R-CNN with the same convolutional layers of the pre-trained model, where the candidate frame is from the RPN network in step 1, then train the detection network Fast R-CNN separately, and after training, the feature parameters of the model and the Fast R-CNN are updated; 3) initialize the RPN model with the model trained in step 2, fix the convolutional layers (i.e. shared convolutional layers) during training, only adjust the feature parameters belonging to the RPN, and secondly train the RPN network; 4) still keep the shared convolutional layers fixed, use the candidate frame output by the RPN adjusted in step 3 as input, and secondly train the Fast R-CNN network, and fine-tune the parameters of the Fast R-CNN.

[0073] In an optional embodiment, the face image feature set described above is respectively input into the RPN regression neural network model and the Fast R-CNN classification neural network model for identification and classification, and a video feature classification evaluation matrix (V Dis , Hap , Qui ) is output, which represents the probabilities of outputting the three emotions of frustration, joy and calm in customer facial expression emotion recognition, and a facial expression classification result is obtained.

[0074] As an optional embodiment, Figure 4 is a flowchart of another optional evaluation method of customer service quality according to an embodiment of the application, as shown in Figure 4 , the face image feature set of the target object is extracted from each frame of the video image, including:

[0075] Step S402, the target classifier is used to perform segmentation processing on each frame of the video image, and a segmented video image is obtained;

[0076] Step S404, the fifth neural network model is used to perform extraction processing on the segmented video image, and a face image feature map of the target object is obtained;

[0077] Step S406, the third neural network model is used to perform target monitoring and accurate positioning processing on the face image feature map, and a candidate frame region corresponding to the face image feature map is obtained;

[0078] Step S408, the fourth neural network model is used to perform maximum pooling processing on the candidate frame region, and the face image feature set is obtained.

[0079] Optionally, the segmented video image comprises at least eye, eyebrow and mouth feature maps of the target object.

[0080] Optionally, the fifth neural network model can be but is not limited to a residual dilated network (RDN) model; the third neural network model can be but is not limited to a regional proposal network (RPN) model; and the fourth neural network model can be but is not limited to a Fast R-CNN model.

[0081] Optionally, the target classifier can be but is not limited to an enhanced classifier based on Gabor features, which is used to identify facial organ feature points of the target object, wherein the facial organ feature points comprise at least inner corner positions of left and right eyes, outer corner positions of left and right eyes, highest point positions of left and right eyes, lowest point positions of left and right eyes, a nose tip position, leftmost and rightmost positions of a mouth corner, and uppermost and lowermost positions of an intersection of a mouth center line and a mouth contour, so as to obtain eye, eyebrow and mouth feature maps of the target object.

[0082] In an optional embodiment, a residual dilated network (RDN) model is used to input the segmented video image into a first layer of convolutional layer (denoted as Conv1) to perform preliminary feature extraction on the image, so as to obtain a facial image feature map of the target object. Taking a 400*400 image as an example, the size of the convolutional kernel of Conv1 is constructed as 3*3, and the number of convolutional kernels is set to 64. Then, the feature map output by Conv1 enters Conv2, the size of the convolutional kernel of which is the same as that of Conv1, and the number of convolutional kernels is set to 128. Then, the feature map output by Conv2 enters Conv3, the size of the convolutional kernel of which is set to 3*3, and the number of convolutional kernels is set to 256. Then, the feature map output by Conv3 enters RDN4, the size of the convolutional kernel of which is set to 3*3, and the number of convolutional kernels is set to 512, and dilated convolution is introduced with dilation parameters d=1 and d=2. Then, the feature map output by RDN4 enters RDN5, the size of the convolutional kernel of which is set to 3*3, and the number of convolutional kernels is still set to 512, and dilated convolution is also introduced with dilation parameters d=2 and d=4 to ensure the detection of micro-expression features. The residual dilated network model improves the robustness of the algorithm by using a deeper residual network structure, and ensures that micro-expression features are retained in the process of continuous convolution by accumulating shallow features and deep features, so as to obtain more accurate output information, which can greatly improve the detection performance of the network without increasing the original model calculation amount.

[0083] Optionally, the feature map is subjected to target detection and accurate positioning by the RPN network model to obtain a candidate frame region, and the candidate frame is subjected to maximum pooling operation by a RoI pooling layer in the Fast R-CNN network model to output a face image feature set including a plurality of feature vectors of the same dimension.

[0084] As an optional embodiment, Figure 5 is a flowchart of another optional evaluation method of customer service quality according to an embodiment of the present application, as shown in Figure 5 The determination of the customer service quality evaluation result corresponding to the target object according to the audio feature classification evaluation result and the video feature classification evaluation result includes:

[0085] Step S502, calculating a first evaluation score corresponding to the evaluation result of each emotion feature;

[0086] Step S504, summing a plurality of the first evaluation scores to obtain a second evaluation score;

[0087] Step S506, determining the customer service quality evaluation result according to the second evaluation score.

[0088] Optionally, the customer service quality evaluation result is represented in the form of an evaluation category, and the evaluation category may include but is not limited to “poor”, “general”, “better” and “very good”.

[0089] In an optional embodiment, the first evaluation score is the degree of conformity of the service quality embodied by a specific emotion feature classification during customer service. Therefore, one emotion feature classification calculates one first evaluation score, and finally M first evaluation scores are calculated. Each first evaluation score is calculated by the following formula:

[0090]

[0091] wherein O wi represents the i-th first evaluation score, 1≤i≤M, and is a positive integer, i.e. i changes from 1 to M. In this example, the audio and video emotions are each divided into 3 categories, so M=6.

[0092] W i represents the weight of the type of the i-th specific emotion feature, for example, the weight of the audio happy emotion classification feature is set to 0.6, the weight of the audio tension emotion classification feature is set to -0.15, the weight of the video expression feature for happy is set to 0.3, the weight of the video expression feature for anger is set to -0.3, and the weight of the video expression feature for disgust is set to -0.2.x A =1 represents the occurrence of the i-th specific feature behavior, and x A=0 indicates that the i-th specific characteristic behavior did not occur. T A T represents the time when a specific emotional characteristic behavior occurs. D Indicates less than T A The dynamic base time can be set arbitrarily by those skilled in the art, for example, based on the start time of each customer service call. K is a time reduction factor, K>1, for example, set to any value between 1.5 and 3. X A T represents A The number of specific emotional characteristic behaviors generated during the pre-examination period. A reference time period is, for example, 1 minute; this invention does not impose a specific limitation. This formula incorporates... Make X A The larger O wi The smaller X A The smaller, the more O wi The larger the value, the less the impact of rigid and unchanging emotional behaviors during customer service on the evaluation, and the more encouraged customer service representatives to respond to customer requests in a timely manner and adapt to changing circumstances.

[0093] As can be seen from the above formula, W i This demonstrates how different emotional states, categorized by a specific emotional characteristic of different types, have varying degrees of impact on customer service quality evaluation. Since the time decrement factor K > 1, for the same type of emotional characteristic, the further the occurrence time is from the dynamic baseline time, the greater the impact on the service quality evaluation results. Furthermore, by setting the time decrement factor, the influence of historical behaviors during the same service period on the final result evaluation is reduced.

[0094] Optionally, the aforementioned second evaluation score represents the degree to which the customer service quality meets the standards within a specific time period. In this embodiment of the invention, the second evaluation score is obtained by calculating the sum of all the first evaluation scores: A higher second-round evaluation score indicates a higher quality of service provided by the customer service representative during the service period. Conversely, a lower second-round evaluation score indicates a lower quality of service provided by the customer service representative. This process assesses the quality of the customer service representative's service. The final evaluation result vector is obtained by summing and averaging the probability vectors of the evaluation categories. The corresponding evaluation category is the final service evaluation category. These categories can be "Poor," "Average," "Good," and "Excellent."

[0095] In an optional embodiment, the above method further includes:

[0096] Step S602: Obtain the first violation count corresponding to the first violation of the target object in the above audio information;

[0097] Step S604, obtaining a second violation frequency corresponding to the second violation behavior of the target object in the video information.

[0098] Step S606, summing the first violation frequency and the second violation frequency to obtain a total violation frequency of the target object.

[0099] Optionally, in a case where it is detected that there is a violation in the audio information of the target object, it is determined that the target object has a first violation behavior; and in a case where it is detected that there is a violation in the video information of the target object, it is determined that the target object has a second violation behavior.

[0100] Optionally, the time feature is taken as a marker, and the score model accumulates the time when the violation marker occurs, the type of the violation, and the accumulated frequency. The database accumulates the frequencies of various violations, and when a preset threshold is exceeded, a monitoring alarm is triggered, and an alarm indication is sent.

[0101] In an optional embodiment, the method further includes:

[0102] Step S702, determining whether an evaluation index of the quality of service of the target object reaches an alarm threshold;

[0103] Step S704, if any of the evaluation indexes reaches the alarm threshold, an alarm indication is sent.

[0104] Optionally, the evaluation index includes at least one of the following: the audio feature classification evaluation result, the video feature classification evaluation result, the quality of service evaluation result, and the total violation frequency.

[0105] In an optional embodiment, the obtaining of the second violation frequency corresponding to the second violation behavior of the target object in the video information includes:

[0106] Step S802, performing frame processing on the video information to obtain a plurality of video images;

[0107] Step S804, determining whether the second violation behavior existing in adjacent two video images is a same violation behavior;

[0108] Step S806, if the determination result is yes, the same violation behavior is recorded as one second violation behavior.

[0109] Optionally, when recording the second violation times, the method further comprises: judging whether the violation behavior existing in the preset time is the same behavior in adjacent two video images, if not, identifying as different violation behaviors, and recording each violation behavior in the background, if yes, not repeating the accumulation. Considering the time domain information, avoiding recording the same violation behavior multiple times due to only considering a single frame of image.

[0110] In an optional embodiment, the method further comprises:

[0111] Step S902, training the first neural network model based on the audio information to obtain a trained first neural network model;

[0112] Step S904, training the second neural network model based on the video information to obtain a trained second neural network model;

[0113] Step S906, updating the first neural network model and the second neural network model respectively according to the trained first neural network model and the trained second neural network model.

[0114] It should be noted that the initial model parameters of the first neural network model (i.e. the feedback neural network model based on cognitive mechanism CIRNN) and the second neural network model (i.e. the Faster R-CNN model) are artificially set. In the process of evaluating the customer service quality of the target object, new audio information, video information and evaluation results are constantly obtained. The model is trained and updated based on the audio information, video information and evaluation results to obtain higher-precision first neural network model and second neural network model (i.e. trained first neural network model and trained second neural network model). The first neural network model and the second neural network model are updated respectively based on the trained first neural network model and the trained second neural network model, so as to achieve the purpose of improving the accuracy of the customer service quality evaluation result.

[0115] Optionally, the training method of the Faster R-CNN network model can be, but is not limited to, as follows: the objective function of the Faster R-CNN network model is a binary cross-entropy function (binary_crossentropy), and the optimization method is Adam; wherein the learning rate of Adam is set to 0.001, the exponential decay rate of the mean of the gradient is set to 0.9, and the exponential decay rate of the uncentered variance of the gradient is set to 0.999; the batch size is set to 200, the training set, the validation set and the test set are set according to a certain proportion, after multiple rounds of training, the test of the validation set is performed every round, the training model of the best generation is saved and used for the test of the test set, and the result is the result of the whole learning. In an optional embodiment, before the above-mentioned obtaining the speech sequence in the above-mentioned audio information, the above-mentioned method comprises:

[0116] Step S1002, segmenting the above-mentioned audio information according to a preset time length to obtain a plurality of audio samples;

[0117] Step S1004, judging the matching degree of the voiceprint feature of the first audio sample and the pre-stored voiceprint feature;

[0118] Step S1006, if the matching degree is greater than or equal to the matching degree threshold, obtaining the speech sequence;

[0119] Step S1008, if the matching degree is less than the matching degree threshold, issuing an alarm instruction.

[0120] Optionally, the above-mentioned audio information is segmented according to a preset time length (for example, one minute per segment) to obtain a plurality of audio samples, which are defined as an audio stream U={U1, U2…U q}, U is an audio stream set, and q is the sample number q=1, 2, …; the first audio sample of about 10 seconds of voice data is input into a voiceprint recognition module, audio feature data (i.e. a preset voiceprint feature) in the call audio file is extracted and sent to a sound comparison module, the sound comparison module compares the audio feature data with the customer service voice feature data stored in the database in advance; the matching degree between the voiceprint feature of the customer service and the pre-stored voiceprint feature information base is judged, if the matching degree is greater than or equal to a predetermined threshold, a response result of passing the verification is returned to the customer service client, and the comparison result and the corresponding worker number of the customer service staff are returned to the client; if the matching degree is less than or equal to the predetermined threshold, an alarm instruction is issued, a response result of failing to pass the verification and a service termination control instruction are returned to the customer service client based on the alarm instruction, a violation mark is registered and uploaded to the cloud platform, and the service is terminated.

[0121] In an optional embodiment, the above-mentioned method further comprises:

[0122] Step S1102, detecting the number of faces in each frame of the video image by the face recognition model;

[0123] Step S1104, determining whether the number of faces is 1;

[0124] Step S1106, if the determination result is yes, extracting the face image feature set from each frame of the video image;

[0125] Step S1108, if the determination result is no, issuing an alarm instruction.

[0126] Optionally, the face recognition model is pre-trained, and the face image information is detected in real time. If the detected number of faces N < 1 or N > 1, a transaction exception instruction is returned to the video client to perform an alarm operation; if the detected number of faces is equal to 1, the subsequent operation is continued.

[0127] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0128] From the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, or network device, etc.) execute the method described in each embodiment of the present application.

[0129] Embodiment 2

[0130] According to the embodiments of the present application, a system embodiment for implementing the evaluation method of customer service quality is also provided, Figure 6 is a structural schematic diagram of the evaluation system of customer service quality according to the embodiments of the present application, as Figure 6 shown, the evaluation system of customer service quality includes: a video customer service client 1, an edge computing platform server 2, and a cloud platform server 3, wherein:

[0131] The video customer service client 1 is used to acquire customer service monitoring information, wherein the customer service monitoring information is audio information and video information generated by a target object during service execution; the edge computing platform server 2 is connected with the video customer service client and is used to calculate an audio feature classification evaluation result based on the audio information by using a first neural network model; the cloud platform server 3 is connected with the edge computing platform server and is used to calculate a video feature classification evaluation result based on the video information by using a second neural network model and a third neural network model, and determine a customer service quality evaluation result corresponding to the target object according to the audio feature classification evaluation result and the video feature classification evaluation result, wherein the audio feature classification evaluation result and the video feature classification evaluation result both include evaluation results of at least one emotional feature.

[0132] Optionally, in the customer service quality evaluation system, each video customer service client, edge node and cloud center has a unique static address, wherein the video customer service client 1 performs data interaction with the edge computing platform server 2 through a communication network; the edge computing platform server 2 performs data interaction with the cloud platform server 3 through a dedicated line network. The video customer service client 1 is provided with an audio and video acquisition system, which is used to acquire audio and video information during video customer service in real time, detect client running state parameters, and upload the running state parameters to the edge computing platform server; the edge computing system 2 includes a physical layer and an application layer, and is used to clean data such as audio and video feature information and client running state parameters, and upload the cleaned parameters to the cloud computing system; the edge computing system 2 also updates the local audio and video management evaluation model by using updated model parameters, calculates the service evaluation result according to the cleaned audio and video feature parameters during customer service and the updated local audio and video service evaluation model, and uploads the service evaluation result to the cloud platform server 3 to evaluate the customer service process. The audio and video management evaluation model in the edge computing system 2 and the cloud computing system 3 has initial model parameters, which are set by artificial when the audio and video management system is produced, but the accuracy of the initial model parameters is low at this time. With the cloud platform continuously receiving the audio and video features and evaluation results sent by the edge computing system 2, the cloud platform trains and updates the audio and video management evaluation model to obtain model parameters with higher accuracy. The model parameters with higher accuracy are then sent to the edge computing system 2, so that the customer service evaluation result with higher accuracy can be calculated.

[0133] Optionally, the specific evaluation method of the customer service quality analysis system is as follows: the customer service personnel accesses the service workbench through the video customer service client 1, the video customer service terminal 1 starts the voice and video collection device; the edge computing platform 2 performs statistics and analysis on the audio and video information uploaded by the video customer service client 1 according to the specified algorithm, and sends the evaluation result to the cloud platform server 3; the background staff sets the related index parameters on the cloud platform server 3 according to the management needs; the cloud platform server 3 displays the monitoring information in real time in the form of visual charts, instrument panels and the like.

[0134] It should be noted that the edge network customer service quality evaluation system provided by the embodiment of the application includes a video customer service client, an edge computing platform server, a cloud platform server and the like, each video customer service client, edge node and cloud center has a unique static address, the video customer service client uploads the running state parameters collected during the service to the edge computing platform server through the audio and video collection device; the edge computing system includes a physical layer and an application layer two-layer structure, which is used for data cleaning, uploading of audio and video feature information, client running state parameters and the like, and calculation of service evaluation results by using a local audio and video management evaluation model and uploading to the cloud platform server. The cloud platform end trains and updates the audio and video management evaluation model, obtains model parameters with higher precision and sends them to the edge computing system, so that the customer service evaluation results with higher precision can be calculated, the evaluation standard is unified, and the calculation pressure of the edge computing system model training is greatly reduced.

[0135] It should be noted that the video customer service client 1, the edge computing platform server 2 and the cloud platform server 3 shown in the application Figure 6 have only a schematic structure, and in specific applications, the customer service quality evaluation system in the application can have more or less structures than the video customer service client 1, the edge computing platform server 2 and the cloud platform server 3 shown in the application. Figure 6

[0136] It should be noted that any one of the optional or preferred customer service quality evaluation methods in the above embodiment 1 can be executed or implemented in the customer service quality evaluation provided by the embodiment.

[0137] In addition, it should be noted that the optional or preferred embodiments of the embodiment can refer to the related description in embodiment 1, which will not be described here.

[0138] Embodiment 3

[0139] According to the embodiment of the application, a device embodiment for implementing the customer service quality evaluation method is also provided, Figure 7 is a structural schematic diagram of a customer service quality evaluation device according to the embodiment of the application, like Figure 7 ​As shown in the above customer service quality evaluation device, comprising: an acquisition module 20, a first calculation module 22, a second calculation module 24, a determination module 26, wherein:

[0140] The acquisition module 20 is configured to acquire customer service monitoring information, wherein the customer service monitoring information is audio information and video information generated by a target object during service execution.

[0141] The first calculation module 22 is configured to calculate an audio feature classification evaluation result based on the audio information using a first neural network model.

[0142] The second calculation module 24 is configured to calculate a video feature classification evaluation result based on the video information using a second neural network model and a third neural network model, wherein the audio feature classification evaluation result and the video feature classification evaluation result each include an evaluation result of at least one emotional feature.

[0143] The determination module 26 is configured to determine a customer service quality evaluation result corresponding to the target object based on the audio feature classification evaluation result and the video feature classification evaluation result.

[0144] It should be noted that each of the above modules can be implemented by software or hardware. For example, for the latter, each of the above modules can be located in the same processor, or any combination of the above modules can be located in different processors.

[0145] It should be noted that the acquisition module 20, the first calculation module 22, the second calculation module 24, and the determination module 26 correspond to steps S102 to S108 in Embodiment 1, and the above modules have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules as part of the device can run in a computer terminal.

[0146] It should be noted that the optional or preferred embodiments of the present embodiment can refer to the related description in Embodiment 1, which will not be repeated here.

[0147] The above customer service quality evaluation device can further include a processor and a memory, and the acquisition module 20, the first calculation module 22, the second calculation module 24, and the determination module 26 are stored in the memory as program units, and the processor executes the above program units stored in the memory to realize the corresponding functions.

[0148] The processor comprises a core, and the core is used to call corresponding program units in the memory. The core can be one or more. The memory can comprise a non-permanent memory in a computer readable medium, a random access memory (RAM), and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory comprises at least one memory chip.

[0149] According to the embodiments of the present application, an embodiment of a non-volatile storage medium is also provided. Optionally, in the embodiment, the non-volatile storage medium comprises a stored program, and when the program runs, the non-volatile storage medium controls a device where the non-volatile storage medium is located to perform the quality evaluation method.

[0150] Optionally, in the embodiment, the non-volatile storage medium can be located in any one of a computer terminal group in a computer network or any one of a mobile terminal group, and the non-volatile storage medium comprises a stored program.

[0151] Optionally, when the program runs, the non-volatile storage medium controls a device where the non-volatile storage medium is located to perform the following functions: obtaining customer service monitoring information, wherein the customer service monitoring information is audio information and video information generated by a target object in a service process; using a first neural network model to calculate an audio feature classification evaluation result based on the audio information; using a second neural network model to calculate a video feature classification evaluation result based on the video information, wherein the audio feature classification evaluation result and the video feature classification evaluation result each comprise at least one emotion feature evaluation result; and determining a customer service quality evaluation result corresponding to the target object based on the audio feature classification evaluation result and the video feature classification evaluation result.

[0152] Optionally, when the program runs, the non-volatile storage medium controls a device where the non-volatile storage medium is located to perform the following functions: obtaining a speech sequence in the audio information; performing frame processing on the speech sequence to extract speech frame features of the speech sequence; performing segmentation processing on the speech frame features to obtain speech segment features of the speech sequence; fitting the speech segment features through a target function to obtain emotion cognition window features of the speech sequence; and calculating the audio feature classification evaluation result through the first neural network model based on the speech frame features, the speech segment features, and the emotion cognition window features.

[0153] Optionally, the program running controls the device where the nonvolatile storage medium is located to perform the following functions: frame processing on the video information to obtain a plurality of video images; extracting a facial image feature set of the target object from each of the video images; and inputting the facial image feature set into the third neural network model and the fourth neural network model respectively to obtain the video feature classification evaluation result.

[0154] Optionally, the program running controls the device where the nonvolatile storage medium is located to perform the following functions: segmenting each of the video images using a target classifier to obtain segmented video images; extracting a facial image feature map of the target object from the segmented video images using a fifth neural network model; performing target monitoring and accurate positioning on the facial image feature map using the third neural network model to obtain a candidate frame region corresponding to the facial image feature map; and performing maximum pooling on the candidate frame region using the fourth neural network model to obtain the facial image feature set.

[0155] Optionally, the program running controls the device where the nonvolatile storage medium is located to perform the following functions: calculating a first evaluation score corresponding to the evaluation result of each of the emotional features; summing the first evaluation scores to obtain a second evaluation score; and determining the customer service quality evaluation result according to the second evaluation score.

[0156] Optionally, the program running controls the device where the nonvolatile storage medium is located to perform the following functions: obtaining a first violation frequency corresponding to the first violation behavior of the target object in the audio information; obtaining a second violation frequency corresponding to the second violation behavior of the target object in the video information; and summing the first violation frequency and the second violation frequency to obtain a total violation frequency of the target object.

[0157] Optionally, the method further includes: determining whether an evaluation index of the customer service quality of the target object reaches an alarm threshold, wherein the evaluation index includes at least one of the following: the audio feature classification evaluation result, the video feature classification evaluation result, the customer service quality evaluation result, and the total violation frequency; and issuing an alarm instruction if any of the evaluation indexes reaches the alarm threshold.

[0158] Optionally, the program running controls the device where the nonvolatile storage medium is located to perform the following functions: frame processing on the video information to obtain a plurality of video images; determining whether the second violation behavior existing in adjacent two of the video images is the same violation behavior; and recording the same violation behavior as one of the second violation behaviors if the determination result is yes.

[0159] Optionally, the device where the non-volatile storage medium is located is controlled to perform the following functions during program execution: training the first neural network model based on the audio information to obtain a trained first neural network model; training the second neural network model based on the video information to obtain a trained second neural network model; and updating the first neural network model and the second neural network model according to the trained first neural network model and the trained second neural network model, respectively.

[0160] Optionally, the device where the non-volatile storage medium is located is controlled to perform the following functions during program execution: segmenting the audio information according to a preset time length to obtain a plurality of audio samples; determining a matching degree of a voiceprint feature of a first audio sample and a pre-stored voiceprint feature; if the matching degree is greater than or equal to a matching degree threshold, obtaining the speech sequence; and if the matching degree is less than the matching degree threshold, issuing an alarm instruction.

[0161] Optionally, the device where the non-volatile storage medium is located is controlled to perform the following functions during program execution: detecting a number of faces in each frame of the video image by a face recognition model; determining whether the number of faces is 1; if the determination result is yes, extracting the face image feature set from each frame of the video image; and if the determination result is no, issuing an alarm instruction.

[0162] According to the embodiments of the present application, an embodiment of a processor is also provided. Optionally, in the present embodiment, the processor is used to run a program, wherein the program performs any of the evaluation methods of the quality of customer service when running.

[0163] According to the embodiments of the present application, an embodiment of a computer program product is also provided, which is adapted to execute a program that initializes the steps of any of the evaluation methods of the quality of customer service when executed on a data processing device.

[0164] Optionally, the computer program product is adapted to execute a program that initializes the following steps when executed on a data processing device: obtaining customer service monitoring information, wherein the customer service monitoring information is audio information and video information generated by a target object during service execution; using a first neural network model to calculate an audio feature classification evaluation result based on the audio information; using a second neural network model to calculate a video feature classification evaluation result based on the video information, wherein the audio feature classification evaluation result and the video feature classification evaluation result each include an evaluation result of at least one emotional feature; and determining a customer service quality evaluation result corresponding to the target object according to the audio feature classification evaluation result and the video feature classification evaluation result.

[0165] According to the embodiments of the present application, an embodiment of an electronic device is also provided, which comprises a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to perform any of the methods for evaluating the quality of customer service.

[0166] The serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0167] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0168] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other manners. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units can be a logical function division, and in actual implementation, there can be another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0169] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple units. According to actual needs, part or all of the units can be selected to achieve the purpose of the present embodiment scheme.

[0170] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0171] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable nonvolatile storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a nonvolatile storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned nonvolatile storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0172] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A method for evaluating customer service quality, characterized in that, include: Obtain customer service monitoring information, wherein the customer service monitoring information is audio and video information generated by the target object during the performance of the service; Using the first neural network model, an audio feature classification evaluation result is calculated based on the audio information; A second neural network model is used to calculate video feature classification evaluation results based on the video information. The audio feature classification evaluation results and the video feature classification evaluation results both include evaluation results of at least one emotion feature. Based on the audio feature classification evaluation results and the video feature classification evaluation results, the customer service quality evaluation result corresponding to the target object is determined, including: calculating the first evaluation score corresponding to the evaluation result of each emotion feature in the following manner: ,in, Let represent the i-th first evaluation score, 1≤i≤M, and be a positive integer; This represents the weight of the type to which the i-th first specific emotion feature belongs; =1 indicates that the i-th specific characteristic behavior has occurred. =0 indicates that the i-th specific characteristic behavior did not occur; Indicates the time when a specific emotional characteristic behavior occurs. Indicates less than The dynamic reference time, where K is the time decrement factor, K>1; express The number of specific emotional characteristic behaviors generated within a reference time period is used to calculate the sum of multiple first evaluation scores to obtain a second evaluation score; the customer service quality evaluation result is determined based on the second evaluation score. The method further includes: obtaining the first violation count corresponding to the first violation of the target object in the audio information; obtaining the second violation count corresponding to the second violation of the target object in the video information, including: performing frame-by-frame processing on the video information to obtain multiple video images; determining whether the second violation in two adjacent video images is the same violation; if the determination result is yes, then the same violation is recorded as 1 second violation; summing the first violation count and the second violation count to obtain the total number of violations of the target object; determining whether the customer service quality evaluation index of the target object reaches the alarm threshold, wherein the evaluation index includes at least one of the following: the audio feature classification evaluation result, the video feature classification evaluation result, the customer service quality evaluation result, and the total number of violations; if any one of the evaluation indexes reaches the alarm threshold, then an alarm indication is issued.

2. The method according to claim 1, characterized in that, The first neural network model is used to calculate the audio feature classification evaluation result based on the audio information, including: Obtain the speech sequence from the audio information; The speech sequence is segmented into frames to extract the speech frame features. The speech frame features are segmented to obtain the speech segment features of the speech sequence; The emotional cognitive window features of the speech sequence are obtained by fitting the features of the speech segment with an objective function; Based on the speech frame features, the speech segment features, and the emotion recognition window features, the audio feature classification evaluation result is calculated using the first neural network model.

3. The method according to claim 1, characterized in that, The second neural network model includes a third neural network model and a fourth neural network model. The step of using the second neural network model to calculate the video feature classification evaluation result based on the video information includes: The video information is divided into frames to obtain multiple video images; Extract the facial image feature set of the target object from each frame of the video image; The facial image feature set is input into the third neural network model and the fourth neural network model respectively to obtain the video feature classification evaluation result.

4. The method according to claim 3, characterized in that, Extracting the facial image feature set of the target object from each frame of the video image, including: Each frame of the video image is segmented using a target classifier to obtain segmented video images. The segmented video images are processed using a fifth neural network model to obtain the facial image feature map of the target object; The third neural network model is used to perform target detection and precise localization processing on the facial image feature map to obtain the candidate box region corresponding to the facial image feature map. The candidate box region is subjected to max pooling processing by the fourth neural network model to obtain the facial image feature set.

5. The method according to claim 1, characterized in that, The method further includes: The first neural network model is trained based on the audio information to obtain the trained first neural network model; The second neural network model is trained based on the video information to obtain the trained second neural network model; The first neural network model and the second neural network model are updated based on the trained first neural network model and the trained second neural network model, respectively.

6. The method according to claim 2, characterized in that, Prior to acquiring the speech sequence from the audio information, the method further includes: The audio information is segmented according to a preset time length to obtain multiple audio samples. Determine the matching degree between the voiceprint features of the first audio sample and the pre-stored voiceprint features; If the matching degree is greater than or equal to the matching degree threshold, then the speech sequence is obtained; If the matching degree is less than the matching degree threshold, an alarm indication is issued.

7. The method according to claim 3, characterized in that, The method further includes: The number of faces in each frame of the video image is detected using a face recognition model; Determine whether the number of faces is 1; If the determination result is yes, then the facial image feature set is extracted from each frame of the video image; If the judgment result is negative, an alarm instruction will be issued.

8. A customer service quality evaluation device, characterized in that, include: The acquisition module is used to acquire customer service monitoring information, wherein the customer service monitoring information is audio and video information generated by the target object during the execution of the service; The first calculation module is used to calculate the audio feature classification evaluation result based on the audio information using a first neural network model; The second calculation module is used to calculate video feature classification evaluation results based on the video information using a second neural network model and a third neural network model. The audio feature classification evaluation results and the video feature classification evaluation results both include evaluation results of at least one emotion feature. The determination module is used to determine the customer service quality evaluation result corresponding to the target object based on the audio feature classification evaluation result and the video feature classification evaluation result, including: calculating the first evaluation score corresponding to the evaluation result of each emotion feature in the following manner: ,in, Let represent the i-th first evaluation score, 1≤i≤M, and be a positive integer; This represents the weight of the type to which the i-th first specific emotion feature belongs; =1 indicates that the i-th specific characteristic behavior has occurred. =0 indicates that the i-th specific characteristic behavior did not occur; Indicates the time when a specific emotional characteristic behavior occurs. Indicates less than The dynamic reference time, where K is the time decrement factor, K>1; express The number of specific emotional characteristic behaviors generated within a reference time period is used to calculate the sum of multiple first evaluation scores to obtain a second evaluation score; the customer service quality evaluation result is determined based on the second evaluation score. The device is further configured to: obtain a first violation count corresponding to a first violation of the target object in the audio information; obtain a second violation count corresponding to a second violation of the target object in the video information, including: performing frame-by-frame processing on the video information to obtain multiple video images; determining whether the second violation in two adjacent video images is the same violation; if the determination result is yes, then recording the same violation as 1 second violation; summing the first violation count and the second violation count to obtain the total number of violations of the target object; determining whether the customer service quality evaluation index of the target object reaches an alarm threshold, wherein the evaluation index includes at least one of the following: the audio feature classification evaluation result, the video feature classification evaluation result, the customer service quality evaluation result, and the total number of violations; if any one of the evaluation indicators reaches the alarm threshold, then issuing an alarm indication.

9. A customer service quality evaluation system, characterized in that, include: A video customer service client is used to obtain customer service monitoring information, wherein the customer service monitoring information is audio and video information generated by the target object during the service execution process; An edge computing platform server is connected to the video customer service client and is used to calculate an audio feature classification evaluation result based on the audio information using a first neural network model. A cloud platform server, connected to the edge computing platform server, is used to calculate video feature classification evaluation results based on the video information using a second neural network model and a third neural network model. Based on the audio feature classification evaluation results and the video feature classification evaluation results, a customer service quality evaluation result corresponding to the target object is determined. Both the audio feature classification evaluation results and the video feature classification evaluation results include evaluation results for at least one emotional feature, including: calculating a first evaluation score corresponding to the evaluation result of each emotional feature in the following manner: ,in, Let represent the i-th first evaluation score, 1≤i≤M, and be a positive integer; This represents the weight of the type to which the i-th first specific emotion feature belongs; =1 indicates that the i-th specific characteristic behavior has occurred. =0 indicates that the i-th specific characteristic behavior did not occur; Indicates the time when a specific emotional characteristic behavior occurs. Indicates less than The dynamic reference time, where K is the time decrement factor, K>1; express The number of specific emotional characteristic behaviors generated within a reference time period is used to calculate the sum of multiple first evaluation scores to obtain a second evaluation score; the customer service quality evaluation result is determined based on the second evaluation score. The cloud platform server is further configured to: obtain the first violation count corresponding to the first violation of the target object in the audio information; obtain the second violation count corresponding to the second violation of the target object in the video information, including: performing frame-by-frame processing on the video information to obtain multiple video images; determining whether the second violation in two adjacent video images is the same violation; if the determination result is yes, then the same violation is recorded as 1 second violation; summing the first violation count and the second violation count to obtain the total number of violations of the target object; determining whether the customer service quality evaluation index of the target object reaches the alarm threshold, wherein the evaluation index includes at least one of the following: the audio feature classification evaluation result, the video feature classification evaluation result, the customer service quality evaluation result, and the total number of violations; if any one of the evaluation indicators reaches the alarm threshold, then an alarm indication is issued.

10. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores multiple instructions adapted for loading and execution by a processor of the customer service quality evaluation method according to any one of claims 1 to 7.

11. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the customer service quality evaluation method according to any one of claims 1 to 7.

Images