Customer service intelligent quality inspection method, device and storage medium

Abstract

The present invention discloses a customer service intelligent quality inspection method, device and storage medium. The method comprises: obtaining customer service voice data; inputting the customer service voice data into a voice keyword detection model to obtain a voice keyword detection result, including: S1, obtaining a first acoustic feature of the customer service voice data; S2, performing a first two-dimensional convolution, maximum pooling, and a first activation function processing on the first acoustic feature; S3, inputting the processing result into a multi-head attention mechanism for keyword capture, and processing the keyword to obtain a voice keyword detection result; inputting the customer service voice data into a voice emotion detection model to obtain a voice emotion detection result, including: A, constructing a three-channel feature map; B, performing a second two-dimensional convolution and batch normalization processing on the feature map; C, obtaining a voice emotion detection result. Performing voice keyword detection and voice emotion detection simultaneously realizes multi-task parallel detection, and improves the quality inspection accuracy by combining the detection results of the two.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a customer service intelligent quality inspection method, device and storage medium. Background Art

[0002] Customer service call quality inspection tasks can be considered as a type of speech emotion recognition. From the perspective of the model, it is to quickly classify the content expressed by the customer through the classification task model, and at the same time grade the customer service quality. Customer service quality inspection can effectively and quickly detect whether the user's expression expresses dissatisfaction, satisfaction, and no emotion, and can also detect whether the customer service attitude and wording are standardized.

[0003] Due to the huge amount of voice data, traditional voice service quality inspection has problems such as low efficiency of manual sampling, low accuracy, poor comprehensiveness, and high voice transcription overhead. At present, many machine learning and deep learning methods have emerged for intelligent quality inspection of customer service voice. The accuracy of customer service voice intelligent quality inspection in the existing technology is poor, and most of them are single quality inspection methods based on voice-to-text conversion, with low quality inspection accuracy. Even if the voice data is monitored, the tasks are unrelated to each other. Such quality inspection methods often obtain single features and the accuracy of the model is limited. Summary of the invention

[0004] The present invention aims to at least solve the technical problems existing in the prior art, and in particular innovatively proposes a customer service intelligent quality inspection method, device and storage medium.

[0005] In order to achieve the above-mentioned purpose of the present invention, according to the first aspect of the present invention, the present invention provides a customer service intelligent quality inspection method, including: obtaining customer service voice data; inputting the customer service voice data into a voice keyword detection model to obtain a voice keyword detection result, and / or, inputting the customer service voice data into a voice emotion detection model to obtain a voice emotion detection result; the inputting the customer service voice data into the voice keyword detection model to obtain the voice keyword detection result specifically includes: step S1, obtaining the first acoustic feature of the customer service voice data; step S2, performing a first two-dimensional convolution processing, a maximum pooling processing, and a first activation function processing on the first acoustic feature; step S3, converting the processing of step S2 into the first linear layer. The result is input into a multi-head attention mechanism for keyword capture, and the captured keywords are processed through a second linear layer to obtain a speech keyword detection result; the customer service voice data is input into a speech emotion detection model to obtain a speech emotion detection result, specifically including: step A, obtaining a second acoustic feature of the customer service voice data, a first-order difference of the second acoustic feature, and a second-order difference of the second acoustic feature, and constructing a feature map with three channels, namely, the second acoustic feature, the first-order difference of the second acoustic feature, and the second-order difference of the second acoustic feature; step B, performing a second two-dimensional convolution and batch normalization on the feature map; step C, processing the processing result of step B through a third linear layer to obtain a speech emotion detection result.

[0006] The above technical solution: directly monitors the voice data of customer service, which can well capture the customer service's urgent emotions, does not need to be converted into text, avoids the introduction of conversion errors, and improves the accuracy of customer service quality detection. In the process of voice keyword detection, the voice keyword detection model can capture the global voice information expression through the multi-head attention mechanism to make the model pay more attention to the characteristics of keywords, thereby improving the detection accuracy of voice keywords. In the process of voice emotion detection, the voice emotion detection model expands the dimension of the second acoustic feature, expands the first-order difference of the second acoustic feature, and the second-order difference of the second acoustic feature, and converts the one-dimensional second acoustic feature into a three-channel feature map. In this way, the feature map is used for subsequent processing to obtain better acoustic features, making the emotion detection result more accurate. The above scheme also discloses that voice keyword detection and voice emotion detection are performed simultaneously, and both are detected at the same time, realizing multi-task parallel detection, and the quality inspection accuracy can be improved by combining the detection results of the two.

[0007] In a preferred embodiment of the present invention, when the step of inputting the customer service voice data into a voice keyword detection model to obtain a voice keyword detection result and the step of inputting the customer service voice data into a voice emotion detection model to obtain a voice emotion detection result exist at the same time, step S3 is replaced by: inputting the processing result of step S2 into a bidirectional long short-term memory network unit for semantic capture; step C is replaced by: fusing the processing result of step B with the semantics captured by the bidirectional long short-term memory network unit in step S3 to obtain a first fusion feature, and obtaining a voice emotion detection result after processing the first fusion feature through a third linear layer; after step S3, step S4 is also included, and step S4 is: fusing the keywords captured by the multi-head attention mechanism with the first fusion feature to obtain a second fusion feature, and obtaining a voice keyword detection result after processing the second fusion feature through a second linear layer.

[0008] The above technical solution: voice keyword detection and voice emotion detection are crossed, and the two parallel detection steps can share part of the data, and the corresponding models can share part of the weights, which can further improve the quality inspection accuracy.

[0009] In order to achieve the above-mentioned purpose of the present invention, according to the second aspect of the present invention, the present invention provides a customer service intelligent quality inspection device, including a customer service voice data acquisition module and a voice quality inspection model module, the voice quality inspection model module includes a voice keyword detection model module and / or a voice emotion detection model module; the voice keyword detection model module includes a first acoustic feature acquisition unit, a first two-dimensional convolution module, a first linear layer, a multi-head attention mechanism unit and a second linear layer connected in sequence; the first acoustic feature acquisition unit acquires the first acoustic feature of the customer service voice data, the first two-dimensional convolution module performs first two-dimensional convolution processing, maximum pooling processing, and activation function processing on the first acoustic feature, and the processing result of the first two-dimensional convolution module is processed by the first linear layer. The multi-head attention mechanism unit is input to capture keywords, and the second linear layer processes the captured keywords to obtain speech keyword detection results; the speech emotion detection model module includes a second acoustic feature acquisition unit, a second two-dimensional convolution module and a third linear layer connected in sequence; the second acoustic feature acquisition unit acquires the second acoustic feature of the customer service voice data, the first-order difference of the second acoustic feature, and the second-order difference of the second acoustic feature, and constructs a feature map of three channels, namely the second acoustic feature, the first-order difference of the second acoustic feature, and the second-order difference of the second acoustic feature; the second two-dimensional convolution module performs second two-dimensional convolution processing and batch normalization processing on the feature map, and the output result of the second two-dimensional convolution module is processed by the third linear layer to obtain the speech emotion detection result.

[0010] The above technical solution: The device directly monitors the voice data of the customer service, and can well capture the customer service's urgent emotions. It does not need to be converted into text, avoids the introduction of conversion errors, and improves the accuracy of customer service quality detection. In the process of voice keyword detection, the voice keyword detection model can capture the global voice information expression through the multi-head attention mechanism to make the model pay more attention to the characteristics of keywords, thereby improving the detection accuracy of voice keywords. In the process of voice emotion detection, the voice emotion detection model expands the dimension of the second acoustic feature, expands the first-order difference of the second acoustic feature, and the second-order difference of the second acoustic feature, and converts the one-dimensional second acoustic feature into a three-channel feature map. In this way, the feature map is used for subsequent processing to obtain better acoustic features, making the emotion detection result more accurate. The above scheme also discloses that voice keyword detection and voice emotion detection are performed simultaneously, and both are detected simultaneously, realizing multi-task parallel detection, and the quality inspection accuracy can be improved by combining the detection results of the two.

[0011] In a preferred embodiment of the present invention, the speech quality inspection model module also includes a CTC loss calculation module, a first loss calculation module, a second loss calculation module and a parameter adjustment module for training; the training process of the speech quality inspection model includes: obtaining speech training samples, setting keyword labels and emotion labels for the speech training samples, the keyword labels include mandatory language labels, prohibited language labels, and polite language labels, the mandatory language labels, prohibited language labels, polite language labels and emotion labels of the service all have two values, and the two values ​​represent abnormality and normality respectively; training the speech keyword detection model and the speech emotion detection model in parallel; during the training process, CTC The TC loss calculation module is used to calculate the error between the phoneme sequence output by the first linear layer and the phoneme sequence pre-extracted from the speech training sample, the first loss calculation module is used to calculate the error between the speech keyword detection result output by the second linear layer and the keyword label, the second loss calculation module is used to calculate the error between the speech emotion detection result output by the third linear layer and the emotion label, and the parameter adjustment module adjusts the model parameters of at least one of the first two-dimensional convolution module, the second two-dimensional convolution module, the bidirectional long short-term memory network unit, and the multi-head attention mechanism unit according to the output results of the CTC loss calculation module, the first loss calculation module, and the second loss calculation module.

[0012] The above technical solution: realizes parallel multi-task joint training of speech keyword detection model and speech emotion detection model, can simultaneously capture acoustic and temporal feature expressions, allows models to share some weights, enables the model to learn some feature information that a single model cannot learn, can adapt to a variety of application scenarios, and avoids large errors in single text quality inspection caused by multiple dialects. The residual memory network based on CTC and attention mechanism is used for the speech quality inspection model, which can improve the accuracy of the current model to a higher level and further improve the detection accuracy.

[0013] In order to achieve the above-mentioned purpose of the present invention, according to the third aspect of the present invention, the present invention provides a customer service intelligent quality inspection method, including step one, and step two and / or step three; step one, obtaining customer voice data and customer service voice data, converting the customer voice data into customer text data, and converting the customer service voice data into customer service text data; step two, obtaining the emotion classification result of the customer text data based on deep learning, recorded as the customer emotion classification result, and in parallel, extracting the emotion classification result of the customer service text data, recorded as the customer service emotion classification result; fusing the customer emotion classification result and the customer service emotion classification result, and obtaining the text emotion detection result after processing the fused result through a fourth linear layer and a second activation function; step three, performing keyword detection on the customer text data to obtain a customer keyword detection result, performing keyword detection on the customer service text data to obtain a customer service keyword detection result, and fusing the customer keyword detection result and the customer service keyword detection result to obtain a text keyword detection result.

[0014] The above technical solution: discloses a text-based customer service quality inspection method, which adopts a multi-task mode to detect emotions and / or keywords on customer texts and customer service texts in parallel, and the final output text emotion detection result integrates the customer emotion classification result and the customer service emotion classification result, and the final output text keyword detection result integrates the customer keyword classification result and the customer service keyword classification result, which helps to improve the customer quality inspection accuracy, and at the same time can integrate the text emotion detection results and the text keyword detection results, which can further improve the quality inspection accuracy.

[0015] In a preferred embodiment of the present invention, in step one, the process of obtaining customer voice data and customer service voice data includes: obtaining original audio data, separating customer audio data and customer service audio data from the original audio data; and performing format conversion and sampling rate conversion on the customer audio data and customer service audio data respectively to obtain customer voice data and customer service voice data.

[0016] The above technical solution realizes the separation of the audio data of the customer and the customer service in the single-track (single-channel) original audio data. Since the single-track audio data requires less storage space, this can save a lot of storage resources.

[0017] In order to achieve the above-mentioned purpose of the present invention, according to the fourth aspect of the present invention, the present invention provides a customer service intelligent quality inspection device, including: a data acquisition module, used to acquire customer voice data and customer service voice data; a text conversion module, converting the customer voice data into customer text data, and converting the customer service voice data into customer service text data; a text emotion detection module and / or a text keyword detection module; the text emotion detection module executes: based on deep learning, obtaining the emotion classification result of the customer text data, recorded as the customer emotion classification result, and in parallel, extracting the emotion classification result of the customer service text data, recorded as the customer service emotion classification result; fusing the customer emotion classification result and the customer service emotion classification result, and obtaining the text emotion detection result after processing the fused result through a fourth linear layer and a second activation function; the text keyword detection module executes: performing keyword detection on the customer text data to obtain a customer keyword detection result, performing keyword detection on the customer service text data to obtain a customer service keyword detection result, and fusing the customer keyword detection result and the customer service keyword detection result to obtain a text keyword detection result.

[0018] The above technical solution: discloses a text-based customer service quality inspection device, which uses a multi-task mode to detect emotions and / or keywords on customer texts and customer service texts in parallel, and the final output text emotion detection result integrates the customer emotion classification result and the customer service emotion classification result, and the final output text keyword detection result integrates the customer keyword classification result and the customer service keyword classification result, which helps to improve the customer quality inspection accuracy, and at the same time can integrate the text emotion detection results and the text keyword detection results, which can further improve the quality inspection accuracy.

[0019] In order to achieve the above-mentioned purpose of the present invention, according to the fifth aspect of the present invention, the present invention provides a customer service intelligent quality inspection method, including: obtaining customer service voice data, and obtaining voice keyword detection results and voice emotion detection results according to the customer service intelligent quality inspection method described in the first aspect of the present invention; obtaining customer service voice data and customer voice data, and obtaining text keyword detection results and text emotion detection results according to the customer service intelligent quality inspection method described in the third aspect of the present invention; fusing the voice keyword detection results and the text keyword detection results to obtain the final keyword detection results, and fusing the voice emotion detection results and the text emotion detection results to obtain the final emotion detection results.

[0020] The above technical solution: This quality inspection method can perform speech detection and text detection at the same time, and the final output detection result is the fusion value of the two. It combines the two modal data of speech and text for comprehensive quality inspection. It can not only capture emotional features but also semantic features well. It is also highly robust, enough to overcome various noise data and emergencies that occur during the training process and have higher accuracy.

[0021] In order to achieve the above-mentioned purpose of the present invention, according to the sixth aspect of the present invention, the present invention provides a computer-readable storage medium storing a computer program, which, when executed, executes the customer service intelligent quality inspection method described in the first aspect of the present invention, or executes the customer service intelligent quality inspection method described in the third aspect of the present invention, or executes the customer service intelligent quality inspection method described in the fifth aspect of the present invention.

[0022] The above technical solution: When the computing program stored in the storage medium is executed, it can perform speech detection and text detection at the same time, and the final output detection result is the fusion value of the two. It combines the two modal data of speech and text for comprehensive quality inspection. It can not only capture emotional features but also semantic features well, and it is also highly robust, enough to overcome various noise data and emergencies that occur during the training process, and has a higher accuracy rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the execution of the first customer service intelligent quality inspection device provided by the present invention in a specific implementation manner;

[0024] Figure 2 is a schematic diagram of the execution of the second customer service intelligent quality inspection device provided by the present invention in a specific implementation manner;

[0025] Figure 3 is a flow chart of a seventh customer service intelligent quality inspection method provided by the present invention in a specific implementation manner;

[0026] Figure 4 is a schematic diagram of the fusion process in step 702 in a specific implementation manner. DETAILED DESCRIPTION

[0027] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0028] In the description of the present invention, it is to be understood that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicating orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0029] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0030] The present invention discloses a first customer service intelligent quality inspection method. In a preferred embodiment, the customer service intelligent quality inspection method includes:

[0031] Step 100, obtaining customer service voice data;

[0032] Step 101, inputting customer service voice data into a voice keyword detection model to obtain voice keyword detection results, specifically including:

[0033] Step S1, obtaining the first acoustic feature of the customer service voice data; the first acoustic feature is preferably, but not limited to, a Mel spectrum feature or a result of taking the logarithm of the Mel spectrum feature. Specifically, in the original audio data, a frame length of 25 ms is formed, and a sliding window is moved 10 ms each time to obtain an 80-dimensional first acoustic feature (which can be a Mel spectrum feature taking the logarithm, i.e., a logfilter bank feature).

[0034] Step S2, performing a first two-dimensional convolution process, a maximum pooling process, and a first activation function process on the first acoustic feature; preferably, including 8 micro-convolutions, each convolution operation is followed by a dropout (discarding rate of the neural network) of 0.05 to 0.2, preferably, a dropout of 0.1, so as to simply and effectively prevent overfitting. Preferably, a maximum pooling process is performed after every two convolution operations. The first activation function is preferably, but not limited to, a RELU activation function.

[0035] Step S3: Input the processing result of step S2 into the MultiHead-Attention mechanism through the first linear layer to capture keywords, and process the captured keywords through the second linear layer to obtain the speech keyword detection result. The second linear layer maps the keyword labels based on the obtained keywords and outputs them. The keyword labels include at least one of the mandatory language labels, prohibited language labels, and polite language labels. The mandatory language labels, prohibited language labels, and polite language labels all have two values, which represent abnormality and normality respectively.

[0036] In this embodiment, preferably, the process of acquiring customer service voice data is: directly obtain data from the hardware storage medium, including the original 8k sampling rate audio data of the customer service answering the phone in each scenario; after obtaining the data, perform corresponding data processing on the data: the first step is to separate the single-track and double-track voice. Since the quality inspection data is mono (hereinafter referred to as single-track) audio data in some scenarios, that is, the customer service and the customer are in the same channel, such voice data cannot be directly separated by code, and the customer service audio data can only be separated by machine learning or deep learning methods; the format and sampling rate of the acquired audio data are converted, because the telephone voice is usually 8k, and the data format has been compressed into mp3, here is written based on the librosa library to convert the format to wav format, further, the wav format data is upsampled to 16k, and then based on VAD (voice activity detection) The silent segment is cut and some noise is removed.

[0037] The present invention discloses a second customer service intelligent quality inspection method. In a preferred embodiment, the customer service intelligent quality inspection method includes:

[0038] Step 200, obtaining customer service voice data;

[0039] Step 201, inputting customer service voice data into a voice emotion detection model to obtain a voice emotion detection result, specifically including:

[0040] Step A, obtain the second acoustic feature of the customer service voice data, the first-order difference of the second acoustic feature, and the second-order difference of the second acoustic feature, and construct a three-dimensional feature map. The three channels of the feature map are the second acoustic feature, the first-order difference of the second acoustic feature, and the second-order difference of the second acoustic feature. In this way, the dimension is expanded and the acoustic features can be better captured. The second acoustic feature is preferably, but not limited to, a Mel cepstral coefficient.

[0041] Step B, performing a second two-dimensional convolution process and a batch normalization process on the feature map; preferably, comprising a plurality of two-dimensional convolutions connected in sequence, and adding a batch normalization layer (BatchNormal) after each two-dimensional convolution.

[0042] Step C: Process the processing result of step B through the third linear layer to obtain the speech emotion detection result. The third linear layer maps the processing result of step B to the corresponding emotion label and outputs it. The emotion label has two values, which represent abnormality and normality respectively.

[0043] The present invention discloses a third customer service intelligent quality inspection method. In a preferred embodiment, the customer service intelligent quality inspection method includes:

[0044] Step 300, obtaining customer service voice data;

[0045] Step 301, inputting customer service voice data into a voice keyword detection model to obtain voice keyword detection results, specifically including:

[0046] Step S1, obtaining the first acoustic feature of the customer service voice data;

[0047] Step S2, performing a first two-dimensional convolution process, a maximum pooling process, and a first activation function process on the first acoustic feature;

[0048] In step S3, the processing result of step S2 is input into the multi-head attention mechanism through the first linear layer to capture keywords, and the captured keywords are processed through the second linear layer to obtain the speech keyword detection result.

[0049] Step 302, inputting the customer service voice data into the voice emotion detection model to obtain the voice emotion detection result, specifically including:

[0050] Step A, obtaining the second acoustic feature, the first-order difference of the second acoustic feature, and the second-order difference of the second acoustic feature of the customer service voice data, and constructing a feature graph having three channels, namely, the second acoustic feature, the first-order difference of the second acoustic feature, and the second-order difference of the second acoustic feature;

[0051] Step B, performing a second two-dimensional convolution process and batch normalization process on the feature map;

[0052] Step C: Process the processing result of step B through a third linear layer to obtain a speech emotion detection result.

[0053] In this embodiment, step 301 and step 302 may be performed in parallel.

[0054] In an application scenario of the present embodiment, preferably, the speech emotion detection result and the speech keyword detection result can be fused as the final customer service quality inspection score. Preferably, the fusion method can be average value fusion or weighted average fusion.

[0055] In a preferred embodiment of the present invention, in the disclosed third customer service intelligent quality inspection method: step S3 is replaced by: inputting the processing result of step S2 into a bidirectional long short-term memory network unit (BiLSTM) for semantic capture;

[0056] Replace step C with: fuse the processing result of step B with the semantics captured by the bidirectional long short-term memory network unit in step S3 to obtain a first fused feature, and obtain the speech emotion detection result after processing the first fused feature through the third linear layer; the fusion method is preferably but not limited to adding values ​​in a specified dimension, for example, the processing result of step B and the semantics captured by the bidirectional long short-term memory network unit are both two-dimensional vectors, then the fusion process can be adding values ​​in the second dimension.

[0057] After step S3, step S4 is also included. Step S4 is: fusing the keywords captured by the multi-head attention mechanism with the first fusion feature to obtain a second fusion feature, and processing the second fusion feature through a second linear layer to obtain a speech keyword detection result. The fusion method can refer to the fusion method in step C, which will not be repeated here.

[0058] The present invention provides a first customer service intelligent quality inspection device. In a preferred embodiment, Figure 1 As shown, the customer intelligent quality inspection device includes a customer service voice data acquisition module and a voice quality inspection model module, and the voice quality inspection model module includes a voice keyword detection model module and / or a voice emotion detection model module; specifically, the voice quality inspection model module includes a voice keyword detection model module, or the voice quality inspection model module includes a voice emotion detection model module, or the voice quality inspection model module includes a voice emotion detection model module and a voice keyword detection model module.

[0059] In this embodiment, the voice keyword detection model module includes a first acoustic feature acquisition unit, a first two-dimensional convolution module, a first linear layer, a multi-head attention mechanism unit and a second linear layer connected in sequence; the first acoustic feature acquisition unit acquires the first acoustic feature of the customer service voice data, the first two-dimensional convolution module performs first two-dimensional convolution processing, maximum pooling processing, and activation function processing on the first acoustic feature, and the processing result of the first two-dimensional convolution module is input into the multi-head attention mechanism unit through the first linear layer for keyword capture, and the second linear layer processes the captured keywords to obtain the voice keyword detection result. Preferably, the first two-dimensional convolution module includes a total of M two-dimensional convolutions, M is a positive integer, preferably 8, the convolution kernel size of the two-dimensional convolution is 30, and each two-dimensional convolution is followed by a dropout of 0.1, and then passes through a relu activation function, and every two convolutions are followed by a maximum pooling layer with a kernel size of 2.

[0060] In this embodiment, the voice emotion detection model module includes a second acoustic feature acquisition unit, a second two-dimensional convolution module and a third linear layer connected in sequence; the second acoustic feature acquisition unit acquires the second acoustic feature of the customer service voice data, the first-order difference of the second acoustic feature, and the second-order difference of the second acoustic feature, and constructs a feature map of three channels, namely the second acoustic feature, the first-order difference of the second acoustic feature, and the second-order difference of the second acoustic feature; the second two-dimensional convolution module performs second two-dimensional convolution processing and batch normalization processing on the feature map, and the output result of the second two-dimensional convolution module is processed by the third linear layer to obtain the voice emotion detection result. Preferably, the second two-dimensional convolution module is modified based on VGG16. Specifically, the last layer of VGG16 is deleted, and a batch normalization layer BatchNormal is added after each layer.

[0061] In a preferred embodiment, the voice keyword detection model module and the voice emotion detection model module both process the customer service voice data in parallel;

[0062] The speech keyword detection model module also includes a bidirectional long short-term memory network unit and a second fusion unit, and the bidirectional long short-term memory network unit semantically captures the output result of the first two-dimensional convolution module;

[0063] The voice emotion detection model module also includes a first fusion unit, which fuses the semantics output by the bidirectional long short-term memory network unit with the output result of the second two-dimensional convolution module and outputs the fusion result to the third linear layer;

[0064] The second fusion unit fuses the output result of the first fusion unit with the output result of the multi-head attention mechanism unit and inputs the fusion result into the second linear layer.

[0065] In a preferred embodiment, the speech quality inspection model module also includes a CTC loss calculation module (connectionist temporal classification, neural network-based temporal classification, referred to as CTC), a first loss calculation module, a second loss calculation module and a parameter adjustment module for training. Here, the role of CTC is to force alignment with the phoneme sequence extracted in advance from the speech training sample, so that our audio features are closer to the text sequence, and the multi-head attention mechanism is to capture the global expression of speech information, so that the model pays more attention to the characteristics of keywords.

[0066] In this embodiment, the training process of the speech quality inspection model includes:

[0067] Step I: obtain speech training samples, set keyword labels and emotion labels for the speech training samples. The keyword labels include mandatory language labels, prohibited language labels, and polite language labels. The mandatory language labels, prohibited language labels, polite language labels, and emotion labels all have two values, which represent abnormality and normality respectively. Due to the limited manual annotation data, the public Mandarin data set on the market is used to integrate speech data from actual scenarios for training, and finally a word error rate that can be used in industry is achieved.

[0068] Step II, training the speech keyword detection model and the speech emotion detection model in parallel; during the training process, the CTC loss calculation module is used to calculate the error between the phoneme sequence output by the first linear layer and the phoneme sequence pre-extracted from the speech training sample, which can be recorded as CTCLoss, the first loss calculation module is used to calculate the error between the speech keyword detection result output by the second linear layer and the keyword label, the second loss calculation module is used to calculate the error between the speech emotion detection result output by the third linear layer and the emotion label, and the parameter adjustment module adjusts the model parameters of at least one of the first two-dimensional convolution module, the second two-dimensional convolution module, the bidirectional long short-term memory network unit, and the multi-head attention mechanism unit according to the output results of the CTC loss calculation module, the first loss calculation module, and the second loss calculation module. The model parameters preferably include, but are not limited to, the Linear.weight, bias and other parameters of the linear layer, the tensor of the two-dimensional convolution module, etc.

[0069] In this implementation, the existing technologies are all single quality inspection methods based on speech-to-text conversion, and the tasks are all unrelated to each other. Such quality inspection methods often obtain single features and the accuracy of the model is limited. The multi-task joint training method based on speech proposed in the present invention can capture the feature expressions in acoustics and time series at the same time, and the weights are shared between models, which can improve the accuracy of the current model to a higher level.

[0070] In this embodiment, preferably, the first loss calculation module and the second loss calculation module both calculate cross entropy loss, and the calculated loss results can be recorded as CELoss1 and CELoss2 respectively. Preferably, the parameter adjustment module adds CTCLoss to CELoss1 and CELoss2 as the final loss function value LOSS to perform back propagation to adjust the model parameters. LOSS = CTCLoss + CELoss1 + CELoss2.

[0071] In this embodiment, preferably, a model iterative update step is also included. During the model iterative update stage, the iterative update needs to be coordinated with manual review, that is, after the quality inspection report is generated, a manual review is first performed to check whether the model score is reasonable in the performance database and the recording database. If it is unreasonable, corresponding modifications are made and feedback is given to the model, so that the model can be trained again with correct labels based on the erroneous data, and the model parameters are continuously iterated to improve the model accuracy.

[0072] The present invention discloses a fourth customer service intelligent quality inspection method. In a preferred embodiment, the customer service intelligent quality inspection method includes:

[0073] Step 400, obtain customer voice data and customer service voice data, convert the customer voice data into customer text data, and convert the customer service voice data into customer service text data; preferably, the voice-to-text conversion adopts the Conformer model.

[0074] Step 401, based on deep learning, obtain the emotion classification result of the customer text data, recorded as the customer emotion classification result, and in parallel, extract the emotion classification result of the customer service text data, recorded as the customer service emotion classification result; fuse the customer emotion classification result and the customer service emotion classification result, and obtain the text emotion detection result after the fusion result is processed by the fourth linear layer and the second activation function. Deep learning preferably but not limited to using the existing BERT emotion classification model. Two BERT emotion classification models can be configured, one for obtaining the emotion classification result of the customer text data, and the other for extracting the emotion classification result of the customer service text data.

[0075] In this embodiment, preferably, in step 400, the process of obtaining the customer voice data and the customer service voice data includes:

[0076] Step 4001, obtain the original audio data, and separate the customer audio data and the customer service audio data from the original audio data; preferably, a fully convolutional time-domain audio separation network (Conv-TasNet) model is used to separate the voice data of the customer and the customer service, and part of the two-channel data is used as a training sample. After the model training is completed, the trained fully convolutional time-domain audio separation network (Conv-TasNet) model can be used to separate the single-track voice, and only the single-track data needs to be saved, thereby saving a lot of storage resources.

[0077] Step 4001, respectively perform format conversion and sampling rate conversion on the customer audio data and the customer service audio data to obtain the customer voice data and the customer service voice data.

[0078] The present invention discloses a fifth customer service intelligent quality inspection method. In a preferred embodiment, the customer service intelligent quality inspection method includes:

[0079] Step 500, obtaining customer voice data and customer service voice data, converting the customer voice data into customer text data, and converting the customer service voice data into customer service text data;

[0080] Step 501, keyword detection is performed on customer text data to obtain customer keyword detection results, keyword detection is performed on customer service text data to obtain customer service keyword detection results, and the customer keyword detection results and customer service keyword detection results are integrated to obtain text keyword detection results. The text-based keyword detection method preferably but not limited to the existing phonetic code matching algorithm is used, which will not be described in detail here.

[0081] The present invention discloses a sixth customer service intelligent quality inspection method. In a preferred embodiment, the customer service intelligent quality inspection method includes:

[0082] Step 600, obtaining customer voice data and customer service voice data, converting the customer voice data into customer text data, and converting the customer service voice data into customer service text data;

[0083] Step 601, based on deep learning, obtain the emotion classification result of the customer text data, recorded as the customer emotion classification result, and in parallel, extract the emotion classification result of the customer service text data, recorded as the customer service emotion classification result; fuse the customer emotion classification result and the customer service emotion classification result, and obtain the text emotion detection result after processing the fused result through the fourth linear layer and the second activation function; preferably, the fusion method can adopt average value fusion or weighted average fusion, and the weight of the customer in the weighted average fusion is lower, so that the customer service quality can be evaluated more accurately.

[0084] Step 602: Perform keyword detection on the customer text data to obtain a customer keyword detection result, perform keyword detection on the customer service text data to obtain a customer service keyword detection result, and fuse the customer keyword detection result and the customer service keyword detection result to obtain a text keyword detection result. Preferably, the fusion method can adopt average value fusion or weighted average fusion. In the weighted average fusion, the weight of the customer is relatively low, so that the customer service quality can be evaluated more accurately.

[0085] The present invention discloses a second customer service intelligent quality inspection device. In a preferred embodiment, Figure 2 As shown, the device comprises:

[0086] Data acquisition module, used to acquire customer voice data and customer service voice data;

[0087] Text conversion module, converting customer voice data into customer text data, and converting customer service voice data into customer service text data;

[0088] Text emotion detection module and / or text keyword detection module;

[0089] The text emotion detection module performs: based on deep learning, the emotion classification results of customer text data are obtained, which are recorded as customer emotion classification results. In parallel, the emotion classification results of customer service text data are extracted, which are recorded as customer service emotion classification results. The customer emotion classification results and the customer service emotion classification results are integrated, and the integrated results are processed through the fourth linear layer and the second activation function to obtain the text emotion detection results.

[0090] The text keyword detection module executes: performing keyword detection on customer text data to obtain customer keyword detection results, performing keyword detection on customer service text data to obtain customer service keyword detection results, and integrating customer keyword detection results and customer service keyword detection results to obtain text keyword detection results.

[0091] The present invention discloses a seventh customer service intelligent quality inspection method. In a preferred implementation, Figure 3 As shown, the customer intelligent quality inspection method includes:

[0092] Step 700, obtaining customer service voice data, and obtaining voice keyword detection results and voice emotion detection results according to the third customer service intelligent quality inspection method provided by the present invention;

[0093] Step 701, obtaining customer service voice data and customer voice data, and obtaining text keyword detection results and text emotion detection results according to the sixth customer service intelligent quality inspection method of the present invention;

[0094] Step 702: The voice keyword detection result and the text keyword detection result are integrated to obtain the final keyword detection result, and the voice emotion detection result and the text emotion detection result are integrated to obtain the final emotion detection result. Preferably, the method further includes a quality inspection report generation step.

[0095] In this embodiment, preferably, the specific fusion process is as follows Figure 4 As shown in the figure, the voice keyword detection results and text keyword detection results are weighted averaged, and the voice emotion detection results and text emotion detection results are weighted averaged, and a quality inspection report is generated based on the two weighted average scores. In the quality inspection report generation step, the online or offline quality inspection stage cooperates with the development of the front-end and back-end, makes the corresponding web-side application, and provides a batch quality inspection programming interface for subsequent calls. During the quality inspection process, a report can be generated in real time and fed back to the mailbox or DingTalk of the specified ID. At the same time, various quality inspection scores and corresponding customer service IDs are entered into the performance database for assessment.

[0096] For text-based quality inspection technology, the current technology usually only has semantic representation after voice-to-text conversion, thus losing the acoustic feature expression, and cannot capture the actual emotions of customer service and agents well. Therefore, the present invention combines the method of simultaneous voice and text quality inspection in the final customer service scoring stage, which can not only capture emotional features but also semantic features well, and is also highly robust, sufficient to overcome various noise data and emergencies that occur during the training process.

[0097] The present invention combines both speech and text modal data for comprehensive quality inspection, making the model more robust and more accurate. In terms of accuracy, a method of joint training of two tasks is adopted, different feature representations are selected for different scenarios, and some weights are shared between models, so that the model can learn some feature information that a single model cannot learn, and can be used in a variety of application scenarios to avoid large errors in single text quality inspection caused by multiple dialects. In terms of model methods, the quality inspection model of the present invention is very innovative, and compared with the baseline model, it has achieved an accuracy improvement of nearly 20%.

[0098] The present invention also discloses a computer-readable storage medium storing a computer program. In a preferred embodiment, when the computer program is executed, the first, second, and third customer service intelligent quality inspection methods of the present invention are executed, or the fourth, fifth, and sixth customer service intelligent quality inspection methods of the present invention are executed, or the seventh customer service intelligent quality inspection method of the present invention is executed.

[0099] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0100] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A customer service intelligent quality inspection method, characterized in that: include: Obtain customer service voice data; Inputting the customer service voice data into a voice keyword detection model to obtain a voice keyword detection result, and / or inputting the customer service voice data into a voice emotion detection model to obtain a voice emotion detection result; The step of inputting the customer service voice data into a voice keyword detection model to obtain a voice keyword detection result specifically includes: Step S1, obtaining a first acoustic feature of the customer service voice data; Step S2, performing a first two-dimensional convolution process, a maximum pooling process, and a first activation function process on the first acoustic feature; Step S3, inputting the processing result of step S2 into the multi-head attention mechanism through the first linear layer to capture keywords, and processing the captured keywords through the second linear layer to obtain speech keyword detection results; The step of inputting the customer service voice data into a voice emotion detection model to obtain a voice emotion detection result specifically includes: Step A, obtaining the second acoustic feature of the customer service voice data, the first-order difference of the second acoustic feature, and the second-order difference of the second acoustic feature, and constructing a feature graph whose three channels are the second acoustic feature, the first-order difference of the second acoustic feature, and the second-order difference of the second acoustic feature; Step B, performing a second two-dimensional convolution process and a batch normalization process on the feature map; Step C: Process the processing result of step B through a third linear layer to obtain a speech emotion detection result.

2. The customer service intelligent quality inspection method according to claim 1, characterized in that: When the step of inputting the customer service voice data into the voice keyword detection model to obtain the voice keyword detection result and the step of inputting the customer service voice data into the voice emotion detection model to obtain the voice emotion detection result exist at the same time, step S3 is replaced by: The processing result of step S2 is input into the bidirectional long short-term memory network unit for semantic capture; and step C is replaced by: The processing result of step B and the semantics captured by the bidirectional long short-term memory network unit in step S3 are fused to obtain a first fused feature, and the first fused feature is processed by a third linear layer to obtain a speech emotion detection result; After step S3, step S4 is also included, and step S4 is: fusing the keywords captured by the multi-head attention mechanism with the first fusion feature to obtain a second fusion feature, and processing the second fusion feature through a second linear layer to obtain a speech keyword detection result.

3. A customer service intelligent quality inspection device, characterized in that: It includes a customer service voice data acquisition module and a voice quality inspection model module, wherein the voice quality inspection model module includes a voice keyword detection model module and / or a voice emotion detection model module; The speech keyword detection model module includes a first acoustic feature acquisition unit, a first two-dimensional convolution module, a first linear layer, a multi-head attention mechanism unit and a second linear layer connected in sequence; the first acoustic feature acquisition unit acquires the first acoustic feature of the customer service voice data, the first two-dimensional convolution module performs first two-dimensional convolution processing, maximum pooling processing, and activation function processing on the first acoustic feature, and the processing result of the first two-dimensional convolution module is input into the multi-head attention mechanism unit through the first linear layer to capture keywords, and the second linear layer processes the captured keywords to obtain speech keyword detection results; The speech emotion detection model module includes a second acoustic feature acquisition unit, a second two-dimensional convolution module and a third linear layer connected in sequence; the second acoustic feature acquisition unit acquires the second acoustic feature of the customer service voice data, the first-order difference of the second acoustic feature, and the second-order difference of the second acoustic feature, and constructs a feature map of three channels, namely the second acoustic feature, the first-order difference of the second acoustic feature, and the second-order difference of the second acoustic feature; the second two-dimensional convolution module performs second two-dimensional convolution processing and batch normalization processing on the feature map, and the output result of the second two-dimensional convolution module is processed by the third linear layer to obtain the speech emotion detection result.

4. The customer service intelligent quality inspection device according to claim 3, characterized in that: The voice keyword detection model module and the voice emotion detection model module both process the customer service voice data in parallel; The speech keyword detection model module also includes a bidirectional long short-term memory network unit and a second fusion unit, and the bidirectional long short-term memory network unit semantically captures the output result of the first two-dimensional convolution module; The speech emotion detection model module also includes a first fusion unit, which fuses the semantics output by the bidirectional long short-term memory network unit with the output result of the second two-dimensional convolution module and outputs the fusion result to the third linear layer; The second fusion unit fuses the output result of the first fusion unit with the output result of the multi-head attention mechanism unit and inputs the fusion result into the second linear layer.

5. The customer service intelligent quality inspection device according to claim 4, characterized in that: The speech quality inspection model module also includes a CTC loss calculation module for training, a first loss calculation module, a second loss calculation module and a parameter adjustment module; The training process of the speech quality inspection model includes: Acquire speech training samples, and set keyword labels and emotion labels for the speech training samples, wherein the keyword labels include mandatory language labels, prohibited language labels, and polite language labels, and the mandatory language labels, prohibited language labels, polite language labels, and emotion labels all have two values, which represent abnormality and normality, respectively; Train the speech keyword detection model and the speech emotion detection model in parallel; During the training process, the CTC loss calculation module is used to calculate the error between the phoneme sequence output by the first linear layer and the phoneme sequence pre-extracted from the speech training sample, the first loss calculation module is used to calculate the error between the speech keyword detection result output by the second linear layer and the keyword label, and the second loss calculation module is used to calculate the error between the speech emotion detection result output by the third linear layer and the emotion label. The parameter adjustment module adjusts the model parameters of at least one of the first two-dimensional convolution module, the second two-dimensional convolution module, the bidirectional long short-term memory network unit, and the multi-head attention mechanism unit according to the output results of the CTC loss calculation module, the first loss calculation module, and the second loss calculation module.

6. A customer service intelligent quality inspection method, characterized in that: include: Acquire customer service voice data, and obtain voice keyword detection results and voice emotion detection results according to the customer service intelligent quality inspection method described in claim 1 or 2; Acquiring customer service voice data and customer voice data, and acquiring text keyword detection results and text emotion detection results, specifically including: step one, and step two and / or step three; Step 1: Acquire customer voice data and customer service voice data, convert the customer voice data into customer text data, and convert the customer service voice data into customer service text data; Step 2: Based on deep learning, obtain the emotion classification result of the customer text data, recorded as the customer emotion classification result, and in parallel, extract the emotion classification result of the customer service text data, recorded as the customer service emotion classification result; fuse the customer emotion classification result and the customer service emotion classification result, and obtain the text emotion detection result after processing the fused result through the fourth linear layer and the second activation function; Step three, performing keyword detection on the customer text data to obtain a customer keyword detection result, performing keyword detection on the customer service text data to obtain a customer service keyword detection result, and integrating the customer keyword detection result and the customer service keyword detection result to obtain a text keyword detection result; The speech keyword detection results and the text keyword detection results are fused to obtain the final keyword detection results, and the speech emotion detection results and the text emotion detection results are fused to obtain the final emotion detection results.

7. The customer service intelligent quality inspection method according to claim 6, characterized in that: In step 1, the process of obtaining customer voice data and customer service voice data includes: Obtaining original audio data, and separating customer audio data and customer service audio data from the original audio data; The customer audio data and the customer service audio data are format converted and the sampling rate converted respectively to obtain the customer voice data and the customer service voice data.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed, the customer service intelligent quality inspection method described in claim 1 or 2 is executed, or the customer service intelligent quality inspection method described in claim 6 or 7 is executed.

Images