Service quality evaluation and improvement method, system and equipment based on bidirectional voice emotion curve interaction analysis, and medium

Abstract

The invention discloses a service quality evaluation and improvement method, system and device based on bidirectional voice emotion curve interaction analysis and a medium, and the method comprises the following steps: separating a dual-track voice stream based on an original customer service call into a seat voice stream and a customer voice stream; extracting acoustic and text features to form a multi-modal feature sequence; outputting probability distribution of the seat and the customer in a predefined emotion dimension; splicing the emotion probability distribution, mapping the emotion probability distribution to a numerical space, generating a seat time sequence emotion curve and a customer time sequence emotion curve, and carrying out index quantification; generating a comprehensive service quality score, and evaluating the service quality; and monitoring a time sequence emotion curve of the customer, pushing an early warning signal and recommending verbal skills when detecting that a negative emotion exceeds a preset threshold value, and generating a service report. According to the method, the problems of fragmentization and interactivity of emotion state analysis, insufficient bidirectional emotion interaction modeling ability between the seat and the client and business disjunction caused by lack of real-time intervention in an existing voice emotion analysis technology are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, speech signal processing and big data analysis, in particular to a service quality evaluation and improvement method, system, device and medium based on bidirectional speech emotion curve interaction analysis. BACKGROUND

[0002] In the market competition of voice interaction as the core, such as customer service, telemedicine, telephone sales, the objective evaluation of service quality (QoS) has become the key to improve the core competitiveness and optimize the operating efficiency of enterprises. The traditional evaluation method relies on manual sampling, which is low in efficiency and subjective. In recent years, with the development of artificial intelligence technology, especially the progress in the field of affective computing, a series of patent technologies aimed at automatically evaluating service quality through speech emotion analysis have emerged. Among them, the evolution of speech emotion analysis method has gone through the process from simple rules to statistical models, and then to the preliminary association of context.

[0003] The early method mainly uses the method based on keyword matching and rules, which is suitable for quick and simple negative tendency screening of call content. Then the method based on static emotion recognition and the method based on pre-trained model are derived.

[0004] 1. Method based on keyword matching and rules The method based on keyword matching and rules identifies specific keywords in the call, such as "complaint" and "unsatisfied", and combines simple acoustic features such as volume and speed to make a rule-based judgment. This method cannot understand the context, and has high false positive and false negative rates for voice recognition with emotional turning points, and cannot capture complex emotional states.

[0005] 2. Method based on static emotion recognition The method based on static emotion recognition uses machine learning or deep learning models such as convolutional neural network CNN or support vector machine SVM to classify the emotion of a single segment of speech. For example, the existing patent extracts MFCC features and inputs them into a classification model to give an overall emotion label for the entire call or a fixed segment. Its limitations are: usually only analyzes the emotion of the customer, ignoring the key influence of the agent's emotion on the interaction process; regards emotion as a static label, which cannot reflect the ups and downs, turning points and interaction effects of emotion in the call process, and cannot evaluate the "emotional guidance" ability of the agent; in addition, the simple classification model cannot understand the reasons for the emotion, and cannot associate sudden changes in emotion with specific content of the conversation, such as the agent's explanation and marketing tactics.

[0006] 3. Method based on pre-trained model The method based on pre-trained model usually adopts the way of fine-tuning pre-trained voice model or fusing text transcription for emotion analysis. Although the effect is improved, there is still a problem of poor model generalization. The model trained in a specific scene is difficult to adapt to the changes of different businesses and different regional accents. In addition, the model has weak interpretability, giving scores but not providing improvement basis, which is difficult for business personnel to understand and trust. The evaluation results are only applicable to post-scoring and early warning, lacking real-time intervention on the service process and personalized empowerment guidance to the agents, failing to maximize the analysis value.

[0007] In summary, based on the in-depth research on the application of existing voice emotion analysis technology in customer service quality evaluation, the main problems and limitations of voice emotion recognition methods can be summarized as follows: 1. Fragmentation of emotion state analysis and lack of interaction. Existing methods discretize continuous emotion dynamics into independent time points for splicing or weighting, failing to effectively model the interaction process as two continuous and coupled emotion curves. For example, when evaluating the service quality of an agent, only the current user emotion is determined to be negative, while the complete downward curve from extreme anger to calmness and the interaction between the curve and the agent's emotion curve of always remaining calm are ignored, resulting in the inability to quantify the true calming effectiveness of the agent. This method is essentially a parallel emotion report rather than an interactive analysis.

[0008] For example, in a complaint call, the customer's initial emotion is "angry". An experienced agent may first show "empathy" and "apology", and the emotion value will briefly show a slight negative value to calm the customer. If the customer is successfully calmed down, their emotion may gradually turn to "calm" or even "satisfaction". Existing technologies may only label the agent as having shown "negative emotion", but cannot understand that this short-lived "empathetic negative" is actually a key positive behavior that contributes to service success. Conversely, if the customer is already showing signs of frustration, the agent still responds in a mechanical and flat tone, even though the agent's emotion is "neutral", the actual service quality is low, but existing technologies cannot capture this "mismatch in emotional interaction".

[0009] 2. Insufficient ability to model two-way emotional interaction. Due to the lack of interaction analysis, existing models cannot effectively depict and quantify the mutual influence mechanism of emotions between the two parties during the service process. Specifically, it is difficult to distinguish whether the user's emotion naturally recovers or the agent's proactive intervention dominates the change in emotion; it is also difficult to measure the guiding force of the agent's emotional stability on the user's emotion. This lack of modeling ability for the causal or driving relationship between emotions makes the evaluation method only show correlation and difficult to reveal causality, greatly limiting its depth value as a tool for optimizing the agent's speech.

[0010] For example, in the first call, the customer remained calm throughout and ultimately rated the customer as satisfied. In the second call, the customer initially expressed extreme anger, but after the agent patiently explained, reassured, and resolved the issue, the customer's emotions eventually turned to basic satisfaction. From the results, the final ratings might be the same, but clearly, the agent in the second call demonstrated stronger communication skills, problem-solving abilities, and emotional guidance skills, creating greater customer value. An evaluation method that only focuses on the final result fails to identify this more valuable ability of the agent in the second call, and might even give a lower rating due to negative emotions during the process, which is undoubtedly inaccurate.

[0011] 3. Lack of real-time intervention and dynamic support capabilities: Existing methods are somewhat lagging and struggle to provide support at "critical turning points" in service interactions. For example, when the system detects that a customer's emotions are escalating towards "anger," and the agent's response pattern fails to effectively alleviate the situation, existing solutions can only mark the call as "high-risk" afterward, lacking dynamic intervention. Summary of the Invention

[0012] The purpose of this invention is to provide a service quality assessment and improvement method, system, device, and medium based on two-way voice emotion curve interaction analysis, in order to solve the problems of fragmentation and lack of interactivity in existing voice emotion analysis technology, insufficient ability to model two-way emotion interaction between agents and customers, and business disconnect caused by lack of real-time intervention.

[0013] To achieve the above objectives, the present invention adopts the following technical solution: A service quality assessment and improvement method based on two-way voice emotion curve interaction analysis includes the following steps: S1: Based on the original customer service call's dual-channel audio stream, the mixed audio stream is separated into the agent's audio stream and the customer's audio stream through sound source separation technology, and the timestamps are aligned; effective audio segments are extracted, and acoustic features and text features are extracted to form a time-aligned multimodal feature sequence; S2: Utilize the temporal model of the Transformer encoder to output the probability distribution of agents and customers on a predefined emotional dimension; splice the emotional probability distributions of each time segment and map them to a continuous numerical space to generate agent temporal emotion curves and customer temporal emotion curves, and perform interactive analysis index quantification; wherein, the multimodal feature sequence formed in step S1 is used as the input of the temporal model. S3: Generate a comprehensive service quality score based on the quantitative results of interactive analysis indicators and a weighted evaluation model to assess service quality; monitor the customer's temporal emotion curve during the call, and when a rapid rise in negative emotions exceeds a preset threshold, push a warning signal and recommended script to the agent's interface; and generate a service report after the call ends, including the agent's temporal emotion curve and the customer's temporal emotion curve, interpretation of quantitative indicators, and improvement suggestions.

[0014] To optimize the above technical solution, the specific limitations also include: In step S1, the sound source separation technique defines a mixed speech signal matrix and uses independent component analysis to solve for the separation matrix, maximizing the statistical independence of the separated signals. The formula is as follows:

[0015] in, The voice sample value of the agent. For customer voice sampling values, For the separation matrix, ; For the original two-channel speech in The observed signal at that moment, These are the original audio sample values ​​from the seat channel. The original customer channel voice sample value. , This represents the total number of time frames. Preferably, in step S1, after the text features are transcribed into text by automatic speech recognition, a pre-trained language model is used to obtain a text embedding vector, which is then concatenated with the feature vector composed of parameters using acoustic features to form a time-aligned multimodal feature sequence.

[0016] Furthermore, in step S2, the temporal model utilizes a multi-head self-attention mechanism to capture long-range contextual dependencies, with the attention function being:

[0017] in, For each attention head dimension, Q, K, and V are the query matrix, key matrix, and value matrix, respectively; Through the fully connected layer and The function outputs the first Probability distribution of frames in the predefined emotion dimension: ,

[0018] in, Represents the probability of each predefined emotion.

[0019] Further, in step S2, the probability distribution of a predefined emotion is mapped onto a continuous numerical space to form a numerical emotion value; the formula for calculating the emotion value is:

[0020] in, For a predefined emotional value weight matrix, Assigning value weights to each predefined emotion. ; Generate time-series emotion curves for agents and customers based on their emotion values:

[0021]

[0022] in, For the customer service representative's emotional curve, For customer sentiment curve, This represents the total number of frames representing valid speech segments.

[0023] Preferably, in step S2, the interaction analysis indicators include: Emotion regulation efficiency is used to measure the agent's ability to regulate the negative emotions of customers. It is calculated by weighting the agent's efficiency in soothing the customer's emotions and the normalized net area of ​​positive and negative emotions throughout the customer's call. Emotional synchronicity is used to measure the degree of synchronization between the agent's and the customer's emotional changes. It is calculated by fusing dynamic time-normalized distance and Pearson correlation coefficient. Key event correlation is used to locate key speech segments with drastic emotional changes by leveraging the attention weights of the Transformer encoder. The attention weights of the frames are extracted, keyframes are determined, and then the corresponding audio and text are associated to map the emotional changes into the corresponding text content.

[0024] Further, in step S3, the comprehensive service quality score is generated based on the quantitative results of the interactive analysis indicators and the weighted evaluation model, using the following formula:

[0025] in, For the effectiveness of emotion regulation, For emotional synchronization, To capture the customer's final emotions at the end of the call. For problem resolution rate, Assigning weights to each indicator. , .

[0026] This invention also proposes a service quality assessment and improvement system based on two-way voice emotion curve interaction analysis, comprising: The data acquisition and preprocessing module is used to separate the mixed speech stream into agent speech stream and customer speech stream based on the original customer service call using sound source separation technology, and align the timestamps; extract effective speech segments, and extract acoustic features and text features to form a time-aligned multimodal feature sequence; The temporal emotion curve module is used to output the probability distribution of agents and customers on a predefined emotion dimension using the temporal model of the Transformer encoder; it splices the emotion probability distribution of each time segment and maps it to a continuous numerical space to generate the agent temporal emotion curve and the customer temporal emotion curve for interactive analysis index quantification; wherein, the multimodal feature sequence formed in step S1 is used as the input of the temporal model. The evaluation and improvement module is used to generate a comprehensive service quality score based on the quantitative results of interactive analysis indicators and a weighted evaluation model to evaluate service quality; it monitors the customer's temporal emotion curve during the call, and when it detects that negative emotions are rising rapidly and exceeding a preset threshold, it pushes a warning signal and recommended scripts to the agent's interface, and generates a service report after the call ends, including the agent's temporal emotion curve and the customer's temporal emotion curve, interpretation of quantitative indicators, and improvement suggestions.

[0027] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the service quality assessment and improvement method based on bidirectional voice emotion curve interaction analysis as described above.

[0028] The present invention also proposes a computer-readable storage medium storing a computer program that enables a computer to execute the service quality assessment and improvement method based on two-way voice emotion curve interaction analysis as described above.

[0029] Compared with the prior art, the beneficial effects of the present invention are: This invention provides a service quality assessment and improvement method based on two-way voice emotion curve interaction analysis. Through sound source separation, temporal alignment, and feature extraction and fusion, it provides a high-quality, multimodal, and consistent data foundation for agent-customer emotion interaction analysis. By constructing a temporal model to generate agent and customer temporal emotion curves and quantifying interaction analysis indicators, it upgrades service quality assessment from static, single-point emotion recognition to dynamic, continuous two-way interaction process analysis, achieving continuous dynamic tracking of emotional states. This overcomes the limitations of traditional fragmented analysis and solves the problem of insufficient modeling capabilities for agent-customer two-way emotion interaction. Through comprehensive evaluation, real-time intervention, and personalized reporting, it realizes a business closed loop from evaluation to empowerment, constructing a complete empowerment system from monitoring, analysis to intervention and improvement, solving the problem of business disconnect caused by a lack of real-time intervention.

[0030] By using sound source separation technology, the problem of speaker voice aliasing in two-channel speech is solved, ensuring that the voice streams of agents and customers are independent and pure, reducing the interference of invalid data on the emotion model. By extracting the acoustic and text features of effective speech segments and fusing them to generate multimodal feature sequences, more comprehensive multi-data representation of emotions is provided, and input is also provided for long-distance context modeling, solving the fragmentation problem of emotion state analysis.

[0031] By stitching together the probability distribution of emotions across different time segments, a continuous emotion curve is formed, fully reflecting the emotional changes of both the agent and the customer during the call. This enables the system to identify trends and patterns of emotional changes, providing a data foundation for assessing the agent's ability to guide emotions. By quantifying interaction analysis indicators, the system objectively measures the agent's ability to proactively regulate customer emotions, transforming service attitude into quantifiable data. This allows the system to distinguish whether emotional changes are a natural decline or the result of the agent's proactive intervention, enabling the assessment of service quality from a process perspective. It also identifies key moments and turning points in the service process, making the assessment results more scientific and comprehensive.

[0032] The weighted evaluation model integrates indicators such as emotion regulation effectiveness and emotion synchronicity to avoid the one-sidedness of a single indicator, making the overall service quality score more scientific and comprehensive. By monitoring the customer's temporal emotion curve, the system can trigger an early warning when the customer's emotion deteriorates, allowing agents to adjust their strategies in time. At the same time, it pushes recommended scripts based on emotion analysis results to help agents take the right actions at critical moments and improve customer satisfaction. After the call ends, a service report is generated, enabling agents to understand their own performance, transforming passive quality inspection into proactive management, and vague evaluation into precise empowerment, continuously driving the improvement of overall service quality and customer satisfaction. Attached Figure Description

[0033] Figure 1 : A flowchart illustrating the service quality assessment and improvement method based on two-way voice emotion curve interaction analysis of the present invention. Figure 2 : A framework diagram of the service quality assessment and improvement method based on two-way voice emotion curve interaction analysis of the present invention.

[0034] Figure 3 : A schematic diagram illustrating the monitoring and early warning method for service quality assessment and improvement based on two-way voice emotion curve interaction analysis of the present invention. Detailed Implementation

[0035] The present invention will be further described in detail below through specific embodiments, but it should not be construed as limiting the scope of the subject matter of the present invention to the following embodiments. All technologies implemented based on the above content of the present invention fall within the scope of the present invention.

[0036] The following is an explanation of some of the terms used in this invention: Mel-Frequency Cepstral Coefficients (MFCC): Acoustic features widely used in speech recognition and emotion analysis. They can simulate the nonlinear perception of human ears for sounds of different frequencies, thus effectively characterizing the spectral properties of speech.

[0037] Transformer architecture: A deep learning model architecture based on self-attention mechanism, which is particularly good at processing sequential data and can capture long-distance contextual dependencies. It is widely used in natural language processing and speech signal processing.

[0038] Multi-head attention mechanism: A core component of Transformer, which enables the model to simultaneously focus on information from different representation subspaces at different positions in the input sequence by running multiple self-attention layers in parallel, thereby capturing contextual information more comprehensively.

[0039] Independent Component Analysis (ICA) is a blind source separation technique. Its core objective is to reverse-engineer the statistically independent original source signals from multiple mixed observation signals. It does not require prior knowledge of the specific parameters of the mixing process and relies solely on the core assumption that the source signals are statistically independent.

[0040] Emotion regulation effectiveness: A quantitative indicator used to measure the ability of customer service agents to proactively regulate customer emotions. It can usually be broken down into sub-indicators such as soothing efficiency (the speed at which negative customer emotions subside) and value gain (the net improvement in customer emotions).

[0041] The technical solution of the present invention will be further described in detail below with reference to specific embodiments: In one embodiment, this invention proposes a service quality assessment and improvement method based on two-way voice emotion curve interaction analysis, the flowchart of which is shown below. Figure 1 As shown, the entire method includes the following steps: S1: Based on the original customer service call's dual-channel audio stream, the mixed audio stream is separated into the agent's audio stream and the customer's audio stream through sound source separation technology, and the timestamps are aligned; effective audio segments are extracted, and acoustic features and text features are extracted to form a time-aligned multimodal feature sequence; S2: Utilize the temporal model of the Transformer encoder to output the probability distribution of agents and customers on a predefined emotional dimension; splice the emotional probability distributions of each time segment and map them to a continuous numerical space to generate agent temporal emotion curves and customer temporal emotion curves, and perform interactive analysis index quantification; wherein, the multimodal feature sequence formed in step S1 is used as the input of the temporal model. S3: Generate a comprehensive service quality score based on the quantitative results of interactive analysis indicators and a weighted evaluation model to assess service quality; monitor the customer's temporal emotion curve during the call, and when a rapid rise in negative emotions exceeds a preset threshold, push a warning signal and recommended script to the agent's interface; and generate a service report after the call ends, including the agent's temporal emotion curve and the customer's temporal emotion curve, interpretation of quantitative indicators, and improvement suggestions.

[0042] like Figure 2 As shown, in step S1, the original two-channel audio stream of the call between the agent and the customer is input, and the agent's audio channels are... Customer voice channel The system outputs a multimodal feature sequence of the call between the agent and the customer. For the dual-channel mixed speech in the agent-customer call scenario, a blind source separation technique based on independent component analysis is used to separate the clean agent speech stream and the customer speech stream, and then align them using timestamp calibration.

[0043] First, define the mixed speech signal matrix. The original two-channel speech signal is at time... The observed signal is:

[0044] in, These are the original audio sample values ​​from the seat channel. The original customer channel voice sample value. ( , This represents the total number of time frames. Then, sound source separation is performed, and the separation matrix is ​​solved using independent component analysis to maximize the statistical independence of the separated signals.

[0045] in, The voice sample value of the agent. For customer voice sampling values, For the separation matrix, Forming a time-aligned speech stream sequence: agent speech stream Customer voice stream ( , They are the first (The audio signal of the frame).

[0046] A dual thresholding method based on energy and zero-crossing rate is used to remove silence segments while retaining valid speech fragments. Frame energy is calculated for the [frame name missing]. Frame speech signal ( Its energy is ,in This represents the number of sampling points per frame.

[0047] The zero-crossing rate is calculated using the following formula:

[0048] in It is a symbolic function.

[0049] Next, determine the valid frames. and ( , If the threshold is based on statistics, it is determined to be a valid speech frame; otherwise, it is a silent frame and is discarded.

[0050] For valid speech frame sequences, acoustic features and text features are extracted in parallel to construct multimodal feature vectors.

[0051] First, eight core acoustic features are extracted to form an acoustic feature vector. ( The formula is as follows: Calculate the 12th order Mel-frequency cepstral coefficients (MFCCs) and the 1st and 2nd order differences, totaling 36 dimensions.

[0052] in, The output of the Mel filter bank, The number of filters. .

[0053] Calculate the fundamental frequency using the autocorrelation method. The calculation formula is as follows:

[0054] in, The hysteresis parameter represents the offset of the signal delay, from... Start increasing, until Taken in the relevant function Maximum corresponding frequency ( (where is the sampling rate).

[0055] The effective speech frame sequence is transcribed into text using the end-to-end speech recognition model Whisper-large. , Given the text length, a pre-trained BERT model is used to extract text embedding vectors, as shown in the following formula:

[0056] in , where is the dimension of the hidden layer in the BERT model.

[0057] Acoustic features and textual features are fused through feature concatenation to obtain the first... Multimodal feature vector of a frame Ultimately, a seat feature sequence is formed. With customer characteristic sequence ,in, It represents the total number of valid frames.

[0058] In step S2, dynamic emotional interaction is achieved through temporal emotion modeling, emotional curve quantification, and interaction feature quantification. The input is a multimodal feature vector, and the output is a bidirectional emotional curve and interaction analysis indicators.

[0059] A Transformer encoder model with positional encoding is employed to capture long-distance dialogue context dependence and output the emotion probability distribution. First, the multimodal feature sequence... Add position encoding ,according to , ,in For feature dimension, Encoding the above features, the output of the multi-head self-attention function is:

[0060] in , For the number of attention heads, , , , The attention function is calculated using the following formula, which represents learnable parameters:

[0061] in, For each attention head dimension, Q, K, and V are the query matrix, key matrix, and value matrix, respectively; Then through the fully connected layer and The function outputs the first... Frames in a predefined set of emotions The probability distribution on: , Finally, the probability sequence of seat emotions was obtained. With customer sentiment probability sequence .

[0062] Mapping discrete emotion probability distribution to continuous value space ( This indicates an extremely negative meaning. (Indicating extremely positive sentiment), constructing a numerical sentiment curve.

[0063] Based on the annotation results from the business scenario, a value weight is assigned to each emotion. The weights are determined using the analytic hierarchy process (AHP). Next, the numerical sentiment score is calculated. The numerical sentiment value of a frame is a weighted sum of a probability distribution and a value weight, calculated as follows:

[0064] in, Represents the probability of each predefined emotion. For a predefined emotional value weight matrix, Assigning value weights to each predefined emotion. Then, time-series sentiment curves are generated for both agents and customers, i.e., agent sentiment curves. Customer sentiment curve ,in This represents the total number of frames representing valid speech segments.

[0065] The core interaction metrics are defined as quantified emotion regulation effectiveness, synchronicity, and association with key events.

[0066] First, assess the agent's ability to manage customers' negative emotions. From "appeasement efficiency" "and value gain" "Weighted composition. Soothing efficiency represents the time difference between the peak and the baseline of a customer's negative emotions, expressed as follows:"

[0067] in , The smaller the size, the more effective the reassurance.

[0068] Value gain represents the net area normalized of positive and negative emotions throughout a customer's call, expressed by the formula:

[0069] The integral value is approximated using the trapezoidal integral method. A higher value indicates a higher net emotional value for the customer.

[0070] The overall calculation of emotion regulation effectiveness is as follows:

[0071] in To appease the efficiency weights, they were obtained through training and adjustment using business data. .

[0072] Secondly, the degree of "synchronization" between changes in customer cleaning habits and those of agents is measured, which is the degree of emotional synchronicity. The dynamic time-warped distance is calculated using a fusion of dynamic time-warped distance and Pearson correlation coefficient. The similarity in shape between two curves is measured by the formula shown below:

[0073] Using Pearson correlation coefficient To measure the degree of linear correlation between two curves:

[0074] in, , , .

[0075] The comprehensive calculation method for emotional synchronicity is shown in the formula:

[0076] in , , , The larger the value, the higher the synchronization.

[0077] This paper utilizes the attention weights of the Transformer encoder to locate key speech segments with dramatic emotional changes, comprising two parts: attention weight extraction and keyframe determination. First, the average weight of the multi-head self-attention mechanism at the top layer of the encoder is obtained. , No. The attention weight of a frame reflects its contribution to the emotion judgment.

[0078] Determine keyframes, if ( (threshold for drastic emotional changes) and If so, it is determined to be a keyframe and associated with the corresponding speech text. This maps emotional changes to corresponding text content.

[0079] In step S3, service quality is quantified through weighted fusion of multiple indicators, and real-time intervention and post-event empowerment are implemented. The input of the module is the various quantitative indicators generated in step S2, and the output is the comprehensive service quality score and specific applications.

[0080] like Figure 3 As shown, firstly, based on core indicators such as emotion regulation efficacy and synchronicity, a weighted evaluation model is constructed to calculate the comprehensive service quality score, as shown in the following formula:

[0081] in, For the effectiveness of emotion regulation, For emotional synchronization, To capture the customer's final emotions at the end of the call. For problem resolution rate, Assigning weights to each indicator. , .

[0082] Then, it is mapped to a percentage score according to the following rules:

[0083] During the call, the slope of the customer's emotion curve is calculated in real time to detect the upward trend of negative emotions. (Within the sliding window...) arrive The mood slope of the frame is:

[0084] like ( If negative emotions rise rapidly to a certain threshold, a real-time alert will be triggered, sending an alert signal and recommended reassurance messages to the agents.

[0085] After the call ends, a service report is generated, including a two-way sentiment curve trend chart, interpretation of quantitative indicators, and suggestions for improvement. The two-way sentiment curve trend chart consists of the agent's time-series sentiment curve and the customer's time-series sentiment curve.

[0086] This invention also proposes a service quality assessment and improvement system based on two-way voice emotion curve interaction analysis, comprising: The data acquisition and preprocessing module is used to separate the mixed speech stream into agent speech stream and customer speech stream based on the original customer service call using sound source separation technology, and align the timestamps; extract effective speech segments, and extract acoustic features and text features to form a time-aligned multimodal feature sequence; The temporal emotion curve module is used to output the probability distribution of agents and customers on a predefined emotion dimension using the temporal model of the Transformer encoder; it splices the emotion probability distribution of each time segment and maps it to a continuous numerical space to generate the agent temporal emotion curve and the customer temporal emotion curve for interactive analysis index quantification; wherein, the multimodal feature sequence formed in step S1 is used as the input of the temporal model. The evaluation and improvement module is used to generate a comprehensive service quality score based on the quantitative results of interactive analysis indicators and a weighted evaluation model to evaluate service quality; it monitors the customer's temporal emotion curve during the call, and when it detects that negative emotions are rising rapidly and exceeding a preset threshold, it pushes a warning signal and recommended scripts to the agent's interface, and generates a service report after the call ends, including the agent's temporal emotion curve and the customer's temporal emotion curve, interpretation of quantitative indicators, and improvement suggestions.

[0087] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-mentioned service quality assessment and improvement method based on two-way voice emotion curve interaction analysis.

[0088] The present invention also proposes a computer-readable storage medium storing a computer program that enables a computer to execute the above-described service quality assessment and improvement method based on two-way voice emotion curve interaction analysis.

[0089] This invention enhances the continuity and contextual awareness of emotion state analysis, overcoming the fragmented judgment problem caused by neglecting temporal dependencies in traditional single-point emotion recognition. It strengthens the quantitative analysis capability of two-way emotion interaction, breaking through the limitations of existing methods that only perform parallel analysis of emotions from both parties and cannot effectively capture the mechanism of mutual influence of emotions. Furthermore, it improves the business applicability and operability of emotion analysis results, transforming abstract numerical indicators into concrete service improvement suggestions, solving the problem of the disconnect between evaluation results and actual operational improvements, and providing a refined management tool for agent performance evaluation and skills training. In practical applications, the service quality assessment and improvement method based on two-way voice emotion curve interaction analysis provided by this invention is suitable for scenarios requiring real-time emotion monitoring and service quality assessment, such as customer service centers, telemedicine consultations, and telephone sales. It features multimodal fusion, temporal dynamic analysis, and business closed-loop guidance.

[0090] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0091] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent substitutions, and improvements made by those skilled in the art to the above embodiments without departing from the scope of the technical solution of the present invention, based on the technical essence of the present invention, shall still fall within the protection scope of the technical solution of the present invention.

Claims

1. A service quality assessment and improvement method based on two-way voice emotion curve interaction analysis, characterized in that, Includes the following steps: S1: Based on the original customer service call's dual-channel audio stream, the mixed audio stream is separated into the agent's audio stream and the customer's audio stream through sound source separation technology, and the timestamps are aligned; effective audio segments are extracted, and acoustic features and text features are extracted to form a time-aligned multimodal feature sequence; S2: Utilize the temporal model of the Transformer encoder to output the probability distribution of agents and customers on a predefined emotional dimension; splice the emotional probability distributions of each time segment and map them to a continuous numerical space to generate agent temporal emotion curves and customer temporal emotion curves, and perform interactive analysis index quantification; wherein, the multimodal feature sequence formed in step S1 is used as the input of the temporal model. S3: Generate a comprehensive service quality score based on the quantitative results of interactive analysis indicators and a weighted evaluation model to assess service quality; monitor the customer's temporal emotion curve during the call, and when a rapid rise in negative emotions exceeds a preset threshold, push a warning signal and recommended script to the agent's interface; and generate a service report after the call ends, including the agent's temporal emotion curve and the customer's temporal emotion curve, interpretation of quantitative indicators, and improvement suggestions.

2. The service quality assessment and improvement method based on two-way voice emotion curve interaction analysis according to claim 1, characterized in that: In step S1, the sound source separation technique defines a mixed speech signal matrix and uses independent component analysis to solve for the separation matrix, maximizing the statistical independence of the separated signals. The formula is as follows: in, The voice sample value of the agent. For customer voice sampling values, For the separation matrix, ; For the original two-channel speech in The observed signal at that moment, These are the original audio sample values ​​from the seat channel. The original customer channel voice sample value. , This represents the total number of time frames.

3. The service quality assessment and improvement method based on two-way voice emotion curve interaction analysis according to claim 1, characterized in that: In step S1, after the text features are transcribed into text by automatic speech recognition, a pre-trained language model is used to obtain the text embedding vector, and then the feature vector composed of parameters using acoustic features is concatenated to form a time-aligned multimodal feature sequence.

4. The service quality assessment and improvement method based on two-way voice emotion curve interaction analysis according to claim 1, characterized in that: In step S2, the temporal model utilizes a multi-head self-attention mechanism to capture long-range contextual dependencies, and the attention function is: in, For each attention head dimension, Q, K, and V are the query matrix, key matrix, and value matrix, respectively; Through the fully connected layer and The function outputs the first Probability distribution of frames in the predefined emotion dimension: ， in, Represents the probability of each predefined emotion.

5. The service quality assessment and improvement method based on two-way voice emotion curve interaction analysis according to claim 4, characterized in that: In step S2, the probability distribution of a predefined emotion is mapped onto a continuous numerical space to form a numerical emotion value; the formula for calculating the emotion value is: in, For a predefined emotional value weight matrix, Assigning value weights to each predefined emotion. ; Generate time-series emotion curves for agents and customers based on their emotion values: in, For the customer service representative's emotional curve, For customer sentiment curve, This represents the total number of frames representing valid speech segments.

6. The service quality assessment and improvement method based on two-way voice emotion curve interaction analysis according to claim 1, characterized in that: In step S2, the interaction analysis metrics include: Emotion regulation efficiency is used to measure the agent's ability to regulate the negative emotions of customers. It is calculated by weighting the agent's efficiency in soothing the customer's emotions and the normalized net area of ​​positive and negative emotions throughout the customer's call. Emotional synchronicity is used to measure the degree of synchronization between the agent's and the customer's emotional changes. It is calculated by fusing dynamic time-normalized distance and Pearson correlation coefficient. Key event correlation is used to locate key speech segments with drastic emotional changes by leveraging the attention weights of the Transformer encoder. The attention weights of the frames are extracted, keyframes are determined, and then the corresponding audio and text are associated to map the emotional changes into the corresponding text content.

7. The service quality assessment and improvement method based on two-way voice emotion curve interaction analysis according to claim 1, characterized in that: In step S3, the comprehensive service quality score is generated based on the quantitative results of the interactive analysis indicators and the weighted evaluation model, using the following formula: in, For the effectiveness of emotion regulation, For emotional synchronization, To capture the customer's final emotions at the end of the call. For problem resolution rate, Assigning weights to each indicator. , .

8. A service quality assessment and improvement system based on two-way voice emotion curve interaction analysis, characterized in that, include: The data acquisition and preprocessing module is used to separate the mixed speech stream into agent speech stream and customer speech stream based on the original customer service call using sound source separation technology, and align the timestamps; extract effective speech segments, and extract acoustic features and text features to form a time-aligned multimodal feature sequence; The temporal emotion curve module is used to output the probability distribution of agents and customers on a predefined emotion dimension using the temporal model of the Transformer encoder; it splices the emotion probability distribution of each time segment and maps it to a continuous numerical space to generate the agent temporal emotion curve and the customer temporal emotion curve for interactive analysis index quantification; wherein, the multimodal feature sequence formed in step S1 is used as the input of the temporal model. The evaluation and improvement module is used to generate a comprehensive service quality score based on the quantitative results of interactive analysis indicators and a weighted evaluation model to evaluate service quality; it monitors the customer's temporal emotion curve during the call, and when it detects that negative emotions are rising rapidly and exceeding a preset threshold, it pushes a warning signal and recommended scripts to the agent's interface, and generates a service report after the call ends, including the agent's temporal emotion curve and the customer's temporal emotion curve, interpretation of quantitative indicators, and improvement suggestions.

9. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the service quality assessment and improvement method based on bidirectional voice emotion curve interaction analysis as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program that causes a computer to perform the service quality assessment and improvement method based on bidirectional voice emotion curve interaction analysis as described in any one of claims 1-7.

Images