Video customer service quality inspection method and device, program product and storage medium

Abstract

The invention discloses a video customer service quality inspection method and device, a program product and a storage medium, and relates to the technical field of customer service quality inspection. The method comprises the following steps: acquiring an audio data stream and a video data stream of a seat and a client in a video interaction process, and obtaining a text data stream according to the audio data stream; calculating first emotion probability distribution according to the audio data stream; calculating second emotion probability distribution according to the video data stream; identifying a business context in the video interaction process according to the text data stream; according to the business context, the first emotion probability distribution and the second emotion probability distribution, calculating a comprehensive emotion state of the seat in the business context; an expected emotion configuration corresponding to the business context is searched, and the matching degree between the comprehensive emotion state and the expected emotion configuration is calculated; and calculating an emotion service quality score of the seat in the video interaction process according to the matching degree. By implementing the technical scheme provided by the invention, the accuracy of video customer service quality inspection can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of customer service quality detection, and in particular to a video customer service quality detection method, device, program product and storage medium. BACKGROUND

[0002] With the rapid development of digital services, video customer service has become an important channel for enterprises to communicate with customers. Compared with traditional telephone customer service, video customer service can provide a more intuitive and personalized service experience. Customers can communicate face-to-face with agents through video calls. This interactive mode has been widely used in financial consulting, medical diagnosis, technical support and other fields.

[0003] In order to ensure the service quality of video customer service, enterprises usually establish a customer service quality detection system to monitor and evaluate the service performance of agents. In the customer service quality detection standard, the emotional performance of agents is one of the core elements of service quality scoring. Specifically, positive emotions such as enthusiasm, patience and friendliness shown by agents during the service process will be given a higher quality detection score according to fixed scoring rules, while negative emotions such as irritability, indifference and impatience will be deducted from the quality detection score according to fixed scoring rules. However, due to the large differences in emotional performance requirements in different business scenarios, the existing technology often uses fixed evaluation standards when scoring the emotional service quality, resulting in inaccurate service quality scores, which affects the accuracy of video customer service quality detection. SUMMARY

[0004] The present application provides a video customer service quality detection method, device, program product and storage medium, which can improve the accuracy of video customer service quality detection.

[0005] In a first aspect of the present application, a video customer service quality detection method is provided, which specifically comprises: real-time acquisition of audio data stream and video data stream of agents and customers during video interaction, and obtaining text data stream according to the audio data stream; extracting acoustic features from the audio data stream, and calculating a first emotional probability distribution according to the acoustic features; extracting facial features from the video data stream, and calculating a second emotional probability distribution according to the facial features; identifying the business context during video interaction according to the text data stream; calculating the comprehensive emotional state of agents in the business context according to the business context, the first emotional probability distribution and the second emotional probability distribution; finding the expected emotional configuration corresponding to the business context from a pre-set service etiquette rule library, and calculating the matching degree between the comprehensive emotional state and the expected emotional configuration; According to the matching degree, an emotional service quality score of the agent in the video interaction process is calculated.

[0006] By adopting the technical solution, the audio and video data streams in the video customer interaction process are acquired in real time and corresponding features are extracted, the emotional probability distribution is calculated from two dimensions of acoustic features and facial features respectively, the business context is identified in combination with the text data stream, and thus the emotional performance of the agent is comprehensively evaluated on the basis of considering the specific business scenario. The scheme realizes the accurate scoring of the emotional service quality of the agent by calculating the matching degree between the emotional state under the audio and video dual modalities and the expected emotional configuration corresponding to the business context. Specifically, the scheme first calculates a first emotional probability distribution and a second emotional probability distribution based on acoustic features and facial features respectively, ensuring the multidimensionality of emotional recognition; secondly, the business context is identified through the text data stream, and the corresponding expected emotional configuration is searched from the service etiquette rule library, ensuring the matching of the scoring standard and the business scenario; finally, the service quality score is calculated based on the matching degree of the comprehensive emotional state and the expected emotional configuration, improving the accuracy of video customer quality inspection.

[0007] Optionally, the calculating the first emotional probability distribution according to the acoustic features comprises: establishing an acoustic feature baseline value of the agent according to the acoustic features; comparing the current acoustic features with the acoustic feature baseline value to obtain emotional features of multiple types; counting the occurrence times and the cumulative duration of the emotional features of each type within a preset analysis period, and calculating the intensity coefficients of the emotional features of each type according to the occurrence times and the cumulative duration; constructing an emotional probability distribution vector within the analysis period according to the intensity coefficients of the emotional features of each type; performing a weighted average operation on the emotional probability distribution vectors of multiple continuous preset analysis periods to obtain the first emotional probability distribution.

[0008] By adopting the technical solution, first, the acoustic feature baseline value of the agent is established, providing a personalized reference standard for emotional feature recognition and effectively eliminating the influence of sound feature differences among different agents; secondly, the current acoustic features are compared with the baseline value in real time, so that subtle features of emotional changes can be captured; thirdly, the intensity coefficients are calculated by counting the occurrence times and the cumulative duration of the emotional features of each type within the preset analysis period, not only considering the frequency of the emotional features, but also focusing on their persistence, so that the emotional evaluation is more comprehensive; finally, the weighted average operation is performed on the emotional probability distribution vectors of multiple continuous analysis periods, not only ensuring the real-time nature of the emotional evaluation, but also avoiding the interference of instantaneous fluctuations, so that a more stable and reliable first emotional probability distribution is obtained.

[0009] Optionally, the step of extracting facial features from the video data stream and calculating a second emotion probability distribution based on the facial features includes: Face detection is performed on the video data stream to determine the facial region of the seat, and key facial feature points are extracted from the facial region. Based on the key facial feature points, the geometric change parameters of the seat's facial expression are calculated. The predicted probability value of each expression category of the agent is calculated based on the geometric change parameters, and the maximum value among the predicted probability values ​​is taken as the confidence level of each expression category. The probability weights of each expression category are calculated based on the confidence level and duration of each expression category, and the probability weights are normalized to obtain the second emotion probability distribution.

[0010] By employing the aforementioned technical solution, firstly, facial expression changes of agents are transformed into quantifiable geometric change parameters through facial detection and key feature point extraction, providing an objective data foundation for expression recognition. Secondly, based on the geometric change parameters, predicted probability values ​​for each expression category are calculated and confidence levels are determined, ensuring the reliability of the expression recognition results. Finally, probability weights are calculated by comprehensively considering the confidence level and duration of each expression category, and normalization is performed to obtain a second emotion probability distribution, reflecting both differences in expression intensity and the impact of expression duration. This multidimensional expression analysis method based on facial geometric features not only improves the accuracy of facial expression recognition but also achieves a more objective and comprehensive assessment of agents' emotional states through dynamic calculation of probability weights, providing reliable data support for subsequent emotion service quality scoring.

[0011] Optionally, identifying the business context of the current interaction process based on the text data stream includes: The text data stream is segmented and keywords are extracted to obtain a set of business keywords; The set of business keywords is matched with a preset business context dictionary to identify the business type identifier; The current interaction stage is determined based on the semantic features in the text data stream, and the business context of the current interaction process is determined based on the business type identifier and the current interaction stage.

[0012] By employing the aforementioned technical solution, a set of business keywords is obtained through word segmentation and keyword extraction of the text data stream. This set is then matched with a pre-defined business context dictionary to identify the business type identifier. Combined with semantic features in the text data stream, the current interaction stage is determined. Finally, based on the business type identifier and the current interaction stage, the business context of the interaction process is accurately determined. This text analysis-based context recognition method transforms unstructured dialogue content into analyzable keyword form, achieves precise business type positioning through dictionary matching, and dynamically tracks the interaction stage by combining semantic feature analysis. This improves processing efficiency while ensuring recognition accuracy, providing a reliable scenario basis for subsequent business context-based emotion service quality assessment.

[0013] Optionally, calculating the agent's overall emotional state in the business context based on the business context, the first emotional probability distribution, and the second emotional probability distribution includes: Calculate the cosine similarity between the first emotion probability distribution and the second emotion probability distribution to obtain the audio-visual emotion consistency index; Based on the business context, the expected value of emotional stability is determined, and the entropy values ​​of the first emotional probability distribution and the second emotional probability distribution are calculated respectively to obtain the audio emotional entropy and the video emotional entropy. Based on the audio-visual emotion consistency index, the audio emotion entropy, and the video emotion entropy, a comprehensive emotion state is calculated using a preset comprehensive emotion calculation formula.

[0014] By employing the aforementioned technical solution, the audio-video emotion consistency index is obtained by calculating the cosine similarity between the first and second emotion probability distributions, effectively measuring the degree of coordination between emotional expressions in both audio and video dimensions. Simultaneously, based on the business context, the expected value of emotional stability is determined, and the audio and video emotion entropies are obtained by calculating the entropy values ​​of the first and second emotion probability distributions, thereby measuring the fluctuation of the agent's emotional expression under different modalities. Finally, the audio-video emotion consistency index, audio emotion entropy, and video emotion entropy are integrated through a pre-defined comprehensive emotion calculation formula, achieving a multi-dimensional comprehensive assessment of the agent's emotional state. This ensures the objectivity of the assessment results while reflecting the differentiated requirements for emotional expression under different business contexts.

[0015] Optionally, the step of calculating the comprehensive emotional state based on the audio-visual emotional consistency index, the audio emotional entropy, and the video emotional entropy using a preset comprehensive emotional calculation formula includes: The comprehensive emotion calculation formula is as follows: ; in, The term refers to the overall emotional state, and S is the audio-visual emotion consistency index. This represents the probability distribution of the first emotion. This represents the probability distribution of the second emotion. This represents the Hadamard product operation. The audio emotion entropy, The video emotion entropy, To prevent extremely small positive numbers with a denominator of zero, D is the KL divergence between the Hadamard product of the first and second emotion probability distributions and the expected value of emotion stability. This is a smoothing factor.

[0016] By employing the aforementioned technical solution, the audio-video emotion consistency index is multiplied by the Hadamard product of the first and second emotion probability distributions, effectively measuring the consistency of emotional expression across the two dimensions of audio and video. By using the audio emotion entropy (audio), video emotion entropy (video), and a minimal positive number ϵ as the denominator, the influencing factor of emotional volatility is introduced, while avoiding the case where the denominator is zero during calculation. Finally, by introducing the KL divergence D between the current emotion probability distribution and the expected value of emotional stability, as well as a smoothing factor δ, a quantitative assessment of emotional stability is achieved. This calculation formula not only comprehensively considers multiple dimensions such as the consistency, volatility, and stability of emotional expression, but also ensures the reliability and interpretability of the calculation results through reasonable mathematical operations, providing a reliable quantitative basis for evaluating customer service quality.

[0017] Optionally, after calculating the emotional service quality score of the interaction process, the method further includes: When the emotional service quality score is lower than the preset quality threshold, a quality inspection anomaly alert is generated, and the time node corresponding to the quality inspection anomaly alert is marked. The emotional service quality scores are statistically analyzed according to each time point to generate a report on the trend of agent emotional performance. Based on the overall emotional state and the business context, corresponding service improvement suggestions are matched from a preset improvement suggestion library; The emotional service quality score, the quality inspection anomaly alert, the trend report, and the service improvement suggestions are stored in the quality inspection database.

[0018] By adopting the above technical solutions, when the emotional service quality score falls below the preset quality threshold, the system automatically generates a quality inspection anomaly alert and records the corresponding time point, achieving real-time monitoring of service quality anomalies. Through time-dimensional statistical analysis of the emotional service quality score, a trend report on agent emotional performance is generated, helping to understand the changing patterns of agents' emotional service levels. Simultaneously, based on comprehensive emotional states and business context, service improvement suggestions are accurately matched from a preset improvement suggestion library, providing targeted improvement guidance for agents. Finally, information such as emotional service quality scores, quality inspection anomaly alerts, trend reports, and service improvement suggestions are uniformly stored in the quality inspection database, facilitating subsequent data tracking and analysis, and providing data support for continuous improvement and training of agents, thus forming a closed-loop service quality management system.

[0019] Secondly, this application provides a video customer service quality inspection device, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the video customer service quality inspection device to perform the method described in the first aspect and any possible implementation thereof.

[0020] Thirdly, this application provides a computer program product containing instructions that, when run on a video customer service quality inspection device, cause the video customer service quality inspection device to perform the method described in the first aspect and any possible implementation thereof.

[0021] Fourthly, this application provides a computer-readable storage medium including instructions that, when executed on a video customer service quality inspection device, cause the video customer service quality inspection device to perform the method described in the first aspect and any possible implementation thereof. Attached Figure Description

[0022] Figure 1 This is a system architecture diagram of a video customer service quality inspection system provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a video customer service quality inspection method provided in an embodiment of this application; Figure 3 This is a schematic diagram of an emotion probability distribution provided in an embodiment of this application; Figure 4 This is an exemplary hardware structure diagram of a video customer service quality inspection device provided in an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0024] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0025] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0026] Figure 1 An architecture for a video customer service quality inspection system is shown. For example... Figure 1 As shown, the system architecture may include a data acquisition device 011, a network 012, and an electronic device 013. The network 012 provides a data transmission link between the data acquisition device 011 and the electronic device 013. The network 012 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0027] Data acquisition device 011 can send audio and video data streams to electronic device 013 via network 012. Data acquisition device 011 is mainly responsible for collecting audio and video data during the interaction between the agent and the customer, and for preprocessing and transmitting the data in real time.

[0028] Data acquisition device 011 is hardware, which can be a terminal device with audio and video acquisition and transmission functions, including but not limited to basic components such as cameras, microphones, audio and video acquisition cards, and customer service workstations.

[0029] Electronic device 013 is responsible for receiving and comprehensively analyzing audio and video data streams, including core functions such as audio emotion feature extraction, video emotion feature extraction, emotion probability distribution calculation, business context recognition, comprehensive emotion state assessment, and quality scoring. Electronic device 013 can monitor customer service quality in real time based on a preset evaluation model, generate quality inspection anomaly alerts and improvement suggestions, and, combined with historical data, ultimately achieve dynamic evaluation of service quality. These analysis and processing results can be used to improve customer service levels.

[0030] It should be noted that electronic devices can be either hardware or software. When an electronic device is hardware, it can be implemented as a distributed cluster of multiple electronic devices or as a single electronic device. When an electronic device is software, it can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed processing) or as a single software program or software module. No specific limitations are set here.

[0031] It should be understood that Figure 1 The number of data acquisition devices 011, networks 012, and electronic devices 013 shown is merely illustrative. Depending on implementation needs, there can be any number of data acquisition devices 011, networks 012, and electronic devices 013. In particular, if audio and video data does not need to be transmitted remotely, the above system architecture may exclude network 012 and include only data acquisition devices 011 or electronic devices 013.

[0032] This application provides a video customer service quality inspection method, for reference. Figure 2 , Figure 2 This is a flowchart illustrating a video customer service quality inspection method provided in an embodiment of this application, including steps S101 to S107, as follows: S101: Real-time acquisition of audio and video data streams during video interaction between agents and customers, and text data streams derived from the audio data streams.

[0033] In this embodiment of the application, the audio data stream refers to a sequence of audio signals continuously collected during video interaction between the agent and the customer, including information such as speech, tone, speech rate, pauses, and background noise. It is usually stored in waveform files or compressed audio formats, such as WAV format audio streams with a sampling rate of 16kHz, which contain 16,000 audio data points per second.

[0034] Specifically, the system continuously monitors the call between the agent and the customer using audio and video acquisition equipment, capturing audio signals in real time and converting them into a digitized audio data stream. Simultaneously, it captures video footage and converts it into a digitized video data stream. For the audio data stream, the continuous audio signal is segmented according to fixed time windows, with each time window's audio data serving as a data unit for subsequent analysis. For the video data stream, video frame sequences are extracted according to frame rate requirements, with each frame containing complete image information for subsequent visual analysis. After acquiring the audio data stream, speech recognition technology is used to convert the audio signal into corresponding text content. The specific process includes: first, preprocessing the audio signal, including noise filtering and volume normalization; then, identifying speech features in the audio using an acoustic model, mapping the sound signal into a phoneme sequence; finally, converting the phoneme sequence into complete text sentences using a language model, thus obtaining a text data stream corresponding to the time sequence of the audio data stream.

[0035] S102: Extract acoustic features from the audio data stream and calculate the first emotion probability distribution based on the acoustic features.

[0036] In the embodiments of this application, acoustic features represent various parameters extracted from audio signals that can reflect the emotional state of speech, including features such as pitch frequency, volume intensity, speech rate variation, and speech energy distribution, such as the range of fundamental frequency variation, short-time energy, zero-crossing rate, Mel frequency cepstral coefficients, and other multi-dimensional feature vectors.

[0037] Specifically, audio signal processing technology is used to extract features from the audio data stream, obtaining acoustic feature parameters including fundamental frequency, formants, spectrograms, and energy distribution. A baseline value for the acoustic features of the agent is established, derived by analyzing the agent's speech characteristics under normal conditions, including statistical indicators such as the mean and standard deviation of each feature parameter. The acoustic features extracted at the current moment are compared with the pre-established baseline value. When a feature parameter deviates from the baseline value by more than a preset threshold, it is identified as the corresponding type of emotional feature; for example, a significant increase in fundamental frequency indicates excitement, and a significant slowdown in speech rate indicates fatigue. Within a preset analysis period, the frequency and cumulative duration of each type of emotional feature are counted. The intensity coefficient of each type of emotional feature is obtained by multiplying the frequency by its weight, adding the product of the cumulative duration by its weight, and then dividing by the total analysis period. The intensity coefficients of various emotional features are combined to form the emotional probability distribution vector for the current analysis period. The emotional probability distribution vectors of multiple consecutive analysis periods are weighted and averaged according to the time decay weight to obtain the first emotional probability distribution that reflects the trend of changes in audio emotions.

[0038] Based on the above embodiments, as an optional embodiment, S102: the step of calculating the first emotion probability distribution based on acoustic features may specifically include the following steps: S201: Establish the acoustic feature baseline value of the agent based on the acoustic features; compare the current acoustic features with the acoustic feature baseline value to obtain multiple types of emotion features.

[0039] In this embodiment of the application, the acoustic feature baseline value represents the standard value of the speech feature parameters of the agent in a normal interactive state. It is obtained by collecting historical audio data of the agent in a calm and neutral emotional state and performing statistical analysis. It includes the baseline values ​​of various acoustic parameters such as the mean fundamental frequency, the standard value of speech rate, and the average level of volume. For example, the baseline parameter set of an agent in a normal state has a mean fundamental frequency of 150 Hz and a speech rate of 180 words per minute.

[0040] Specifically, by analyzing audio samples from calls where the agent was in a calm state, various acoustic feature parameters are extracted and statistically calculated to obtain the mean and standard deviation of features including fundamental frequency, speech rate, volume, and pitch variation, thus establishing a personalized acoustic feature baseline model for the agent. During real-time interaction, the extracted acoustic feature parameters at the current moment are compared with the established baseline values ​​to calculate the degree of deviation between the current feature value and the baseline value. When the fundamental frequency deviates from the baseline value by more than a preset threshold range, it is determined to be an emotional feature of excitement or frustration based on the direction and magnitude of the deviation; when the speech rate is significantly faster or slower than the baseline value, it is identified as an emotional feature of tension or fatigue; when the volume intensity deviates from the baseline level, it is determined to be an emotional feature of excitement or depression. Through this baseline comparison method, continuous acoustic feature data is converted into discrete emotional features of multiple types, including different types of emotional feature classification results such as positive emotions, negative emotions, and neutral emotions.

[0041] S202: Statistically analyze the frequency and cumulative duration of each type of emotional characteristic within the preset analysis period, and calculate the intensity coefficient of each type of emotional characteristic based on the frequency and cumulative duration.

[0042] In this embodiment, the preset analysis period refers to a fixed time window used for emotion statistical analysis, such as every 30 seconds or every minute as an analysis period, during which the identified emotion characteristics of each type are quantitatively analyzed. The frequency of occurrence refers to the frequency at which a certain type of emotion characteristic is detected within the analysis period, and the cumulative duration refers to the total duration of existence of that type of emotion characteristic within the analysis period.

[0043] Specifically, within each preset analysis period, the system monitors and records the triggering of various emotional characteristics in real time, and counts the occurrence frequency of different emotional characteristics such as excitement, tension, and fatigue. It also records the duration of each detected emotional characteristic and accumulates the durations of multiple occurrences of the same emotional characteristic to obtain the cumulative duration of that emotional type within the current analysis period. The frequency of occurrence is normalized by dividing the number of occurrences of that emotional characteristic by the total number of occurrences of all emotional characteristics within the analysis period to obtain the frequency percentage. Then, the cumulative duration is normalized by dividing the cumulative duration of that emotional characteristic by the total duration of the analysis period to obtain the time percentage. Next, weighting factors are assigned to both the frequency percentage and the time percentage, where the frequency weight measures the importance of the frequency of emotional fluctuations, and the time weight measures the importance of the stability of emotional persistence. Finally, the product of the frequency percentage and the frequency weight is added to the product of the time percentage and the time weight to obtain the intensity coefficient of that emotional characteristic within the current analysis period.

[0044] S203: Construct an emotion probability distribution vector within the analysis period based on the intensity coefficients of each type of emotion feature; perform a weighted average operation on the emotion probability distribution vectors of multiple consecutive preset analysis periods to obtain the first emotion probability distribution.

[0045] In this embodiment of the application, the emotion probability distribution vector represents a probability distribution array formed by the intensity coefficients of various types of emotion features within a single analysis period. Each element of the vector corresponds to a probability value of an emotion type, such as a three-dimensional vector containing a probability of 0.3 for excitement, 0.4 for tension, and 0.3 for calmness.

[0046] Specifically, based on the calculated intensity coefficients of each emotion type, the intensity coefficients of all emotion types are normalized so that the sum of all intensity coefficients equals 1, forming the emotion probability distribution vector for the current analysis period. During normalization, the intensity coefficient of each emotion type is divided by the sum of the intensity coefficients of all emotion types to obtain the probability value of that emotion type in the current period. For multiple consecutive preset analysis periods, the emotion probability distribution vector corresponding to each period is obtained and arranged in chronological order to form an emotion probability distribution vector sequence. When performing a weighted average calculation on this vector sequence, a corresponding weight coefficient is assigned to the probability distribution vector of each period, with the weight of analysis periods closer to the current time being larger and the weight of analysis periods farther from the current time being smaller. During the weighted average calculation, the emotion probability distribution vector of each period is multiplied by its corresponding weight coefficient, then all weighted vectors are summed according to element position, and finally divided by the sum of all weight coefficients to obtain a first emotion probability distribution that integrates the emotion change trends of multiple consecutive analysis periods.

[0047] S103: Extract facial features from the video data stream and calculate the second emotion probability distribution based on the facial features.

[0048] In this embodiment of the application, facial features refer to key feature information within the facial area detected from the video image, including feature data for emotion recognition such as the location of facial key points and expression change parameters, such as geometric feature parameters such as eye opening and closing, corner of mouth curvature, and eyebrow height.

[0049] Specifically, each frame of the video data stream is processed using a face detection algorithm to locate and select the facial region of the agent. Pre-defined key facial feature points are extracted from this region, including the coordinates of the corners of the eyes, mouth, eyebrow tips, and nostrils. Based on the spatial relationships of these key facial feature points, geometric change parameters of facial expressions are calculated, including quantitative indicators reflecting the expression state such as the aspect ratio of the eyes, mouth, and the distance between the eyebrows and eyes. These calculated geometric change parameters are then input into a pre-trained expression recognition model for inference calculations, yielding the predicted probability values ​​for each expression category at the current moment. The maximum probability value is extracted from the predicted probability value sequence for each expression category as the confidence score for that category. The frequency and duration of each expression category within a certain time window are statistically analyzed, and the probability weight value for each expression category is obtained by multiplying its confidence score by its duration. The probability weights of all expression categories are normalized. The probability weight of each expression category is divided by the sum of the probability weights of all categories to ensure that the sum of the probability weights of each category is equal to 1, thus forming a second emotion probability distribution that represents the distribution of each emotion type in the current time period.

[0050] like Figure 3 As shown, Figure 3 This is a schematic diagram of an emotion probability distribution provided in an embodiment of this application. The horizontal dimension of the upper part of the diagram represents different emotion categories (such as happiness, calmness, anxiety, and anger), and the height of the vertical bars represents the probability value of that emotion in the corresponding modality.

[0051] The graph contains two different colored bars, representing the emotion probability distributions analyzed from different data streams: the blue bar represents the first emotion probability distribution calculated based on acoustic features from the audio data stream, and the green bar represents the second emotion probability distribution calculated based on facial features from the video data stream. The core of this approach is not to analyze a single emotion with the highest probability, but rather to comprehensively consider the complete emotion probability distribution vectors from both audio and video dimensions.

[0052] Based on the above embodiments, as an optional embodiment, S103: the step of extracting facial features from the video data stream and calculating the second emotion probability distribution based on the facial features may specifically include the following steps: S301: Perform face detection on the video data stream, determine the facial region of the agent, extract key facial feature points from the facial region, and calculate the geometric change parameters of the agent's facial expression based on the key facial feature points.

[0053] In the embodiments of this application, geometric change parameters refer to numerical indicators that quantify facial expression changes based on the spatial positional relationship between key facial feature points. These parameters are used to represent the degree of geometric deformation of facial expression states, such as the aspect ratio of the eyes, the angle of the corners of the mouth, and the distance the eyebrows are raised.

[0054] Specifically, a deep learning facial detection algorithm is used to scan and analyze each frame of the video data stream. Bounding box regression and classification confidence calculation are used to determine the rectangular region of the seated person's face, obtaining the coordinates of the upper left and lower right corners of the facial region. Within the determined facial region, a facial keypoint detection algorithm is run to extract a preset number of key facial feature points, including the two-dimensional coordinates of key feature points such as the inner and outer corners of the eyes, upper and lower eyelids, eyebrow contours, nose wings and tip, lip contours, and facial contours. Based on the extracted key facial feature point coordinates, geometric change parameters reflecting facial expression states are calculated. These include calculating the aspect ratio of the left and right eyes, the ratio of the distance between the upper and lower lips to the width of the corners of the mouth, the vertical distance between the center points of the eyebrows and eyes, the angular offset of the corners of the mouth relative to the center of the mouth, and the geometric deformation of the cheek region, forming multi-dimensional geometric change parameters characterizing the facial expression features of the current frame.

[0055] S302: Calculate the predicted probability value of each expression category of the agent based on the geometric change parameters, and take the maximum value of the predicted probability value as the confidence level of each expression category.

[0056] In this embodiment of the application, the predicted probability value refers to the probability value of the agent belonging to each emotional expression category calculated based on the geometric change parameters, which represents the degree of matching between the current facial expression state and different emotional types.

[0057] Specifically, the calculated geometric change parameter vectors are subjected to numerical analysis and classification calculations. Based on the numerical range and combination characteristics of the geometric change parameters, the matching degree of the expression category is determined through preset classification rules and judgment conditions. Specifically: the judgment conditions for a happy expression include geometric changes such as the upward movement of the corners of the mouth exceeding a preset threshold, the appearance of wrinkles at the corners of the eyes, and the lifting of cheek muscles; the judgment conditions for an angry expression include geometric changes such as lowered eyebrows, reduced distance between eyebrows, downturned corners of the mouth, and widened nostrils; the judgment conditions for a sad expression include a combination of features such as raised inner eyebrows, a downward angle of the corners of the mouth reaching a set range, and drooping eyelids; the judgment conditions for a surprised expression include geometric changes such as significantly raised eyebrows, increased eye opening, and an elliptical mouth opening; and the judgment condition for a neutral expression is that the changes in the positions of each key point are within the normal fluctuation range. Based on the matching degree between the current geometric change parameters and the judgment conditions for each expression category, the predicted probability value of each expression category for the agent at the current moment is calculated and output. Within a preset time window, the predicted probability value sequence of each expression category at each moment is continuously collected to form a time series of probability values ​​for each expression category. Extract the maximum probability value from the time series of predicted probability values ​​for each emoji category, and use this maximum value as the confidence level for the corresponding emoji category.

[0058] S303: Calculate the probability weight of each expression category based on the confidence level and duration of each expression category, and normalize each probability weight to obtain the second emotion probability distribution.

[0059] In this embodiment of the application, the duration refers to the length of time during which the predicted probability value of each expression category exceeds a preset threshold, reflecting the duration to which the agent displays a specific expression.

[0060] Specifically, the duration of each expression category is first calculated. For each expression category, the number of times its predicted probability value exceeds a set threshold (e.g., 0.5) within a preset time window is counted. This number is multiplied by the sampling interval to obtain the corresponding duration. For example, if the predicted probability value of a happy expression exceeds the threshold 15 times within a 30-second time window, and the sampling interval is 1 second, then the duration of the happy expression is 15 seconds. Next, the probability weight of each expression category is calculated using a weighted combination of confidence level and duration. The specific formula is: Probability Weight = α × Confidence Level + β × (Duration / Time Window Length), where α and β are preset weight coefficients, and α + β = 1. Confidence level reflects the intensity of the expression, and the duration percentage reflects the stability of the expression. Combining the two allows for a more comprehensive assessment of the importance of each expression category. Finally, the probability weights are normalized. The sum of the probability weights for all expression categories is calculated, and the probability weight of each expression category is divided by the sum to ensure that the sum of the probability values ​​for all expression categories equals 1, resulting in a standardized second emotion probability distribution.

[0061] S104: Identify the business context in the video interaction process based on the text data stream.

[0062] In this embodiment of the application, the business context refers to a comprehensive description of the specific business scenario and interaction steps discussed between the agent and the customer during the video interaction, including multi-dimensional information such as business type, processing stage, and nature of the problem. For example, the business context could be "insurance claims consultation stage," "bank account opening process," or "complaint handling and resolution stage," etc.

[0063] Specifically, the text data stream is first segmented and keywords are extracted. Continuous text content is divided into semantic units, identifying words with business meaning, filtering out stop words and irrelevant words, and extracting core business-related words to form a set of business keywords. Then, the extracted set of business keywords is matched one by one with a pre-defined business context dictionary. This dictionary contains standard vocabulary and synonym mappings from different business domains. Through keyword matching degree calculation and word frequency statistical analysis, the business type identifier that best matches the current dialogue content is identified. Next, the progress status of the current dialogue is analyzed based on the semantic features in the text data stream. By identifying language pattern features such as greetings, inquiries, confirmations, and closing remarks, the specific stage of the current interaction (consultation, processing, confirmation, or termination) is determined. Finally, based on the identified business type identifier and the determined current interaction stage, the business type and interaction stage are combined and matched according to pre-defined business context mapping rules to determine the complete business context of the current interaction process.

[0064] Based on the above embodiments, as an optional embodiment, S104: the step of identifying the business context in the video interaction process based on the text data stream may specifically include the following steps: S401: Perform word segmentation and keyword extraction on the text data stream to obtain a set of business keywords.

[0065] In this embodiment, the business keyword set refers to an ordered combination of core words extracted from the text data stream that are related to a specific business domain and service content. It includes word units with clear business meanings, such as business names, operational verbs, service objects, and time adverbs. For example, in an insurance business scenario, the business keyword set may include words such as "claims," ​​"policy," "beneficiary," "audit," and "payment."

[0066] Specifically, the input text data stream is first segmented into sentences according to preset word segmentation rules, breaking down continuous character sequences into independent lexical units, identifying lexical boundaries, and processing punctuation and special characters. Then, the segmentation results are subjected to part-of-speech tagging and semantic analysis, filtering out words with practical meaning such as nouns, verbs, and adjectives, while removing functional words such as auxiliary words, conjunctions, and modal particles. Next, the frequency statistics and importance weight of each word are calculated, and the relative importance of words in the current text is determined through TF-IDF value calculation, setting an importance threshold to filter low-frequency and irrelevant words. Simultaneously, matching and verification are performed against a preset business dictionary, retaining professional terms and key expressions relevant to the business domain. Finally, the selected high-weight business-related words are sorted and organized according to their importance, forming a set of business keywords containing word content and weight information.

[0067] S402: Match the set of business keywords with the preset business context dictionary to identify the business type identifier.

[0068] In this embodiment, the business type identifier refers to the specific business category code of the current interaction process determined through dictionary matching analysis, used to indicate the main business categories and service types involved in the interaction between the agent and the customer. For example, the business type identifier can be "INS_CLAIM" to represent insurance claims, "BANK_ACCOUNT" to represent bank account opening, or "LOAN_APPLY" to represent loan application, etc.

[0069] Specifically, firstly, a pre-defined business context dictionary is loaded. This dictionary contains multiple business type categories and their corresponding feature word mappings. Each business type corresponds to a set of standard keywords and weight configurations. Then, word elements from the business keyword set are extracted one by one and matched with the feature words of each business type in the business context dictionary using string and semantic matching. The number of successful matches and the corresponding weight value for each word are recorded. Next, the matching score for each business type is calculated using a weighted summation method: Matching Score = ∑(Matching Word Weight × Word Importance Coefficient), where the matching word weight is obtained from the business context dictionary, and the word importance coefficient is extracted from the business keyword set. Then, the matching scores of all business types are sorted and compared, and the business type with the highest score is selected as the candidate result. Simultaneously, it is verified whether this score exceeds a pre-defined recognition threshold. Finally, the identifier code corresponding to the business type with the highest score exceeding the threshold is output as the final business type identifier.

[0070] S403: Determine the current interaction stage based on the semantic features in the text data stream, and determine the business context of the current interaction process based on the business type identifier and the current interaction stage.

[0071] In this embodiment, the current interaction stage refers to the specific stage and processing status of the dialogue between the agent and the customer during the video interaction process, and is used to represent the time node and operation step in the entire service process. For example, the current interaction stage can be "problem consultation stage", "information verification stage", "business processing stage", "result confirmation stage" or "service end stage", etc.

[0072] Specifically, the process begins by extracting semantic features from the text data stream. Natural language processing (NLP) techniques are used to identify linguistic features such as sentence structure, tone vocabulary, tense changes, and discourse shifts, analyzing the distribution patterns of interrogative, declarative, and imperative sentences in the dialogue. Then, based on a pre-defined interaction stage identification model, the semantic features are classified. This model includes typical language patterns and feature vocabularies for different interaction stages. Pattern matching and feature weight calculation determine the interaction stage type that best matches the current dialogue content. Next, the identified business type identifier is combined and mapped with the current interaction stage by querying a pre-defined business context mapping table. This table defines the specific business context identifiers corresponding to different business types at each interaction stage. Finally, the business context matching degree is calculated according to the mapping rules, using the formula: Business Context Matching Degree = Business Type Confidence × Interaction Stage Confidence × Contextual Relationship Coefficient. The combination with the highest matching degree is selected as the business context of the current interaction process.

[0073] S105: Calculate the agent's overall emotional state in the business context based on the business context, the first emotional probability distribution, and the second emotional probability distribution.

[0074] In this embodiment, the comprehensive emotional state refers to the comprehensive evaluation result of the agent's emotions calculated by fusing audio voice information and video facial expression information. It is used to represent the agent's true emotional tendency and stability in a specific business context. For example, the comprehensive emotional state can be described as "happy_stable", "anxious_fluctuating", "calm_consistent", or "tense_unstable".

[0075] Specifically, firstly, the cosine similarity between the first and second emotion probability distributions is calculated. The cosine of the angle between the two emotion probability vectors is measured by dividing the vector dot product by the product of their magnitudes, yielding the audio-video emotion consistency index. This index reflects the degree of matching between spoken and facial emotions. Next, a pre-defined emotion stability expectation mapping table is queried based on the business context to obtain the standard emotion fluctuation range for the current business scenario. The information entropy of the first and second emotion probability distributions is calculated separately. A logarithmic weighted summation method is used to quantify the uncertainty of the emotion distribution, resulting in audio and video emotion entropy, which reflect the complexity and concentration of the emotion distribution. Finally, based on the audio-video emotion consistency index, audio emotion entropy, and video emotion entropy, a pre-defined comprehensive emotion calculation model is used for weighted fusion. The consistency index, the reciprocal of the normalized emotion entropy, and the business context weights are linearly combined according to pre-defined weighting coefficients to calculate the comprehensive emotion state.

[0076] Based on the above embodiments, as an optional embodiment, S105: the step of calculating the comprehensive emotional state of the agent in the business context according to the business context, the first emotion probability distribution, and the second emotion probability distribution may specifically include the following steps: S501: Calculate the cosine similarity between the first emotion probability distribution and the second emotion probability distribution to obtain the audio-visual emotion consistency index.

[0077] In this embodiment, the audio-visual emotion consistency index is a quantitative indicator obtained by calculating the similarity between the probability distribution of audio speech emotion and the probability distribution of video facial emotion. It is used to assess whether the emotions expressed by the agent through different perceptual channels at the same time are consistent. The index ranges from 0 to 1, with a value closer to 1 indicating greater consistency in audio and video emotion expression, and a value closer to 0 indicating significant differences in emotion expression.

[0078] Specifically, firstly, the first and second emotion probability distributions are constructed as multi-dimensional probability vectors, with each dimension corresponding to a specific emotion category and its corresponding probability value, ensuring that the two vectors have the same dimensional structure and emotion category mapping relationship. Then, the cosine similarity algorithm is used to calculate the cosine value of the angle between the two emotion probability vectors. The cumulative correlation of the two vectors across each emotion dimension is obtained through vector dot product operation, and then normalized by dividing by the product of their respective vector magnitudes to eliminate the influence of vector magnitude differences. Next, the calculation results are validated for numerical range and outlier handling. When an emotion probability vector is zero, a default minimum consistency threshold is set to avoid undefined results. Finally, the normalized cosine similarity value is used as the audio-visual emotion consistency index.

[0079] S502: Determine the expected value of emotional stability based on the business context, calculate the entropy values ​​of the first emotional probability distribution and the second emotional probability distribution respectively, and obtain the audio emotional entropy and the video emotional entropy.

[0080] In this embodiment, the expected value of emotional stability refers to a quantitative standard for the ideal stability that the distribution of agent emotions should achieve in a specific business context, used to judge whether the current emotional state meets the expected requirements of the business scenario. Audio emotion entropy and video emotion entropy represent the degree of uncertainty of the probability distribution of voice emotion and the probability distribution of facial emotion, respectively. The higher the entropy value, the more dispersed the emotion distribution and the more unstable the emotional state; the lower the entropy value, the more concentrated the emotion distribution and the clearer the emotion expression.

[0081] Specifically, firstly, a pre-defined emotional stability expectation value configuration table is queried based on the business context type. This configuration table pre-establishes a mapping relationship between different business scenarios and expected emotional stability levels. For example, customer complaint handling scenarios correspond to lower emotional stability expectation values, routine consultation service scenarios correspond to medium stability expectation values, and important customer maintenance scenarios correspond to higher stability expectation values. Then, information entropy is calculated for the first and second emotional probability distributions respectively. A negative weighted summation of probability logarithms is used to quantify the degree of disorder in the emotional distribution. This is achieved by iterating through the probability values ​​of each emotional category, calculating the product of each probability value and its logarithm, and summing the results. Next, the calculated raw entropy values ​​are normalized, mapping them to a standardized numerical range to obtain the audio emotional entropy and video emotional entropy.

[0082] S503: Based on the audio-visual emotion consistency index, audio emotion entropy, and video emotion entropy, the comprehensive emotion state is calculated using a preset comprehensive emotion calculation formula.

[0083] Specifically, the audio-visual emotion consistency index, audio emotion entropy, and video emotion entropy are substituted into a preset comprehensive emotion calculation formula to obtain the comprehensive emotion state. The preset comprehensive emotion calculation formula is as follows: The formula employs a three-layer nested structure. The first layer uses the audio-visual emotion consistency index S as a global weighting factor to regulate overall credibility. When the audio-visual emotion expression is highly consistent, this weight approaches 1, making the overall emotional state more credible. When there is obvious conflict, this weight is smaller, thus reducing the assessment confidence. The core part of the second layer uses the Hadamard product operation. This method achieves deep fusion of two emotion probability distributions by multiplying corresponding elements of the first and second emotion probability distributions one by one. This highlights the emotional components that are commonly reinforced in both distributions while suppressing abnormal fluctuations in a single channel. (The denominator part...) The third layer of stability compensation factor performs entropy normalization adjustment. The sum of audio and video emotion entropy reflects the dispersion of the overall emotion distribution. The larger the entropy value, the more unstable the emotion, which inhibits the comprehensive evaluation. The minimum positive number ϵ is used to prevent mathematical anomalies when both entropy values ​​are zero simultaneously. In this context, D represents the KL divergence between the fused sentiment probability distribution and the expected value of sentiment stability in the business context, where the fused sentiment probability distribution is... The comprehensive probability distribution obtained after normalizing the Hadamard product is represented by the KL divergence, which measures the degree to which the fused sentiment distribution deviates from the expected distribution. A larger D value indicates a more severe deviation. A smoothing factor δ avoids computational anomalies when the KL divergence is zero. This logarithmic structure ensures that the compensation factor is larger when the sentiment distribution is close to the expected value and smaller when it is far from the expected value, realizing a dynamic adjustment mechanism based on business context. Finally, the comprehensive sentiment state is calculated. .

[0084] S106: Search for the expected emotional configuration corresponding to the business context from the preset service etiquette rule base, and calculate the matching degree between the comprehensive emotional state and the expected emotional configuration.

[0085] In this embodiment, the service etiquette rule base is a pre-built knowledge base containing standard emotional expression requirements for different business scenarios. Based on service standards and best practices in the customer service industry, this rule base establishes corresponding expected emotional configuration templates for various business contexts. The expected emotional configurations are stored in the form of multi-dimensional emotional vectors, including ideal distribution ratios of various emotions such as positive, negative, and neutral, while also considering the differentiated requirements for emotional expression in different business scenarios.

[0086] Specifically, the system first performs a precise matching query in the service etiquette rule base based on the type identifier of the current business context to obtain the expected emotional configuration vector corresponding to that context. This configuration vector defines the ideal emotional distribution state that the agent should exhibit in a specific business scenario. For example, in a customer consultation scenario, the expected behavior is high patience and moderate enthusiasm; in a customer complaint handling scenario, the expected behavior is high empathy and stable emotional control; and in a product recommendation scenario, the expected behavior is moderate enthusiasm and professionalism. Then, a vector similarity algorithm is used to calculate the degree of matching between the overall emotional state and the expected emotional configuration, quantifying the closeness of the two emotional vectors in multidimensional space through metrics such as cosine similarity or Euclidean distance. The system also considers the importance weights of different emotional dimensions, assigning higher matching weights to key emotional indicators to ensure that core service requirements receive priority attention. Through the above calculation process, the final matching degree between the overall emotional state and the expected emotional configuration is obtained.

[0087] S107: Calculate the agent's emotional service quality score during video interaction based on the matching degree.

[0088] In this embodiment of the application, the emotional service quality score refers to a quantitative evaluation index based on the degree to which the actual emotional performance of the agent meets the service standard requirements. It is used to measure the agent's emotional management and service performance level during video interaction. For example, the score range can be set to 0-100 points, where 90 points or above indicates excellent emotional service quality, 70-89 points indicates a good level, 60-69 points indicates a qualified level, and below 60 points indicates service quality that needs improvement.

[0089] Specifically, based on the matching score obtained in the previous step, a pre-defined scoring conversion algorithm maps the matching score to a standardized emotional service quality score. First, the matching score is normalized, adjusting its value range to a uniform interval of zero to one. Then, a linear transformation is used to proportionally expand the normalized matching score into a percentage score. Next, a piecewise function strategy is applied to refine the score, using corresponding calculation weights based on the different numerical intervals of the matching score. A steeper score growth curve is used for high matching scores to reflect excellent performance, a moderate linear growth pattern is used for medium matching scores, and a relatively flat score distribution is used for low matching scores. The influence of time is also introduced; for scenarios with longer interaction durations, an exponential decay model is used to correct the base score, with the decay rate negatively correlated with the interaction duration. Finally, the emotional service quality score of the agent during the video interaction is output.

[0090] Based on the above embodiments, as an optional embodiment, S107: after the step of calculating the agent's emotional service quality score during video interaction based on the matching degree, a step of improving the service is also included, which may specifically include the following steps: S701: When the emotional service quality score is lower than the preset quality threshold, generate a quality inspection anomaly reminder and mark the time node corresponding to the quality inspection anomaly reminder.

[0091] In this embodiment, the quality inspection anomaly alert refers to an alarm notification mechanism automatically triggered by the system when the agent's emotional service quality score fails to meet the preset standard. It is used to indicate that there is a deviation in service quality that requires manual intervention or improvement. For example, when an agent shows impatience while handling customer complaints, resulting in a score below 70, the system will generate a quality inspection anomaly alert of the type "inappropriate emotional expression". Or when an agent lacks enthusiasm during product recommendation, resulting in a score below 75, the system will generate a quality inspection anomaly alert of the type "insufficient service enthusiasm".

[0092] Specifically, the system first compares the calculated emotion service quality score with a pre-set quality threshold. This threshold is configured differently based on various business scenarios and service level requirements, typically set between 60 and 80 points. When the emotion service quality score is detected to be lower than the corresponding threshold, the system immediately initiates anomaly detection. Based on the degree of score deviation and the specific emotion dimension, it automatically matches the appropriate anomaly type template to generate detailed quality inspection anomaly alerts. Simultaneously, the system obtains the real-time timestamp of the current video interaction, precisely marking and recording the exact moment the anomaly occurred.

[0093] S702: Statistically analyze the emotional service quality scores according to each time point to generate a report on the trend of agent emotional performance.

[0094] In this embodiment of the application, the agent's emotional performance trend report refers to a visualized statistical document formed by comprehensively analyzing the emotional service quality score data at different time points. It is used to represent a detailed record of the agent's emotional change patterns and service performance fluctuations throughout the entire video interaction process. For example, the report can show the complete trajectory of the agent's emotional positivity at the beginning of the interaction with a score of 85, the decline in emotional stability in the middle due to handling complex issues with a score of 72, and the recovery to a good state with a score of 80 after adjustments. Alternatively, it can show the differences in the agent's emotional adaptability in different types of customer interactions.

[0095] Specifically, the system first collects emotional service quality score data for each time point during the entire video interaction process, arranging and organizing it chronologically. Then, the timeline is divided into several statistical intervals, which can be divided at fixed intervals or dynamically segmented based on important nodes in the interaction event. Next, multi-dimensional statistical calculations are performed on the score data within each time interval, including key indicators such as the average score, highest score, lowest score, and score variance. The system further analyzes the magnitude and direction of score changes between adjacent time intervals, identifying periods where scores significantly increase, decrease, or remain stable. Simultaneously, by combining the marking information of abnormal time nodes, the system performs correlation analysis between quality inspection anomaly alerts and score fluctuations in the corresponding time periods. Finally, the system integrates all statistical analysis results into a structured data format, generating a report on the agent's emotional performance trend that includes a timeline display, score curve graphs, statistical data tables, and anomaly event annotations.

[0096] S703: Based on the overall emotional state and business context, match the corresponding service improvement suggestions from the preset improvement suggestion library; store the emotional service quality score, quality inspection anomaly reminder, trend report and service improvement suggestions in the quality inspection database.

[0097] In this application embodiment, service improvement suggestions refer to targeted optimization guidance schemes provided based on the specific emotional performance issues of agents and the characteristics of business scenarios. These schemes represent specific operational suggestions and training content to help agents improve their emotional management capabilities and enhance service quality. For example, when an agent is detected to be showing anxiety during customer complaint handling, the system will match improvement suggestions such as "maintaining a steady tone, using empathetic listening skills, and appropriately using soothing language." Or, when an agent is found to lack enthusiasm during product introductions, the system will provide targeted guidance schemes such as "enhancing the intonation of the voice, enriching facial expressions, and highlighting the advantages and features of the product."

[0098] Specifically, the system first acquires comprehensive emotional state information identified during the current interaction, including key features such as specific emotion type, intensity, and duration. Simultaneously, it extracts corresponding business context information, covering contextual elements such as the business type, customer characteristics, processing stage, and specific scenario of the current interaction. Then, the system accesses a pre-built improvement suggestion library, which is structured and categorized according to different emotional state categories and business scenario dimensions. Next, using a multi-condition matching algorithm, the system retrieves the most suitable improvement suggestion from the library, using the current comprehensive emotional state features and business context information as query conditions. The matching process considers the priority weight of emotional states and the applicability of the business scenario. After completing the improvement suggestion matching, the system uniformly organizes the emotional service quality score data, quality inspection anomaly alert records, agent emotional performance trend reports, and the matched service improvement suggestions generated during this interaction, and writes them into the corresponding data tables in the quality inspection database according to a predetermined data structure format.

[0099] The following describes an exemplary video customer service quality inspection device provided in an embodiment of this application. Figure 4 This is an exemplary hardware structure diagram of a video customer service quality inspection device provided in an embodiment of this application.

[0100] In some embodiments, the video customer service quality inspection device is a computer device or includes a computer device. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device stores data. The network interface of the computer device is used to communicate with other external terminals or servers via a network connection. In some embodiments, the network interface can be a wired network interface; in some embodiments, the network interface can also be a wireless network interface. When the computer program is executed by the processor, it implements the methods in the embodiments of this application.

[0101] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0102] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0103] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0104] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0105] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for video customer service quality inspection, characterized in that, The method comprises: real-time acquisition of audio data stream and video data stream of the agent and the customer in the video interaction process, and obtaining text data stream according to the audio data stream; extracting acoustic features in the audio data stream, and calculating a first emotion probability distribution according to the acoustic features; extracting facial features in the video data stream, and calculating a second emotion probability distribution according to the facial features; identifying a business context in the video interaction process according to the text data stream; calculating a comprehensive emotion state of the agent in the business context according to the business context, the first emotion probability distribution and the second emotion probability distribution; finding a corresponding expected emotion configuration of the business context from a preset service etiquette rule library, and calculating a matching degree between the comprehensive emotion state and the expected emotion configuration; calculating an emotion service quality score of the agent in the video interaction process according to the matching degree.

2. The method of claim 1, wherein, The method comprises: establishing an acoustic feature baseline value of the agent according to the acoustic features; comparing the current acoustic features with the acoustic feature baseline value to obtain emotion features of multiple types; statistically counting the occurrence times and cumulative duration of the emotion features of each type within a preset analysis period, and calculating the intensity coefficients of the emotion features of each type according to the occurrence times and the cumulative duration; constructing an emotion probability distribution vector within the analysis period according to the intensity coefficients of the emotion features of each type; performing weighted average operation on the emotion probability distribution vectors of multiple continuous preset analysis periods to obtain a first emotion probability distribution.

3. The method of claim 1, wherein, The method comprises: performing face detection on the video data stream to determine the face area of the agent, and extracting key facial feature points from the face area to calculate the geometric change parameters of the facial expressions of the agent based on the key facial feature points; calculating the prediction probability values of each expression category of the agent according to the geometric change parameters, and taking the maximum value in the prediction probability values as the confidence of each expression category; calculating the probability weights of each expression category according to the confidence of each expression category and the duration, and performing normalization processing on each probability weight to obtain a second emotion probability distribution.

4. The method of claim 1, wherein, The method comprises: performing word segmentation processing and keyword extraction on the text data stream to obtain a business keyword set; matching the business keyword set with a preset business context dictionary to identify a business type identifier; determining a current interaction stage based on the semantic features in the text data stream, and determining the business context of the current interaction process according to the business type identifier and the current interaction stage.

5. The method of claim 1, wherein, The method comprises: calculate a cosine similarity between the first emotion probability distribution and the second emotion probability distribution to obtain an audio-video emotion consistency index; determine an emotion stability expectation value according to the business context, and calculate an audio emotion entropy and a video emotion entropy according to entropies of the first emotion probability distribution and the second emotion probability distribution respectively; calculate a comprehensive emotion state according to the audio-video emotion consistency index, the audio emotion entropy, and the video emotion entropy through a preset comprehensive emotion calculation formula.

6. The method of claim 5, wherein, The comprehensive emotion calculation formula comprises: The comprehensive emotion calculation formula comprises: ； wherein, S is the audio-video emotional consistency index, is the first emotional probability distribution, is the second emotional probability distribution, denotes the Hadamard product operation, is the audio emotional entropy, is the video emotional entropy, is a small positive number to prevent the denominator from being zero, D is the KL divergence between the Hadamard product of the first emotional probability distribution and the second emotional probability distribution and the emotional stability expectation value, is a smoothing factor.

7. The method of claim 1, wherein, After the emotion service quality score of the agent in the video interaction process is calculated according to the matching degree, the method further comprises: when the emotion service quality score is lower than a preset quality threshold, generating a quality inspection abnormality reminder and marking a time node corresponding to the quality inspection abnormality reminder; statistically analyzing the emotion service quality score according to each time node to generate an agent emotion performance trend report; based on the comprehensive emotion state and the business context, matching a corresponding service improvement suggestion from a preset improvement suggestion library; storing the emotion service quality score, the quality inspection abnormality reminder, the trend report, and the service improvement suggestion into a quality inspection database.

8. A video customer care quality inspection device, characterized by, The video customer service quality inspection device comprises one or more processors and a memory; the memory is coupled with the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors invoke the computer instructions to enable the video customer service quality inspection device to perform the method according to any one of claims 1-7.

9. A computer program product comprising instructions, characterized in that, When the computer program product runs on the video customer service quality inspection device, the video customer service quality inspection device performs the method according to any one of claims 1-7.

10. A computer-readable storage medium comprising instructions, characterized in that, When the instructions run on the video customer service quality inspection device, the video customer service quality inspection device performs the method according to any one of claims 1-7.

Images