Service strategy adjustment method and device based on emotion recognition, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a service strategy adjustment method based on emotion recognition, which comprises the following steps: acquiring voice data in a service session, performing segmentation processing, extracting feature data, inputting the feature data into an emotion classification model to recognize emotion tags, and generating an emotion tag sequence; and further generating an emotional fluctuation graph, analyzing a service anomaly probability, generating a risk early warning signal when the service anomaly probability reaches a preset threshold, and adjusting a dialogue strategy of the service session based on the risk early warning signal. The emotion label is identified through the emotion classification model, and the service abnormity probability is analyzed in combination with the emotion fluctuation graph, so that the risk early warning signal is generated when the service abnormity probability reaches the preset threshold value, the service strategy is adjusted in time, real-time monitoring and service dynamic adjustment of the customer emotion state are realized, the customer satisfaction can be effectively improved, and the customer experience is improved. And economic loss and reputation loss caused by complaints are reduced.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a service strategy adjustment method, device, equipment and storage medium based on emotion recognition. Background Art

[0002] Artificial intelligence technology has been widely used in the insurance field, especially in the claims process, where the application of intelligent claims systems has gradually become popular. The current intelligent claims system is mainly used in small claims scenarios such as auto insurance and medical insurance. Through automated process processing, it achieves rapid case closure and efficient claims, effectively reducing claims time and labor costs. However, although the existing intelligent claims system has made significant progress in claims efficiency, there are still some shortcomings that have not been effectively resolved.

[0003] Existing intelligent claims technology mainly focuses on the physical and financial aspects of claims, such as loss assessment, calculation of claims amount and automated processing of claims process, but ignores the emotional changes and psychological state of the insured during the claims process. During the claims process, due to information asymmetry, poor communication or claims results that are not as expected, the insured is prone to negative emotions or complaints, and these emotional fluctuations often lead to higher complaint risks for insurance companies. However, the existing intelligent claims system lacks real-time monitoring and analysis of the insured's emotional state, cannot accurately identify the insured's emotional fluctuations, and lacks a corresponding early warning mechanism to indicate potential complaint risks.

[0004] The lack of existing technologies for emotion monitoring and early warning can easily lead to insurance institutions ignoring the psychological changes of policyholders during the claims process and failing to take timely adjustment measures, which may result in a decline in customer satisfaction and an increase in complaints, leading to fines, penalties, loss of reputation and other problems. Summary of the invention

[0005] The main purpose of the present invention is to provide a service strategy adjustment method, device, equipment and storage medium based on emotion recognition, aiming to solve the technical problem that the existing technology lacks means to monitor and analyze the emotional state of customers during the service process, and is unable to timely identify potential complaint risks and adjust service strategies.

[0006] To achieve the above object, the present invention provides a service strategy adjustment method based on emotion recognition, comprising:

[0007] Collect voice data from service sessions;

[0008] Segmenting the voice data according to a preset time length to obtain a plurality of voice segments;

[0009] Extract features from each speech segment to obtain feature data of each speech segment;

[0010] Inputting feature data of each speech segment into a sentiment classification model, and identifying a sentiment label of each speech segment through the sentiment classification model;

[0011] Combining the emotion labels of each speech segment to generate an emotion label sequence of the speech data, and generating an emotion fluctuation graph according to the emotion label sequence;

[0012] Analyzing the probability of service abnormality based on the sentiment fluctuation graph;

[0013] When the service abnormality probability reaches a preset threshold, a risk warning signal is generated;

[0014] The service strategy of the service session is adjusted based on the risk warning signal.

[0015] Furthermore, to achieve the above-mentioned purpose, the present invention provides a service strategy adjustment device based on emotion recognition, comprising:

[0016] A voice collection module, used to collect voice data in service sessions;

[0017] A speech segmentation module, used for segmenting the speech data according to a preset time length to obtain a plurality of speech segments;

[0018] A feature extraction module is used to extract features from each speech segment to obtain feature data of each speech segment;

[0019] The emotion classification module is used to input the feature data of each speech segment into the emotion classification model, and identify the emotion label of each speech segment through the emotion classification model;

[0020] The emotion analysis module is used to combine the emotion labels of each speech segment to generate an emotion label sequence of the speech data, and generate an emotion fluctuation graph according to the emotion label sequence;

[0021] A service anomaly analysis module, used to analyze the probability of service anomaly based on the sentiment fluctuation graph;

[0022] An early warning generation module is used to generate a risk early warning signal when the service abnormality probability reaches a preset threshold;

[0023] A service policy adjustment module is used to adjust the service policy of the service session based on the risk warning signal.

[0024] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a service strategy adjustment program based on emotion recognition stored in the memory and executable on the processor, and the service strategy adjustment program based on emotion recognition, when executed by the processor, implements the steps of the service strategy adjustment method based on emotion recognition as described above.

[0025] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which is stored a service strategy adjustment program based on emotion recognition, and when the service strategy adjustment program based on emotion recognition is executed by a processor, the steps of the service strategy adjustment method based on emotion recognition as described above are implemented.

[0026] Beneficial effects: The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a service strategy adjustment method based on emotion recognition, which collects voice data in service conversations, performs segmentation processing and extracts feature data, inputs the feature data into an emotion classification model to identify emotion tags, and generates an emotion tag sequence; further generates an emotion fluctuation graph, analyzes the service abnormality probability, and generates a risk warning signal when the service abnormality probability reaches a preset threshold, and adjusts the dialogue strategy of the service conversation based on the risk warning signal. The present invention collects and analyzes voice data in service conversations, identifies emotion tags based on an emotion classification model, and analyzes the service abnormality probability in combination with an emotion fluctuation graph, thereby generating a risk warning signal when the service abnormality probability reaches a preset threshold, and adjusting the service strategy in a timely manner, thereby realizing real-time monitoring of the customer's emotional state and dynamic adjustment of services, which can effectively reduce the risk of complaints, improve customer satisfaction, and reduce economic losses and reputation losses caused by complaints. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0028] Figure 1 A schematic diagram of an application environment of a service strategy adjustment method based on emotion recognition in an embodiment of the present invention;

[0029] Figure 2 A flow chart of an embodiment of a method for adjusting a service strategy based on emotion recognition according to the present invention;

[0030] Figure 3 This is a functional module diagram of a preferred embodiment of the service strategy adjustment device based on emotion recognition of the present invention;

[0031] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0032] Figure 5FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0033] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0034] The service strategy adjustment method based on emotion recognition provided by the embodiment of the present invention can be applied in Figure 1 In the application environment, the user terminal communicates with the server terminal through the network. The server terminal can collect voice data in the service session through the user terminal, perform segmentation processing and extract feature data, input the feature data into the emotion classification model to identify the emotion tag, and generate an emotion tag sequence; further generate an emotion fluctuation graph, analyze the service abnormality probability, and when the service abnormality probability reaches a preset threshold, generate a risk warning signal, and adjust the dialogue strategy of the service session based on the risk warning signal. The present invention collects and analyzes voice data in the service session, identifies the emotion tag based on the emotion classification model, and analyzes the service abnormality probability in combination with the emotion fluctuation graph, thereby generating a risk warning signal when the service abnormality probability reaches a preset threshold, and adjusting the service strategy in time, realizing real-time monitoring of the customer's emotional state and dynamic adjustment of the service, which can effectively reduce the risk of complaints, improve customer satisfaction, and reduce economic losses and reputation losses caused by complaints. Among them, the user terminal can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0035] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a service strategy adjustment method based on emotion recognition provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0036] like Figure 2 As shown, the service strategy adjustment method based on emotion recognition proposed by the present invention includes the following steps:

[0037] S10, collecting voice data in the service session;

[0038] In this embodiment, during the service session, in order to accurately obtain the voice conversation between the insured and the service personnel, the system needs to configure a voice acquisition device or a related software module. The voice acquisition device can be a recording function built into the telephone system, an external high-quality microphone, or an audio acquisition module in a network communication scenario. These devices need to have a high sampling rate (for example, 16kHz or higher) and a high bit depth (usually 16 bits) to ensure the quality of the audio data. The parameters of the recording function or the audio acquisition module can be set in a way that is integrated with the telephone system or network communication protocol to collect audio signals in the service session. In specific implementation, the existing voice acquisition API interface or hardware driver can be used to complete data acquisition. At the same time, according to the requirements of the acquisition scenario, noise reduction microphone equipment that meets industry standards can be deployed to reduce the interference of environmental noise.

[0039] During the voice collection process, the system should be able to capture the audio data in the service session in real time. Real-time collection means that audio data can be continuously collected, transmitted and recorded throughout the service session to avoid data loss or interruption. After the audio signal is collected, it should be cached or directly saved to the database to ensure the continuity of subsequent processing. Use streaming processing and use buffer technology to store the audio stream in real time, or write it directly to the storage device. A common method is to use an audio stream processing library (such as FFmpeg, PortAudio, etc.) to process the reading and real-time writing of the audio stream through a programming interface. For the storage format, a lossless audio format (such as WAV) is usually selected to ensure data quality.

[0040] When collecting voice data, the data format and sampling parameters need to be set. Usually, the WAV format with PCM encoding is used, the sampling rate is set to 16kHz, and the bit depth is 16 bits. These parameter settings are to ensure the sampling quality of the audio data so that subsequent emotion recognition can be carried out accurately. In the implementation process, the format, sampling rate and bit depth of the audio data can be set in the initialization configuration of the voice acquisition module. Specific configuration is performed through the hardware driver layer or the audio acquisition software interface, and the configuration file is loaded when the acquisition starts to ensure that the entire acquisition process meets the preset standards.

[0041] Example description: In the telephone service scenario of insurance claims, the system uses the built-in voice acquisition function of the telephone system to capture the voice conversation between the customer and the customer service staff in real time during the conversation. By storing the voice signal in high-quality WAV format, the insurance company can accurately extract the customer's voice features for emotion recognition and complaint warning analysis. For example, when the customer shows anxiety or dissatisfaction in the conversation, the system can use the high-quality collected data for subsequent emotion analysis, thereby helping the customer service staff to adjust the communication strategy more timely and effectively.

[0042] By deploying high-quality voice acquisition equipment or integrating audio acquisition modules, voice data in service conversations can be captured in real time and accurately, and the interference of background noise on voice signals can be effectively reduced. By setting appropriate sampling rates and data formats, the collected audio data can be guaranteed to have high fidelity and integrity, providing a high-quality data foundation for subsequent voice emotion recognition, thereby improving the accuracy and stability of emotion analysis.

[0043] S20, segmenting the voice data according to a preset time length to obtain a plurality of voice segments;

[0044] In this embodiment, before the speech data is segmented, the time length of the segmentation needs to be set. The choice of time length is usually determined according to the needs of speech emotion analysis, and the common time length range is between 1 and 3 seconds. The appropriate time length is selected to balance the emotional integrity of the speech segment and the requirements of real-time processing. The preset time length can be achieved through a system configuration file or parameter setting. In specific implementation, the system can allow users to flexibly adjust the time length according to the actual application scenario. For different application scenarios, the segmentation time length can be dynamically adjusted according to experience or model requirements. For example, a fixed time period of 1 second may be set in a service session, while in other scenarios with more stable emotions, it can be extended to 3 seconds.

[0045] Segmentation requires the use of a specific algorithm to cut continuous speech data into multiple speech segments. Common segmentation methods include fixed time window segmentation and segmentation based on endpoint detection. Fixed time window segmentation refers to evenly cutting speech data according to a preset time length, while segmentation based on endpoint detection automatically identifies the start and end points of speech segments based on the energy characteristics of speech signals. When implementing fixed time window segmentation, you can use a time sliding window algorithm to set a fixed time length and gradually cut the audio data. The step size of each sliding is consistent with the length of the time window, so that non-overlapping speech segments can be formed. If a segmentation method based on endpoint detection is used, the start and end points of speech can be determined by analyzing the energy threshold and short-time zero crossing rate (ZCR) of the speech signal. The endpoint detection algorithm can be implemented by combining short-time energy analysis and short-time zero crossing rate calculation, and the boundaries of speech segments can be automatically located by jointly analyzing energy changes and signal frequency changes.

[0046] After the segmentation is completed, the system needs to store or cache each speech segment in real time for subsequent feature extraction and sentiment analysis. The storage format of the speech segment should be consistent with the previous acquisition format to ensure consistency in subsequent processing. Each speech segment can be stored or cached directly in memory through the audio processing library or the system-defined data structure. When storing, the system saves each segmented speech segment according to the preset audio format and path. For real-time processing scenarios, the segmented audio clips can be directly passed to the subsequent feature extraction module to avoid the overhead caused by multiple read and write operations.

[0047] Example: In the insurance claims service process in the financial field, the conversation between the customer and the customer service may last for a long time. In order to accurately monitor the emotional changes of the customer, the system uses a fixed time window of 1 second to cut the continuous voice data. For segments containing obvious silence or invalid voice, the system uses an endpoint detection algorithm to automatically filter and store the valid voice segments for subsequent emotional feature extraction and analysis. This segmented processing method helps to monitor customer emotional fluctuations in real time, identify potential service anomalies in a timely manner, and help customer service personnel quickly formulate response strategies.

[0048] By properly segmenting the speech data, the continuous speech signal can be cut into multiple independent speech segments, which is convenient for subsequent feature extraction and emotion recognition. Reasonable segment length setting can balance the emotional integrity of the speech segment and the requirements of real-time processing of the system. At the same time, the segmentation method combined with endpoint detection can automatically identify and separate valid speech segments, effectively eliminating the interference of silent segments or irrelevant audio data, thereby improving the accuracy of emotion recognition.

[0049] S30, extracting features from each speech segment to obtain feature data of each speech segment;

[0050] In this embodiment, feature extraction is to analyze each speech segment and extract core feature data that can reflect the emotional information of the speech segment. Feature data usually includes information such as frequency, energy, and time domain changes of the audio, which is used to describe the emotional characteristics of the speech segment.

[0051] Extract pitch change information from speech segments. Pitch is very important in emotional expression and can reflect the speaker's emotional ups and downs (e.g., pitch rises when angry and is relatively stable when calm). Use audio processing tools (such as librosa or praat) to detect the fundamental frequency (F0) of each speech segment and record the dynamic changes of pitch.

[0052] Time domain features focus on the change of the amplitude of speech signals over time, reflecting the energy distribution and fluctuation of the signal. Typical examples of time domain features include short-time energy and short-time zero crossing rate (ZCR). The short-time energy and short-time zero crossing rate are calculated for each speech segment. The short-time energy represents the energy change in the speech segment, and the zero crossing rate is used to reflect the frequency content of the speech signal. These features are calculated by writing relevant algorithms or calling existing audio processing libraries.

[0053] More dimensional information can be extracted by jointly analyzing time domain and frequency domain features. For example, more frequency details can be obtained by windowing and transforming the spectrum, and the time domain energy can be combined to enhance the extraction of emotional information. In implementation, time domain and frequency domain methods can be combined, using technologies such as continuous wavelet transform (CWT) or short-time Fourier transform (STFT) to extract joint features in the time and frequency domains.

[0054] Use pre-trained emotion feature extraction models, which can be trained based on emotion datasets of speech and are specifically used to extract emotion-related features. Use existing emotion recognition models, such as open source openSMILE or DeepSpeech, to directly extract emotion-related features from each speech segment. This avoids the complex process of traditional audio feature extraction and directly obtains high-level features related to emotion recognition.

[0055] Example: In the insurance claims service in the financial field, when the customer communicates with the customer service by voice, the system can extract the pitch features and time domain features of each voice segment. For example, when it is detected that the customer's pitch suddenly rises and the short-term energy increases, the system may infer that the customer's emotions fluctuate greatly, and then generate an early warning signal. This combination of multiple feature extraction methods can help the system accurately identify the emotional changes of customers during the service process, adjust the service strategy in time, and avoid the escalation of potential complaints or dissatisfaction.

[0056] Feature extraction can achieve a comprehensive representation of the emotional information of each speech segment, which can not only capture the frequency domain features of the speech signal, but also extract the time domain, pitch and other emotion-related information. These rich feature data can provide high-quality input for the subsequent emotion classification model, thereby significantly improving the accuracy and precision of emotion recognition, helping the system to better predict the emotional state of the insured, and thus optimize the service process.

[0057] S40, inputting the feature data of each speech segment into a sentiment classification model, and identifying the sentiment label of each speech segment through the sentiment classification model;

[0058] In this embodiment, the emotion classification model is a core module for receiving feature data of speech segments and outputting corresponding emotion tags. The emotion classification model can be selected according to the needs and data characteristics, and a common machine learning model or a deep learning model can be selected. Usually, the emotion classification model can be a support vector machine (SVM), a random forest (RF), a convolutional neural network (CNN), a long short-term memory network (LSTM) or a hybrid model. A suitable classification model can be selected according to the needs of different application scenarios and data volumes. For example, for smaller data sets, traditional machine learning methods such as SVM, KNN, etc. can be selected; and for large-scale emotional speech data sets, deep learning models such as LSTM, CNN or Transformer models can be used. The model can be implemented by training and deploying common machine learning frameworks (such as Scikit-learn, TensorFlow, PyTorch, etc.).

[0059] The data input to the sentiment classification model is the feature data extracted from each speech segment. The input data usually includes multiple features, such as MFCC, pitch, energy, frequency components, etc. In order to enhance the learning effect of the model, the feature data usually needs to be preprocessed by normalization or standardization. Before the model is deployed, the feature data of each speech segment is organized into a vector or matrix form and input into the model. At the same time, in order to eliminate the dimensional differences between features, the feature data can be normalized. In the specific implementation, tools such as StandardScaler and MinMaxScaler can be used to preprocess the data.

[0060] After receiving the feature data, the sentiment classification model classifies it according to the trained model and outputs the corresponding sentiment label. Sentiment labels are usually divided according to the sentiment categories set by the system, such as anger, sadness, happiness, trust, anxiety, etc. During the training phase, the model learns the mapping relationship between feature data and sentiment labels. When the trained model inputs new feature data, it can output the sentiment label of each speech segment through forward propagation or classification algorithm calculation. The label can be represented by digital coding (such as 0-5 represents different emotions) or directly output text labels. In the implementation process, the prediction method of the deep learning framework (such as model.predict in PyTorch) is used for forward reasoning to obtain the sentiment classification result.

[0061] The training of sentiment classification models is carried out through a large amount of training data with sentiment labels. The optimization of the model is to improve the classification accuracy of the model by adjusting the model parameters (such as learning rate, number of network layers, regularization parameters, etc.). Use the labeled emotional speech data set for model training, select the appropriate loss function (such as cross entropy loss), and adjust the hyperparameters according to the training results. You can use methods such as grid search or Bayesian optimization to automatically adjust the model parameters to improve the classification effect.

[0062] Example description: In the insurance claims scenario in the financial field, when a customer communicates with a customer service representative over the phone, the system converts the voice data into feature data through feature extraction and inputs it into a pre-trained sentiment classification model. Assuming that an LSTM model is used to process time series data, the LSTM model can capture the gradual changes in the customer's emotions during the conversation. When the system recognizes that the emotion label in certain voice segments is "angry" or "anxious", it can judge the customer's emotional state based on the context and make early warnings and adjustments to the service strategy. This helps customer service personnel adjust the communication method in a timely manner, alleviate customer dissatisfaction, and reduce the possibility of complaints.

[0063] By inputting the feature data of each speech segment into the sentiment classification model, the sentiment information in the speech segment can be accurately identified. The sentiment classification model can effectively deal with complex and changeable emotional expressions by learning the sentiment patterns in a large amount of training data. Selecting a suitable sentiment classification model (such as SVM, LSTM, etc.) can significantly improve the accuracy of sentiment recognition, thus providing a reliable basis for subsequent sentiment analysis and service strategy adjustment.

[0064] S50, combining the emotion labels of each speech segment to generate an emotion label sequence of the speech data, and generating an emotion fluctuation graph according to the emotion label sequence;

[0065] In this embodiment, after the emotion classification model performs emotion recognition on each speech segment, the emotion tags of each speech segment need to be combined in chronological order to generate a complete emotion tag sequence. The emotion tag sequence can reflect the continuity and trend of emotion changes during the entire service session. The emotion tags of each speech segment are arranged in the chronological order of their appearance to form a time series of emotion tags. In implementation, a list or array data structure can be used to store the emotion tag sequence, and newly recognized emotion tags can be gradually appended in chronological order to form an emotion tag sequence.

[0066] The purpose of analyzing the sequence of emotional labels is to identify the changing patterns and trends of emotional states, so as to discover the key nodes of emotional fluctuations. Emotional label sequence analysis can reveal the dynamic information of emotional changes by counting the changing frequency of emotional labels, the pattern of continuous labels, the time interval between emotional labels and other features. Count and count the emotional states in the emotional label sequence, and identify the emotional states with higher frequency and their continuous appearance patterns. For example, you can count the number of consecutive appearances of "anger-anxiety-anger" in the label sequence, or calculate the conversion frequency between adjacent emotional labels. The statistical results can serve as an important basis for emotional fluctuation analysis.

[0067] Based on the emotion tag sequence and its analysis results, an emotion fluctuation graph is generated. The emotion fluctuation graph is a visual representation that shows the changing trend of the emotion state throughout the service session. The vertical axis of the fluctuation graph usually represents different emotion tags, while the horizontal axis represents the time or speech segment number. The amplitude of the fluctuation can represent the intensity of emotion or the stability of the emotion state. The emotion fluctuation graph is drawn through programming tools (such as Matplotlib, Seaborn and other data visualization libraries). In the chart, each speech segment is marked on the timeline in turn, and the corresponding emotion label is drawn at each time point. At the same time, the intensity or conversion frequency of the emotion state is represented by the change of color, line or symbol.

[0068] The generated emotion fluctuation graph is dynamically analyzed to identify mutation points or high-frequency change areas of emotion. This can help the system determine whether the fluctuation of customer emotions exceeds the normal range, thereby warning of possible complaint risks. When implemented, the mutation points in the emotion fluctuation graph can be marked, and the frequency of emotion changes, the rate of change of emotion intensity and other characteristics can be calculated. By setting a certain threshold, when the emotion fluctuation exceeds the threshold, an early warning prompt is automatically generated.

[0069] Example: In the insurance claim telephone service in the financial field, the system identifies the emotion tags of each voice segment of the call and combines these tags into a complete time series. In actual operation, each tag can represent states such as "calm", "anxious", and "angry". The emotion fluctuation graph shows the emotional change curve of the customer throughout the conversation, such as showing that the customer's emotion gradually changes from "calm" to "angry" in a certain period of time. Through the emotion fluctuation graph, customer service can promptly notice the change point of the customer's emotion and quickly adjust the communication strategy to prevent the emotion from further deteriorating.

[0070] By combining the emotion labels of each speech segment, a complete emotion label sequence can be generated, which can fully reflect the changing trend of the insured's emotions during the service session. The emotion fluctuation graph visually displays the dynamic changes of emotions, allowing service personnel to quickly identify the changes and fluctuations of the insured's emotions, so as to take timely response strategies when potential risk emotional states appear.

[0071] S60, analyzing the probability of service abnormality based on the emotion fluctuation graph;

[0072] In this embodiment, the emotion fluctuation graph reflects the change of emotion labels over time during the service session. By extracting the key features in the emotion fluctuation graph, a basis can be provided for the subsequent abnormal probability analysis. The key features include the rate of change of emotion intensity, the duration of the emotion state, the frequency of the change of the emotion state, etc. The emotion fluctuation graph is analyzed to extract the following features:

[0073] Sentiment intensity change rate: Calculate the gradient of the emotion label in the fluctuation graph over time, and record the speed and amplitude of the intensity change.

[0074] Emotional state transition frequency: Count the number of transitions between different emotional states to obtain the transition frequency of each state and the transition status of adjacent emotional states.

[0075] Emotional state duration: record the duration of each emotional state and count the distribution of each emotional state.

[0076] When implementing, data processing tools (such as Pandas, NumPy, etc.) can be used to analyze the emotional fluctuation graph, extract the above features and organize them into a feature data table.

[0077] The analysis of service anomaly probability depends on the classification or regression analysis of the feature data extracted from the emotion fluctuation graph. An anomaly discrimination model (such as a classification model or a regression model) is established by correlating the feature data of the emotion fluctuation graph with known service anomaly cases. First, the features in the emotion fluctuation graph are quantified, and the training set and validation set are constructed using the labeled anomaly case data in the historical service records. Next, a classification algorithm (such as logistic regression, random forest, SVM, etc.) or a regression algorithm (such as linear regression, decision tree regression, etc.) is used for modeling. The degree of correlation between the features of the current emotion fluctuation graph and the service anomaly is analyzed through the prediction results of the model.

[0078] By matching the emotion fluctuation graph features with the service anomaly discrimination model, the service anomaly probability corresponding to the current emotion fluctuation features is calculated. The service anomaly probability indicates the possibility of an abnormal event occurring under the customer's current emotional state. Using the trained anomaly discrimination model, the feature data in the emotion fluctuation graph is input into the model, and the corresponding anomaly probability value is output. The anomaly probability can be calculated using a machine learning framework (such as Scikit-learn or TensorFlow), and the need to issue an early warning can be determined based on the preset probability threshold.

[0079] Example: In the insurance claims scenario in the financial field, the system can continuously monitor the customer's emotional changes through the emotional fluctuation map. Assume that in a conversation between a customer and a customer service representative, the emotional fluctuation map shows that the customer's emotional state gradually changes from "calm" to "anxious" and frequently switches to "angry". After analyzing these features, the system determines that the probability of service abnormality is high, so it sends an early warning to the customer service representative. Based on the early warning signal, the customer service representative quickly adjusted the communication strategy, successfully calmed the customer's emotions, and avoided the occurrence of complaints.

[0080] Through feature extraction based on emotional fluctuation graphs and service anomaly probability analysis, the potential connection between customer emotional changes and service anomalies can be accurately identified. This analysis method helps to monitor the customer's emotional state in real time during the service process, and predict and evaluate the probability of service anomalies in advance. The system can send early warning signals to service personnel based on the level of anomaly probability, thereby reducing the risk of customer complaints and improving service quality.

[0081] S70, when the service abnormality probability reaches a preset threshold, generating a risk warning signal;

[0082] In this embodiment, the system calculates the service abnormality probability by analyzing the emotional fluctuation graph and compares it with a preset threshold. The threshold is set based on historical data and business needs to determine the severity of the abnormal situation. When the calculated abnormal probability is greater than or equal to the preset threshold, the current emotional state is considered to have a higher service risk. The system sets a preset threshold for the abnormal probability based on previous statistical analysis results and empirical data from historical cases. For example, if historical data shows that when the abnormal probability reaches more than 70%, the possibility of customer complaints increases significantly, the threshold is set to 0.7. In a specific implementation, a configuration file can be used to store the threshold, or the threshold can be flexibly adjusted through the management interface of the business system.

[0083] When the probability of service anomalies reaches or exceeds the preset threshold, the system will automatically generate a risk warning signal to alert relevant service personnel. Risk warning signals can be issued in a variety of forms, such as prompt sounds, message pushes, pop-up warnings, etc. In the system, threshold comparison operations can be implemented through conditional judgment statements (such as if statements). When the conditions are met, the early warning mechanism module is called to generate and send a risk warning signal. For example, a push message can be sent through an API interface, or a prompt sound can be played through an audio playback module. The warning signal can contain preset text descriptions, risk levels, and basic customer information.

[0084] Different warning levels (such as low, medium, and high) can be set according to the probability of service anomalies. The warning level reflects the severity of the service anomaly and helps service personnel make priority and response strategies. By grading the probability of service anomalies, for example, the probability interval is divided into 0.6-0.7 (low level), 0.7-0.8 (medium level), and 0.8-1.0 (high level). During implementation, these grading standards can be dynamically adjusted by configuring parameters. When the system generates a warning signal, different warning levels will be attached according to the probability of service anomalies, and the information will be pushed to relevant personnel through the message notification module.

[0085] Example: In the insurance claims service process in the financial sector, the system continuously monitors the customer's emotional state and calculates the probability of service anomalies. When the customer's emotions gradually fluctuate and show anxiety, anger and other emotions during the communication between the customer service staff and the customer, the system calculates that the probability of service anomalies reaches the threshold of 0.8 and immediately sends a high-level risk warning signal to the customer service staff. After receiving the warning, the customer service staff quickly adjusts the words or related service strategies and guides the customer's emotions, thereby effectively avoiding the occurrence of service complaints.

[0086] By setting a preset threshold for the probability of service anomalies, the system can effectively monitor the emotional state of the insured during the service process, and issue a risk warning signal in a timely manner when the probability of anomalies reaches or exceeds the threshold. In this way, service personnel can quickly make response adjustments based on the prompts of the warning signal to prevent further expansion of service anomalies, reduce the risk of customer complaints, and improve service quality.

[0087] S80: Adjust the service strategy of the service session based on the risk warning signal.

[0088] In this embodiment, the risk warning signal generated by the system contains information such as risk level, emotional state description, and recommended service adjustment strategy. The adjustment of the service strategy requires first parsing the warning signal to understand the specific risk level and customer emotional state, so as to determine the direction of adjustment. After the system receives the risk warning signal, it determines the priority of service strategy adjustment by parsing the risk level and emotional state information in the signal. During implementation, a structured warning signal data format (such as JSON or XML) can be used for information transmission, and the service strategy adjustment module parses key fields in the signal data, such as risk level, emotional label, etc.

[0089] Based on the analyzed risk level and emotional state, the system automatically selects or recommends the corresponding service strategy. Different risk levels and emotional states correspond to different strategy adjustment plans, such as adjusting the tone of conversation, suggesting words, terminating the conversation, or transferring the call to a customer service representative with higher authority. The system has multiple preset service strategies, which can be selected according to different risk levels. For example:

[0090] Low risk level: It is recommended that customer service staff use a friendlier tone to express understanding and comfort.

[0091] Medium risk level: prompt customer service staff to proactively ask customers about their specific needs and seek solutions.

[0092] High risk level: It is recommended that customer service personnel listen to customer opinions calmly and report or transfer the case to senior customer service in a timely manner.

[0093] In specific implementation, different policy templates can be preset in the system, and appropriate policies can be dynamically selected based on risk levels. The policy templates contain specific operation steps or speech suggestions, and customer service staff can adjust the dialogue in real time based on the template content.

[0094] During the service session adjustment process, the system continuously monitors changes in the customer's emotional state and adjusts the strategy in real time based on the monitoring results. The adjustment of the service strategy should be dynamic and continuous so that new responses can be made when the customer's emotions change further. When implemented, the sentiment analysis module and the early warning signal analysis module can be combined to monitor the service session in real time. Every time a significant change in the emotional state is detected, the service strategy module will adjust the corresponding service strategy based on the new emotional information and risk signals, and provide real-time feedback of the adjustment suggestions to the customer service staff.

[0095] Example description: In the insurance claims service in the financial field, during the service conversation between the customer and the customer service, the system will recommend the customer service to adjust the service strategy based on real-time sentiment analysis and risk warnings, and can flexibly optimize the service content according to the customer's emotional state.

[0096] For example, during the claims process, the system discovered through sentiment analysis that the customer's emotions gradually changed from "calm" to "anxious" and continued to show "disappointment" or "anger". After the system generated a mid-level risk warning signal, it pushed a suggestion message to the customer service staff, prompting them to proactively provide customers with additional discounts or service upgrades, such as extending the policy validity period, fast-track claims, and giving away additional policy benefits. In specific cases, the customer service staff can proactively tell the customer based on the push prompts: "We attach great importance to your problem. To show our sincerity, we are willing to provide you with additional service discounts." This strategic adjustment can effectively alleviate customer dissatisfaction.

[0097] In addition, if the system analyzes that the customer's mood is gradually deteriorating and generates a high-level risk warning signal, in addition to suggesting upgrading the session processing permissions, it can also suggest that the customer service staff proactively provide higher-value services, such as immediately starting the VIP customer fast claims process, providing exclusive customer service follow-up, etc. Customer service staff can proactively inform customers based on the system's prompts: "In order to better help you solve the problem, we have provided you with a fast channel, and your claim will be directly followed up and handled by a specialist."

[0098] By flexibly adjusting service strategies and combining them with specific measures such as offering discounts, upgrading services, and granting exclusive benefits, we can better meet the personalized needs of customers, thereby improving service satisfaction, enhancing customers' trust in insurance companies, and reducing potential complaints and risks.

[0099] The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a service strategy adjustment method based on emotion recognition, which collects voice data in service conversations, performs segmentation processing and extracts feature data, inputs the feature data into an emotion classification model to identify emotion tags, and generates an emotion tag sequence; further generates an emotion fluctuation graph, analyzes the service abnormality probability, generates a risk warning signal when the service abnormality probability reaches a preset threshold, and adjusts the dialogue strategy of the service conversation based on the risk warning signal. The present invention collects and analyzes voice data in service conversations, identifies emotion tags based on an emotion classification model, and analyzes the service abnormality probability in combination with an emotion fluctuation graph, thereby generating a risk warning signal when the service abnormality probability reaches a preset threshold, and adjusting the service strategy in a timely manner, thereby realizing real-time monitoring of the customer's emotional state and dynamic adjustment of services, which can effectively reduce the risk of complaints, improve customer satisfaction, and reduce economic losses and reputation losses caused by complaints.

[0100] In one embodiment, the above S30 includes:

[0101] S301, performing windowing processing on each speech segment to obtain each windowed speech segment;

[0102] S302, performing a fast Fourier transform on each windowed speech segment to obtain spectrum data of each speech segment;

[0103] S303, processing the spectrum data of each speech segment through a Mel filter bank to obtain Mel frequency data of each speech segment;

[0104] S304, performing logarithmic transformation and discrete cosine transformation on the Mel frequency data of each speech segment to obtain Mel frequency cepstral coefficient feature data of each speech segment;

[0105] S305, determining the first-order difference and the second-order difference of the Mel frequency data of each speech segment, and obtaining the dynamic feature data of each speech segment based on the first-order difference and the second-order difference;

[0106] S306, combining the Mel-frequency cepstral coefficient feature data and the dynamic feature data of each speech segment to obtain feature data of each speech segment.

[0107] In this embodiment, each speech segment is windowed to reduce the influence of spectrum leakage. By smoothing the signal, the influence of edge mutation is reduced. Commonly used window functions include Hamming window and Hanning window. Use Hamming Window function for windowing. For example, the window function can be applied to each speech segment through Python's scipy.signal.hamming function. The specific steps are: multiply the sampled data of each speech segment by the window function coefficient point by point to obtain the windowed speech segment.

[0108] The windowed speech segment is converted from the time domain to the frequency domain through the fast Fourier transform to obtain the spectrum data of the speech segment. FFT can represent the time signal as a linear combination of a series of frequency components, and the windowed signal is fast Fourier transformed. The converted frequency domain signal represents the energy distribution of each speech segment in the form of frequency, that is, the spectrum data.

[0109] The spectral data is weighted by the Mel filter bank, and the frequency components are mapped to the Mel scale. The Mel filter bank simulates the human ear's perception of frequency. The spectral data of each speech segment is mapped to the Mel scale using the Mel filter bank function. The parameters of the Mel filter bank (such as the number of filters and the frequency range) need to be adjusted according to the application requirements. The specific implementation can be used in the librosa.filters.mel function in the librosa library.

[0110] Perform a logarithmic transformation on the Mel frequency data to compress its dynamic range. Then generate Mel frequency cepstral coefficients (MFCC) through discrete cosine transform (DCT). After taking the logarithm of the Mel frequency data, use the DCT function (such as scipy.fftpack.dct) to perform discrete cosine transform. Take the first several MFCC coefficients (such as 12 or 13) as the main components of the feature data.

[0111] Calculate the first-order difference and second-order difference of the Mel frequency cepstral coefficients of each speech segment to obtain dynamic features. Dynamic features reflect the rate (speed) and acceleration of change of MFCC over time. Perform differential calculations on the MFCC feature data of each speech segment. The first-order difference represents the rate information, and the second-order difference represents the acceleration information. These calculations can be implemented using differential formulas or related library functions (such as numpy.diff).

[0112] Combine the static features (MFCC feature data) and dynamic features (differential features) of each speech segment to form complete feature data. The combined feature data contains the static frequency domain features and dynamic change information of each speech segment. Matrix concatenation or vector merging of MFCC features and differential features is performed to form a feature vector for each speech segment. You can use numpy.concatenate or similar concatenation functions in Python to achieve the combination of feature data.

[0113] This embodiment can capture the static and dynamic information of each speech segment comprehensively and meticulously by extracting features from each speech segment. Static features such as MFCC reflect the basic spectral characteristics of the speech segment, while dynamic features such as first-order and second-order differences can reflect the trend of the speech segment changing over time. This feature extraction method can enhance the robustness and accuracy of emotion recognition and provide high-quality input data for subsequent emotion classification models.

[0114] In one embodiment, before the above S40, the method further includes:

[0115] S401, pre-collecting speech data of various emotional states, and constructing an emotional speech database based on the speech data of various emotional states;

[0116] S402, extracting features from the speech data in the emotional speech database to obtain training feature data;

[0117] S403: training a sentiment classification model using the training feature data.

[0118] In this embodiment, before building the emotion classification model, it is necessary to collect speech data containing multiple emotional states in advance. These emotional states can include anger, happiness, sadness, calmness, anxiety, etc. The collected data samples should cover as many emotional expressions as possible to ensure the generalization ability of the classification model. Through a special speech data collection project, volunteers can be invited to simulate speech under various emotional states, or a public emotional speech data set can be used. During the collection, each speech segment needs to be emotionally labeled, and the amount of data for each emotional state must be relatively balanced.

[0119] The collected voice data is sorted and stored to form an emotional voice database. The database should contain voice files and corresponding emotional label information to facilitate subsequent feature extraction and model training. The collected voice data is organized through a database management system (such as MySQL, SQLite, etc.) or a file system. Each voice segment can be bound to its emotional label by file naming or data table association. The data storage format can be selected as needed, for example, a voice file in WAV format corresponds to a label file in CSV format.

[0120] Perform feature extraction on each speech data segment in the emotional speech database. The extraction process is consistent with the feature extraction method mentioned above. The extracted feature data and the emotional label together constitute the training feature data for training the emotional classification model. Use the aforementioned feature extraction techniques (such as MFCC extraction, pitch feature extraction, etc.) to process each speech data segment in the emotional speech database and extract the corresponding feature data. Combine these feature data with their corresponding emotional labels to form a training feature data table. The Pandas library can be used to store feature data and labels in CSV files or DataFrame format for model training.

[0121] Use the training feature data to train the sentiment classification model so that it can learn the mapping relationship between feature data and sentiment labels. The sentiment classification model can be a machine learning model (such as SVM, random forest) or a deep learning model (such as LSTM, CNN). Choose a suitable model architecture based on the size of the dataset and the complexity of the sentiment classification task. For small-scale datasets, traditional machine learning models such as SVM and random forest can be used; for large-scale datasets, deep learning models such as LSTM and CNN can be used. In specific implementation, use tools such as Scikit-learn, TensorFlow, or PyTorch to train the model by calling the fit method, and use the validation set to evaluate the accuracy of the model.

[0122] This embodiment can significantly improve the training quality of the emotion classification model by pre-collecting speech data of various emotion states, building a complete emotion speech database, and extracting feature data. The emotion classification model can learn from these diverse training data, thereby mastering the characteristic laws under various emotion states and improving the accuracy and robustness of classification. In practical applications, such a model can efficiently and accurately identify emotion labels in speech segments, providing reliable support for subsequent emotion analysis.

[0123] In one embodiment, the above S50 includes:

[0124] S501, arranging the emotion labels of each speech segment in chronological order to obtain an emotion label sequence;

[0125] S502, determining the time interval between every two adjacent emotion tags in the emotion tag sequence to form a time interval sequence;

[0126] S503, analyzing the change of the emotion label sequence in combination with the time interval sequence to determine the transition point of the emotion state;

[0127] S504: Generate the emotion fluctuation graph based on the transition points and the time interval sequence.

[0128] In this embodiment, the emotion tags of each speech segment are arranged in chronological order to form a complete emotion tag sequence. Each emotion tag corresponds to a specific time period and represents the emotional state of the insured during the time period. The system needs to record the timestamp and corresponding emotion tag of each speech segment and sort these emotion tags in chronological order. When implemented, the emotion tags can be sorted in chronological order in the form of a list or a data table to form an emotion tag sequence.

[0129] Determine the time interval between each two adjacent emotion tags in the emotion tag sequence and generate a time interval sequence. The time interval sequence reflects the duration and transition rhythm between each emotion state. The time interval sequence is formed by calculating the difference between the end time of each two adjacent speech segments and the start time of the next speech segment. The timestamp information can be used to calculate the time difference between adjacent tags by traversing the emotion tag sequence, and the result is recorded as a time interval sequence.

[0130] Combined with the time interval sequence, the changing trend of the emotion tag sequence is analyzed to determine the transition point of the emotion state. The emotion state transition point reflects the moment when the customer's emotion changes from one state (such as calm) to another state (such as anger). In the emotion tag sequence, the system compares the changes of adjacent tags one by one, combined with the length of the time interval, to identify significant changes in the emotion state. For example, if there is a clear change in emotion category between adjacent emotion tags and the time interval is short, it can be determined that this is a transition point. The information of the transition point can be stored in the form of a timestamp to facilitate subsequent analysis.

[0131] Based on the identified emotional state transition points and time interval sequences, an emotional fluctuation graph that reflects the trend of emotional changes is generated. The emotional fluctuation graph can visualize the changes in the customer's emotional state over time. Use data visualization tools (such as Matplotlib, Seaborn, etc.) to plot the emotional label sequence and the time interval sequence into a fluctuation graph. The horizontal axis represents time, and the vertical axis represents the value or classification of the emotional label. The transition points of the emotional state can be highlighted by special marks or colors.

[0132] This embodiment can accurately identify the turning point of customer emotions by arranging the emotion labels of each speech segment in chronological order, generating an emotion label sequence, and analyzing the emotion change trend in combination with the time interval sequence. The generated emotion fluctuation graph can intuitively display the change process of the customer's emotional state, which helps customer service personnel to understand the dynamic changes of the customer's emotional state in real time, take timely response strategies, and reduce the risk of customer complaints.

[0133] In one embodiment, the above S60 includes:

[0134] S601, calculating the emotion intensity change rate and the emotion state conversion frequency based on the emotion fluctuation graph;

[0135] S602, determining the emotion fluctuation characteristics according to the emotion intensity change rate and the emotion state conversion frequency;

[0136] S603: Determine the service abnormality probability according to the emotion fluctuation characteristics.

[0137] In this embodiment, the emotion intensity change rate is an indicator to measure the magnitude of the change of the emotion state within a certain period of time. It reflects the degree of emotion fluctuation, that is, the speed of change of the customer's emotion state. The system calculates the change rate of emotion intensity based on the emotion label sequence in the emotion fluctuation graph. The change amount of emotion intensity of adjacent emotion states can be calculated using the numerical difference method, and combined with the corresponding time interval to obtain the change rate. For example, for two adjacent emotion state labels S1 and S2, if their emotion intensities are I1 and I2, and the time interval is T, the emotion intensity change rate calculation formula is:

[0138]

[0139] The emotional state transition frequency indicates how often the customer's emotional state changes during the service session, and can reflect the degree of fluctuation of the customer's emotions. The system counts the emotional label sequence in the emotional fluctuation graph and calculates the number of transitions between different emotional states over a period of time. The emotional state transition frequency is determined by counting the duration of each emotional state. For example, if the emotional state switches from "calm" to "angry" twice over a period of time, the conversion frequency within the period can be calculated. When implementing, you can use the Pandas library or a custom algorithm to traverse and count the emotional label sequence, and record the number of switches and time intervals between adjacent labels.

[0140] Based on the rate of change of emotion intensity and the frequency of emotion state transitions, combined with other statistical information of the emotion fluctuation graph, the system can determine the characteristics of emotion fluctuations. Emotion fluctuation characteristics can include the overall fluctuation of emotion intensity, the trend of frequent transitions, etc. The system combines the rate of change of emotion intensity and the frequency of emotion state transitions to form a multi-dimensional emotion fluctuation feature vector. The information of each feature can be combined by weighted averaging, normalization, etc. to form a quantitative feature value. In specific implementation, NumPy or other data processing libraries can be used to complete the combination and calculation of feature vectors.

[0141] The system determines the probability of service anomalies based on the characteristics of emotional fluctuations through pre-trained models or set rules. The probability of service anomalies indicates the possibility of service anomalies occurring under the current customer emotional state. Machine learning models (such as logistic regression, random forest, neural network, etc.) can be used to predict the probability of service anomalies. The system inputs the emotional fluctuation feature vector into the trained model, and the model outputs a probability value. This probability value indicates the degree of correlation between the customer's emotional fluctuation characteristics and historical abnormal service events.

[0142] If the machine learning model is not used, the preset rules can be used for analysis. For example, when the rate of change of emotion intensity and the frequency of emotion state conversion exceed a certain threshold, the probability of service abnormality is considered to be high.

[0143] This embodiment can comprehensively describe the fluctuation of the customer's emotional state by calculating the emotion intensity change rate and emotion state conversion frequency based on the emotion fluctuation graph. The system can accurately assess the probability of service abnormality through comprehensive analysis of emotion fluctuation characteristics. When the probability of service abnormality is high, the system can issue a warning in advance to help customer service personnel take appropriate countermeasures to avoid customers' emotional out-of-control and reduce the risk of service quality problems.

[0144] In one embodiment, before the above S70, the method further includes:

[0145] S701, based on historical data, analyzing the correlation between sentiment characteristics and service abnormal events;

[0146] S702, taking the emotional features whose correlation exceeds a preset correlation threshold as key emotional features;

[0147] S703, performing statistical analysis on the key sentiment features to determine sentiment indicators including mean, standard deviation and / or quantile;

[0148] S704: Determine the preset threshold according to the emotion index.

[0149] In this embodiment, the degree of correlation between emotional features (such as emotional intensity, emotional change rate, emotional conversion frequency, etc.) and service abnormal events (such as customer complaints, negative feedback, etc.) is determined by analyzing historical service data. This step is to identify which emotional features are closely related to service abnormalities. For retrospective analysis of historical data, statistical analysis methods (such as correlation coefficient analysis, chi-square test, t-test, etc.) or machine learning methods (such as feature importance analysis, decision tree analysis, etc.) can be used. In specific implementation, the correlation coefficient can be calculated through Pandas and SciPy libraries, and the correlation value of each emotional feature and service abnormal events can be output.

[0150] The analyzed emotional features that are significantly correlated with service abnormality events are taken as key emotional features. These features play a key role in subsequent threshold setting and model training. A correlation threshold (such as 0.5) is set, and the system compares the correlation coefficients of all emotional features with the threshold, and selects emotional features that exceed the preset correlation threshold as key emotional features. For example, if the correlation coefficient between a feature and a service abnormality event is 0.7, which exceeds the threshold of 0.5, it will be marked as a key emotional feature.

[0151] Perform statistical analysis on key sentiment features to determine sentiment indicators including mean, standard deviation and / or quantiles. Sentiment indicators are calculated to better understand the distribution and changes of key sentiment features. Perform statistical analysis on each key sentiment feature to calculate common statistical indicators such as mean, standard deviation, quartiles, etc. You can use the describe method of the Pandas library to quickly obtain these statistical information. At the same time, the calculation of sentiment indicators can help identify the abnormal range and normal range of key sentiment features, providing a basis for subsequent threshold setting.

[0152] Based on the statistical indicators of key emotional features, determine the preset thresholds for judging the probability of service anomalies. The preset thresholds are usually within the statistical distribution range of key emotional features and are set based on historical experience or business needs. By analyzing the distribution of key emotional features, the system can set reasonable preset thresholds. For example, the threshold can be set using the mean plus or minus a certain multiple of the standard deviation, or it can be defined based on quantiles (such as the 75th quantile). In specific implementations, the settings of these thresholds can be dynamically adjusted and saved in the system configuration file or managed through the interface.

[0153] In this embodiment, through the emotional feature analysis based on historical data, the system can identify the key emotional features that are significantly related to service abnormality events, and determine the reasonable range of these features through statistical analysis. By setting preset thresholds based on the statistical indicators of key emotional features, the accuracy of predicting service abnormality events can be improved. When the system sets reasonable preset thresholds based on the analysis results, it can better judge the risk of service abnormality caused by customer emotional fluctuations, thereby issuing early warning signals in a timely manner at an early stage.

[0154] In one embodiment, after the above S10, the method further includes:

[0155] S101, performing denoising processing on the speech data to obtain denoised speech data;

[0156] S102, performing normalization processing on the denoised speech data to obtain normalized speech data;

[0157] S103, filtering the normalized speech data to obtain filtered speech data;

[0158] S104, detecting the starting point and the ending point of the effective speech in the filtered speech data according to a preset energy threshold and a short-time zero-crossing rate;

[0159] S105, removing the silent segment in the voice data according to the starting point and the ending point to obtain valid voice data.

[0160] In this embodiment, the collected speech data is subjected to denoising to eliminate background noise and unnecessary interference signals and improve the quality of the speech data. Common denoising methods include low-pass filtering, spectral subtraction, Wiener filtering, etc. Using the spectral subtraction method, by sampling the background noise, the noise power spectrum is calculated and subtracted from the spectrum of the speech. Spectral subtraction denoising can be performed with the help of Python's scipy.signal module or librosa library.

[0161] Normalize the denoised speech data to make the amplitude range of the speech signal uniform and reduce the impact of amplitude differences on subsequent feature extraction. Common normalization methods include amplitude normalization and zero mean normalization. Normalize the amplitude of the denoised speech data, for example, limit the amplitude range of the speech data to [-1,1], or adjust the mean of each sample to zero and normalize the variance. This can be achieved through Numpy's normalization function.

[0162] Filter the normalized speech data to remove unnecessary frequency components. Common filtering methods include bandpass filters, low-pass filters, and high-pass filters. Use a bandpass filter to filter out noise components outside the frequency range of the speech signal, leaving only the main frequency range of the speech. In specific implementation, you can use the scipy.signal.butter function to design a bandpass filter, and then use the scipy.signal.filtfilt function to filter the signal.

[0163] The starting and ending points of valid speech in the filtered speech data are detected according to the preset energy threshold and short-time zero-crossing rate. The short-time zero-crossing rate reflects the number of times the signal crosses the zero value, which can be used to distinguish speech segments from silence segments. Calculate the short-time energy and short-time zero-crossing rate of the filtered speech data, and set an energy threshold to judge the validity of the signal. If the energy of the signal exceeds the threshold and the zero-crossing rate meets the preset conditions, it is considered to be valid speech. The starting and ending positions of valid speech can be detected by traversing each segment of the signal, and Numpy or librosa.feature.zero_crossing_rate function can be used to assist in the implementation.

[0164] According to the detected starting point and ending point of the valid speech, the silent segments in the speech data are removed to obtain valid speech data. The purpose of this step is to reduce data redundancy and retain the meaningful speech part. Using the detected starting point and ending point, the speech signal is intercepted from the starting point to the ending point, and the silent segments at the beginning and end are removed. In the specific implementation, the valid speech signal can be intercepted by array slicing.

[0165] This embodiment can significantly improve the quality of speech signals and effectively remove redundant silent segments by sequentially performing operations such as denoising, normalization, filtering, and start and end point detection. The processed speech data has clearer features, which is helpful for subsequent sentiment analysis and model training, and improves the accuracy and robustness of sentiment recognition.

[0166] In one embodiment, a service strategy adjustment device based on emotion recognition is provided, and the service strategy adjustment device based on emotion recognition corresponds one-to-one to the service strategy adjustment method based on emotion recognition in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of a service strategy adjustment device based on emotion recognition of the present invention. Voice collection module 10, voice segmentation module 20, feature extraction module 30, emotion classification module 40, emotion analysis module 50, service anomaly analysis module 60, warning generation module 70 and service strategy adjustment module 80. The functional modules are described in detail as follows:

[0167] A voice collection module 10, used to collect voice data in a service session;

[0168] A speech segmentation module 20, configured to segment the speech data according to a preset time length to obtain a plurality of speech segments;

[0169] A feature extraction module 30 is used to extract features from each speech segment to obtain feature data of each speech segment;

[0170] The emotion classification module 40 is used to input the feature data of each speech segment into the emotion classification model, and identify the emotion label of each speech segment through the emotion classification model;

[0171] The emotion analysis module 50 is used to combine the emotion labels of each speech segment to generate an emotion label sequence of the speech data, and generate an emotion fluctuation graph according to the emotion label sequence;

[0172] A service anomaly analysis module 60, for analyzing the probability of service anomaly based on the sentiment fluctuation graph;

[0173] The warning generation module 70 is used to generate a risk warning signal when the service abnormality probability reaches a preset threshold;

[0174] The service policy adjustment module 80 is configured to adjust the service policy of the service session based on the risk warning signal.

[0175] In one embodiment, the feature extraction module 30 is specifically used for:

[0176] Performing windowing processing on each speech segment to obtain each windowed speech segment;

[0177] Performing a fast Fourier transform on each windowed speech segment to obtain spectrum data of each speech segment;

[0178] The spectral data of each speech segment is processed by a Mel filter bank to obtain Mel frequency data of each speech segment;

[0179] Performing logarithmic transformation and discrete cosine transformation on the Mel frequency data of each speech segment to obtain Mel frequency cepstral coefficient feature data of each speech segment;

[0180] Determine the first-order difference and the second-order difference of the Mel frequency data of each speech segment, and obtain the dynamic feature data of each speech segment based on the first-order difference and the second-order difference;

[0181] The Mel-frequency cepstral coefficient feature data of each speech segment and the dynamic feature data are combined to obtain feature data of each speech segment.

[0182] In one embodiment, the sentiment classification module 40 is specifically used for:

[0183] Pre-collecting speech data of various emotional states, and building an emotional speech database based on the speech data of various emotional states;

[0184] Extracting features from the speech data in the emotional speech database to obtain training feature data;

[0185] The sentiment classification model is trained using the training feature data.

[0186] In one embodiment, the sentiment analysis module 50 is specifically used to:

[0187] Arrange the emotion labels of each speech segment in chronological order to obtain an emotion label sequence;

[0188] Determine the time interval between every two adjacent emotion tags in the emotion tag sequence to form a time interval sequence;

[0189] Analyzing the changes of the emotion label sequence in combination with the time interval sequence to determine the transition point of the emotion state;

[0190] The emotion fluctuation map is generated based on the transition points and the time interval sequence.

[0191] In one embodiment, the service anomaly analysis module 60 is specifically used to:

[0192] Based on the emotion fluctuation graph, calculate the emotion intensity change rate and emotion state transition frequency;

[0193] Determining emotional fluctuation characteristics according to the emotional intensity change rate and the emotional state conversion frequency;

[0194] The service abnormality probability is determined according to the emotion fluctuation characteristics.

[0195] In one embodiment, the warning generation module 70 is specifically used to:

[0196] Based on historical data, analyze the correlation between sentiment characteristics and service abnormal events;

[0197] The sentiment features whose relevance exceeds a preset relevance threshold are regarded as key sentiment features;

[0198] Performing statistical analysis on the key sentiment features to determine sentiment indicators including mean, standard deviation and / or quantile;

[0199] The preset threshold is determined according to the emotion index.

[0200] In one embodiment, the voice collection module 10 is specifically used for:

[0201] Performing denoising on the speech data to obtain denoised speech data;

[0202] Normalizing the denoised speech data to obtain normalized speech data;

[0203] Performing filtering processing on the normalized speech data to obtain filtered speech data;

[0204] Detecting the starting point and the ending point of the effective speech in the filtered speech data according to the preset energy threshold and the short-time zero-crossing rate;

[0205] The silent segment in the voice data is removed according to the starting point and the ending point to obtain valid voice data.

[0206] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the service side of a service strategy adjustment method based on emotion recognition.

[0207] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a user-side method for adjusting a service strategy based on emotion recognition

[0208] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0209] Collect voice data from service sessions;

[0210] Segmenting the voice data according to a preset time length to obtain a plurality of voice segments;

[0211] Extract features from each speech segment to obtain feature data of each speech segment;

[0212] Inputting feature data of each speech segment into a sentiment classification model, and identifying a sentiment label of each speech segment through the sentiment classification model;

[0213] Combining the emotion labels of each speech segment to generate an emotion label sequence of the speech data, and generating an emotion fluctuation graph according to the emotion label sequence;

[0214] Analyzing the probability of service abnormality based on the sentiment fluctuation graph;

[0215] When the service abnormality probability reaches a preset threshold, a risk warning signal is generated;

[0216] The service strategy of the service session is adjusted based on the risk warning signal.

[0217] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0218] Collect voice data from service sessions;

[0219] Segmenting the voice data according to a preset time length to obtain a plurality of voice segments;

[0220] Extract features from each speech segment to obtain feature data of each speech segment;

[0221] Inputting feature data of each speech segment into a sentiment classification model, and identifying a sentiment label of each speech segment through the sentiment classification model;

[0222] Combining the emotion labels of each speech segment to generate an emotion label sequence of the speech data, and generating an emotion fluctuation graph according to the emotion label sequence;

[0223] Analyzing the probability of service abnormality based on the sentiment fluctuation graph;

[0224] When the service abnormality probability reaches a preset threshold, a risk warning signal is generated;

[0225] The service strategy of the service session is adjusted based on the risk warning signal.

[0226] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0227] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0228] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0229] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A service strategy adjustment method based on emotion recognition, characterized in that: The following steps are involved: Collect voice data from service sessions; Segmenting the voice data according to a preset time length to obtain a plurality of voice segments; Extract features from each speech segment to obtain feature data of each speech segment; Inputting feature data of each speech segment into a sentiment classification model, and identifying a sentiment label of each speech segment through the sentiment classification model; Combining the emotion labels of each speech segment to generate an emotion label sequence of the speech data, and generating an emotion fluctuation graph according to the emotion label sequence; Analyzing the probability of service abnormality based on the sentiment fluctuation graph; When the service abnormality probability reaches a preset threshold, a risk warning signal is generated; The service strategy of the service session is adjusted based on the risk warning signal.

2. The service strategy adjustment method based on emotion recognition according to claim 1, characterized in that: Perform feature extraction on each speech segment to obtain feature data of each speech segment, including: Performing windowing processing on each speech segment to obtain each windowed speech segment; Performing a fast Fourier transform on each windowed speech segment to obtain spectrum data of each speech segment; The spectral data of each speech segment is processed by a Mel filter bank to obtain Mel frequency data of each speech segment; Performing logarithmic transformation and discrete cosine transformation on the Mel frequency data of each speech segment to obtain Mel frequency cepstral coefficient feature data of each speech segment; Determine the first-order difference and the second-order difference of the Mel frequency data of each speech segment, and obtain the dynamic feature data of each speech segment based on the first-order difference and the second-order difference; The Mel-frequency cepstral coefficient feature data of each speech segment and the dynamic feature data are combined to obtain feature data of each speech segment.

3. The service strategy adjustment method based on emotion recognition according to claim 1, characterized in that: Before inputting the feature data of each speech segment into the emotion classification model and identifying the emotion label of each speech segment through the emotion classification model, the method further includes: Pre-collecting speech data of various emotional states, and building an emotional speech database based on the speech data of various emotional states; Extracting features from the speech data in the emotional speech database to obtain training feature data; The sentiment classification model is trained using the training feature data.

4. The service strategy adjustment method based on emotion recognition according to claim 1, characterized in that: Combining the emotion labels of each speech segment to generate an emotion label sequence of the speech data, and generating an emotion fluctuation graph according to the emotion label sequence, including: Arrange the emotion labels of each speech segment in chronological order to obtain an emotion label sequence; Determine the time interval between every two adjacent emotion tags in the emotion tag sequence to form a time interval sequence; Analyzing the changes of the emotion label sequence in combination with the time interval sequence to determine the transition point of the emotion state; The emotion fluctuation map is generated based on the transition points and the time interval sequence.

5. The service strategy adjustment method based on emotion recognition according to claim 1, characterized in that: Based on the sentiment fluctuation graph, the probability of service abnormality is analyzed, including: Based on the emotion fluctuation graph, calculate the emotion intensity change rate and emotion state transition frequency; Determining emotional fluctuation characteristics according to the emotional intensity change rate and the emotional state conversion frequency; The service abnormality probability is determined according to the emotion fluctuation characteristics.

6. The service strategy adjustment method based on emotion recognition according to claim 1, characterized in that: When the service abnormality probability reaches a preset threshold, before generating a risk warning signal, the method further includes: Based on historical data, analyze the correlation between sentiment characteristics and service abnormal events; The sentiment features whose relevance exceeds a preset relevance threshold are regarded as key sentiment features; Performing statistical analysis on the key sentiment features to determine sentiment indicators including mean, standard deviation and / or quantile; The preset threshold is determined according to the emotion index.

7. The service strategy adjustment method based on emotion recognition as described in claim 1, characterized in that: After collecting the voice data in the service session, it also includes: Performing denoising on the speech data to obtain denoised speech data; Normalizing the denoised speech data to obtain normalized speech data; Performing filtering processing on the normalized speech data to obtain filtered speech data; Detecting the starting point and the ending point of the effective speech in the filtered speech data according to the preset energy threshold and the short-time zero-crossing rate; The silent segment in the voice data is removed according to the starting point and the ending point to obtain valid voice data.

8. A service strategy adjustment device based on emotion recognition, characterized in that: The service strategy adjustment device based on emotion recognition includes: A voice collection module, used to collect voice data in service sessions; A speech segmentation module, used for segmenting the speech data according to a preset time length to obtain a plurality of speech segments; A feature extraction module is used to extract features from each speech segment to obtain feature data of each speech segment; The emotion classification module is used to input the feature data of each speech segment into the emotion classification model, and identify the emotion label of each speech segment through the emotion classification model; The emotion analysis module is used to combine the emotion labels of each speech segment to generate an emotion label sequence of the speech data, and generate an emotion fluctuation graph according to the emotion label sequence; A service anomaly analysis module, used to analyze the probability of service anomaly based on the sentiment fluctuation graph; An early warning generation module is used to generate a risk early warning signal when the service abnormality probability reaches a preset threshold; A service policy adjustment module is used to adjust the service policy of the service session based on the risk warning signal.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a service policy adjustment program based on emotion recognition stored in the memory and executable on the processor. When the service policy adjustment program based on emotion recognition is executed by the processor, the steps of the service policy adjustment method based on emotion recognition as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a service strategy adjustment program based on emotion recognition, and when the service strategy adjustment program based on emotion recognition is executed by the processor, the steps of the service strategy adjustment method based on emotion recognition as described in any one of claims 1-7 are implemented.

Images