User portrait outbound analysis system based on emotion recognition

Abstract

The invention relates to the technical field of intelligent communication, in particular to a user portrait call-out analysis system based on emotion recognition, which comprises a communication and audio processing module, a voice recognition module, an emotion recognition module, an intelligent dialogue management module, a voice synthesis module and a user portrait and quality analysis module. The intelligent dialogue management module is used for fusing the recognized text content and emotion data and then inputting the fused text content and emotion data into a large language model so as to generate a reply text matched with the current user emotion state; the voice synthesis module is used for receiving the reply text and the real-time emotion data, dynamically adjusting voice synthesis parameters according to the emotion data, synthesizing the reply text into voice and outputting the voice; according to the invention, the emotion of the user can be recognized in real time, the voice interaction strategy is dynamically adjusted, the user portrait is updated based on the multi-modal data, and the service quality and the user satisfaction can be improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent communication technology, and in particular to a user profile outbound call analysis system based on emotion recognition. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent customer service systems have been widely applied in many fields such as finance, telecommunications, and e-commerce. Traditional customer service systems mainly rely on predefined rule bases or simple keyword matching technology to achieve basic self-service question answering and task processing functions. While these systems can reduce labor costs for enterprises, they are rigid in handling complex and ever-changing user needs, especially lacking the ability to perceive and adapt to users' emotional states. In practical applications, fluctuations in user emotions often directly affect communication efficiency and service quality, and most existing systems cannot identify and respond to such changes in real time, resulting in a poor user experience and potentially exacerbating user dissatisfaction.

[0003] In recent years, interactive customer service systems based on Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) technologies have gradually become mainstream. These systems convert user speech into text using an Automatic Speech Recognition module, then utilize Natural Language Processing (NLP) technology to understand user intent and generate responses, and finally use a TTS module to convert the text responses into speech output. Some advanced systems have also introduced deep learning models, such as Recurrent Neural Networks (RNNs) or the Transformer architecture, to improve the accuracy of dialogue understanding. However, these systems still have significant shortcomings in terms of emotional interaction: on the one hand, most systems only focus on the correctness of semantic content, ignoring the emotional information contained in the speech; on the other hand, the system's response generation and speech output are often static and cannot be dynamically adjusted according to the user's emotional state during the conversation.

[0004] In terms of emotion recognition technology, existing solutions are mostly based on feature extraction from speech signals (such as fundamental frequency, energy, and spectrum) or text sentiment analysis (such as sentiment classification based on dictionaries or machine learning). These methods typically analyze isolated speech segments or single sentences of text, making it difficult to achieve continuous emotion tracking during dialogue. Furthermore, emotion recognition modules are often independent of dialogue management modules, lacking effective coordination mechanisms, which prevents the system from optimizing dialogue strategies in real time based on emotion analysis results. For example, when a user exhibits anger or anxiety, the system may still proceed with the dialogue according to the established process instead of prioritizing emotional reassurance, which can easily further deteriorate the user experience.

[0005] In outbound calling scenarios, intelligent customer service systems not only need to complete tasks such as business promotion or information collection, but also bear the important responsibility of maintaining customer relationships and improving service satisfaction. Existing outbound calling systems typically use fixed scripted dialogue flows, which, while ensuring the standardization of task execution, lack in-depth mining and utilization of personalized user characteristics (such as age, region, and historical interaction records). At the same time, the system's evaluation of call quality relies heavily on simple indicators (such as call duration and task completion rate), failing to comprehensively consider soft indicators such as emotional interaction quality and user satisfaction, resulting in evaluation results that cannot fully reflect service quality.

[0006] Furthermore, existing systems rely heavily on post-hoc analysis of historical data for user profiling, lacking the ability to update profiles in real-time during conversations. This prevents the system from dynamically adjusting dialogue strategies based on current interactions and from generating targeted service suggestions immediately after a call ends. For example, if the system could promptly adjust its recommendation strategy or reassurance language when it detects strong interest or resistance towards a particular service, it would significantly improve outbound call effectiveness. However, current technologies have not yet achieved an efficient closed loop between emotion recognition, dialogue management, and user profile updates.

[0007] In summary, existing intelligent customer service systems still have significant limitations in terms of emotion perception, dynamic interaction, and personalized service. Therefore, there is an urgent need for an outbound call analysis system that can identify user emotions in real time, dynamically adjust voice interaction strategies, and update user profiles based on multimodal data to improve service quality and user satisfaction. Summary of the Invention

[0008] The purpose of this invention is to provide a user profile outbound call analysis system based on emotion recognition, which can identify user emotions in real time, dynamically adjust voice interaction strategies, and update user profiles based on multimodal data, thereby improving service quality and user satisfaction.

[0009] To achieve the above objectives, the present invention provides a user profile outbound call analysis system based on emotion recognition, including a communication and audio processing module, a speech recognition module, an emotion recognition module, an intelligent dialogue management module, a speech synthesis module, and a user profile and quality analysis module; The speech recognition module is connected to the communication and audio processing module; the emotion recognition module is connected to the communication and audio processing module; the intelligent dialogue management module is connected to the speech recognition module and the emotion recognition module respectively; the speech synthesis module is connected to the communication and audio processing module, the emotion recognition module, and the intelligent dialogue management module respectively; the user profiling and quality analysis module is connected to the speech recognition module, the emotion recognition module, and the intelligent dialogue management module respectively. The communication and audio processing module is used to establish a call link and synchronously distribute the received uplink audio stream to the speech recognition module and the emotion recognition module. The speech recognition module is used to recognize the upstream audio stream in real time and convert it into corresponding text content; The emotion recognition module is used to perform continuous real-time emotion analysis on the uplink audio stream, generate and output emotion data to the intelligent dialogue management module and the speech synthesis module. The emotion recognition module and the speech recognition module process the same uplink audio stream in parallel. The intelligent dialogue management module is used to fuse the identified text content and the emotion data and input them into a large language model to generate a response text that matches the current user's emotional state. The speech synthesis module is used to receive the reply text and the real-time emotion data, and dynamically adjust the speech synthesis parameters according to the emotion data to synthesize the reply text into speech and output it. The user profiling and quality analysis module is used to generate a service quality assessment report and update the user profile after a single call, based on the conversation content, user emotion change curve, and business results of the call.

[0010] The communication and audio processing module is configured as follows: By using bidirectional real-time transmission protocol stream mirroring technology, the uplink audio stream from the user is input in one channel and distributed in two channels, and pushed to the speech recognition module and the emotion recognition module in real time respectively. At the same time, the downlink audio stream from the speech synthesis module is seamlessly received and played to the user.

[0011] The emotion recognition module is configured as follows: The system performs sliding analysis on the uplink audio stream within a predetermined time window, extracts emotion feature vectors based on a pre-trained emotion representation model, and predicts emotion trends for short-term emotion sequences. Finally, it outputs structured emotion data, including discrete emotion labels, emotion dimension values, and emotion trend indicators.

[0012] The pre-trained emotion representation model uses the pre-trained emotion2vec_plus_large model.

[0013] The intelligent dialogue management module is configured as follows: The text content, the emotion data, the current dialogue history, and the predefined business rules and dialogue goals are dynamically combined into a highly structured prompt word. The prompt words are input into a large language model to generate response text that matches the current sentiment strategy.

[0014] The speech synthesis module is configured as follows: A preset emotion-timbre parameter mapping table maps emotion labels and intensities to specific, quantifiable TTS driving parameters; Before each speech synthesis, the corresponding target acoustic parameter value is obtained by querying the emotion-timbre parameter mapping table based on the latest output received from the emotion recognition module. The speech synthesis engine is driven by the target acoustic parameter values ​​to synthesize the response text.

[0015] The user profiling and quality analysis module is configured as follows: After a single call ends, data from the entire call is collected via a data bus, and timestamps are used to align user statements, system responses, and changes in sentiment tags for that call. Conduct multi-dimensional intelligent evaluation of customer service quality and generate quantitative service quality reports; The user characteristics revealed in this interaction will be incrementally updated into the user profile as tags.

[0016] This invention discloses a user profile outbound call analysis system based on emotion recognition. By constructing an integrated intelligent interactive closed loop encompassing perception, decision-making, execution, and feedback, it achieves real-time dynamic response and strategy adjustment to user emotions during outbound calls. Furthermore, it automatically performs multi-dimensional service quality assessments and dynamically updates user profiles after the call ends. This system elevates emotion recognition from a post-event analysis tool to a core driving element throughout the entire interaction process, enabling it to proactively adjust the system's dialogue strategy and voice output in real time. This invention can identify user emotions in real time, dynamically adjust voice interaction strategies, and update user profiles based on multimodal data, thereby improving service quality and user satisfaction. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0018] Figure 1 This is a schematic diagram of the overall structure of the user profile outbound call analysis system based on emotion recognition of the present invention.

[0019] Figure 2 This is the overall architecture diagram of the user profile outbound call analysis system based on emotion recognition of the present invention.

[0020] Figure 3 This is a flowchart of the user profile outbound call analysis system based on emotion recognition of the present invention.

[0021] 1-Communication and audio processing module, 2-Speech recognition module, 3-Emotion recognition module, 4-Intelligent dialogue management module, 5-Speech synthesis module, 6-User profiling and quality analysis module. Detailed Implementation

[0022] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0023] Please see Figures 1-3 This invention provides a user profile outbound call analysis system based on emotion recognition, including a communication and audio processing module 1, a speech recognition module 2, an emotion recognition module 3, an intelligent dialogue management module 4, a speech synthesis module 5, and a user profile and quality analysis module 6. Through the aforementioned solution, user emotions can be identified in real time, voice interaction strategies can be dynamically adjusted, and user profiles can be updated based on multimodal data, thereby improving service quality and user satisfaction.

[0024] In this specific embodiment, the speech recognition module 2 is connected to the communication and audio processing module 1; the emotion recognition module 3 is connected to the communication and audio processing module 1; the intelligent dialogue management module 4 is connected to the speech recognition module 2 and the emotion recognition module 3 respectively; the speech synthesis module 5 is connected to the communication and audio processing module 1, the emotion recognition module 3, and the intelligent dialogue management module 4 respectively; and the user profiling and quality analysis module 6 is connected to the speech recognition module 2, the emotion recognition module 3, and the intelligent dialogue management module 4 respectively. The communication and audio processing module 1 is used to establish a call link and synchronously distribute the received uplink audio stream to the speech recognition module 2 and the emotion recognition module 3. The speech recognition module 2 is used to recognize the upstream audio stream in real time and convert it into corresponding text content; The emotion recognition module 3 is used to perform continuous real-time emotion analysis on the uplink audio stream, generate and output emotion data to the intelligent dialogue management module 4 and the speech synthesis module 5. The emotion recognition module 3 and the speech recognition module 2 process the same uplink audio stream in parallel. The intelligent dialogue management module 4 is used to fuse the identified text content and the emotion data and input them into the large language model to generate a reply text that matches the current user's emotional state. The speech synthesis module 5 is used to receive the reply text and the real-time emotion data, and dynamically adjust the speech synthesis parameters according to the emotion data to synthesize the reply text into speech and output it. The user profiling and quality analysis module 6 is used to generate a service quality assessment report and update the user profile after a single call, based on the dialogue content, user emotion change curve, and business results of the call.

[0025] In this embodiment, by constructing an integrated intelligent interactive closed loop encompassing perception, decision-making, execution, and feedback, real-time dynamic response to user emotions and strategy adjustment are achieved during outbound calls. Furthermore, the system automatically performs multi-dimensional service quality assessments and dynamically updates user profiles after the call ends. This system elevates emotion recognition from a post-event analysis tool to a core driving element throughout the entire interaction process, enabling it to proactively adjust the system's dialogue strategy and voice output in real time. This invention can identify user emotions in real time, dynamically adjust voice interaction strategies, and update user profiles based on multimodal data, thereby improving service quality and user satisfaction.

[0026] The communication and audio processing module 1 is configured to: input the uplink audio stream from the user through a two-way real-time transmission protocol stream mirroring technology, distribute it in two ways, and push it to the speech recognition module 2 and the emotion recognition module 3 in real time, respectively; at the same time, it seamlessly receives the downlink audio stream from the speech synthesis module 5 and plays it to the user.

[0027] In this embodiment, the communication and audio processing module 1 serves as the communication foundation of the system, built upon the mature open-source communication platform FreeSWITCH. Its technical implementation involves deeply customizing FreeSWITCH's media processing logic, enabling it not only to handle call control (initiating / answering outbound calls) but, more importantly, to act as a high-performance audio routing center. It utilizes bidirectional Real-Time Transport Protocol (RTP) streaming mirroring technology to input the user's uplink audio stream into one channel and distribute it in two channels, pushing it in real-time to the Automatic Speech Recognition (ASR) module and the Emotion Recognition (ERR) module 3, respectively, thus ensuring the synchronization and low latency of audio analysis. Simultaneously, this module seamlessly receives the downlink audio stream from the Text-to-Speech (TTS) module and plays it back to the user. The advantages of this architecture are: leveraging FreeSWITCH's high stability and high concurrency processing capabilities to ensure the reliability of the call link; and avoiding redundant audio data acquisition and transmission through streaming mirroring technology, reducing system complexity and latency, and laying a solid foundation for subsequent real-time emotion analysis.

[0028] Secondly, the emotion recognition module 3 is configured to: perform sliding analysis on the uplink audio stream within a predetermined time window, extract emotion feature vectors based on a pre-trained emotion representation model, and predict emotion trends in short-term emotion sequences, ultimately outputting structured emotion data, including discrete emotion labels, emotion dimension values, and emotion trend indicators. The pre-trained emotion representation model uses a pre-trained emotion2vec_plus_large model.

[0029] In this embodiment, the speech recognition module 2 (ASR, FireRedASR) and the emotion recognition module 3 constitute the parallel perception layer of the system. A key design feature of this invention is that the speech recognition module 2 and the emotion recognition module 3 operate independently and in parallel, rather than in series (i.e., converting to text first and then analyzing the text's emotion). The significant advantages of this approach are: ① Information fidelity: The emotion recognition module 3 directly processes the raw audio, capturing rich paralinguistic features in the speech (such as pitch, velocity, energy spectrum, pauses, etc.), which are crucial emotional information that text cannot convey. ② Complementarity and redundancy: The parallel structure allows the system to accurately perceive user emotions through the audio stream even when the ASR deviates in recognizing certain accents or noisy environments, providing another dimension of more robust interactive context.

[0030] The technical implementation scheme of emotion recognition module 3 is as follows: ① Input and Frame Processing: The emotion recognition module 3 uses audio frames of fixed duration (e.g., 3 seconds) as the basic processing unit and continuously analyzes the audio stream using an overlapping sliding window (e.g., with a step size of 1 second). This high-frequency sampling can capture more subtle emotional fluctuations.

[0031] ② Core Model and Feature Extraction: The pre-trained emotion2vec_plus_large model is used. This model is a general speech representation model based on self-supervised learning. Its advantage lies in its ability to extract clean, high-dimensional emotion embedding vectors from audio that are independent of speaker identity and content. Compared with traditional models based on hand-designed features (such as MFCC), emotion2vec has stronger representational power and better generalization ability.

[0032] ③ Temporal Context Modeling and Prediction: The emotion recognition module 3 does not analyze each 3-second frame in isolation, but includes a temporal context analyzer. This analyzer maintains a short-term emotion state queue (such as the emotion sequence of the past 30 seconds) and applies lightweight time series analysis algorithms (such as calculating differences and simple linear regression) to assess the trend (rising, falling, stable) and volatility of emotions. This allows the system not only to report "the user is currently angry," but also to predict "the user's anger is rapidly escalating," providing a valuable early warning window for the system.

[0033] ④ Output: The emotion recognition module 3 finally outputs structured emotion metadata, including: discrete emotion tags, emotion dimension values ​​(such as valence, arousal) and emotion trend indicators (such as "stable", "intensified", "relaxed").

[0034] Meanwhile, the intelligent dialogue management module 4 is configured to: dynamically combine the text content, the emotion data, the current dialogue history, and the predefined business rules and dialogue goals into a highly structured prompt word; input the prompt word into a large language model to generate a response text that conforms to the current emotion strategy.

[0035] In this embodiment, the intelligent dialogue management module 4 integrates a large language model (Qwen3-large). This module is the decision-making center of the system, and its innovation lies in constructing an emotion information injection mechanism, which includes: (1) Input for Multimodal Information Fusion: The intelligent dialogue management module 4 internally designs a prompt word engineering engine. This engine dynamically combines text from the speech recognition module 2, all emotion metadata from the emotion recognition module 3, the current dialogue history, and predefined business rules and dialogue goals into a highly structured prompt word. Specifically, it constructs a structured prompt word template, which contains instruction fields for describing the user's current emotional state and specifying the response strategy, such as: "[System Role]: You are a customer service specialist. Current business goal: Explain billing questions. [Latest User Statement]: {ASR Text}. [User Emotional State]: Emotion Label = Anger, Trend = Intensified, Volatility = High. [Instruction]: Based on the current strong negative emotion, you must prioritize empathy and apology, provide brief reassurance, promise to resolve the issue, and avoid directly delving into technical details." (2) Emotion-guided dialogue strategy: Through the above prompts, emotional information is no longer an external reference, but rather a strong prior condition that directly shapes the generation process of the large language model. The core advantage of this design is that it perfectly combines the general dialogue capabilities of the large language model with domain-specific emotion response strategies, ensuring that the response content is not only semantically correct, but also has high emotional intelligence and adaptability in terms of emotional strategies, achieving a leap from "response" to "dialogue".

[0036] In addition, the speech synthesis module 5 is configured to: preset an emotion-timbre parameter mapping table, which maps emotion tags and intensities to specific, quantifiable TTS driving parameters; before each speech synthesis, based on the latest output received from the emotion recognition module 3, query the emotion-timbre parameter mapping table to obtain the corresponding target acoustic parameter values; and drive the speech synthesis engine based on the target acoustic parameter values ​​to synthesize the response text.

[0037] In this embodiment, the speech synthesis module 5 (TTS, such as CosyVoice) is the system's emotion expression actuator. Its innovation lies in realizing real-time control of speech parameters based on emotion. The system internally pre-defines a configurable emotion-timbre parameter mapping table, which is essentially an emotional speech science rule engine. It maps emotion labels and intensity to specific, quantifiable TTS driving parameters. Specifically, it maps emotion labels to specific acoustic parameter values ​​such as speech rate, pitch, and energy, for example: (1) Anger / High intensity: speech_rate=0.8 (slow down by 20%), pitch=0.9 (lower pitch), energy=0.7 (lower energy), emphasis keyword.

[0038] (2) Happy / moderate intensity: speech_rate=1.1, pitch=1.05, energy=1.0.

[0039] Before each synthesis, the speech synthesis module 5 queries the latest output of the emotion recognition module 3 and retrieves the corresponding acoustic parameters from the mapping table in real time to drive the synthesis engine. The direct benefit of this is that the voice output by the system maintains a high degree of consistency with the dialogue strategy in terms of timbre, achieving empathy at the acoustic level, making the intelligent customer service more "human" and greatly enhancing the affinity and persuasiveness of communication.

[0040] Finally, the user profiling and quality analysis module 6 is configured to: after a single call ends, collect all-link data through a data bus, and align the user statements, system responses, and changes in emotion tags of this call using timestamps; perform multi-dimensional intelligent evaluation of customer service quality, and generate a quantitative service quality report; and incrementally update the user profile with the user characteristics revealed in this interaction in the form of tags.

[0041] In this embodiment, the user profiling and quality analysis module 6 serves as the system's feedback and optimization engine, enabling closed-loop learning. The operation of the user profiling and quality analysis module 6 includes: (1) Data aggregation and alignment: After the call ends, the module starts up, gathers the data of the entire link through a data bus, and uses timestamps to accurately align all events (user statements, system replies, changes in emotion tags) to form a complete "digital twin" of an interaction.

[0042] (2) Multi-dimensional intelligent evaluation of customer service quality: Service quality assessment specifically includes quantitative indicators such as the conversion efficiency of user emotions from the start to the end of the call and the duration of negative emotions; the evaluation algorithm not only includes hard indicators such as "whether the problem is solved", but also innovatively introduces indicators of emotion conversion efficiency, such as: analyzing the emotion curve and calculating "the duration of negative emotions" and "the time required for emotion to ease after system intervention". A weighted scoring model is used to generate a quantitative service quality report.

[0043] (3) Dynamic updating of user profile: The user characteristics revealed in this interaction (such as emotional sensitivity, comprehension ability, preferred communication style) are incrementally updated into the user profile in the form of tags. The advantage of this mechanism is that it enables subsequent interactions (whether intelligent or human) to be based on richer context, so as to achieve truly personalized services.

[0044] Please refer to the figure. To better understand this invention, the workflow of the emotion recognition-based user profile outbound call analysis system of this invention is described below. The workflow steps are as follows: Step 1: Initiate a call and establish a two-way audio channel (corresponding to the diagram: Initiate an outbound call and connect). (1) Action: The system automatically dials and connects the user's phone number through the FreeSWITCH communication core.

[0045] (2) Technical foundation: This step establishes the most critical two-way real-time audio stream (RTP stream). The user's voice (upstream audio stream) is continuously transmitted in, and the system's response (downstream audio stream) is broadcast out. This is the "source" of all real-time processing.

[0046] Step Two: Parallel Processing, Simultaneously Understanding "Content" and "Emotion" (corresponding to: Real-time Parallel Perception and Processing in the diagram). The system doesn't wait for the user to finish speaking before processing; instead, it processes the same upstream audio stream simultaneously, like two production lines. ① Pipeline A (Understanding the "literal meaning"): The ASR (Audio Recognition) module converts the user's speech stream into a text stream in real time. For example, if the user says "Your service is terrible," the ASR will output the corresponding text.

[0047] ② Pipeline B (Understanding the "Subtext"): The emotion recognition module 3 processes the same audio stream simultaneously. It doesn't care about specific words, but uses a dedicated model like emotion2vec_plus_large to analyze the user's tone, intonation, and speaking speed every 3 seconds to determine the emotional state (such as "anger", "calm", "confused") and calculate an intensity index of the emotional fluctuation.

[0048] ③ The advantages of this design: Parallel processing ensures that sentiment analysis and semantic understanding are equally fast and without delay. Moreover, emotions come directly from speech signals, which is more accurate and direct than sentiment analysis using translated text, and can capture anger, helpless sighs, etc., that cannot be expressed in text.

[0049] Step 3: The brain makes a decision and generates a "high emotional intelligence" response (corresponding to the image: integrating emotions and text to generate a context-aware response). The specific process is as follows: (1) When the system’s “brain” (the dialogue management module that integrates a large language model) is thinking about how to respond, it will simultaneously refer to the “text” from pipeline A and the “emotion” from pipeline B.

[0050] (2) Specific actions: The module will combine these two pieces of information into a "prompt word" with clear instructions and give it to the large model. For example: "The user said: 'Your service is terrible' (text). The user's current emotion is 'anger' and the fluctuation index is very high (emotion). As customer service, you must first soothe the emotion, sincerely apologize, and then try to solve the problem. Please generate a reply." (3) Effects: The large model will no longer generate cold, standard, procedural responses (such as "Please describe the specific problem"), but will generate more empathetic responses (such as "Sir / Madam, I am very sorry for the bad experience you had. I can tell you are really angry. Please give me a moment, and I will check and handle it for you immediately"). This means that emotional signals directly determine the strategy and content direction of the conversation.

[0051] Step 4: Speech synthesis, speaking with the "right tone" (corresponding to the diagram: dynamically adjusting TTS synthesis parameters according to emotions). (1) The system must not only speak the correct words, but also speak them with the appropriate "tone". Before converting the text generated in the previous step into speech, the TTS (speech synthesis) module will receive the latest emotion tags sent by the emotion recognition module 3 in real time.

[0052] (2) Specific actions: The system has an “emotion-tone parameter mapping table”. For example, if the system detects that the user is “angry”, the TTS will automatically speak in a slower, lower, and softer tone to achieve a soothing effect; if the system detects that the user is “calm”, it will communicate in a normal and clear tone.

[0053] (3) Effects: It achieves “empathy” at the voice level, avoiding the awkward situation where the content is very comforting but the tone is cold or cheerful, and greatly improves the naturalness and credibility of the interaction.

[0054] Step 5: Looping Interaction and End Analysis (corresponding to the following in the diagram: Playing Emotionally Adapted Voice -> Determining Loop -> Post-Call Analysis and Summary), the specific process is as follows: (1) Loop: The emotion-adapted speech generated by TTS is played to the user via FreeSWITCH to complete one round of interaction. Then the system immediately returns to step two to continue listening to and processing the user's next sentence, and so on, until the call ends.

[0055] (2) Post-call analysis and summary (closed-loop feedback): As soon as the call ends, the user profile and quality analysis module 6 is immediately activated. It combines the "audio transcript" (full text of the conversation), "emotional electrocardiogram" (emotional change curve) of the entire call process and the final business result (whether the problem was solved) for analysis.

[0056] (3) Customer service quality evaluation: It not only looks at "whether the business is resolved", but also evaluates "emotional management ability", such as: whether the user was successfully calmed down from anger? Was there any sentence that caused the user's emotions to fluctuate drastically? Thus, a "service quality report" of intelligent customer service is given.

[0057] (4) User profile update: Record the user characteristics discovered in this interaction, such as "this user is particularly sensitive to cost issues" and "more patient explanation is needed during communication", to enrich the user's profile tags.

[0058] (5) Decision support: Finally, the system will automatically generate two reports: one for the customer service supervisor, which evaluates the performance of the robot customer service and suggests whether a second follow-up call by human customer service is needed and the key points to focus on during the follow-up call; the other for human customer service, which provides updated user profiles and communication suggestions to ensure the continuity and personalization of subsequent services (if necessary).

[0059] The key features of the emotion recognition-based user profiling outbound call analysis system of this invention are as follows: I. Real-time Emotion Perception Architecture Based on Parallel Pipeline: This architecture abandons the traditional serial, high-latency approach of first converting speech to text and then analyzing the text's emotion. Instead, it innovatively employs an architecture where ASR (Aspect-Recognition) and emotion recognition modules 3 process the same raw uplink audio stream in parallel. This ensures that emotion analysis and semantic understanding have equal real-time performance and can directly capture richer paralinguistic features (such as intonation, speech rate, and energy) from the speech signal, laying the technical foundation for subsequent real-time intervention.

[0060] Second, emotional state is injected as prior knowledge into the decision-making mechanism of large language models: Real-time identified user emotion tags, fluctuation indices, and trend predictions are used as key, prioritized contextual information and deeply integrated with the prompts of large language models (such as ChatGLM and Qwen). This makes the model's response generation no longer purely semantically driven, but rather emotion-strategy-prioritized. For example, the system mandates that when "anger" is detected, the model must prioritize generating empathetic and reassuring content before delegating business logic. This is an innovative approach that combines emotional intelligence rules with generative AI capabilities.

[0061] III. Real-time Control Loop of Emotion Signals on TTS Tone: A direct control channel is established from the emotion recognition module 3 to the TTS (speech synthesis) module. Through a preset "emotion-tone parameter mapping table," the system can dynamically and automatically adjust the acoustic parameters (such as speech rate, pitch, and rhythm) of the TTS output based on real-time emotion analysis results, achieving "empathy" at the speech level. This ensures that the system's speech output is emotionally consistent with the dialogue content, greatly improving the naturalness and credibility of the interaction.

[0062] IV. Closed-Loop Evaluation and Profile Update Based on Multimodal Data Fusion: After a call ends, the system does not solely evaluate based on the business outcome. Instead, it innovatively incorporates the "user sentiment curve" as a core evaluation indicator, integrating it with the dialogue text and business resolution status for multi-dimensional analysis. This not only generates more comprehensive customer service quality reports but, more importantly, enables dynamic and incremental updates to user profiles. For example, it marks users' emotional sensitivities and communication style preferences, providing data support and decision-making suggestions for subsequent personalized services (such as whether a second manual follow-up is needed).

[0063] This invention presents a user profile outbound call analysis system based on emotion recognition. By employing an advanced emotion2vec_plus_large emotion recognition model and performing micro-analysis of user emotion fluctuations every 3 seconds, this invention can track user emotion curves in real time and continuously, and predict emotion trends. This provides a data foundation for early system intervention, fundamentally solving the problem of slow response and achieving dynamic perception and prediction of emotions. Furthermore, by deeply integrating the emotion recognition module 3 with a TTS module (such as CosyVoice), the system can dynamically adjust the timbre characteristics of the synthesized speech (such as gentle, calm, positive, and empathetic) based on real-time emotion analysis results. This allows the intelligent customer service's "tone" to adapt to the user's emotional state, enhancing the affinity and persuasiveness of communication, and achieving real-time emotional adaptation of timbre. Simultaneously, this invention inputs emotional state as a key prior knowledge into the decision-making process of a large language model (such as ChatGLM or Qwen), guiding the model to generate high-EQ dialogue that conforms to the current emotional context. This ensures that responses not only accurately answer questions but also effectively manage user emotions, guiding the conversation towards a positive outcome and achieving intelligent dialogue decision-making based on prior emotional information. Furthermore, after the call ends, the system can integrate the emotional curve of the interaction, the business resolution status, and the user's existing prior information in real time to generate a dynamically updated user profile and a multi-dimensional customer service quality evaluation report. This report not only objectively reflects the service level but also accurately determines whether a second call from a human agent is necessary, providing clear problem summaries and follow-up suggestions for human customer service representatives. This forms a complete closed loop from perception to action, significantly improving outbound call efficiency and customer satisfaction, and achieving real-time profiling and intelligent evaluation based on multi-dimensional data.

[0064] The above-disclosed embodiments are merely one or more preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments and equivalent changes made in accordance with the claims of this application still fall within the scope of this application.

Claims

1. A user profile outbound call analysis system based on emotion recognition, characterized in that, It includes a communication and audio processing module, a speech recognition module, an emotion recognition module, an intelligent dialogue management module, a speech synthesis module, and a user profiling and quality analysis module; The speech recognition module is connected to the communication and audio processing module; the emotion recognition module is connected to the communication and audio processing module; the intelligent dialogue management module is connected to the speech recognition module and the emotion recognition module respectively; the speech synthesis module is connected to the communication and audio processing module, the emotion recognition module, and the intelligent dialogue management module respectively; the user profiling and quality analysis module is connected to the speech recognition module, the emotion recognition module, and the intelligent dialogue management module respectively. The communication and audio processing module is used to establish a call link and synchronously distribute the received uplink audio stream to the speech recognition module and the emotion recognition module. The speech recognition module is used to recognize the upstream audio stream in real time and convert it into corresponding text content; The emotion recognition module is used to perform continuous real-time emotion analysis on the uplink audio stream, generate and output emotion data to the intelligent dialogue management module and the speech synthesis module. The emotion recognition module and the speech recognition module process the same uplink audio stream in parallel. The intelligent dialogue management module is used to fuse the identified text content and the emotion data and input them into a large language model to generate a response text that matches the current user's emotional state. The speech synthesis module is used to receive the reply text and the real-time emotion data, and dynamically adjust the speech synthesis parameters according to the emotion data to synthesize the reply text into speech and output it. The user profiling and quality analysis module is used to generate a service quality assessment report and update the user profile after a single call, based on the conversation content, user emotion change curve, and business results of the call.

2. The user profile outbound call analysis system based on emotion recognition as described in claim 1, characterized in that, The communication and audio processing module is configured as follows: By using bidirectional real-time transmission protocol stream mirroring technology, the uplink audio stream from the user is input in one channel and distributed in two channels, and pushed to the speech recognition module and the emotion recognition module in real time respectively. At the same time, the downlink audio stream from the speech synthesis module is seamlessly received and played to the user.

3. The user profile outbound call analysis system based on emotion recognition as described in claim 1, characterized in that, The emotion recognition module is configured as follows: The system performs sliding analysis on the uplink audio stream within a predetermined time window, extracts emotion feature vectors based on a pre-trained emotion representation model, and predicts emotion trends for short-term emotion sequences. Finally, it outputs structured emotion data, including discrete emotion labels, emotion dimension values, and emotion trend indicators.

4. The user profile outbound call analysis system based on emotion recognition as described in claim 3, characterized in that, The pre-trained emotion representation model uses the pre-trained emotion2vec_plus_large model.

5. The user profile outbound call analysis system based on emotion recognition as described in claim 1, characterized in that, The intelligent dialogue management module is configured as follows: The text content, the emotion data, the current dialogue history, and the predefined business rules and dialogue goals are dynamically combined into a highly structured prompt word. The prompt words are input into a large language model to generate response text that matches the current sentiment strategy.

6. The user profile outbound call analysis system based on emotion recognition as described in claim 1, characterized in that, The speech synthesis module is configured as follows: A preset emotion-timbre parameter mapping table maps emotion labels and intensities to specific, quantifiable TTS driving parameters; Before each speech synthesis, the corresponding target acoustic parameter value is obtained by querying the emotion-timbre parameter mapping table based on the latest output received from the emotion recognition module. The speech synthesis engine is driven by the target acoustic parameter values ​​to synthesize the response text.

7. The user profile outbound call analysis system based on emotion recognition as described in claim 1, characterized in that, The user profiling and quality analysis module is configured as follows: After a single call ends, data from the entire call is collected via a data bus, and timestamps are used to align user statements, system responses, and changes in sentiment tags for that call. Conduct multi-dimensional intelligent evaluation of customer service quality and generate quantitative service quality reports; The user characteristics revealed in this interaction will be incrementally updated into the user profile as tags.

Images