Intelligent outbound call robot dialogue control system integrating multimodal emotion recognition

Abstract

This invention discloses an intelligent outbound call robot dialogue control system integrating multimodal emotion recognition. This invention relates to the field of artificial intelligence technology and consists of three main modules: an audio perception and structuring module, a multimodal emotion feature deconstruction module, and a dynamic emotion interaction control module. First, the system uses a recording device to record the target person's voice in real time, and determines speech activity frames through short-time energy and zero-crossing rate, accurately segmenting them into speech segment sequences. Second, it extracts speech rate and loudness features from each audio segment and constructs an emotion processing feature sequence by combining normalization processing. Finally, based on the comparison of the emotion feature sequence with a preset threshold table, it judges the target person's emotional state in real time and automatically adjusts the outbound call robot's dialogue emotion control parameters according to a dynamic control strategy, achieving intelligent and humanized outbound call dialogue control.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically, it relates to an intelligent outbound call robot dialogue control system that integrates multimodal emotion recognition. Background Technology

[0002] With the rapid development of intelligent outbound call robot technology, voice interaction has become an important bridge in modern business services. In outbound call scenarios such as sales promotion and customer service, the application of voice technology is no longer limited to simple voice recognition and command execution, but is gradually moving towards an emotional and intelligent dialogue experience.

[0003] Existing technologies for emotion recognition and control in intelligent outbound calling robots have significant shortcomings, mainly in that emotion recognition relies on single-modal features, the processing is not real-time, and the control mechanism is static and rigid. Traditional systems typically perform isolated analysis based on simple audio parameters such as volume or speech rate, lacking the fusion and correlation of multimodal features. This results in low accuracy in emotion judgment, susceptibility to environmental noise or individual differences, and an inability to comprehensively capture the user's emotional state. For example, existing methods may use fixed frame intervals to process audio without combining dynamic segmentation with speech activity detection, easily misjudging non-speech parts, and insufficient processing of long speech segments, causing delays or omissions in emotion analysis. Furthermore, existing technologies often generate a fixed response after a single emotion assessment at the beginning of the conversation, without continuously monitoring the trend of emotion changes, making dynamic adjustments impossible. This makes the robot's interaction appear stiff and unnatural, unable to adapt to user emotional fluctuations, thus reducing the effectiveness of the conversation and the user experience. In feature extraction, existing methods mostly rely on absolute threshold comparisons without correlation analysis, resulting in a lack of robustness in emotion state judgment and an inability to adapt to baseline differences among different users. For example, the independent use of loudness and speech rate ignores their synergistic effect.

[0004] To address the aforementioned issues, this invention proposes an intelligent outbound call robot dialogue control system that integrates multimodal emotion recognition. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an intelligent outbound call robot dialogue control system that integrates multimodal emotion recognition, solving the problems of inaccurate emotion recognition, non-real-time control, and lack of adaptability in existing technologies.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A smart outbound call robot dialogue control system integrating multimodal emotion recognition, the system comprising:

[0008] The audio perception and structuring module records the human voice audio associated with the target person during the dialogue between the intelligent outbound call robot and the target person in real time based on a pre-built recording device. It segments the human voice audio, determines the human voice audio segment sequence associated with the time sequence, and updates the human voice audio segment sequence in real time.

[0009] The multimodal emotion feature decomposition module decomposes the human voice audio segments in the human voice audio segment sequence, determines the human voice loudness sequence and human voice speech rate sequence associated with the corresponding human voice audio segment and the time sequence, performs correlation analysis on the human voice loudness and human voice speech rate associated with the same human voice audio segment, and constructs the emotion processing feature sequence.

[0010] The dynamic emotion interaction control module analyzes the first emotion processing feature sequence in the dialogue process, determines the emotional state of the target person in this voice audio segment, and generates preliminary control instructions to regulate the dialogue emotion control parameters of the intelligent outbound call robot.

[0011] Then, by continuously acquiring the emotion processing feature sequences associated with the emotion processing features in the human voice audio segment sequence, and performing correlation analysis, the emotional state of the target person is continuously determined, and the dialogue emotion regulation parameters of the intelligent outbound call robot are continuously adjusted.

[0012] As a further aspect of the present invention, the specific method for segmenting human voice audio in the audio perception and structuring module is as follows:

[0013] Using pre-built recording equipment, the audio of the target person's voice during the conversation between the intelligent outbound calling robot and the target person can be recorded individually and in real time.

[0014] The human voice audio is continuously framed at a preset frame interval T1 to generate a temporally correlated human voice audio frame sequence.

[0015] Calculate the short-time energy and zero-crossing rate associated with each human voice audio frame in the human voice audio frame sequence, and determine whether there is a human voice in the human voice audio frame based on the Gaussian mixture model;

[0016] If a human voice is detected in any human voice audio frame, then this human voice audio frame is marked as a speech activity frame; otherwise, it is marked as a non-speech activity frame.

[0017] The continuous speech activity frames are aggregated into a single speech segment. When a non-speech activity frame is detected to continuously exceed a preset first time threshold, the current speech segment is terminated, and an independent human voice audio segment is generated.

[0018] As a further aspect of the present invention, in the audio perception and structuring module, if the duration of any speech segment exceeds a preset second time threshold, the speech segment is forcibly terminated to generate an independent human voice audio segment.

[0019] As a further aspect of the present invention, the specific method for determining the human voice audio segment sequence associated with time sequence and updating the human voice audio segment sequence in real time in the audio perception and structuring module is as follows:

[0020] Obtain the first human voice audio segment during the dialogue and label it D1;

[0021] The subsequent human voice audio segments are obtained in chronological order and summarized together with human voice audio segment D1, and denoted as human voice audio segment sequence D1, D2, ..., Dj, where j represents the total number of human voice audio segments in the human voice audio segment sequence;

[0022] When a new human voice audio segment is detected, it is inserted at the end of the human voice audio segment sequence, and j is updated.

[0023] As a further aspect of the present invention, the specific method for determining the corresponding human voice audio segment and the time-series associated human voice loudness sequence and human voice speech rate sequence in the multimodal emotion feature deconstruction module is as follows:

[0024] Extract any human voice audio segment Di from the sequence of human voice audio segments D1, D2, ..., Dj, where i is the counting index and 1 ≤ i ≤ j;

[0025] Determine the total duration of the human voice audio segment Di, denoted as m seconds, where m seconds does not exceed the total duration of the second time threshold;

[0026] Construct the time series t1, t2, ..., tm associated with the human voice audio segment Di;

[0027] The human voice audio segment Di is divided into time series t1, t2, ..., tm to obtain the human voice audio segment segment sequence Di_t1, Di_t2, ..., Di_tm;

[0028] Determine any human voice audio segment Di_tn in the human voice audio segment segment sequence, where n is the counting index, 1≤n≤m;

[0029] Perform a text conversion operation on the human voice audio segment Di_tn, determine the total number of text characters, and calculate the human voice speech rate SPn associated with the human voice audio segment Di_tn using the formula: total number of text characters / 1 second = SPn.

[0030] Similarly, determine the speech rate associated with the remaining audio segments and arrange them in chronological order, denoted as the speech rate sequence SP1, SP2, ..., SPm associated with audio segment Di;

[0031] Determine the average loudness in the human voice audio segment Di_tn, and use it as the human voice loudness LOn associated with the human voice audio segment Di_tn;

[0032] Similarly, determine the loudness of the remaining human voice audio segments and arrange them in chronological order, denoted as the loudness sequence LO1, LO2, ..., LOm associated with the human voice audio segment Di;

[0033] By analogy, determine the human voice loudness sequence and human voice speech rate sequence associated with each human voice audio segment in the human voice audio segment sequence D1, D2, ..., Dj.

[0034] As a further aspect of the present invention, the specific method for constructing an emotion processing feature sequence by performing correlation analysis on the loudness and speech rate of the same human voice audio segment in the multimodal emotion feature deconstruction module is as follows:

[0035] Determine the human voice loudness sequence LO1, LO2, ..., LOm and the human voice speech rate sequence SP1, SP2, ..., SPm associated with any human voice audio segment Di in the human voice audio segment sequence D1, D2, ..., Dj;

[0036] Normalization is performed on the human voice loudness sequence and the human voice speech rate sequence to obtain the normalized human voice loudness sequence LO1',LO2',...,LOm' and the normalized human voice speech rate sequence SP1',SP2',...,SPm'. The normalization process adopts the minimum-maximum normalization method to map the sequence values ​​to the [0,1] interval.

[0037] For any time tn in the time series t1, t2, ..., tm, take the normalized human voice loudness LON' and normalized human voice speech rate SPn' associated with this time;

[0038] Determine the values ​​of human voice loudness LON' and human voice speech rate SPn';

[0039] Construct a two-dimensional coordinate system with the human voice loudness LON' as the horizontal axis and the human voice speech rate SPn' as the vertical axis. Mark the human voice loudness LON' and human voice speech rate SPn' as the horizontal and vertical axes in the constructed two-dimensional coordinate system to obtain a data point. Construct a straight line L1 perpendicular to the horizontal axis and parallel to the vertical axis and a straight line L2 perpendicular to the vertical axis and parallel to the horizontal axis through this data point. Determine the area value of the closed region formed by the straight line L1, the straight line L2, the horizontal axis, and the vertical axis as the emotion processing feature associated with the human voice audio segment Di, and label it as EFn.

[0040] Similarly, determine the emotion processing feature sequence EF1,EF2,...,EFm associated with the human voice audio segment Di;

[0041] By analogy, the emotion processing feature sequence associated with each human voice audio segment in the sequence D1, D2, ..., Dj is determined.

[0042] As a further aspect of the present invention, the specific method by which the dynamic emotion interaction control module generates preliminary control instructions to regulate the dialogue emotion control parameters of the intelligent outbound call robot is as follows:

[0043] Extract the emotion processing feature sequence EF1,EF2,...,EFm associated with the first human voice audio segment D1 in the dialogue process;

[0044] Calculate the arithmetic mean of the emotion processing feature sequence, denoted as the average emotion feature value EF_avg;

[0045] The average emotional feature value EF_avg is compared with a pre-constructed emotional state threshold table, which includes a first emotional state threshold Th1 and a second emotional state threshold Th2, where Th1 < Th2.

[0046] If EF_avg≤Th1, the target person's emotional state is determined to be negative.

[0047] If Th1 < EF_avg < Th2, the target person's emotional state is determined to be neutral.

[0048] If EF_avg≥Th2, the target person's emotional state is determined to be positive.

[0049] Based on the determined emotional state, the corresponding preliminary control instructions are obtained from the control instruction mapping table predefined by the operator, wherein the control instruction mapping table defines the correspondence between the emotional state and the dialogue emotional control parameters of the intelligent outbound call robot.

[0050] The initial control instructions are sent to the dialogue generation module of the intelligent outbound call robot, and the dialogue emotion control parameters of the intelligent outbound call robot are adaptively adjusted.

[0051] As a further aspect of the present invention, the specific method for continuously determining the emotional state of the target person and continuously adjusting the dialogue emotion control parameters of the intelligent outbound call robot in the dynamic emotion interaction control module is as follows:

[0052] Continuously acquire the first human voice audio segment D1 and the subsequent human voice audio segments Dk in the sequence of human voice audio segments D1, D2, ..., Dj, where k is the counting index and 2≤k≤j;

[0053] Determine the average emotional feature value associated with the human voice audio segment Dk, denoted as EF_avg_k;

[0054] Based on the established control instruction mapping table, control instructions are continuously acquired to continuously adjust the dialogue emotion control parameters of the intelligent outbound call robot.

[0055] The beneficial effects of this invention are:

[0056] (1) This invention constructs a dynamic emotion recognition system based on human voice loudness and speech rate through real-time audio perception and multimodal emotion deconstruction technology. Its core advantage lies in realizing the continuous and accurate capture and adaptive control of the user's emotional state during outbound calls. Firstly, it generates preliminary control instructions through the first round of emotion analysis to ensure that emotional resonance is established in the early stage of the dialogue. Secondly, it forms a closed-loop feedback mechanism through the correlation analysis of continuous feature sequences, enabling the robot to dynamically adjust the dialogue strategy and significantly improve the naturalness of the interaction and user acceptance. This real-time and progressive emotion response mechanism effectively overcomes the mechanical and rigid drawbacks of traditional outbound call systems, and strengthens the ability of emotional communication while improving service efficiency.

[0057] (2) This invention records and processes human voice audio in real time and frames it, and combines short-time energy, zero-crossing rate and Gaussian mixture model to accurately detect human voice, which effectively improves the accuracy and robustness of voice activity frame determination and reduces environmental noise interference. Secondly, by aggregating continuous voice activity frames and automatically segmenting them when non-voice activity exceeds the threshold, the efficiency and rationality of audio segmentation are ensured, and a forced termination mechanism is introduced to avoid excessively long voice segments from affecting processing efficiency. In addition, the dynamic updating of human voice audio segment sequence maintains temporal correlation and realizes real-time tracking and optimization of the dialogue process. Based on this, the automation level of audio processing is enhanced, and the response speed and data analysis capabilities in intelligent outbound calling scenarios are improved.

[0058] (3) This invention extracts temporally correlated loudness and speech rate sequences from human voice audio, and innovatively performs correlation analysis on the normalized loudness and speech rate in a two-dimensional coordinate system to calculate the area of ​​the closed region as an emotion processing feature. This method effectively integrates the physical properties of sound with the dynamic changes over time, and can capture emotional fluctuations more comprehensively and precisely, significantly improving the accuracy and robustness of emotion recognition. Its advantage lies in overcoming the limitations of single feature analysis through multimodal feature fusion and mathematical modeling, and enhancing the system's ability to analyze complex emotional states.

[0059] (4) This invention extracts and analyzes the emotional features of the audio during the dialogue process, realizes the dynamic identification and classification of the target person's emotional state, and generates corresponding robot emotional control instructions based on the preset control instruction mapping table, thereby achieving adaptive interaction; its core is that it can continuously adjust the emotional expression parameters of the outbound robot according to the real-time emotional feedback of the target person, making the interaction process more emotionally intelligent and flexible, effectively improving the naturalness and affinity of human-computer dialogue, and enhancing communication efficiency and user experience; secondly, through continuous tracking and feedback control of the target person's emotional state, the dynamic optimization of the interaction strategy is realized. Attached Figure Description

[0060] The invention will now be further described with reference to the accompanying drawings.

[0061] Figure 1 This is a schematic diagram of the system described in this invention;

[0062] Figure 2 This is a flowchart illustrating the method described in Embodiment 2 of the present invention;

[0063] Figure 3 This is a flowchart illustrating the method described in Embodiment 3 of the present invention;

[0064] Figure 4 This is a flowchart illustrating the method described in Embodiment 4 of the present invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] Example 1

[0067] An intelligent outbound call robot dialogue control system integrating multimodal emotion recognition, such as... Figure 1 As shown, this system includes the following:

[0068] This system, an intelligent outbound call robot dialogue control system integrating multimodal emotion recognition, mainly consists of three modules: audio perception and structuring module, multimodal emotion feature deconstruction module, and dynamic emotion interaction control module.

[0069] The essence of this system is a real-time human-computer interaction feedback loop based on multimodal emotion computing. The core goal is to enable the intelligent outbound call robot to no longer mechanically broadcast preset scripts, but to "read between the lines" like a human, and dynamically adjust its tone, speaking speed, and content (summarized as dialogue emotion regulation parameters) according to the emotional state of the caller (i.e. the target person), thereby achieving more natural, more effective, and more humane communication.

[0070] In the audio perception and structuring module, the system records the voice audio associated with the target person during the dialogue between the intelligent outbound call robot and the target person in real time using a pre-built recording device. The system then segments the voice audio to determine the time-series-related voice audio segment sequence and updates the sequence in real time, as follows:

[0071] First, the target person's voice audio is acquired individually using a pre-built recording device, which includes active noise reduction functionality;

[0072] The recording device records human voice audio in an uninterrupted and continuously updated manner; therefore, the duration of the human voice audio increases over time.

[0073] Then, the human voice audio is segmented to obtain a sequence of human voice audio segments associated with time sequence. That is, the human voice audio is divided into several human voice audio segments, and then all human voice audio segments are sorted in chronological order, and the sorted result is recorded as the human voice audio segment sequence.

[0074] The purpose of this segmentation is to adapt to the fluidity of human emotions. By dividing continuous dialogue into temporal segments, it is to capture subtle changes in emotions. For example, when a target user says "I'll think about it," the first half of the sentence may be calm, while the second half may be spoken at a slower pace due to hesitation. Segmentation analysis can capture this subtle dynamic.

[0075] Secondly, real-time updates of the human voice audio segment sequence provide a data foundation for subsequent real-time control.

[0076] In the multimodal emotion feature deconstruction module, the human voice audio segments in the human voice audio segment sequence determined by the above module are decomposed to determine the human voice loudness sequence and human voice speech rate sequence associated with the corresponding human voice audio segment and the time sequence. The human voice loudness and human voice speech rate associated with the same human voice audio segment are analyzed to construct the emotion processing feature sequence.

[0077] This module can be understood as performing emotion decoding operations on the human voice audio segments in the human voice audio segment sequence in turn, transforming the original audio signal into quantifiable feature indicators that represent emotions, namely the human voice loudness sequence and the human voice speech rate sequence.

[0078] The purpose of selecting these two sound association parameters is:

[0079] Human voice loudness: the intensity of sound. Loudness usually increases when excited or angry, while it decreases when sad or frustrated.

[0080] Speech rate: the number of syllables (or words) spoken per unit of time. Speech rate increases when nervous or excited, and decreases when thinking, hesitating, or sad.

[0081] By fusing the parameters associated with two sounds, the accuracy and robustness of emotion recognition can be further improved, and false positives can be reduced.

[0082] In the dynamic emotion interaction control module, the first emotion processing feature sequence of the dialogue process is analyzed to determine the emotional state of the target person in this voice audio segment, and a preliminary control instruction is generated to control the dialogue emotion control parameters of the intelligent outbound call robot.

[0083] Then, by continuously acquiring emotion processing feature sequences associated with emotion processing features in human voice audio segments and performing correlation analysis, the emotional state of the target person is continuously determined, and the dialogue emotion regulation parameters of the intelligent outbound call robot are continuously adjusted. Specifically:

[0084] At the start of the conversation, the user's initial emotional baseline is determined based on the first valid emotional feature vector, which determines the starting point for the intelligent outbound call robot to interact with the target person.

[0085] Subsequently, by performing correlation analysis on the emotion processing feature sequences associated with the emotion processing features in the human voice audio segment sequence, the intelligent outbound call robot is continuously adjusted to achieve a global adaptive effect.

[0086] In this way, through the collaboration of the three modules, a feedback loop of perception-analysis-decision-execution-reperception is achieved, enabling the intelligent outbound call robot to adjust its behavior according to the emotional state of the target person, thereby successfully completing the outbound call task.

[0087] Example 2

[0088] This embodiment, based on Embodiment 1, further discloses the method for determining the sequence of human voice audio segments involved in the audio perception and structuring module, such as... Figure 2 As shown, it specifically includes the following:

[0089] Based on the content described in Embodiment 1, it can be seen that the audio perception and structuring module uses a recording device pre-built by the operator to record the human voice audio associated with the target person during the dialogue between the intelligent outbound call robot and the target person in real time and individually.

[0090] The real-time recorded human voice audio is then extracted, and the human voice audio is continuously segmented into frames starting from the first time point by the operator's preset frame interval T1. Each human voice audio frame obtained from the frame segmentation is sorted in chronological order to obtain a human voice audio frame sequence. It should be noted that the frame interval T1 is generally taken as a small value, such as 10 frames, while one frame is usually 20-30ms. The operator can make appropriate adjustments according to the actual situation.

[0091] Next, each human voice audio frame in the human voice audio frame sequence is obtained, and the short-time energy of each human voice audio frame is determined by calculating the sum of squares of the signal amplitude within the frame.

[0092] Then, the number of times the intra-frame signal of each human voice audio frame crosses the zero axis is calculated to determine the zero-crossing rate;

[0093] The short-time energy and zero-crossing rate mentioned are both covered by existing technologies. Then, the presence of human voice in the corresponding human voice audio frame can be determined by using a Gaussian mixture model (existing technologies can already achieve this function, and the audio perception and structuring module can directly integrate this function).

[0094] If a human voice is detected in any human voice audio frame, then this human voice audio frame is marked as a speech activity frame; otherwise, this human voice audio frame is marked as a non-speech activity frame.

[0095] Then, the consecutive speech activity frames are aggregated and merged into meaningful speech segments to represent a complete speech unit.

[0096] When consecutive voice activity frames are aggregated and a consecutive non-voice activity frame duration exceeds a first time threshold preset by the operator is detected, the current voice segment is determined to have ended, and the determined voice segment is treated as an independent human voice audio segment.

[0097] During the aggregation of continuous voice activity frames, if the duration of any voice segment exceeds the second time threshold preset by the operator, a forced termination operation is performed on the voice segment, and an independent human voice audio segment is generated to avoid the voice segment being too long and to ensure real-time processing efficiency.

[0098] At this point, independent human voice audio segments are generated. The first human voice audio segment determined during the dialogue between the intelligent outbound call robot and the target person is marked as D1. Then, the subsequent human voice audio segments are marked in chronological order and summarized. The summarized result is recorded as the human voice audio segment sequence, represented as: D1, D2, ..., Dj, where j represents the total number of human voice audio segments in the human voice audio segment sequence, which changes over time.

[0099] After a new human voice audio segment is generated, the generated human voice audio segment is inserted into the tail of the human voice audio segment sequence, and j is updated (incremented by one).

[0100] The core objective of this embodiment is to record the target person's voice audio in real time using a recording device. Then, by utilizing features such as short-time energy and zero-crossing rate, combined with a Gaussian mixture model, it accurately identifies the speech activity frames where human voices are present and aggregates them into independent human voice audio segments, avoiding misjudgments caused by prolonged silence or non-human voice interference. When generating each audio segment, the duration is automatically detected. If it exceeds a preset threshold, the generation of the independent audio segment is forcibly terminated, ensuring processing efficiency and real-time performance. Finally, these independent audio segments are summarized in chronological order to form a human voice audio segment sequence for subsequent speech analysis or dialogue optimization.

[0101] Example 3

[0102] This embodiment, based on Embodiment 2, further discloses a method for constructing emotion processing feature sequences based on human voice audio segment sequences in the multimodal emotion feature deconstruction module, such as... Figure 3 As shown, it specifically includes the following:

[0103] Based on the content described in Example 2, a sequence of human voice audio segments D1, D2, ..., Dj can be obtained. Any human voice audio segment is selected from it and labeled as Di, where i is a counting index with a value range of 1 to j. Example processing is performed on the human voice audio segment Di. The remaining human voice audio segments in the sequence of human voice audio segments D1, D2, ..., Dj are processed in the same way as follows. In the actual processing, the processing is carried out according to the order of time.

[0104] First, calculate the total duration (in seconds) of the time difference between the start and end times of the human voice audio segment Di, denoted as m seconds, where m seconds does not exceed the total duration (in seconds) of the second time threshold.

[0105] Next, based on the determined m seconds, the time series associated with the human voice audio segment Di is constructed, represented as: t1, t2, ..., tm, where t1 represents the first second, t2 represents the second second, and so on. The purpose of this step is to discretize the total duration m seconds into an integer time point sequence, which facilitates the alignment and analysis of time series features.

[0106] Then, the human voice audio segment Di is divided into m human voice audio segments of 1 second length according to the time sequence t1, t2, ..., tm. All human voice audio segments are sorted in chronological order and recorded as the human voice audio segment segment sequence, represented as: Di_t1, Di_t2, ..., Di_tm.

[0107] Determine any human voice audio segment from the determined sequence of human voice audio segments Di_t1, Di_t2, ..., Di_tm, denoted as Di_tn, where n is the counting index, with a value ranging from 1 to m.

[0108] Perform a text-to-text operation on the human voice audio segment Di_tn (based on ASR speech recognition technology), determine the total number of characters in the human voice audio segment Di_tn, and calculate the human voice speech rate SPn associated with the human voice audio segment Di_tn by using: total number of characters / 1 second = SPn.

[0109] By repeating the above steps, the speech rate associated with other voice audio segments in the human voice audio segment sequence Di_t1, Di_t2, ..., Di_tm can be determined. Then, the human voice speech rate sequence is obtained by sorting the segments according to the order of the human voice audio segment sequence Di_t1, Di_t2, ..., Di_tm, and is represented as SP1, SP2, ..., SPm.

[0110] Then, the average loudness (that is, the average decibel level of the voice volume) in the human voice audio segment Di_tn is calculated, and the average loudness is used as the human voice loudness associated with the human voice audio segment Di_tn, denoted as LON.

[0111] By repeating the above steps, the loudness of other human voice audio segments in the human voice audio segment sequence Di_t1, Di_t2, ..., Di_tm can be determined. Then, the loudness of the human voice can be sorted according to the order of the human voice audio segment sequence Di_t1, Di_t2, ..., Di_tm to obtain the human voice loudness sequence, represented as: LO1, LO2, ..., LOm.

[0112] By analogy, we can determine the human voice loudness sequence and human voice speech rate sequence associated with each human voice audio segment in the sequence D1, D2, ..., Dj.

[0113] Based on the above steps, obtain the human voice loudness sequence LO1, LO2, ..., LOM associated with any human voice audio segment Di in the human voice audio segment sequence D1, D2, ..., Dj, and the human voice speech rate sequence SP1, SP2, ..., SPm for example processing.

[0114] The loudness sequences LO1, LO2, ..., LOm and speech rate sequences SP1, SP2, ..., SPm associated with the human voice audio segment Di are normalized to eliminate the dimensional differences between loudness and speech rate, making them comparable. The normalization process uses the min-max normalization method, mapping each loudness and speech rate in the loudness sequences LO1, LO2, ..., LOm and speech rate sequences SP1, SP2, ..., SPm to the interval [0,1], resulting in the normalized loudness sequences LO1', LO2', ..., LOm' and speech rate sequences SP1', SP2', ..., SPm'.

[0115] Obtain the time series t1, t2, ..., tm determined in the above steps, extract any time tn in the time series t1, t2, ..., tm, and obtain the normalized human voice loudness LON' and normalized human voice speech rate SPn' associated with time tn.

[0116] Extract the numerical components of the normalized human voice loudness LON' and the normalized human voice speed SPn', and construct a two-dimensional coordinate system with the human voice loudness LON' value as the horizontal axis and the human voice speed SPn' value as the vertical axis.

[0117] Then, using the human voice loudness LON' as the horizontal axis and the human voice speed SPn' as the vertical axis, the two-dimensional coordinates formed by the normalized human voice loudness LON' and normalized human voice speed SPn' associated with time tn are labeled in the constructed two-dimensional coordinate system. Thus, a data point is obtained in the constructed two-dimensional coordinate system. Then, a straight line L1 perpendicular to the horizontal axis and parallel to the vertical axis is constructed through this data point, and then a straight line L2 perpendicular to the vertical axis and parallel to the horizontal axis is constructed.

[0118] At this point, lines L1 and L2 will form a closed region with the horizontal and vertical axes of the two-dimensional coordinate system. The area of ​​this closed region is calculated, and then this area value is used as the emotion processing feature associated with the human voice audio segment Di, labeled as EFn.

[0119] Similarly, based on the human voice loudness sequence LO1, LO2, ..., LOM and the human voice speech rate sequence SP1, SP2, ..., SPm, all the emotional processing features associated with the human voice audio segment Di can be determined and recorded as the emotional processing feature sequence in chronological order, denoted as: EF1, EF2, ..., EFm.

[0120] By repeating the above steps, the emotion processing feature sequence associated with each human voice audio segment in the sequence D1, D2, ..., Dj can be determined.

[0121] Example 4

[0122] This embodiment, based on embodiment 3, further discloses a method for adjusting the dialogue emotion control parameters of the intelligent outbound call robot in the dynamic emotion interaction control module, such as... Figure 4 As shown, it specifically includes the following:

[0123] Based on the content of Example 3, the sequence of human voice audio segments D1, D2, ..., Dj associated with the intelligent outbound call robot and the target person during the dialogue process, as well as the emotion processing feature sequence associated with any human voice audio segment;

[0124] The first human voice audio segment D1 and its associated emotion processing feature sequences EF1, EF2, ..., EFm in the human voice audio segment sequence D1, D2, ..., Dj are obtained and processed as follows:

[0125] First, calculate the arithmetic mean of the emotion processing feature sequences EF1, EF2, ..., EFm associated with the human voice audio segment D1, and record the calculated arithmetic mean as the average emotion feature value EF_avg.

[0126] Then, the average emotional feature value EF_avg is compared with the emotional state threshold table pre-constructed by the operator. The emotional state threshold table includes a first emotional state threshold Th1 and a second emotional state threshold Th2. The first emotional state threshold Th1 < the second emotional state threshold Th2, and both the first emotional state threshold Th1 and the second emotional state threshold Th2 are pre-set by the operator (determined based on the results of big data statistics).

[0127] If the average emotional feature value EF_avg is less than or equal to the first emotional state threshold Th1, then the target person's emotional state is marked as negative (e.g., sadness, grief).

[0128] If the average emotional feature value EF_avg is greater than the first emotional state threshold Th1 and less than the second emotional state threshold Th2, then the target person's emotional state is marked as neutral (e.g., calm, neutral).

[0129] If the average emotional feature value EF_avg is greater than or equal to the second emotional state threshold Th2, then the target person's emotional state is marked as positive (e.g., excitement, anger).

[0130] Next, based on the emotional state of the identified target person, the corresponding preliminary control instructions are obtained from the control instruction mapping table predefined by the operator. The control instruction mapping table defines the correspondence between emotional state and the emotional control parameters of the intelligent outbound call robot.

[0131] The control instruction mapping table is defined by the operator as a structured set of rules, defining the mapping relationship from the input user's emotional state to the set of dialogue emotion control parameters that the intelligent outbound call robot should adopt. Specifically:

[0132] By determining the target person's emotional state (including negative, neutral, and positive states), the system provides corresponding matching results. For example, when the target person is speaking rapidly due to anger, the intelligent outbound call robot can help guide the target person to calm down by slowing down their speech.

[0133] Among them, the dialogue emotion regulation parameters of the intelligent outbound call robot are all preset by the operator. This is the part that can be achieved by existing technologies, such as TTS technology based on Wavenet and GAN. This solution will not elaborate further.

[0134] Then, the determined preliminary control instructions are sent to the dialogue generation module of the intelligent outbound call robot to automatically adjust the dialogue emotion control parameters of the intelligent outbound call robot.

[0135] Next, continue to obtain the subsequent human voice audio segments Dk of the first human voice audio segment D1 in the sequence of human voice audio segments D1, D2, ..., Dj, where k is the counting index, and the value range is from 2 to j.

[0136] Then, the average emotional feature value EF_avg_k associated with the human voice audio segment Dk is determined to identify the emotional state of the target person. Based on the pre-determined control instruction mapping table, control instructions are continuously obtained, and the dialogue emotion control parameters of the intelligent outbound call robot are continuously controlled.

[0137] All data in the formulas described above are numerical calculations performed with dimensions removed. Furthermore, any content not described in detail in this specification is existing technology known to those skilled in the art.

[0138] This embodiment aims to achieve real-time perception and dynamic response of the target person's emotions by an intelligent outbound call robot. By extracting the emotional processing feature sequence of human voice, calculating the average emotional feature value and comparing it with a preset threshold, it automatically determines whether the target person is in a negative, neutral, or positive state. Based on this, it retrieves the corresponding control instructions from the mapping table and adjusts the robot's speech rate, pitch, and other dialogue parameters in real time, forming a closed-loop feedback of "emotion recognition - strategy matching - parameter control". Its technical effect is to enable the intelligent outbound call robot to have emotional empathy capabilities similar to human customer service, and can continuously soothe anger, alleviate sadness, or maintain calmness during the conversation, reduce user resistance, improve service experience and communication success rate, while reducing human intervention and achieving efficient and personalized intelligent outbound calls.

[0139] The above description is merely an example and illustration of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the invention or exceed the scope defined in the claims, all of which should fall within the protection scope of the present invention.

[0140] It should be stated that all user data collected in this application was collected with the user's consent and authorization. Furthermore, the uses of user data are legal and compliant, and the use and processing of user data comply with the relevant laws, regulations, and standards of the relevant regions.

Claims

1. An intelligent outbound call robot dialogue control system integrating multimodal emotion recognition, characterized in that, The system includes: The audio perception and structuring module records the human voice audio associated with the target person during the dialogue between the intelligent outbound call robot and the target person in real time based on a pre-built recording device. It segments the human voice audio to determine the human voice audio segment sequence D1, D2, ..., Dj associated with the time sequence, where j represents the total number of human voice audio segments in the human voice audio segment sequence. The human voice audio segment sequence is updated in real time. The multimodal emotion feature decomposition module decomposes the human voice audio segments in the human voice audio segment sequence, determines the corresponding human voice loudness sequence and human voice speech rate sequence associated with the time sequence of the human voice audio segment, and performs correlation analysis on the human voice loudness and human voice speech rate associated with the same human voice audio segment. The specific method for constructing the emotion processing feature sequence is as follows: Determine the human voice loudness sequence LO1, LO2, ..., LOm and the human voice speech rate sequence SP1, SP2, ..., SPm associated with any human voice audio segment Di in the human voice audio segment sequence D1, D2, ..., Dj, where m is the total duration of the human voice audio segment Di, representing m seconds; Normalization is performed on the human voice loudness sequence and the human voice speech rate sequence to obtain the normalized human voice loudness sequence LO1',LO2',...,LOm' and the normalized human voice speech rate sequence SP1',SP2',...,SPm'. The normalization process adopts the minimum-maximum normalization method to map the sequence values ​​to the [0,1] interval. For any time tn in the time series t1, t2, ..., tm, take the normalized human voice loudness LON' and normalized human voice speech rate SPn' associated with this time, where n is the counting index, 1≤n≤m; Determine the values ​​of human voice loudness LON' and human voice speech rate SPn'; Construct a two-dimensional coordinate system with the human voice loudness LON' as the horizontal axis and the human voice speech rate SPn' as the vertical axis. Mark the human voice loudness LON' and human voice speech rate SPn' as the horizontal and vertical axes in the constructed two-dimensional coordinate system to obtain a data point. Construct a straight line L1 perpendicular to the horizontal axis and parallel to the vertical axis and a straight line L2 perpendicular to the vertical axis and parallel to the horizontal axis through this data point. Determine the area value of the closed region formed by the straight line L1, the straight line L2, the horizontal axis, and the vertical axis as the emotion processing feature associated with the human voice audio segment Di, and label it as EFn. Similarly, determine the emotion processing feature sequence EF1,EF2,...,EFm associated with the human voice audio segment Di; By analogy, determine the emotion processing feature sequence associated with each human voice audio segment in the human voice audio segment sequence D1, D2, ..., Dj; The dynamic emotion interaction control module analyzes the first emotion processing feature sequence in the dialogue process, determines the emotional state associated with the target person and the audio segment of the voice, and generates preliminary control instructions to regulate the dialogue emotion control parameters of the intelligent outbound call robot. Then, by continuously acquiring the emotion processing feature sequences associated with the emotion processing features in the human voice audio segment sequence, and performing correlation analysis, the emotional state of the target person is continuously determined, and the dialogue emotion regulation parameters of the intelligent outbound call robot are continuously adjusted.

2. The system according to claim 1, characterized in that, The specific method for segmenting human voice audio in the audio perception and structuring module is as follows: Using pre-built recording equipment, the audio of the target person's voice during the conversation between the intelligent outbound calling robot and the target person can be recorded individually and in real time. The human voice audio is continuously framed at a preset frame interval T1 to generate a temporally correlated human voice audio frame sequence. Calculate the short-time energy and zero-crossing rate associated with each human voice audio frame in the human voice audio frame sequence, and determine whether there is a human voice in the human voice audio frame based on the Gaussian mixture model; If a human voice is detected in any human voice audio frame, then this human voice audio frame is marked as a speech activity frame; otherwise, it is marked as a non-speech activity frame. The continuous speech activity frames are aggregated into a single speech segment. When a non-speech activity frame is detected to continuously exceed a preset first time threshold, the current speech segment is terminated, and an independent human voice audio segment is generated.

3. The system according to claim 2, characterized in that, In the audio perception and structuring module, if the duration of any speech segment exceeds a preset second time threshold, the speech segment is forcibly terminated, and an independent human voice audio segment is generated.

4. The system according to claim 2, characterized in that, In the audio perception and structuring module, the specific method for determining the time-series-related human voice audio segment sequence and updating the human voice audio segment sequence in real time is as follows: Obtain the first human voice audio segment during the dialogue and label it D1; Obtain subsequent human voice audio segments in chronological order, and summarize them together with human voice audio segment D1, denoted as human voice audio segment sequence D1, D2, ..., Dj; When a new human voice audio segment is detected, it is inserted at the end of the human voice audio segment sequence, and j is updated.

5. The system according to claim 4, characterized in that, In the multimodal emotion feature deconstruction module, the specific method for determining the corresponding human voice audio segment and the time-series associated human voice loudness sequence and human voice speech rate sequence is as follows: Extract any human voice audio segment Di from the sequence of human voice audio segments D1, D2, ..., Dj, where i is the counting index and 1 ≤ i ≤ j; Construct the time series t1, t2, ..., tm associated with the human voice audio segment Di, where m seconds is the total duration not exceeding the second time threshold; The human voice audio segment Di is divided into time series t1, t2, ..., tm to obtain the human voice audio segment segment sequence Di_t1, Di_t2, ..., Di_tm; Determine any human voice audio segment Di_tn in the human voice audio segmentation sequence; Perform a text conversion operation on the human voice audio segment Di_tn, determine the total number of text characters, and calculate the human voice speech rate SPn associated with the human voice audio segment Di_tn using the formula: total number of text characters / 1 second = SPn. Similarly, determine the speech rate associated with the remaining audio segments and arrange them in chronological order, denoted as the speech rate sequence SP1, SP2, ..., SPm associated with audio segment Di; Determine the average loudness in the human voice audio segment Di_tn, and use it as the human voice loudness LOn associated with the human voice audio segment Di_tn; Similarly, determine the loudness of the remaining human voice audio segments and arrange them in chronological order, denoted as the loudness sequence LO1, LO2, ..., LOm associated with the human voice audio segment Di; By analogy, determine the human voice loudness sequence and human voice speech rate sequence associated with each human voice audio segment in the human voice audio segment sequence D1, D2, ..., Dj.

6. The system according to claim 5, characterized in that, In the dynamic emotion interaction control module, the specific method for generating preliminary control instructions to regulate the dialogue emotion control parameters of the intelligent outbound call robot is as follows: Extract the emotion processing feature sequence EF1,EF2,...,EFm associated with the first human voice audio segment D1 in the dialogue process; Calculate the arithmetic mean of the emotion processing feature sequence, denoted as the average emotion feature value EF_avg; The average emotional feature value EF_avg is compared with a pre-constructed emotional state threshold table, which includes a first emotional state threshold Th1 and a second emotional state threshold Th2, where Th1 < Th2. If EF_avg≤Th1, the target person's emotional state is determined to be negative. If Th1 < EF_avg < Th2, the target person's emotional state is determined to be neutral. If EF_avg≥Th2, the target person's emotional state is determined to be positive. Based on the determined emotional state, the corresponding preliminary control instructions are obtained from the control instruction mapping table predefined by the operator, wherein the control instruction mapping table defines the correspondence between the emotional state and the dialogue emotional control parameters of the intelligent outbound call robot. The initial control instructions are sent to the dialogue generation module of the intelligent outbound call robot, and the dialogue emotion control parameters of the intelligent outbound call robot are adaptively adjusted.

7. The system according to claim 6, characterized in that, In the dynamic emotion interaction control module, the specific method for continuously determining the emotional state of the target person and continuously controlling the dialogue emotion control parameters of the intelligent outbound call robot is as follows: Continuously acquire the first human voice audio segment D1 and the subsequent human voice audio segments Dk in the sequence of human voice audio segments D1, D2, ..., Dj, where k is the counting index and 2≤k≤j; Determine the average emotional feature value associated with the human voice audio segment Dk, denoted as EF_avg_k; Based on the established control instruction mapping table, control instructions are continuously acquired to continuously adjust the dialogue emotion control parameters of the intelligent outbound call robot.

Images