A method and system for analyzing multi-modal negotiation of mediation services

Abstract

The application discloses a mediation business multi-modal negotiation analysis method and system, and relates to the technical field of data processing. In the method, first, synchronized voice signals and facial image data are acquired and aligned to construct an initial sequence; voice features are extracted to identify voice instability events and trigger time stamps; according to this, facial image segments are intercepted, geometric features are extracted, and facial tension indexes are calculated to identify facial tension events; based on the time sequence coupling relationship and the tension indexes of the two, emotional suppression states and coupling strengths are determined; facial and voice features, suppression states and coupling strengths are fused to calculate comprehensive emotional evaluation scores; further, emotional dynamic prediction sequences are generated, and finally emotional risk judgment results are output and mediation communication rhythm parameters are dynamically adjusted. The technical scheme provided by the application can improve the accuracy of identifying emotional states in the mediation process.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically a method and system for multimodal negotiation analysis in mediation business. Background Technology

[0002] With the rapid development of video encoding and decoding technology and the deepening application of intelligent video analysis, video decoding technology is increasingly widely used in judicial practice, labor arbitration, and commercial dispute mediation. In the recording and analysis of mediation videos, decoding video data allows for the extraction of multimodal information such as facial expressions and body language, thereby assessing the parties' emotional state. Research indicates that emotional information contained in mediation scene videos is associated with various risk factors. Emotional suppression, as a common psychological defense mechanism, shows an increasing potential risk of eruption in mediation practice. The formation of emotional suppression involves multiple factors, including psychological defense, social expectations, and the communication environment. Sustained psychological stress during mediation, particularly the long-term separation between emotional expression and inner feelings, is considered a significant risk trigger. Existing video decoding technologies, when processing mediation scene videos, primarily focus on video data compression and reconstruction, with less consideration given to the targeted extraction of emotion-related features and the effective analysis of risk information. Therefore, establishing an effective emotional state monitoring and mediation risk early warning system based on video decoding technology is crucial for the smooth conduct of mediation work.

[0003] Currently, emotional risk warnings in mediation scenarios primarily involve collecting voice signals and facial image data, using emotion recognition algorithms to assess the emotional state of the parties involved, and combining this with emotion classification results for risk assessment. This method determines the emotion category by analyzing changes in voice and facial expression features during monitoring, and is relatively convenient to operate, thus finding some application in the field of intelligent mediation assistance.

[0004] However, with the increasing demand for precise mediation and personalized communication strategies, the limitations of existing early warning methods are becoming increasingly apparent. In practical applications, due to the complexity and concealment of emotional expressions in mediation scenarios, and the fact that the evolution of emotional risk is the result of dynamic interaction of multimodal information, existing methods lack effective mechanisms for tracking inconsistencies in emotional expressions and the dynamic evolution of risk, often leading to biased risk assessment results. This assessment bias results in insufficient accuracy in predicting the degree of risk, thereby reducing the accuracy of identifying emotional states during the mediation process. Summary of the Invention

[0005] To address the above problems, this invention provides a multimodal negotiation analysis method and system for mediation services, which solves the problem of inaccurate judgment of the dynamic evolution trend of emotions due to bias in the assessment of emotional risk in mediation scenarios in the prior art, and can improve the accuracy of identifying emotional states during the mediation process.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: Acquire synchronized speech signals and facial image data in the mediation environment, and construct an initial speech sequence and an initial facial image sequence after aligning the timestamps of the synchronized speech signals and the facial image data. Extract the speech feature set of the initial speech sequence, identify speech instability events based on the energy features and intonation features in the speech feature set, and extract the trigger timestamps; Based on the trigger timestamp, extract the corresponding set of facial image segments from the initial facial image sequence; Extract facial geometric features from the set of facial image segments, calculate the facial tension index based on the facial geometric features, and identify facial tension events and their corresponding timestamps based on the facial tension index; Based on the trigger timestamp and the occurrence timestamp, the temporal coupling relationship between the voice instability event and the facial tension event is determined, and the emotional suppression state and coupling strength are determined in combination with the facial tension index. A comprehensive emotion assessment score is calculated by integrating the facial geometric features, the emotional suppression state, the coupling strength, and the speech temporal change features in the speech feature set. Based on the comprehensive emotion assessment score, an emotion dynamic prediction sequence is generated, and the emotion risk judgment result is output according to the emotion dynamic prediction sequence, and the mediation communication rhythm parameters are adjusted.

[0007] By employing the aforementioned technical solution, a unified temporal reference for multimodal data is established by constructing synchronized initial speech sequences and initial facial image sequences through timestamp alignment. This enables subsequent precise temporal correlation analysis of speech signals and facial image data. Based on this, the invention identifies speech instability events and extracts trigger timestamps based on energy and intonation features. According to the trigger timestamps, corresponding facial image fragment sets are extracted from the initial facial image sequence, achieving directional facial data acquisition triggered by speech emotion fluctuations. This avoids indiscriminate analysis of facial data across all time periods and improves the efficiency of capturing key emotional events. Furthermore, the invention identifies facial tension events and their corresponding occurrence timestamps by calculating a facial tension index. Based on the trigger timestamps and occurrence timestamps, the temporal coupling relationship between speech instability events and facial tension events is determined. This effectively captures the inconsistencies between the individual's speech expression and facial expressions. The establishment of this temporal coupling relationship overcomes the limitations of existing technologies that analyze each modality separately, enabling the system to identify emotional suppression states where the individual is deliberately controlling their outward expression but experiencing strong internal emotional fluctuations. By combining facial tension index to determine the state of emotional suppression and coupling strength, this invention can not only determine whether emotional suppression exists, but also quantify the degree of suppression, providing a more accurate basis for subsequent risk assessment. Based on this, the invention integrates facial geometric features, emotional suppression state, coupling strength, and speech temporal change features to calculate a comprehensive emotion assessment score. This integrates multi-dimensional emotional information into a unified assessment index, and generates a dynamic emotion prediction sequence based on the comprehensive emotion assessment score. This enables effective tracking of the dynamic evolution of emotional risk, and can predict the trend of emotions transitioning from a suppressed state to an explosive state. This solves the assessment bias problem caused by the lack of an effective tracking mechanism for inconsistent emotional expression and the dynamic evolution of risk in existing technologies, improving the accuracy of identifying emotional states during mediation.

[0008] Optionally, during a preset mediation silence period, voice signals and facial images are simultaneously acquired as reference data; the timbre reference value and the statistical mean of the sampling point coordinates of the reference data are extracted to construct a neutral reference parameter set; the synchronous voice signal during the mediation process is converted from analog to digital using a preset sampling frequency, and the facial image data is timestamped in conjunction with the neutral reference parameter set to obtain the initial voice sequence and the initial facial image sequence corresponding to the time.

[0009] Optionally, the initial speech sequence is segmented into frames, the peak energy intensity of each frame is extracted as the energy feature, the fundamental frequency fluctuation amplitude is extracted as the pitch feature, and the speech rate change rate between adjacent frames is calculated as the speech temporal change feature, thus forming the speech feature set; the deviation of the pitch feature of the target frame from the average pitch feature in the historical stable period is calculated; when the energy feature of the target frame is detected to be greater than a preset energy threshold, and the deviation is greater than a preset fundamental frequency fluctuation threshold, the target frame is marked as the speech instability event, and the time node corresponding to the target frame is recorded as the trigger timestamp.

[0010] Optionally, by combining the neutral reference parameter set, the eyebrow raising angle, mouth opening and closing amplitude, and eye squinting degree of each frame in the facial image segment set are extracted and combined into the facial geometric features; the indicators in the facial geometric features are weighted and calculated to obtain the facial tension index; the number of consecutive frames in which the facial tension index is greater than a preset tension threshold is counted; when the number of consecutive frames multiplied by the single frame duration is greater than a preset duration threshold, it is determined as the facial tension event; the time node corresponding to the image frame in which the facial tension index reaches the target peak is recorded as the occurrence timestamp.

[0011] Optionally, the time difference between the trigger timestamp and the occurrence timestamp is calculated; when the absolute value of the time difference is less than or equal to a preset time window threshold, the temporal coupling relationship is determined to be in a synchronous coupling state; when the absolute value of the time difference is greater than the preset time window threshold, if the time difference is positive, the temporal coupling relationship is determined to be in a delayed coupling state, and if the time difference is negative, the temporal coupling relationship is determined to be in a premature coupling state; the facial tension index corresponding to the occurrence timestamp is extracted, and the facial tension index is divided by the sum of the absolute value of the time difference and the duration of a single frame to calculate the coupling strength; when the temporal coupling relationship is in the delayed coupling state and the coupling strength is greater than a preset suppression threshold, the current object is determined to be in the emotional suppression state.

[0012] Optionally, an emotion suppression coefficient is calculated based on the frequency of occurrence of the emotion suppression state; the corresponding emotion weight is determined according to the temporal coupling relationship, and the cumulative emotion pressure index is calculated in combination with the coupling strength; the speech emotion activation amount is calculated based on the speech temporal change features in the speech feature set, and the facial expression emotion activation amount is calculated based on the facial geometric features; the emotion suppression coefficient, the cumulative emotion pressure index, the speech emotion activation amount and the facial expression emotion activation amount are weighted and fused to obtain the comprehensive emotion evaluation score.

[0013] Optionally, the comprehensive emotion assessment score within the historical time window is extracted to construct an emotion assessment historical trajectory; the fluctuation slope and acceleration of the emotion assessment historical trajectory at the current moment are calculated; the preset time period is divided into multiple consecutive prediction time steps; for each prediction time step, the comprehensive emotion assessment score at the current moment, the product of the fluctuation slope and the prediction time step, and the product of the acceleration and the square of the prediction time step are added together to calculate the predicted emotion score at the corresponding moment; the predicted emotion scores corresponding to each prediction time step are arranged in ascending order of time to form the emotion dynamic prediction sequence.

[0014] Optionally, the emotional stress accumulation rate is calculated based on the emotional dynamic prediction sequence; the comprehensive emotional assessment score, the emotional suppression coefficient, and the emotional stress accumulation rate are input into a preset emotional risk classification mapping table, and the current emotional risk level is matched and output as the emotional risk judgment result; based on the emotional risk level, combined with the speech temporal change characteristics and the change trend of the facial tension index, a mediation communication rhythm parameter including the speaking frequency adjustment value, communication buffer time, and intervention response interval is generated.

[0015] Optionally, the baseline emotional suppression coefficient and baseline emotional pressure accumulation rate at the moment the mediation communication rhythm parameter is generated are recorded; if the emotional dynamic prediction sequence is detected to show a continuous upward trend, the voice feature set of the current time window is extracted and the comprehensive emotional assessment score is updated; the target emotional risk level is determined based on the updated comprehensive emotional assessment score; if the emotional suppression coefficient of the current time window is greater than the baseline emotional suppression coefficient and the emotional pressure accumulation rate of the current time window is greater than the baseline emotional pressure accumulation rate, the adjustment range of the mediation communication rhythm parameter is increased by a preset step size.

[0016] Secondly, embodiments of this application provide a mediation business multimodal negotiation analysis system, the mediation business multimodal negotiation analysis system comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the mediation business multimodal negotiation analysis system to perform the method as described in the first aspect and any possible implementation thereof.

[0017] In summary, one or more technical solutions provided in this application have at least the following technical effects or advantages: By constructing synchronized initial speech and facial image sequences through timestamp alignment, a unified temporal benchmark for multimodal data is established, enabling precise temporal correlation analysis of speech signals and facial image data. Based on this, the invention identifies speech instability events and extracts trigger timestamps based on energy and intonation features. According to the trigger timestamps, corresponding facial image fragment sets are extracted from the initial facial image sequence, achieving targeted facial data acquisition triggered by speech emotion fluctuations. This avoids indiscriminate analysis of facial data across all time periods and improves the efficiency of capturing key emotional events. Furthermore, the invention identifies facial tension events and their corresponding occurrence timestamps by calculating a facial tension index. Based on the trigger and occurrence timestamps, the temporal coupling relationship between speech instability events and facial tension events is determined. This effectively captures the inconsistencies between the individual's speech expression and facial expressions. The establishment of this temporal coupling relationship overcomes the limitations of existing technologies that analyze each modality separately, enabling the system to identify emotional suppression states where the individual is deliberately controlling their outward expression but experiencing strong internal emotional fluctuations. By combining facial tension index to determine the state of emotional suppression and coupling strength, this invention can not only determine whether emotional suppression exists, but also quantify the degree of suppression, providing a more accurate basis for subsequent risk assessment. Based on this, the invention integrates facial geometric features, emotional suppression state, coupling strength, and speech temporal change features to calculate a comprehensive emotion assessment score. This integrates multi-dimensional emotional information into a unified assessment index, and generates a dynamic emotion prediction sequence based on the comprehensive emotion assessment score. This enables effective tracking of the dynamic evolution of emotional risk, and can predict the trend of emotions transitioning from a suppressed state to an explosive state. This solves the assessment bias problem caused by the lack of an effective tracking mechanism for inconsistent emotional expression and the dynamic evolution of risk in existing technologies, improving the accuracy of identifying emotional states during mediation. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a multimodal negotiation analysis method for mediation services disclosed in an embodiment of this application; Figure 2 This is another flowchart illustrating a multimodal negotiation analysis method for mediation services disclosed in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a system provided in an embodiment of this application.

[0019] In the diagram: 301, Central Processing Unit; 302, Read-Only Memory; 303, Random Access Memory; 304, Bus; 305, Input / Output Interface; 306, Input Section; 307, Output Section; 308, Storage Section; 309, Communication Section; 310, Driver; 311, Removable Media. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solution, the present invention will be described in detail below with reference to embodiments. The description in this part is only exemplary and explanatory, and should not be used to limit the scope of protection of the present invention in any way.

[0021] It should be noted that, in this document, the terms "comprising," "including," and any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Specific examples have been used in this document to illustrate the principles and implementation methods of the present invention. These examples are merely for the purpose of helping to understand the method and core ideas of the present invention. The above are only preferred embodiments of the present invention. It should be pointed out that, due to the limitations of written expression and the objective existence of infinite specific structures, those skilled in the art can make several improvements, modifications, or variations without departing from the principles of the present invention, and can also combine the above technical features in an appropriate manner. These improvements, modifications, variations, or combinations, or the direct application of the concept and technical solution of the present invention to other situations without modification, should all be considered within the scope of protection of the present invention.

[0022] This application provides a multimodal negotiation analysis method for mediation services, referring to... Figure 1 , Figure 1 This is a flowchart illustrating a multimodal negotiation analysis method for mediation services provided in an embodiment of this application. The method is applied to a system, which refers to a hardware and software integrated platform capable of executing a multimodal negotiation analysis program for mediation services. The system can execute a multimodal negotiation analysis program for mediation services. The method includes steps 101 to 107, as follows: Step 101: Obtain synchronized speech signals and facial image data in the mediation environment, and construct the initial speech sequence and initial facial image sequence after aligning the timestamps of the synchronized speech signals and facial image data.

[0023] In this embodiment, the mediation environment refers to a specific location where dispute mediation activities are conducted, such as a court mediation room, a labor arbitration room, or a community mediation center—a physical space equipped with audio and video capture equipment. Synchronized voice signal refers to the real-time audio data of participants' speech collected during the mediation process, including information such as speaking content, tone, and volume. Facial image data refers to visual information such as facial expressions and facial muscle changes obtained through camera equipment. A timestamp represents the precise time point at which each voice segment and facial image was captured, used to achieve time synchronization between different data sources. The initial voice sequence refers to the set of original voice data segments arranged in chronological order. The initial facial image sequence refers to the set of original facial image data arranged in chronological order.

[0024] Specifically, voice signals and facial image data during the mediation process are collected using a microphone array and a high-definition camera set up in the mediation room. The microphone array collects the voice signals of all parties involved in the mediation process, with a sampling rate set to 16kHz and a sampling precision of 16bit; the high-definition camera captures facial images of the parties at a frequency of 25 frames per second. For the collected raw data, their respective time information is first extracted. Since there may be slight time differences in the acquisition from different devices, timestamp alignment processing is required. The alignment process calculates the acquisition time difference between the two types of data and performs time interpolation or downsampling operations on the facial image data to ensure that a corresponding facial image can be found for each voice segment. After time alignment is completed, the voice signals are organized into an initial voice sequence in chronological order, and the facial images are similarly organized into an initial facial image sequence in chronological order, establishing a one-to-one correspondence between the two data sequences.

[0025] In one possible implementation, synchronized speech signals and facial image data in the mediation environment are acquired, and an initial speech sequence and an initial facial image sequence are constructed after aligning the timestamps of the synchronized speech signals and facial image data. Specifically, this includes steps 1011-1012, as follows: Step 1011: During the preset mediation silence period, simultaneously collect voice signals and facial images as reference data; extract the timbre reference value and the statistical mean of the sampling point coordinates of the reference data to construct a neutral reference parameter set.

[0026] Specifically, a pre-set 30-second silent period is arranged before the formal commencement of mediation, requiring participants to remain relaxed. During this time, participants' voice signals are captured using a microphone at a frequency of 16kHz, while facial images are captured by a high-definition camera at a rate of 25 frames per second. The captured voice signals are first preprocessed, including background noise removal and signal normalization. Then, a short-time Fourier transform is used to frame the voice signal, with each frame length set to 25 milliseconds and a frame shift of 10 milliseconds. Next, the fundamental frequency, the first three formant frequencies, and the spectral envelope features are extracted from each frame. These feature values ​​from all frames during the silent period are averaged to obtain the timbre baseline value. For the facial images, a facial landmark detection algorithm is first applied to each frame to locate the coordinates of 68 facial landmarks. Then, the average value of the coordinates of each landmark in all frames during the silent period is calculated to obtain the statistical mean of the sampled point coordinates. Finally, the timbre baseline value and the statistical mean of the sampled point coordinates are combined to construct a neutral baseline parameter set, providing a reference benchmark for subsequent emotion change analysis.

[0027] Step 1012: The synchronous speech signal during the mediation process is converted from analog to digital using a preset sampling frequency, and the facial image data is timestamped in combination with the neutral reference parameter set to obtain the initial speech sequence and the initial facial image sequence corresponding to the time.

[0028] Specifically, during the mediation process, a preset sampling frequency of 16kHz is used to collect synchronous speech signals from the environment to ensure sufficient frequency information in the speech signal. The collected analog speech signal is converted from analog to digital using 16-bit quantization precision. The conversion process includes three steps: sampling, quantization, and encoding. First, the continuous speech signal is discretely sampled according to the sampling frequency, extracting 16,000 sampling points per second from the continuous signal. Then, the amplitude of each sampling point is quantized, mapping it to one of 65,536 possible discrete values. Finally, the quantization result is converted into binary digital encoding to obtain the digitized speech signal. Simultaneously, the real-time acquired facial image data is processed, marking each frame with a precise acquisition timestamp. Since the speech signal and facial image data are acquired by different devices, there is a time synchronization discrepancy, requiring timestamp alignment. The alignment process first calculates the time correspondence between speech sampling points and image frames, calibrating by comparing the acquisition time difference between the two types of data. Using a constructed neutral reference parameter set as a reference, the mapping function between the image time series and the speech time series is calculated. Based on this mapping function, the timestamps of the facial image data are adjusted to precisely correspond with the timestamps of the speech signal. After timestamp alignment, the speech data is organized into an initial speech sequence in chronological order, and the facial images are organized into an initial facial image sequence in chronological order, ensuring that each element in the sequence corresponds completely in time, which facilitates subsequent multimodal emotion analysis.

[0029] Step 102: Extract the speech feature set of the initial speech sequence, identify speech instability events based on the energy features and intonation features in the speech feature set, and extract the trigger timestamps.

[0030] In this embodiment, the speech feature set refers to the combination of multidimensional acoustic feature parameters extracted from the initial speech sequence, used to characterize various attributes of speech. Energy features represent the energy distribution of the speech signal in the time domain, reflecting changes in the speaker's volume. Pitch features refer to the fundamental frequency of the speech and its variations, representing changes in the pitch of the speaker's voice. Speech instability events refer to the time periods during dialogue where speech features exhibit abnormal fluctuations, indicating that the speaker may be in a state of emotional fluctuation. The trigger timestamp represents the precise time point at which the speech instability event was detected, serving as a time anchor point for subsequent analysis.

[0031] Specifically, the initial speech sequence is segmented into frames, with each frame length set to 25 milliseconds and a frame shift set to 10 milliseconds to ensure sufficient overlap between adjacent frames to capture continuous speech changes. For each frame of speech signal, its short-time energy value is calculated as an energy feature by normalizing the sum of squares of all sampling points within the frame. Simultaneously, the fundamental frequency value of each frame is extracted as a pitch feature using autocorrelation or cepstral analysis, and the rate of change of speech rate between adjacent frames is calculated, i.e., the degree of acceleration or deceleration of speech rhythm between adjacent frames. These features together constitute the speech feature set. When identifying speech instability events, the ratio of the energy feature of the current target frame to the energy feature of recent historical frames is calculated. When this ratio exceeds a preset energy threshold, it is simultaneously detected whether the pitch feature of the target frame deviates from the average pitch feature of historical stable periods to a preset fundamental frequency fluctuation threshold, such as 20%. If both conditions are met simultaneously, the target frame is marked as a speech instability event, and the corresponding time point of the frame is recorded as a trigger timestamp.

[0032] In one possible implementation, a speech feature set of the initial speech sequence is extracted. Based on the energy features and intonation features in the speech feature set, speech instability events are identified and trigger timestamps are extracted. Specifically, this includes steps 1021-1023, as follows: Step 1021: Perform frame segmentation on the initial speech sequence, extract the peak energy intensity of each frame as energy feature, extract the fundamental frequency fluctuation amplitude as pitch feature, and calculate the speech rate change rate between adjacent frames as speech temporal change feature to form a speech feature set.

[0033] Specifically, the initial speech sequence is first processed by frame segmentation, using a fixed frame length of 25 milliseconds and a frame shift of 10 milliseconds to ensure a 15-millisecond overlap between adjacent frames, facilitating the capture of continuous speech variation features. For each frame of speech signal, the maximum value of the sum of squared amplitudes of all sampling points within the frame is calculated as the peak energy intensity of that frame, i.e., the energy feature. Simultaneously, the periodicity of each frame signal is analyzed using the autocorrelation method to extract the fundamental frequency value. The difference between the maximum and minimum values ​​of the fundamental frequency within five consecutive frames is calculated as the fundamental frequency fluctuation amplitude, i.e., the pitch feature. Furthermore, the speech rate variation rate is obtained by calculating the zero-crossing rate change ratio between two adjacent frames, serving as the speech temporal variation feature. These three features are organized according to frame correspondence to form a complete speech feature set, where each frame of speech corresponds to a set of three-dimensional feature vectors for subsequent speech emotion analysis processing.

[0034] Step 1022: Calculate the deviation of the intonation features of the target frame from the mean of intonation features during historical stable periods.

[0035] Specifically, the first step is to determine the range of historically stable periods, selecting a 3-second speech segment preceding the target frame, roughly corresponding to 300 speech frames. From these historical frames, frames with relatively stable energy characteristics are selected, i.e., those frames whose energy variation is less than 20% of the historical maximum energy, forming the historically stable frame sample set. The arithmetic mean of the intonation features of these historically stable frames is calculated, which is obtained by adding the fundamental frequency fluctuations of all historically stable frames and dividing by the number of frames. Then, the difference between the intonation features of the target frame and this mean is calculated, and the difference is divided by the intonation feature mean to obtain the normalized deviation. This deviation value reflects the degree of change of the intonation features of the target frame relative to the normal state; the larger the value, the more abnormal the intonation change, providing a quantitative indicator for subsequent judgment of speech instability events.

[0036] Step 1023: When the energy characteristics of the target frame are detected to be greater than the preset energy threshold and the deviation is greater than the preset baseband fluctuation threshold, the target frame is marked as a speech instability event, and the time node corresponding to the target frame is recorded as the trigger timestamp.

[0037] Specifically, firstly, preset energy thresholds and preset fundamental frequency fluctuation thresholds are determined. The energy threshold is set to 1.5 times the average energy feature value within historical stable periods, and the fundamental frequency fluctuation threshold is set to 0.3, indicating that a deviation exceeding 30% is considered significant fluctuation. For the target frame being processed, its energy feature is compared with the preset energy threshold. Simultaneously, it is checked whether the calculated deviation exceeds the preset fundamental frequency fluctuation threshold. When the energy feature of the target frame is greater than the preset energy threshold, and the deviation is also greater than the preset fundamental frequency fluctuation threshold, the frame is considered to meet the criteria for a speech instability event. This target frame is marked as a speech instability event, meaning that an abnormal fluctuation in speech emotion was detected at that time point. Simultaneously, the corresponding time information of the target frame, i.e., the time position of the frame in the initial speech sequence, is extracted and recorded as a trigger timestamp. This trigger timestamp serves as a time marker for speech emotion changes, used for subsequent temporal correlation analysis with facial expression changes, providing a time synchronization point for multimodal emotion recognition.

[0038] Step 103: Extract the corresponding set of facial image segments from the initial facial image sequence based on the trigger timestamp.

[0039] In this embodiment, the facial image fragment set refers to a series of time-related facial image frames extracted from the initial facial image sequence based on the trigger timestamp of the speech instability event, used to analyze the facial expression changes corresponding to changes in speech emotion. These image fragments typically include consecutive facial images within a certain time window before and after the trigger timestamp, ensuring that the complete process of facial expression changes can be captured.

[0040] Specifically, facial images are captured 2 seconds before and 3 seconds after the trigger timestamp, forming a 5-second set of facial image segments. This setup aims to simultaneously capture the facial state before and after the emotional fluctuations in the speech. During the capture process, the timestamps of each frame in the initial facial image sequence are precisely matched, identifying all image frames within the time window before and after the trigger timestamp. For each captured image frame, its original resolution and clarity are preserved, while its time offset relative to the trigger timestamp is recorded to facilitate subsequent analysis of the temporal characteristics of facial expression changes. After capture, these facial images are organized into a set of facial image segments in chronological order.

[0041] Step 104: Extract facial geometric features from the set of facial image segments, calculate the facial tension index based on the facial geometric features, and identify facial tension events and their corresponding timestamps based on the facial tension index.

[0042] In this embodiment, facial geometric features refer to the shape, position, and motion parameters of key facial regions extracted from facial images, used to characterize changes in facial expressions. Common facial geometric features include eyebrow raising angle, mouth opening and closing amplitude, and eye squinting degree. The facial tension index represents a quantitative value of facial muscle tension, which is a comprehensive index calculated based on the weighted average of multiple facial geometric features. A facial tension event refers to a state where the facial tension index exceeds a specific threshold and persists for a certain period of time, indicating that the individual's facial muscles are in a state of high tension. The occurrence timestamp indicates the precise time point at which the facial tension event was detected.

[0043] Specifically, a facial landmark detection algorithm is applied to each frame of the image to locate 68 facial landmarks, including key locations in areas such as eyebrows, eyes, nose, and mouth. Based on these landmarks, the following geometric features are calculated: eyebrow upward angle, measured by the angle between the line connecting the outer and inner points of the eyebrow and the horizontal line; mouth opening and closing amplitude, measured by the vertical distance between the midpoints of the upper and lower lips; and eye squinting degree, measured by the ratio of the vertical distance between the upper and lower eyelids to the width of the eye. These geometric parameters are compared with a neutral reference parameter set to calculate the deviation of each parameter. The facial tension index is calculated using a weighted summation method, that is, the deviation of the eyebrow upward angle is multiplied by a weight of 0.4, the deviation of the mouth opening and closing amplitude is multiplied by a weight of 0.35, and the deviation of the eye squinting degree is multiplied by a weight of 0.25, and the three are added together to obtain the facial tension index. When identifying facial tension events, the number of frames in which the facial tension index is greater than a preset tension threshold is counted. When the number of consecutive frames multiplied by the single frame duration is greater than a preset duration threshold, it is determined to be a facial tension event. The time point corresponding to the image frame where the facial tension index reaches its local maximum value is recorded as the occurrence timestamp.

[0044] In one possible implementation, facial geometric features are extracted from a set of facial image segments. A facial tension index is calculated based on the facial geometric features, and facial tension events and their corresponding timestamps are identified based on the facial tension index. Specifically, this includes steps 1041-1043, as follows: Step 1041: Combine the neutral reference parameter set to extract the eyebrow raising angle, mouth opening and closing amplitude, and eye squinting degree of each frame in the facial image fragment set, and combine them into facial geometric features.

[0045] Specifically, firstly, the reference values ​​of facial keypoint coordinates from the neutral baseline parameter set are loaded as the comparison benchmark for subsequent feature extraction. A facial keypoint detection algorithm is applied to each frame in the facial image fragment set to accurately locate 68 facial landmarks, including key positions in areas such as the eyes, eyebrows, nose, and mouth. Based on these keypoint coordinates, the eyebrow upward angle is calculated: four keypoint coordinates representing the inner and outer points of the left and right eyebrows are extracted, and the angle formed by the line connecting the left and right eyebrows and the horizontal line is calculated. Then, the average of the angles of both eyebrows is taken. The mouth opening amplitude is calculated: the coordinates of two keypoints representing the midpoints of the upper and lower lip are extracted, the vertical distance between them is calculated, and then normalized by dividing by the reference value of this distance in the neutral state. The degree of eye squinting is calculated: the keypoint coordinates of the midpoints of the upper and lower eyelids and the endpoints of the eyes are extracted for both eyes. The ratio of the vertical distance between the midpoints of the upper and lower eyelids to the width of the eye is calculated, and the average of the ratios for both eyes is taken. The three geometric feature values ​​are combined into a feature vector to form a complete facial geometric feature, which is used for subsequent facial tension calculation and emotional state analysis.

[0046] Step 1042: Perform weighted calculations on each index in the facial geometric features to obtain the facial tightness index.

[0047] Specifically, the weighting coefficients for each facial geometric feature indicator are first determined. Based on professional knowledge and experience in facial expression analysis, the eyebrow raising angle indicator is assigned a weight of 0.4, the mouth opening amplitude indicator is assigned a weight of 0.35, and the eye squinting degree indicator is assigned a weight of 0.25. Then, each geometric feature indicator is normalized: the degree of difference between the current feature value and the corresponding reference value in the neutral baseline parameter group is calculated to obtain the relative change. For the eyebrow raising angle, the absolute value of the difference between the current angle value and the neutral state angle value is calculated; for the mouth opening amplitude, the ratio of the current amplitude value to the neutral state amplitude value is calculated; for the eye squinting degree, the absolute value of the difference between the current ratio and the neutral state ratio is calculated. These three normalized feature changes are multiplied by their corresponding weighting coefficients, and then the three weighted results are summed to obtain the facial tension index. This index value is usually limited to between 0 and 1, where 0 represents a completely relaxed state and 1 represents an extremely tense state.

[0048] Step 1043: Count the number of consecutive frames in which the facial tension index is greater than the preset tension threshold. When the number of consecutive frames multiplied by the duration of a single frame is greater than the preset duration threshold, it is determined to be a facial tension event. Record the time node corresponding to the image frame in which the facial tension index reaches the target peak as the occurrence timestamp.

[0049] Specifically, a preset tension threshold is first set, typically 0.65, indicating that a facial tension index exceeding this value is considered a significant tension state. For each frame in the facial image segment set, its facial tension index is compared to the preset tension threshold. When the facial tension index of the current frame is detected to be greater than the preset tension threshold, the number of consecutive frames meeting the condition begins to be counted. Subsequent frames are processed, and as long as the facial tension index continues to be greater than the preset tension threshold, the consecutive frame count counter is incremented. The duration of a single frame is calculated based on the frame rate of the camera device; for example, for a video of 25 frames per second, the duration of a single frame is 40 milliseconds. The accumulated number of consecutive frames is multiplied by the duration of a single frame to obtain the duration of the facial tension state. This duration is compared with a preset duration threshold, typically set to 500 milliseconds. When the duration exceeds the preset duration threshold, it is determined to be a valid facial tension event. Among all the frames in which this facial tension event lasts, the frame with the highest facial tension index is identified, i.e., the target peak frame. The time information corresponding to this target peak frame, i.e., the time position of this frame in the initial facial image sequence, is extracted and recorded as an occurrence timestamp. This timestamp serves as a time marker for facial emotion changes, which is then used for subsequent temporal correlation analysis with vocal emotion changes.

[0050] Step 105: Based on the trigger timestamp and occurrence timestamp, determine the temporal coupling relationship between voice instability events and facial tension events, and combine the facial tension index to determine the emotional suppression state and coupling strength.

[0051] In this embodiment, temporal coupling refers to the temporal correlation between speech instability events and facial tension events, reflecting the degree of coordination between speech and facial expression changes. Synchronous coupling indicates that speech and facial expression change almost simultaneously; delayed coupling indicates that facial expression change lags behind speech change; and advanced coupling indicates that facial expression change precedes speech change. Emotional suppression refers to a psychological state where the individual experiences internal emotional fluctuations but conceals them by controlling facial expressions; this is typically more pronounced in delayed coupling. Coupling strength indicates the closeness of the correlation between speech instability events and facial tension events; a higher value indicates a stronger correlation.

[0052] Specifically, the time difference between the trigger timestamp and the occurrence timestamp is calculated. If the absolute value of the time difference is less than or equal to a preset time window threshold, the temporal coupling relationship is determined to be synchronous coupling. If the absolute value of the time difference is greater than the preset time window threshold, the sign of the time difference is further determined. When the time difference is positive, the temporal coupling relationship is determined to be delayed coupling; when the time difference is negative, the temporal coupling relationship is determined to be advanced coupling. The coupling strength is calculated by dividing the facial tension index corresponding to the occurrence timestamp by the sum of the absolute value of the time difference and the duration of a single frame. When the temporal coupling relationship is delayed coupling and the calculated coupling strength is greater than a preset suppression threshold, the current object is determined to be in a state of emotional suppression, indicating that the person may be consciously controlling their facial expressions to conceal their inner emotional fluctuations.

[0053] In one possible implementation, the temporal coupling relationship between the speech instability event and the facial tension event is determined based on the trigger timestamp and the occurrence timestamp, and the emotional suppression state and coupling strength are determined by combining the facial tension index. Specifically, this includes steps 1051-1053, as follows: Step 1051: Calculate the time difference between the trigger timestamp and the occurrence timestamp; when the absolute value of the time difference is less than or equal to the preset time window threshold, the timing coupling relationship is determined to be synchronous coupling; when the absolute value of the time difference is greater than the preset time window threshold, if the time difference is positive, the timing coupling relationship is determined to be delayed coupling; if the time difference is negative, the timing coupling relationship is determined to be advanced coupling.

[0054] Specifically, the trigger timestamps of speech instability events and occurrence timestamps of facial tension events are first extracted. These two timestamps record the precise moments when changes in speech emotion and facial expression are detected, respectively. The time difference between these two timestamps is calculated by subtracting the trigger timestamp from the occurrence timestamp, resulting in a signed time difference value. A preset time window threshold of 200 milliseconds is set as the critical value for determining synchronous and asynchronous states. The absolute value of the time difference is calculated and compared with the preset time window threshold. If the absolute value of the time difference is less than or equal to the preset time window threshold, the temporal coupling relationship is determined to be synchronous coupling, indicating that changes in speech emotion and facial expression occur almost simultaneously. If the absolute value of the time difference is greater than the preset time window threshold, the sign of the time difference is further determined. When the time difference is positive, it means that the occurrence timestamp is later than the trigger timestamp, i.e., facial expression changes lag behind changes in speech emotion; in this case, the temporal coupling relationship is determined to be delayed coupling. When the time difference is negative, it means that the occurrence timestamp is earlier than the trigger timestamp, that is, the change in facial expression occurs before the change in voice emotion. At this time, the temporal coupling relationship is judged as an advanced coupling state.

[0055] Step 1052: Extract the facial tension index corresponding to the occurrence timestamp, divide the facial tension index by the sum of the absolute value of the time difference and the duration of a single frame, and calculate the coupling strength.

[0056] Specifically, firstly, the facial tension index corresponding to the timestamp of the facial tension event is extracted; that is, the index value at which the facial tension reaches its peak during the facial tension event. Then, the calculated time difference is extracted, and its absolute value is taken to represent the absolute time interval between the speech instability event and the facial tension event. The duration of a single frame is determined based on the frame rate of the camera equipment; for a video of 25 frames per second, the duration of a single frame is 40 milliseconds. The absolute value of the time difference is added to the duration of the single frame to obtain a time correction value. This correction ensures that even when the time difference is zero, the calculation result has a reasonable numerical range. The coupling strength is calculated by dividing the facial tension index corresponding to the timestamp by the time correction value. This calculation method shows that the higher the facial tension index and the smaller the time interval, the greater the coupling strength, reflecting a comprehensive assessment of the intensity and temporal synchronicity of emotional expression.

[0057] Step 1053: When the sequential coupling relationship is in a delayed coupling state and the coupling strength is greater than the preset suppression threshold, it is determined that the current object is in an emotional suppression state.

[0058] Specifically, the first step is to examine the type of temporal coupling relationship. If the temporal coupling relationship is determined to be a delayed coupling state, further analysis is conducted to determine if emotional suppression exists. The coupling strength value calculated in step 1052 is extracted and compared with a preset suppression threshold. The preset suppression threshold is typically set to 0.8, an empirical value determined based on psychological research and experimental data. When the coupling strength is greater than the preset suppression threshold, it indicates that although facial expression changes lag behind vocal emotion changes, there is still a strong correlation between the two. This situation usually reflects that the person is trying to control or conceal their facial expressions, but their inner emotional fluctuations have already been expressed through vocal features. In this case, the current subject is determined to be in a state of emotional suppression, and this determination will be used for subsequent emotion assessment and adjustment of mediation strategies. If the temporal coupling relationship is not a delayed coupling state, or the coupling strength is not greater than the preset suppression threshold, the current subject is determined not to be in a state of emotional suppression.

[0059] Step 106: Integrate facial geometric features, emotional suppression state, coupling strength, and speech temporal change features from the speech feature set to calculate a comprehensive emotion assessment score.

[0060] In this embodiment, the comprehensive emotion assessment score refers to a numerical value calculated by fusing multiple emotion-related features, used to represent a comprehensive quantitative assessment of the individual's emotional state. The emotion suppression coefficient represents the frequency and intensity of emotional suppression, reflecting the degree to which the individual conceals their emotions. The cumulative emotional stress index is a cumulative stress value calculated based on temporal coupling relationships and coupling strength, representing the degree of accumulated emotional stress. The speech emotion activation level represents the degree of emotional activation in speech, extracted through speech temporal change features. The facial expression emotion activation level represents the degree of emotional expression in facial expressions, calculated through facial geometric features.

[0061] Specifically, firstly, the emotional suppression coefficient is calculated based on the frequency of occurrence of emotional suppression states. The number of times emotional suppression states were identified in the last 10 minutes is counted and divided by the total number of speech instability events during that time period to obtain the emotional suppression frequency ratio. This ratio is then multiplied by the most recently detected coupling strength to calculate the emotional suppression coefficient. Secondly, corresponding emotional weights are determined based on temporal coupling relationships: synchronous coupling states are weighted at 0.6, lagging coupling states at 0.9, and leading coupling states at 0.4. The emotional weights are multiplied by the weighted average of the last 5 coupling strengths to calculate the cumulative emotional stress index. Then, the speech emotional activation level is calculated based on the speech temporal change features in the speech feature set. Specifically, this is done by extracting the product of the variance and mean of the speech rate change rate over the last 30 seconds. Simultaneously, the facial expression emotional activation level is calculated based on facial geometric features. Specifically, this is done by calculating the weighted sum of the changes in the amplitude of each component of the facial geometric features over the last 30 seconds. Finally, the emotional suppression coefficient, cumulative emotional stress index, voice emotional activation and facial emotional activation were weighted at 0.3, 0.3, 0.2 and 0.2 respectively, and the results were normalized to the range of 0-100 to obtain a comprehensive emotional assessment score.

[0062] In one possible implementation, facial geometric features, emotional suppression state, coupling strength, and speech temporal variation features from the speech feature set are fused to calculate a comprehensive emotion assessment score, specifically including steps 1061-1063, as follows: Step 1061: Calculate the emotional suppression coefficient based on the frequency of occurrence of emotional suppression state; determine the corresponding emotional weight according to the temporal coupling relationship, and calculate the cumulative emotional stress index in combination with the coupling strength.

[0063] Specifically, firstly, the number of times emotional suppression was identified within the last 10 minutes is counted. By retrospectively analyzing historical data, the total number of events judged as emotional suppression within this time window is counted. Simultaneously, the total number of speech instability events within this time window is counted as the baseline denominator. The frequency of emotional suppression is calculated by dividing the number of occurrences of emotional suppression by the total number of speech instability events. This frequency is then multiplied by the most recently detected coupling strength value to obtain the emotional suppression coefficient. This coefficient comprehensively considers the historical frequency and current intensity of suppression, forming an overall evaluation index for emotional suppression. Next, based on the temporal coupling relationship type determined in step 1051, corresponding emotional weights are determined: for synchronous coupling, the emotional weight is set to 0.6; for delayed coupling, the emotional weight is set to 0.9; and for advanced coupling, the emotional weight is set to 0.4. These weight values ​​are determined based on psychological research, reflecting the different degrees of influence of different coupling types on emotional states. Then, the coupling strength values ​​of the five most recently identified coupling events are extracted, and weights are assigned decreasing weights from most recent to oldest, calculating a weighted average to obtain the historical average coupling strength. The cumulative emotional stress index is calculated by multiplying the emotional weight by the mean of historical coupling strength. This index reflects the cumulative effect of emotional stress over time.

[0064] Step 1062: Calculate the speech emotion activation based on the speech temporal change features in the speech feature set, and calculate the facial expression emotion activation based on the facial geometric features.

[0065] Specifically, firstly, the temporal variation features of speech over the past 30 seconds are extracted to form a temporal feature sequence. The statistical properties of this sequence are calculated, including the mean and variance of the speech rate change rate. The mean speech rate change rate reflects the overall trend of speech speed increasing or decreasing, while the variance reflects the degree of fluctuation in speech rate change. The product of the variance and mean of the speech rate change rate is used as the speech rate change index, which considers both the direction and intensity of speech rate change. This index is normalized and mapped to the range of 0-1 to obtain the speech emotion activation level. The larger the activation level, the more active the emotional components carried in the speech. Simultaneously, facial geometric feature data over the past 30 seconds is extracted, including the temporal variation sequences of eyebrow raising angle, mouth opening and closing amplitude, and eye squinting degree. The maximum change amplitude of each geometric feature within this time window is calculated, i.e., the difference between the maximum and minimum values. Weights were assigned to the variations in these three geometric features: the variation in eyebrow raising angle was weighted at 0.4, the variation in mouth opening and closing angle at 0.35, and the variation in eye squinting at 0.25. The three weighted variations were then summed and normalized to a range of 0-1 to obtain the emotional activation value. A higher activation value indicates a richer emotional component in the facial expression.

[0066] Step 1063: Weight and fuse the emotion suppression coefficient, cumulative emotional stress index, voice emotion activation amount, and facial emotion activation amount to obtain a comprehensive emotion assessment score.

[0067] Specifically, the weighting coefficients for each emotion parameter are first determined. Based on psychological research and experimental verification results, the following weights are set: the emotion suppression coefficient is weighted at 0.3, representing the proportion of emotion suppression's contribution to the overall emotional state; the cumulative emotional stress index is weighted at 0.3, representing the degree of influence of historically accumulated emotional stress on the current state; the voice emotion activation is weighted at 0.2, representing the proportion of voice features' contribution to the emotion assessment; and the facial expression emotion activation is weighted at 0.2, representing the proportion of facial expressions' contribution to the emotion assessment. Then, the calculated emotion suppression coefficient and cumulative emotional stress index, as well as the calculated voice emotion activation and facial expression emotion activation, are extracted. The emotion suppression coefficient is multiplied by its weight of 0.3 to obtain the weighted emotion suppression contribution value; the cumulative emotional stress index is multiplied by its weight of 0.3 to obtain the weighted emotion stress contribution value; the voice emotion activation is multiplied by its weight of 0.2 to obtain the weighted voice emotion contribution value; and the facial expression emotion activation is multiplied by its weight of 0.2 to obtain the weighted facial expression emotion contribution value. These four weighted contribution values ​​are added together to obtain the raw score for the comprehensive emotion assessment. The raw score is transformed to a standardized score range of 0-100 using a linear mapping function, resulting in a final comprehensive emotion assessment score. A higher score indicates a more tense or unstable emotional state, providing a quantitative basis for subsequent emotion risk assessment and intervention strategy development.

[0068] Step 107: Generate an emotional dynamic prediction sequence based on the comprehensive emotion assessment score, and output the emotion risk judgment result and adjust the mediation communication rhythm parameters according to the emotional dynamic prediction sequence.

[0069] In this embodiment, the emotional dynamic prediction sequence refers to time-series data that predicts the trend of emotional changes over a future period based on current and historical emotional assessment scores. The emotional risk assessment result represents a graded assessment of the potential risk of the party's emotional state, typically categorized as low risk, medium risk, and high risk. The mediation communication rhythm parameters refer to a set of specific parameters used to guide mediators in adjusting their communication strategies, including speaking frequency adjustment values, communication buffer time, and intervention response intervals.

[0070] Specifically, firstly, the comprehensive emotion assessment score within the most recent 5-minute window is extracted to construct an emotion assessment historical trajectory. A linear regression method is applied to this historical trajectory to calculate the fluctuation slope at the current moment; simultaneously, the acceleration of emotion change is obtained by calculating the second difference of the historical trajectory. The next 3-minute time period is divided into 6 prediction time steps, each lasting 30 seconds. For each prediction time step, a second-order prediction model is used to calculate the predicted emotion score for that time step by adding the comprehensive emotion assessment score at the current moment, the product of the fluctuation slope and the prediction time step, and the product of the acceleration and the square of the prediction time step. These 6 prediction emotion scores are arranged in ascending order to form a dynamic emotion prediction sequence. Based on this prediction sequence, the emotion stress accumulation rate is calculated, which is the difference between the last value and the first value in the prediction sequence divided by the total time span. Using the comprehensive emotion assessment score, emotion suppression coefficient, and emotion stress accumulation rate as input parameters, a preset emotion risk classification mapping table is queried, and the current emotion risk level is matched and output as the emotion risk judgment result. Based on the emotional risk level, combined with the characteristics of speech temporal changes and the trend of facial tension index changes, the parameters for the adjustment communication rhythm are generated: for the high risk level, the speaking frequency adjustment value is set to -30%, the communication buffer time is set to 90 seconds, and the intervention response interval is set to 5 seconds; for the medium risk level, the parameters are set to -15%, 60 seconds, and 3 seconds, respectively; for the low risk level, the parameters are set to 0%, 30 seconds, and 1 second, respectively.

[0071] In one possible implementation, a dynamic emotion prediction sequence is generated based on a comprehensive emotion assessment score, specifically including steps 1071-1073, as follows: Step 1071: Extract the comprehensive emotion assessment score within the historical time window and construct the historical trajectory of emotion assessment; calculate the slope and acceleration of the fluctuation of the historical trajectory of emotion assessment at the current moment.

[0072] Specifically, the historical time window size is first determined to be 5 minutes, representing a period of 5 minutes backward from the current moment. All comprehensive emotion assessment scores within this time window are extracted from the historical database of the emotion analysis system and sorted by timestamp from earliest to latest. If the historical data collection is uneven, the data is resampled using linear interpolation to ensure a uniform distribution of time points; for example, one data point every 10 seconds, generating a total of 30 evenly distributed data points. These time-uniform emotion assessment scores constitute the historical emotion assessment trajectory. Next, a linear regression method is used to calculate the slope of this historical trajectory at the current moment. Specifically, time is used as the independent variable and the emotion assessment score as the dependent variable. The least squares method is applied to the data points within the most recent 60 seconds to calculate the slope of the regression line; this slope is the fluctuation slope, expressed in fractions per second. A positive fluctuation slope indicates a tendency towards increased tension, while a negative slope indicates a tendency towards decreased relaxation. Then, the acceleration is obtained by calculating the second difference of the historical trajectory. First, the first-order difference of the emotion assessment scores between adjacent time points is calculated, which is the change in score between every two adjacent data points divided by the time interval, resulting in a series of velocity values. Then, the first-order difference of these velocity values ​​is calculated, which is the change in velocity value between adjacent values ​​divided by the time interval, resulting in a series of acceleration values. The average of the acceleration values ​​over the most recent 30 seconds is taken as the emotion change acceleration at the current moment, in fractions per second. 2 A positive acceleration value indicates that the rate of emotional change is accelerating, while a negative value indicates that the rate of emotional change is slowing down.

[0073] Step 1072: Divide the preset time period into multiple consecutive prediction time steps; for each prediction time step, add the current comprehensive emotion assessment score, the product of the fluctuation slope and the prediction time step, and the product of the acceleration and the square of the prediction time step to calculate the prediction emotion score for the corresponding time.

[0074] Specifically, first, a preset time period is set, typically 3 minutes, representing the prediction of emotional changes within the next 3 minutes from the current moment. This preset time period is divided into multiple evenly distributed prediction time steps, usually set to one prediction point every 30 seconds. Therefore, the 3 minutes are divided into 6 prediction time steps, corresponding to 30 seconds, 60 seconds, 90 seconds, 120 seconds, 150 seconds, and 180 seconds after the current moment. For each prediction time step, a second-order prediction model is applied to calculate its corresponding predicted emotional score. The prediction model includes three components: an initial value component, a linear change component, and an accelerated change component. For the initial value component of the prediction model, the comprehensive emotional assessment score at the current moment is directly used. For the linear change component, the product of the fluctuation slope obtained in step 1071 and the prediction time step is calculated. For example, for the second prediction point, the linear change component equals the fluctuation slope multiplied by 60. For the accelerated change component, the product of the acceleration obtained in step 1071 and the square of the prediction time step is calculated. For example, for the second prediction point, the accelerated change component equals the acceleration multiplied by 60 squared, which is 3600. The three components are summed to obtain the predicted sentiment score for that prediction time step. If the prediction result exceeds the valid range of the sentiment score, it is truncated to ensure that the predicted value is within the valid range. The above calculation process is repeated to calculate the predicted sentiment scores for each of the six prediction time steps, forming a complete prediction result set.

[0075] Step 1073: Arrange the predicted sentiment scores corresponding to each prediction time step in ascending order of time to form a dynamic sentiment prediction sequence.

[0076] Specifically, the predicted sentiment scores for all calculated prediction time steps are first collected, including the timestamp and corresponding predicted score for each time point. These prediction results are then sorted from earliest to latest according to their timestamps to ensure that the time order of the data points is consistent with the actual timeline of sentiment changes. After sorting, the actual comprehensive sentiment assessment score at the current moment is taken as the first data point in the sequence, and then the predicted sentiment scores for each prediction time step are added sequentially to form a complete time series. Each data point in the sequence contains two basic attributes: the time point, the time offset relative to the current moment, and the predicted sentiment score. To facilitate subsequent analysis, auxiliary attributes can be added to each data point, such as prediction confidence, calculated based on the distance between the predicted time point and the current time point; the greater the distance, the lower the confidence. The sequence data is then standardized to ensure a consistent format and ease of storage and analysis. The final constructed dynamic emotion prediction sequence includes emotion assessment values ​​at seven time points from the current moment to the next 3 minutes, namely the current moment, 30 seconds, 60 seconds, 90 seconds, 120 seconds, 150 seconds, and 180 seconds. It fully presents the expected trajectory of emotion changes and can be used for subsequent emotion risk assessment and adjustment of mediation strategies.

[0077] In the above embodiments, a basic assessment of emotional state was achieved through multimodal signal acquisition and feature extraction. To further improve the accuracy and early warning capability of real-time identification of high-risk emotional states in complex psychological fluctuation situations, and to establish a quantitative correlation assessment model between dynamic emotional changes and risk warnings, this application also provides a multimodal negotiation analysis method for mediation operations. This method constructs a mapping relationship between multidimensional emotional characteristics and risk levels by analyzing the synergistic effect of emotional stress accumulation rate, emotional suppression coefficient, and comprehensive emotional assessment score, and performs time-series dynamic parameter optimization calculations. This enables the system to more comprehensively and accurately assess the psychological state during the mediation process while considering multiple emotional fluctuation factors, effectively identifying potential risk situations that may be overlooked in single-indicator evaluations. The following section combines... Figure 2 Another method for multimodal negotiation analysis of mediation services in this application embodiment is described below: Please see Figure 2 This is another flowchart illustrating the multimodal negotiation analysis method for mediation services in this application embodiment.

[0078] Step 201: Calculate the rate of accumulation of emotional stress based on the dynamic prediction sequence of emotions.

[0079] Specifically, the predicted emotion scores at the first and last two time points are extracted from the constructed dynamic emotion prediction sequence. The first point of the sequence represents the overall emotion assessment score at the current moment, and the last point represents the predicted emotion score at the end of a preset time period, typically the predicted value 180 seconds after the current moment. The difference between these two emotion scores is calculated by subtracting the first point score from the last point score, yielding the net change in emotion score over the entire prediction time period. This net change is divided by the total duration of the preset time period, which is 180 seconds, to obtain the average change in emotion score per second, i.e., the rate of emotional stress accumulation. For example, if the current overall emotion assessment score is 65 points, and the predicted score 3 minutes later is 83 points, the net change is 18 points, and the rate of emotional stress accumulation is 18 divided by 180, or 0.1 points per second. This accumulation rate reflects the average increase or decrease in emotional stress per unit time, providing a key input parameter for subsequent emotion risk assessment.

[0080] Step 202: Input the comprehensive emotion assessment score, emotion suppression coefficient and emotion stress accumulation rate into the preset emotion risk classification mapping table, match and output the current emotion risk level as the emotion risk judgment result.

[0081] Specifically, a three-dimensional emotion risk classification mapping table is first pre-defined. This table takes the comprehensive emotion assessment score, emotion suppression coefficient, and emotion stress accumulation rate as input dimensions and outputs the corresponding emotion risk level. The mapping table is structured using interval division: the comprehensive emotion assessment score is divided into multiple intervals, for example, 0-40 is the low score interval, 40-70 is the medium score interval, and 70-100 is the high score interval; the emotion suppression coefficient is divided into low suppression interval, medium suppression interval, and high suppression interval; and the emotion stress accumulation rate is divided into decreasing interval, stable interval, and increasing interval. The current comprehensive emotion assessment score, emotion suppression coefficient, and emotion stress accumulation rate are obtained, and these three parameters together constitute the query vector. This query vector is used to search for the corresponding risk level in the emotion risk classification mapping table. The search process is based on the nearest neighbor principle, that is, finding the mapping table entry that is closest to the query vector. For example, when the comprehensive emotion assessment score is 75, the emotion suppression coefficient is 0.6, and the emotion stress accumulation rate is 0.1 points per second, the table search may match the "high risk" level. The risk level obtained from the query will be output as the result of the emotional risk assessment, which will be used to generate subsequent mediation and communication rhythm parameters.

[0082] Step 203: Based on the emotional risk level, combined with the characteristics of speech temporal changes and the trend of facial tension index changes, generate mediation communication rhythm parameters including speech frequency adjustment value, communication buffer duration and intervention response interval.

[0083] Specifically, the emotional risk level is first obtained as the primary basis for generating the mediation communication rhythm parameters. Then, the speech temporal variation features within the last 30 seconds are extracted, and their trend is calculated, i.e., the linear regression slope of the feature within the time window. Simultaneously, the facial tension index value sequence within the last 30 seconds is extracted, and its trend is calculated, i.e., the linear regression slope of the index value within the time window. Based on the emotional risk level, speech temporal variation features, and the trend of the facial tension index, the mediation communication rhythm parameters are generated. For a high-risk level, the speaking frequency adjustment value is set to -30%, indicating a recommendation to reduce speaking frequency; the communication buffer time is set to 90 seconds, indicating the buffer time required before and after key communications; and the intervention response interval is set to 5 seconds, indicating that at least 5 seconds should be waited after intervention before proceeding to the next interaction. For a medium-risk level, the speaking frequency adjustment value is set to -15%, the communication buffer time is set to 60 seconds, and the intervention response interval is set to 3 seconds. For a low-risk level, the speaking frequency adjustment value is set to 0%, the communication buffer time is set to 30 seconds, and the intervention response interval is set to 1 second. If both the speech timing characteristics and facial tension index show an upward trend, the communication buffer time will be increased by an additional 20%; if both show a downward trend, the communication buffer time will be reduced by 10%. The final generated mediation communication rhythm parameters serve as specific guidelines for adjusting the communication rhythm during the mediation process.

[0084] In one possible implementation, after generating the mediation communication rhythm parameters, which include the speaking frequency adjustment value, communication buffer duration, and intervention response interval, the process further includes steps 2031-2033, as follows: Step 2031: Record the baseline emotional suppression coefficient and baseline emotional stress accumulation rate at the moment when the mediation communication rhythm parameters are generated.

[0085] Specifically, after generating the initial mediation and communication rhythm parameters, the current emotional indicators are immediately recorded as benchmark values ​​for subsequent dynamic adjustments. First, the current emotional suppression coefficient is extracted, reflecting the overall trend and intensity of the individual's emotional concealment. This coefficient value is recorded as the benchmark emotional suppression coefficient and stored in the system's benchmark parameter storage area. Simultaneously, the currently calculated emotional stress accumulation rate is extracted, reflecting the changing trend of emotional stress. This rate value is recorded as the benchmark emotional stress accumulation rate and also stored in the system's benchmark parameter storage area. To ensure data timeliness and traceability, the current system timestamp is also recorded as a timestamp for these benchmark parameters. These benchmark values ​​will serve as reference standards during subsequent monitoring to determine the magnitude of changes in emotional state, thereby deciding whether to adjust the mediation and communication rhythm parameters.

[0086] Step 2032: If the dynamic emotion prediction sequence is detected to show a continuous upward trend, extract the speech feature set of the current time window and update the comprehensive emotion assessment score.

[0087] Specifically, the constructed emotional dynamic prediction sequence is first subjected to trend analysis. The difference between the predicted scores at adjacent time points in the sequence is calculated. If the difference is positive for multiple consecutive time points (at least 3), the emotional dynamic prediction sequence is determined to have a continuous upward trend, indicating that the emotional state is expected to continue to deteriorate. When this continuous upward trend is detected, the real-time data update process is immediately initiated. The current time window is redefined, typically using the data from the most recent 2 minutes as the observation range. Within this time window, speech signals are reacquired, and the latest set of speech features is extracted, including energy features, intonation features, and speech temporal change features. Simultaneously, the latest facial image data is acquired, facial geometric features are extracted, and the facial tension index is calculated. Based on these updated feature data, the calculation process from steps 1061 to 1063 is re-executed to obtain the updated emotional suppression coefficient, cumulative emotional stress index, speech emotional activation level, and facial emotional activation level. Then, following the weighted fusion method in step 1063, these four indicators are weighted and combined to calculate the updated comprehensive emotional assessment score. This updated score reflects the latest emotional state assessment result and will be used for subsequent dynamic adjustment of mediation parameters.

[0088] Step 2033: Determine the target emotional risk level based on the updated comprehensive emotional assessment score. If the emotional suppression coefficient of the current time window is greater than the baseline emotional suppression coefficient, and the emotional pressure accumulation rate of the current time window is greater than the baseline emotional pressure accumulation rate, then increase the adjustment range of the mediation communication rhythm parameter by the preset step size.

[0089] Specifically, firstly, based on the updated comprehensive emotion assessment score in step 2032, the risk classification mapping query in step 202 is re-executed to obtain the current target emotion risk level. Simultaneously, the latest calculated emotion suppression coefficient and emotion stress accumulation rate are extracted from the current time window. The current emotion suppression coefficient is compared with the baseline emotion suppression coefficient recorded in step 2031, and the difference between the two is calculated. Similarly, the current emotion stress accumulation rate is compared with the baseline emotion stress accumulation rate, and the difference between the two is calculated. When both conditions are met simultaneously—the current emotion suppression coefficient is greater than the baseline emotion suppression coefficient, and the current emotion stress accumulation rate is greater than the baseline emotion stress accumulation rate—the dynamic adjustment process of the mediation communication rhythm parameters is triggered. The adjustment adopts a preset step size method: for the speaking frequency adjustment value, it is reduced by an additional 10 percentage points from the current value; for the communication buffer time, it is increased by 20 seconds from the current value; for the intervention response interval, it is increased by 2 seconds from the current value. If the target emotion risk level is higher than the original risk level, the adjustment magnitude is doubled to cope with a rapid deterioration. The adjusted parameters must be within a reasonable range. For example, the speaking frequency adjustment value should not be lower than -50%, the communication buffer time should not exceed 180 seconds, and the intervention response interval should not exceed 10 seconds. The final updated mediation communication rhythm parameters will replace the original parameters and be used to guide the subsequent mediation communication process.

[0090] It should be noted that, in this embodiment, preset parameter tuning refers to the process of determining various thresholds, weights, and benchmark values ​​through a systematic approach to ensure the accuracy and stability of the emotion analysis algorithm. Preset parameters include various types such as sampling frequency, time threshold, feature weights, and risk classification boundaries. The tuning method represents the technical path for determining the optimal values ​​of these parameters, typically combining expert knowledge, statistical analysis, and iterative optimization. Specifically, the preset parameter tuning in this application employs a three-stage approach. The first stage is initial value setting, where theoretically reasonable initial values ​​are set for each parameter based on psychological research literature and industry experience. For example, the initial time window threshold for emotion analysis is set to 200 milliseconds, a value derived from the average delay time of emotional responses in psychological research; the weight allocation of the facial tension index is based on the ranking of the importance of emotional expression in each facial expression unit in the Facial Behavior Coding System (FACS). The second stage is data-driven calibration, collecting a set of labeled multimodal emotion datasets containing at least 500 hours of mediation scene audio and video data and professional psychological assessment results. Cross-validation is used to test the performance of the parameters under different values, and the optimal parameter values ​​are determined by minimizing the prediction error. For example, the tension threshold of the facial tension index was determined to be at an optimal balance point of 0.65 through ROC curve analysis. The third stage is on-site fine-tuning. After deploying the system in actual mediation environments, feedback data is continuously collected, and minor adjustments are made based on different types of cases and the characteristics of the parties involved. A parameter adaptive mechanism is established, enabling the system to dynamically adjust certain thresholds according to the characteristics of specific mediation scenarios. For example, for high-conflict family disputes, the sensitivity of emotional risk assessment is automatically increased, and the preset suppression threshold is lowered; for commercial disputes, the threshold is correspondingly increased to reduce false alarms. The entire parameter tuning process follows evidence-based principles, and all parameter adjustments are recorded in the system log for subsequent analysis and improvement, ensuring that the system can provide reliable emotion analysis results in various mediation scenarios.

[0091] The following describes a multimodal negotiation analysis system for mediation services from the perspective of hardware processing. Please refer to [link to relevant documentation]. Figure 3 This is a schematic diagram of the structure of a multimodal negotiation analysis system for mediation services in an embodiment of this application.

[0092] It should be noted that, Figure 3 The structure of a multimodal negotiation analysis system for mediation business shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0093] like Figure 3As shown, a multimodal negotiation analysis system for mediation services includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 302 or a program loaded from storage portion 308 into random access memory (RAM) 303, such as executing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0094] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including a hard disk, etc.; and communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

Claims

1. A multimodal negotiation analysis method for mediation services, characterized in that, The method includes: Acquire synchronized speech signals and facial image data in the mediation environment, and construct an initial speech sequence and an initial facial image sequence after aligning the timestamps of the synchronized speech signals and the facial image data. Extract the speech feature set of the initial speech sequence, identify speech instability events based on the energy features and intonation features in the speech feature set, and extract the trigger timestamps; Based on the trigger timestamp, extract the corresponding set of facial image segments from the initial facial image sequence; Extract facial geometric features from the set of facial image segments, calculate the facial tension index based on the facial geometric features, and identify facial tension events and their corresponding timestamps based on the facial tension index; Based on the trigger timestamp and the occurrence timestamp, the temporal coupling relationship between the voice instability event and the facial tension event is determined, and the emotional suppression state and coupling strength are determined in combination with the facial tension index. A comprehensive emotion assessment score is calculated by integrating the facial geometric features, the emotional suppression state, the coupling strength, and the speech temporal change features in the speech feature set. Based on the comprehensive emotion assessment score, an emotion dynamic prediction sequence is generated, and the emotion risk judgment result is output according to the emotion dynamic prediction sequence, and the mediation communication rhythm parameters are adjusted.

2. The method according to claim 1, characterized in that, The process of acquiring synchronized speech signals and facial image data in the mediation environment, aligning the timestamps of the synchronized speech signals and facial image data, and constructing an initial speech sequence and an initial facial image sequence includes: During the preset mediation silence period, voice signals and facial images are collected simultaneously as reference data; Extract the timbre reference value and the statistical mean of the sampling point coordinates from the reference data to construct a neutral reference parameter set; The synchronous speech signal during the mediation process is converted from analog to digital using a preset sampling frequency, and the facial image data is timestamped in conjunction with the neutral reference parameter set to obtain the initial speech sequence and the initial facial image sequence corresponding to the time.

3. The method according to claim 1, characterized in that, The step of extracting the speech feature set of the initial speech sequence, identifying speech instability events based on the energy features and intonation features in the speech feature set, and extracting the trigger timestamps includes: The initial speech sequence is segmented into frames, the peak energy intensity of each frame is extracted as the energy feature, the fundamental frequency fluctuation amplitude is extracted as the pitch feature, and the speech rate change rate between adjacent frames is calculated as the speech temporal change feature, thus forming the speech feature set. Calculate the deviation of the intonation features of the target frame from the mean intonation features during historical stable periods; When the energy characteristics of the target frame are detected to be greater than a preset energy threshold and the deviation is greater than a preset baseband fluctuation threshold, the target frame is marked as the speech instability event, and the time node corresponding to the target frame is recorded as the trigger timestamp.

4. The method according to claim 2, characterized in that, The process of extracting facial geometric features from the set of facial image segments, calculating a facial tension index based on the facial geometric features, and identifying facial tension events and their corresponding timestamps based on the facial tension index includes: Combining the neutral reference parameter set, the eyebrow raising angle, mouth opening and closing amplitude, and eye squinting degree of each frame in the facial image segment set are extracted and combined into the facial geometric features; The facial tightness index is obtained by weighting the various indicators in the facial geometric features. The number of consecutive frames in which the facial tension index is greater than a preset tension threshold is counted. When the number of consecutive frames multiplied by the duration of a single frame is greater than a preset duration threshold, it is determined to be the facial tension event. The time point corresponding to the image frame where the facial tension index reaches the target peak is recorded as the occurrence timestamp.

5. The method according to claim 1, characterized in that, The process of determining the temporal coupling relationship between the voice instability event and the facial tension event based on the trigger timestamp and the occurrence timestamp, and determining the emotional suppression state and coupling strength in conjunction with the facial tension index, includes: Calculate the time difference between the trigger timestamp and the occurrence timestamp; When the absolute value of the time difference is less than or equal to a preset time window threshold, the temporal coupling relationship is determined to be in a synchronous coupling state. When the absolute value of the time difference is greater than the preset time window threshold, if the time difference is positive, the time coupling relationship is determined to be a delayed coupling state; if the time difference is negative, the time coupling relationship is determined to be a leading coupling state. Extract the facial tension index corresponding to the occurrence timestamp, divide the facial tension index by the sum of the absolute value of the time difference and the duration of a single frame, and calculate the coupling strength. When the temporal coupling relationship is in the state of delayed coupling and the coupling strength is greater than the preset suppression threshold, it is determined that the current object is in the state of emotional suppression.

6. The method according to claim 1, characterized in that, The process of fusing the facial geometric features, the emotional suppression state, the coupling strength, and the speech temporal variation features in the speech feature set to calculate a comprehensive emotion assessment score includes: The emotional suppression coefficient is calculated based on the frequency of occurrence of the aforementioned emotional suppression state. The corresponding emotional weights are determined based on the temporal coupling relationship, and the cumulative emotional stress index is calculated in combination with the coupling strength. The speech emotion activation level is calculated based on the speech temporal change features in the speech feature set, and the facial expression emotion activation level is calculated based on the facial geometric features. The comprehensive emotion assessment score is obtained by weighting and fusing the emotion suppression coefficient, the cumulative emotion stress index, the voice emotion activation level, and the facial expression emotion activation level.

7. The method according to claim 1, characterized in that, The generation of the dynamic emotion prediction sequence based on the comprehensive emotion assessment score includes: Extract comprehensive emotion assessment scores within historical time windows to construct historical emotion assessment trajectories; Calculate the slope and acceleration of the fluctuation of the historical trajectory of the emotion assessment at the current moment; Divide the preset time period into multiple consecutive prediction time steps; For each predicted time step, the comprehensive emotion assessment score at the current moment, the product of the fluctuation slope and the predicted time step, and the product of the acceleration and the square of the predicted time step are added together to calculate the predicted emotion score at the corresponding moment. The predicted emotion scores corresponding to each prediction time step are arranged in ascending order of time to form the dynamic emotion prediction sequence.

8. The method according to claim 1, characterized in that, The step of outputting an emotional risk assessment result based on the emotional dynamic prediction sequence and adjusting the mediation communication rhythm parameters includes: The rate of emotional stress accumulation is calculated based on the aforementioned emotional dynamic prediction sequence. The comprehensive emotion assessment score, the emotion suppression coefficient, and the emotion stress accumulation rate are input into a preset emotion risk classification mapping table, and the current emotion risk level is matched and output as the emotion risk judgment result. Based on the emotional risk level, combined with the speech temporal change characteristics and the change trend of the facial tension index, a mediation communication rhythm parameter is generated, which includes the speech frequency adjustment value, communication buffer duration, and intervention response interval.

9. The method according to claim 8, characterized in that, After generating the mediation communication rhythm parameters, which include speaking frequency adjustment values, communication buffer duration, and intervention response intervals, the method further includes: Record the baseline emotional suppression coefficient and baseline emotional stress accumulation rate at the moment when the mediation communication rhythm parameters are generated; If the emotion dynamic prediction sequence is detected to show a continuous upward trend, the voice feature set of the current time window is extracted and the comprehensive emotion assessment score is updated. The target emotional risk level is determined based on the updated comprehensive emotional assessment score. If the emotional suppression coefficient of the current time window is greater than the baseline emotional suppression coefficient, and the emotional pressure accumulation rate of the current time window is greater than the baseline emotional pressure accumulation rate, then the adjustment range of the mediation and communication rhythm parameter is increased by a preset step size.

10. A multimodal negotiation analysis system for mediation services, characterized in that, The mediation business multimodal negotiation analysis system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the mediation business multimodal negotiation analysis system to perform the method as described in any one of claims 1-9.

Images