Emotion state perception quantization method based on voice continuous emotion space

Abstract

This invention provides a method for quantifying emotion state perception based on a continuous emotion space in speech, relating to the field of mental health assistance. Its features include the following steps: audio acquisition and preprocessing, speech representation extraction, continuous emotion state regression, derived dimension calculation, global emotion output, fixed-period emotion timeline generation, emotion state transition quantification, and structured interface output. The advantages of this invention are: it transforms the original speech signal into a continuous emotion state vector, replacing discrete emotion labels; it generates an emotion timeline sequence based on a fixed time window, enabling real-time tracking of dynamic emotion changes; it adapts to mental health scenarios, generating multi-dimensional derived dimensions to improve the scenario adaptability of emotion perception; and it constructs a full-link language emotion recognition system, achieving continuous quantification, dynamic tracking, and transfer calculation of language emotions, providing highly reliable language emotion recognition output for psychological AI.

Description

Technical Field

[0001] This invention relates to the field of mental health support, and in particular to a method for quantifying the perception of emotional states based on a continuous emotional space of speech. Background Technology

[0002] Existing AI systems for psychological dialogue mostly use text as their core input, and their speech and language emotion recognition only employs discrete label classification methods such as joy, anger, sorrow, and happiness. This approach has significant shortcomings in psychological health scenarios: discrete labels have poor stability across different groups and contexts, making it impossible to achieve refined expression of language emotions; single classification results are discrete values, making it difficult to directly use them as continuous control signals for AI dialogue strategies; and they only output a single emotion conclusion for the entire speech segment, lacking the dynamic trajectory of language emotion changes and failing to capture emotional transitions and fluctuations. This results in low accuracy in language emotion recognition, an inability to perform quantitative analysis of language emotion state transitions, and difficulty in supporting risk identification, empathy intervention, and safety strategy triggering in psychological scenarios.

[0003] In the field of psychology, PAD is an ideal framework for high-quality language emotion recognition, which can completely describe emotional state through multiple continuous values. However, existing technologies have not combined this framework with speech signal processing to form an emotion perception solution that is feasible, quantifiable, and compatible with psychological AI.

[0004] Therefore, there is an urgent need for a method that can output continuous emotion parameters, fixed time window trajectories, and state transition quantities based on voice output to solve the core problems of coarse granularity of emotion representation, lack of dynamic process, and disconnection from AI decision-making systems in existing technologies. Summary of the Invention

[0005] The purpose of this invention is to provide a method for quantifying emotion state perception based on a continuous emotion space in speech, solving the problems of coarse-grained emotion representation, lack of dynamic process quantification, and difficulty in effective linkage with AI decision-making systems in existing technologies. This invention breaks through the limitations of traditional speech emotion recognition, constructing a full-link language emotion recognition system oriented towards mental health. Through a continuous emotion space, it achieves refined representation, dynamic tracking, and state transition quantification of language emotions, providing highly usable language emotion recognition output for psychological dialogue AI.

[0006] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0007] A method for quantifying emotion state perception based on a continuous speech emotion space includes the following steps:

[0008] S1. Audio Acquisition and Preprocessing: Acquire user voice audio, unify sampling rate and channel format, complete duration segmentation, silence processing, and voice activity detection, providing a clean and standard voice signal for high-precision language emotion recognition. The audio is uniformly transcoded to a 16kHz sampling rate, mono floating-point waveform. 16kHz is the international standard sampling rate for voice emotion recognition, fully preserving emotional features such as rhythm, intonation, and timbre. Mono mode eliminates interference from stereo channel differences, ensuring consistency in feature extraction. It supports wav, mp3, and m4a formats and is compatible with multiple terminals such as mobile phones, microphones, and recording devices. When the duration is insufficient for the analysis window, only the global language emotion and sentiment results are output.

[0009] S2. Speech Representation Extraction: The preprocessed audio is input into the speech representation network to extract deep features carrying linguistic emotions from the speech, obtaining a time-series latent vector. This provides core feature input for linguistic emotion recognition. The speech representation network is a WAV2VEC2 self-supervised speech encoder, which does not require a large amount of manually labeled data and can autonomously learn emotional-related features such as prosody, rhythm, timbre, and voiceprint. It is currently the optimal solution for speech emotion feature extraction, efficiently extracting latent linguistic emotion features from speech and outputting a time-series latent vector. , where T is the number of time steps, representing the temporal emotional changes of the speech, and Ti is the number of time steps, corresponding to the emotional features of each frame of the speech;

[0010] S3. Continuous Emotional State Regression: Pooling is performed on the time-series latent vector H to obtain a segment-level representation of language emotion z. The regression head outputs a continuous emotional state vector representing valence, arousal, and sense of control, completing the continuous quantization output of core language emotion recognition. z is input into the regression head, and the output is a continuous emotional state vector e = ...<a,d,v> 'a' represents valence, 'd' represents arousal, and 'v' represents control. The output values ​​are mapped to the range of [0,1] or [-1,1] to adapt to the continuous quantification standard of language emotion.

[0011] The higher the valence value, the more pleasant and positive the mood; the lower the valence value, the more negative and depressed the mood.

[0012] The higher the arousal value, the more excited and agitated the emotions; the lower the arousal value, the calmer and more depressed the emotions.

[0013] The higher the sense of control score, the more dominant the user is; the lower the sense of control score, the more passive and helpless the user is.

[0014] S4. Derived Dimension Calculation: Based on continuous emotional state vectors, calculate the derived dimensions of tension and stress, enrich the dimensional system of language emotion recognition, and improve the comprehensiveness of language emotion analysis in mental health scenarios.

[0015] The formula for calculating tension is as follows: Tension directly reflects a user's psychological tension and is a core indicator for psychological risk identification. `clip` is a numerical clipping function to prevent results from exceeding the [0,1] interval. `λ∈[0,1]` is a weighting coefficient, with a default value of 0.7, emphasizing the interaction between arousal and valence. Higher arousal (A) and lower valence (V) result in stronger tension; conversely, high arousal combined with low control (D) also significantly increases tension. The parameter `λ` is used to balance the contributions of the two pathways.

[0016] S5. Global Emotion Output: Generates the global average emotion state vector and derived dimensions of the entire speech, outputs the global language emotion recognition result of the entire speech, and intuitively reflects the user's overall language emotion and emotional state. The global emotion is obtained by taking the arithmetic average of the emotion vectors of all effective time windows.

[0017] S6. Fixed-Period Emotion Timeline Generation: Speech is segmented according to fixed time windows, and the emotional state vector sequence of each time window is output to achieve real-time tracking and temporal presentation of dynamic changes in language emotion. The fixed time window length L is 3 seconds. In psychological scenarios, the minimum perceptible period of human emotional changes is 3 seconds, ensuring tracking accuracy. The shorter L is, the higher the temporal resolution, but the speech information within a single window may be insufficient, leading to unstable regression. The longer L is, the more robust the emotion estimation, but it may smooth out rapid emotional fluctuations. The step size H is 1-3 seconds. When H is less than L, the windows overlap, which can improve the smoothness of the timeline. A step size of 1 second with overlapping windows is high-resolution tracking; a step size of 3 seconds with no overlap is efficient computation. The temporal resolution of language emotion tracking can be flexibly adjusted. Each window outputs a separate timeline. and This forms an emotional timeline sequence;

[0018] S7. Emotional state transition quantification: Smooth the emotional timeline sequence, calculate the emotional transition vector, transition intensity and transition direction of adjacent periods, and complete the precise quantification of the evolution of language emotional state.

[0019] The smoothing process uses an exponential moving average, with the following formula: Where α is the smoothing coefficient; the larger α is, the weaker the smoothing effect, and the smaller α is, the stronger the smoothing effect, thus eliminating emotional spikes and noise. The smoothed emotion vector, This is the original emotion vector;

[0020] The formula for calculating the transition vector is: It directly quantifies changes in emotional values;

[0021] The method for calculating the transfer intensity is a weighted Euclidean distance, and the formula is as follows: ,in , The calculation results are cropped to the [0,1] interval. The weight allocation prioritizes valence and arousal, with each accounting for 40% of the weight. Both have a significant impact on psychological state assessment and AI intervention strategies. The sense of control accounts for 20% of the weight, which helps to understand changes in users' coping abilities.

[0022] The transfer direction angle is calculated in the AV plane, and the formula is: To judge the trend of emotional changes;

[0023] S8. Structured Interface Output: Outputs global emotion, timeline emotion, and state transition data in a structured format for use by the psychological dialogue AI, enabling seamless integration between language emotion recognition results and the AI ​​decision-making system.

[0024] As an improvement, step S7 also includes speaker baseline correction, which uses the average of the user's historical stable state emotions as the baseline to offset the current emotion vector, thereby eliminating the interference of individual differences on language emotion recognition.

[0025] As an improvement, the output data in step S8 includes the confidence level of language emotion recognition. The confidence level is determined by the effective speech duration, signal-to-noise ratio, and model stability. When the confidence level is lower than the threshold, a conservative language emotion conclusion is output to ensure the reliability of recognition.

[0026] The beneficial effects of this invention are as follows:

[0027] 1. By outputting continuous V / A / D values, it replaces discrete emotion labels, delicately expressing mixed emotions and gradual processes, perfectly adapting to the complexity of language emotions in psychological scenarios, and significantly improving the accuracy of language emotion recognition.

[0028] 2. By using fixed time windows and state transition quantification techniques, the dynamic evolution of language emotion is transformed into calculable indicators of transition intensity and direction, providing a new dimension for psychological state analysis.

[0029] 3. The continuous emotion parameters and transfer quantities output by language emotion recognition are directly used as control signals for decision-making logic such as dialogue AI strategy routing, empathy intensity adjustment, and security risk triggering, realizing a closed loop of the entire link from language emotion perception to intelligent interaction.

[0030] 4. The intensity of transfer can serve as a trigger condition for language and emotion risks, supporting the system to proactively intervene before emotions escalate, implement de-arousal strategies or rebuild the user's sense of control, and significantly improve the safety of language and emotion interaction in psychological dialogue AI.

[0031] 5. The method is modular and the output interface is standardized, making it easy to integrate into existing multimodal psychological AI dialogue pipelines and significantly reducing the deployment cost of language emotion recognition functions. Attached Figure Description

[0032] Figure 1 This is a flowchart of the audio acquisition, preprocessing, and speech representation extraction process of the present invention.

[0033] Figure 2 This is a schematic diagram of the continuous emotion space and derived tension index of the present invention.

[0034] Figure 3 This is a schematic diagram of the 3-second time window emotion timeline of the present invention.

[0035] Figure 4 This is a flowchart illustrating the calculation of the transfer vector, transfer intensity, and orientation angle in this invention.

[0036] Figure 5 This is a schematic diagram of the structured output data packet for AI in this invention. Detailed Implementation

[0037] To make the content of this invention easier to understand, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Identical components are indicated by the same reference numerals.

[0038] Example 1:

[0039] S1. Audio Acquisition and Preprocessing: The system acquires 16kHz mono WAV format audio input from the user. It performs speech activity detection, removing silent segments at the beginning and end to ensure signal quality for emotion recognition.

[0040] S2. Feature Extraction and Regression: The processed audio is input into a pre-trained wav2vec2 model, which outputs a time-series latent vector. , where T is the number of time steps;

[0041] Pooling the time series data yields a segment-level representation: ;

[0042] Mean pooling is used to obtain a fixed-length segment vector, which is then passed through a fully connected regression layer to output three values: V, A, and D. The values ​​are then restricted to the [0,1] interval by the Sigmoid function.

[0043] Continuous emotional states are output through the regression head: ,in It can be tanh, ReLU, or other activation functions;

[0044] The output is mapped to the standard interval [0,1][0,1][0,1], and the formula is as follows: ;

[0045] S3. Time Axis and Transition Quantization: Set the window length L to 3 seconds and the step size H to 3 seconds (no overlap). Repeat the above operation for each 3-second segment to obtain the emotion sequence. Perform smoothing on the emotion time axis sequence and calculate the emotion transition vector, transition intensity, and transition direction of adjacent cycles.

[0046] The smoothing process uses an exponential moving average, with the following formula: ;

[0047] The formula for calculating the transition vector is: ;

[0048] The method for calculating the transfer intensity is a weighted Euclidean distance, and the formula is as follows: ,in , The calculation results are then cropped to the [0,1] interval.

[0049] The transfer direction angle is calculated in the AV plane, and the formula is: ;

[0050] S4. Interface Output: Outputs JSON format data, including global sentiment and timeline sequence, to complete the standardized output of language sentiment recognition results.

[0051] Example 2:

[0052] S1. Audio Acquisition and Preprocessing: The system acquires 16kHz mono WAV format audio input from the user. It performs speech activity detection, removing silent segments at the beginning and end to purify the signal for high-precision speech emotion recognition.

[0053] S2. Feature Extraction and Regression: The processed audio is input into a pre-trained wav2vec2 model, which outputs a time-series latent vector. , where T is the number of time steps;

[0054] Pooling the time series data yields a segment-level representation: ;

[0055] Mean pooling is used to obtain a fixed-length segment vector, which is then passed through a fully connected regression layer to output three values: V, A, and D. The values ​​are then restricted to the [0,1] interval by the Sigmoid function.

[0056] Continuous emotional states are output through the regression head: ,in It can be tanh, ReLU, or other activation functions;

[0057] The output is mapped to the standard interval [0,1]. The linear mapping is [−1,1], such as x′=2x−1, to adapt to different PAD scales.

[0058] S3. Derived Dimension Calculation: Based on the continuous emotional state vector, calculate the derived dimensions of tension and stress.

[0059] When λ=0.7, the formula for calculating tension is: This is used to quantify users' tension or psychological burden and improve the multi-dimensional indicators of language emotion recognition.

[0060] S4. Fixed period: Set the window length L=3 seconds and the step h=1 second (with overlap) to obtain a time axis sequence with higher time resolution and generate a high time resolution language sentiment time sequence.

[0061] S5. Inertial Smoothing: Apply exponential moving average smoothing to the time-axis series, using a smoothing factor α = 0.6, to obtain the smoothed series. It effectively reduces noise spikes, calculates the language sentiment transfer vector, weighting intensity and direction angle, and generates semantic labels for sentiment changes;

[0062] S6. Transfer Quantization: Calculate the transfer vector. ;

[0063] Calculate the weighted transition strength, and set the weights. , ; , , ; transfer intensity ;

[0064] Calculate the direction angle on the AV plane And generate semantic labels: if Δak>0.2 and In the first quadrant, the label is "excitement-driven"; if Δak < −0.2, the label is "moderation / calmness".

[0065] Output global fluctuation statistics for overall evaluation: ;

[0066] Speaker baseline correction: The system stores the average emotion value of the user's 10 most recent stable states (transition intensity <0.1) as the individual baseline. Before each output, the current emotion value is subtracted from the baseline to obtain the offset relative to the user's normal state, so as to eliminate the systematic bias caused by individual timbre differences.

[0067] Confidence output: Calculate the percentage of valid speech within the current analysis window. If the percentage of valid speech is less than 50%, the confidence level is set to low and low confidence is noted in the output interface. It is recommended to obtain more speech.

[0068] S7. Interface Output: The final output JSON data structure includes version, user ID, global summary, detailed timeline (including confidence level), and fluctuation statistics (such as the standard deviation of the transition intensity of the entire audio segment), completing the entire process of language emotion recognition.

[0069] Example 3:

[0070] S1. Audio Acquisition and Preprocessing: The user records a 12-second MP3 voice message via mobile phone. The system automatically transcodes it to 16kHz mono WAV format. If the user hesitates and pauses for 1 second before speaking, VAD will automatically remove this segment to ensure that only valid voice is analyzed, retaining 10 seconds of valid voice. If the user only records 2 seconds of voice, the system directly outputs the global emotion result without generating a timeline sequence.

[0071] S2. Speech Representation Extraction: The speech input after 10 seconds of preprocessing is processed by the Wav2Vec2 model. Features are extracted at 20ms / frame to generate a time-series latent vector of T=500, which accurately captures the subtle emotional changes in speech at every moment.

[0072] S3. After performing mean pooling on the temporal latent vector, the regression head outputs V=0.7, A=0.6, and D=0.5, which means that the user's language emotion is more pleasant, moderately excited, and has a certain degree of control, rather than a simple discrete label of "happy".

[0073] S4. Calculation of derived dimensions: Taking λ=0.7, user A=0.8, V=0.3, D=0.4, substituting into the formula, we get: tension=clip(0.7×0.8×(1−0.3)+0.3×0.8×(1−0.4),0,1)=0.536, which means that the user is in a moderate state of tension and there is a clear sense of tension in their language and emotions;

[0074] S5. Global Emotion Output: The global emotion of a 10-second voice message is V=0.65, A=0.58, D=0.52, and tension=0.48, which directly reflects the user's overall stable and slightly pleasant emotions with mild tension.

[0075] S6. Fixed-period emotion timeline generation: Set a 3-second window and a 1-second step size, and generate 8 time window emotion sequences in 10 seconds of voice, accurately capturing the ups and downs and transitions of emotions at every moment;

[0076] S7. Quantification of Emotional State Transition: Smoothing coefficient α = 0.6, with the previous window smoothing the emotion. The original mood of the current window After smoothing The transition vector Δ = (0.06, 0.06, 0.06), transition strength = 0.06, and direction angle 45° represent a slight positive increase in user sentiment.

[0077] S8. Structured Interface Output: The system outputs a JSON data packet with a confidence level of 0.9 (high confidence). The psychological AI directly reads the emotional data and generates an empathetic response such as "I feel that your mood is slowly getting better."

[0078] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for quantifying the perception of emotional state based on a continuous emotional space of speech, characterized in that, Includes the following steps: S1. Audio Acquisition and Preprocessing: Acquire user voice audio, unify sampling rate and channel format, complete duration segmentation, silence processing and voice activity detection, distinguish between silent and noisy segments in speech segments, and avoid subsequent models extracting invalid features from silent segments. S2. Speech Representation Extraction: Input the preprocessed audio into the speech representation network to obtain the time series latent vector; S3. Continuous Emotional State Regression: Pooling is performed on the latent vector of the time series, and the continuous emotional state vector of valence, arousal and control is output through the regression head. The role of pooling is to compress the variable time series into a fixed-length segment-level representation. S4. Derived Dimension Calculation: Based on the continuous emotional state vector, the derived dimensions of tension and stress are calculated. Tension is a combined indicator that reflects the intensity of a user's emotions under anxiety, tension, or stress. S5. Global Emotion Output: Generates the global average emotion state vector and derived dimensions of the entire speech, outputs the global language emotion recognition result of the entire speech, and intuitively reflects the user's overall language emotion and emotional state. The global emotion is obtained by taking the arithmetic average of the emotion vectors of all effective time windows. S6. Fixed-period emotion timeline generation: Speech is segmented according to fixed time windows, and the emotion state vector sequence of each time window is output to realize real-time tracking and temporal presentation of dynamic changes in language emotion. S7. Emotional state transition quantification: Smooth the emotional timeline sequence, calculate the emotional transition vector, transition intensity and transition direction of adjacent periods, and complete the precise quantification of the evolution of language emotional state. S8. Structured Interface Output: Outputs global emotion, timeline emotion, and state transition data in a structured format for use by the psychological dialogue AI, enabling seamless integration between language emotion recognition results and the AI ​​decision-making system.

2. The method for quantifying emotion state perception based on a continuous speech emotion space according to claim 1, characterized in that: The audio in step S1 is uniformly transcoded to a 16kHz sampling rate, mono floating-point waveform, supporting wav, mp3, and m4a formats; when the duration is insufficient for the analysis window, only the global emotion is output.

3. The method for quantifying emotion state perception based on a continuous speech emotion space according to claim 1, characterized in that: The speech representation network mentioned in step S2 is the wav2vec2 self-supervised speech encoder. This model learns general acoustic representations on a large amount of unlabeled speech data and can capture acoustic cues closely related to emotions, such as intonation, stress, speech rate, and energy changes, and efficiently extract deep language emotion latent vectors from speech.

4. The method for quantifying emotion state perception based on a continuous speech emotion space according to claim 1, characterized in that: The pooling method described in step S3 is one of mean pooling, attention pooling, or statistical pooling. Mean pooling calculates the average value of all time steps, which is simple and effective. Attention pooling can automatically focus on time periods with richer emotional information. Statistical pooling introduces additional statistical measures such as variance and skewness to preserve the characteristics of emotional fluctuations. The output of the regression head is mapped to the [0,1] or [-1,1] interval through the Sigmoid or Tanh activation function to replace discrete emotion labels and refine the description of language emotions.

5. The method for quantifying emotion state perception based on a continuous speech emotion space according to claim 1, characterized in that: The formula for calculating tension in step S4 is as follows: Where a represents arousal, v represents valence, and d represents sense of control. The higher the arousal A and the lower the valence V, the stronger the tension. At the same time, high arousal combined with low sense of control D will also significantly increase tension. clip is a numerical clipping function that accurately quantifies the verbal and emotional expressions of psychological tension.

6. The method for quantifying emotion state perception based on a continuous speech emotion space according to claim 1, characterized in that: The fixed time window length mentioned in step S6 is 3 seconds, with a step size of 1-3 seconds, which balances the accuracy of language sentiment tracking with computational efficiency. The shorter the fixed time window length, the higher the temporal resolution; the longer the fixed time window length, the more robust the sentiment estimation.

7. The method for quantifying emotion state perception based on a continuous speech emotion space according to claim 1, characterized in that: The smoothing process described in step S7 uses an exponential moving average, with the following formula: Where α is the smoothing coefficient; the larger α is, the weaker the smoothing effect; the smaller α is, the stronger the smoothing effect. The smoothed emotion vector, This is the original emotion vector.

8. The method for quantifying emotion state perception based on a continuous speech emotion space according to claim 1, characterized in that: The formula for calculating the transition vector in step S7 is as follows: ; The method for calculating the transfer intensity is a weighted Euclidean distance, and the formula is as follows: ,in , The calculation results are then cropped to the [0,1] interval. The shift direction angle is calculated in the AV plane and is used to determine the main type of emotional change.

9. The method for quantifying emotion state perception based on a continuous speech emotion space according to claim 1, characterized in that: Step S7 also includes speaker baseline correction, which uses the average of the user's historical stable state emotions as the baseline to offset the current emotion vector, thereby eliminating the interference of individual differences on language emotion recognition.

10. The method for quantifying emotion state perception based on a continuous speech emotion space according to claim 1, characterized in that: The output data in step S8 includes confidence level, which is determined by the effective speech duration, signal-to-noise ratio, and model stability. When the confidence level is lower than the threshold, a conservative conclusion is output.

Images