Real-time psychological state recognition method and system based on voice emotion intensity analysis

Abstract

The invention discloses a real-time psychological state recognition method and system based on voice emotion intensity analysis, and relates to the technical field of voice recognition. According to the specific implementation scheme, after front-end preprocessing is carried out on an original audio stream, enhanced streaming audio is obtained, in the frame division process, audio multiple judgment is carried out in real time to distinguish silence and muting, optimization real-time performance evaluation is synchronously carried out to determine whether audio optimization judgment is called or not, and the audio optimization judgment is called or not. Then hierarchical feature extraction is carried out to output a time sequence feature matrix, the time sequence feature matrix serves as input of an emotion intensity estimation model, finally time sequence smoothing and multi-scale fusion are carried out on multi-dimensional emotion intensity vectors output by the emotion intensity estimation model to obtain an emotion intensity time sequence, and the emotion intensity time sequence is input into a psychological state inference model. And mapping to obtain a psychological state and outputting. According to the technical scheme of the invention, the problem that the psychological state recognition credibility is reduced due to inaccurate emotion intensity caused by weak emotion sound and low mute precision of voice recognition is solved.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a method and system for real-time psychological state recognition based on speech emotion intensity analysis. Background Technology

[0002] By analyzing people's voice signals in real time, identifying and assessing their emotional state and mental health, this technology can provide individuals with accurate emotional feedback, mental health assessments, and personalized emotion regulation solutions. This can improve service quality, optimize experience, and enhance emotional interaction in multiple fields (such as mental health, customer service, and human-computer interaction). The implementation of this technology can detect mental health problems earlier, provide timely intervention, and support more natural emotional communication.

[0003] For example, Chinese invention patent CN115101053B discloses a dialogue processing method, device, terminal, and storage medium based on emotion recognition, which includes: during communication between a customer service representative and a customer service representative based on a business scenario, collecting the customer's first voice information and the customer service representative's second voice information; identifying the customer's emotion based on the first voice information; if the customer's emotion is negative, generating dialogue speech based on the first and second voice information according to the time point when the customer's negative emotion occurs; then performing voice analysis on the dialogue speech to obtain voice analysis results; performing text analysis on the dialogue speech to obtain text analysis results; and generating customer reassurance strategy information based on the voice analysis results and text analysis results.

[0004] For example, Chinese invention patent CN115101074B discloses a speech recognition method, device, medium, and equipment based on user speaking emotions. This includes: acquiring user speech, performing frequency response analysis on the user speech to generate a frequency response curve corresponding to the user speech, performing text recognition on the user speech to generate text data corresponding to the user speech, determining a target scene type based on multiple historical text data and the text data corresponding to the user speech, determining a preset emotion frequency response curve for the user speech under the target scene type based on the target scene type, comparing the frequency response curve with the preset emotion frequency response curve to determine the emotion attribute of the frequency response curve, determining the user's intent based on the text data when the emotion attribute is normal, and supplementing the text data based on emotion color when the emotion is abnormal, and determining the user's intent based on the supplemented text data.

[0005] Generally, the implementation of this type of technology first requires the acquisition and preprocessing of speech signals. This involves using devices such as microphones to acquire human speech signals. In real-world environments, speech signals are often interfered with by background noise. Therefore, preprocessing of the speech, such as noise reduction and echo cancellation, is usually necessary to improve speech quality. Next, feature extraction is performed. Common features include audio features, speakerprint features, time-domain features, and frequency-domain features. Then, emotion classification and psychological state inference are performed. This means that the extracted features are then fed into an emotion recognition model to infer the speaker's emotions. Commonly used models include traditional machine learning models and deep learning models. Finally, real-time recognition and feedback are performed. Through a real-time speech emotion analysis system, the speaker's psychological state can be quickly identified, such as emotional state (e.g., happiness, anger, sadness, fear) and emotional intensity (e.g., the strength of the emotion). Psychological stress can also be inferred by analyzing features such as tension, urgency, and trembling in the speech.

[0006] The above-mentioned technology has at least the following technical problems: On the one hand, real-time emotion recognition systems often need to use streaming ASR (Automatic Speech Recognition) to convert speech content into text in real time, providing real-time data for subsequent emotion analysis. However, it suffers from problems such as network latency, segmented frame loss, and inaccurate speech segmentation, which can cause "jumps" or "delays" in emotion intensity, affecting real-time psychological state monitoring.

[0007] On the other hand, VAD (Voice Activity Detection) is sensitive to weak voices or background noise. Low-energy voices such as soft voices, trembling, and crying are easily mistaken for silence. Background noise (wind noise, crowd noise, mechanical noise) may trigger incorrect "voice segments", emotional key points may be accidentally deleted or segmented incorrectly, and intensity curves may be broken.

[0008] Because VAD (Voice Activity Detection) mainly relies on "energy threshold and simple spectral features" to determine whether speech exists, while emotional weak speech has too low acoustic energy and indistinct features; conversely, background noise has stable energy and some frequency bands are similar to human voice, so it is easy to be misjudged.

[0009] Meanwhile, weak-energy emotional speech has low energy, weak spectral characteristics, and does not conform to normal speech patterns, while background noise often has high energy and human-like frequency characteristics. In addition, VAD is based on energy and simple spectral judgment, which makes weak speech easy to be misjudged as silence and noise easy to be misjudged as speech, thus causing emotional key points to be covered or segmented incorrectly. Summary of the Invention

[0010] This invention provides a real-time psychological state recognition method and system based on speech emotion intensity analysis. It can improve the accuracy of emotion intensity analysis by increasing the precision of distinguishing between weak emotional sounds and silence during speech recognition, thereby enhancing the reliability of psychological state recognition. The technical solution provided by this application is as follows: According to a first aspect of this application, a real-time psychological state recognition method based on speech emotion intensity analysis is provided. The method includes: performing front-end preprocessing on the acquired raw audio stream to obtain enhanced streaming audio, and then dividing it into frames to obtain individual audio streams. The front-end preprocessing is used to perform noise reduction, echo removal, and anti-saturation processing on the raw audio stream. During frame division, real-time audio multi-judgment is performed to distinguish between silence and weak sounds, and a real-time optimization assessment is performed simultaneously to determine whether to call the corresponding audio optimization judgment, thereby improving the accuracy of audio processing. Based on the speech signal obtained after audio multi-judgment, hierarchical feature extraction is performed, and the corresponding time-series feature matrix is ​​output as input to an emotion intensity estimation model. The multi-dimensional emotion intensity vector output by the emotion intensity estimation model is subjected to temporal smoothing and multi-scale fusion to obtain an emotion intensity time series, which is then input into a psychological state inference model to map and output the psychological state.

[0011] According to another aspect of this application, a real-time psychological state recognition system based on speech emotion intensity analysis is provided. This system applies a method for real-time psychological state recognition based on speech emotion intensity analysis. The system includes: a raw audio preprocessing module, a silence / weak voice differentiation module, an emotion intensity estimation module, and a psychological state inference output module. The raw audio preprocessing module performs front-end preprocessing on the acquired raw audio stream to obtain enhanced streaming audio, and then divides it into frames to obtain individual audio streams. The front-end preprocessing is used to perform noise reduction, echo removal, and anti-saturation processing on the raw audio stream. The silence / weak voice differentiation module performs real-time audio multi-judgment to distinguish between silence and weak voices during frame division, and simultaneously performs real-time optimization assessment to determine whether to call the corresponding audio optimization judgment, thereby improving the accuracy of audio processing. The emotion intensity estimation module performs hierarchical feature extraction on the speech signal obtained after audio multi-judgment, outputting a corresponding time-series feature matrix as input to the emotion intensity estimation model. The psychological state inference output module performs temporal smoothing and multi-scale fusion on the multi-dimensional emotion intensity vector output by the emotion intensity estimation model to obtain an emotion intensity time series, which is then input into the psychological state inference model to map and output the psychological state.

[0012] The beneficial effects of the technical solutions provided by the embodiments of the present invention include at least the following: 1. Enhanced streaming audio is obtained by preprocessing the acquired raw audio stream. This not only helps avoid the problem of unstable dynamic range of the raw audio causing difficulty in generalizing subsequent feature extraction, but also improves the signal-to-noise ratio of weak sounds. Frame segmentation is performed to obtain audio streams of each frame. During frame segmentation, multi-level audio judgment is performed in real time to distinguish between silence and weak sounds, which helps avoid misjudging weak emotional signals as silence. This solves the problem that existing VADs cannot recognize weak sounds (breathing, sobbing, choking) and treat them as silence. Simultaneously, real-time optimization evaluation is performed to determine whether to call the corresponding audio optimization judgment, which helps reduce latency and reduce computing resource consumption. Then, hierarchical feature extraction is performed on the speech signal obtained after multi-level audio judgment, which is more conducive to capturing emotional nuances. The gradient of emotion changes is used to output the corresponding time-series feature matrix, which serves as input to the emotion intensity estimation model. This provides multi-channel input in both time and feature dimensions, enabling the emotion model to utilize the complete temporal context. Finally, the multi-dimensional emotion intensity vector output by the emotion intensity estimation model is subjected to temporal smoothing and multi-scale fusion to obtain the emotion intensity time series. This solves the problems of high noise and strong fluctuations in the output of real-time models and also helps to improve the stability and reliability of psychological state inference. This data is then input into the psychological state inference model to map and output the psychological state. This addresses the lack of sensitivity to weak emotion changes in existing technologies and helps to improve the accuracy of distinguishing between silence and weak sounds in speech recognition, thereby improving the accuracy of mild emotion analysis and ensuring the reliability of psychological state inference.

[0013] 2. Verifying the real-time evaluation conditions for real-time evaluation metrics and real-time judgment parameters not only solves the problem of traditional voice emotion analysis systems being insensitive to changes in device computing power and prone to stuttering or uncontrolled latency under high load, but also helps to comprehensively characterize the current computing resource load through multi-dimensional performance metrics. This avoids misjudging the system's real-time capabilities due to relying on a single metric. Furthermore, if all real-time evaluation metrics meet the set real-time evaluation conditions, the corresponding audio optimization judgment is invoked for voice emotion intensity analysis. This improves the recognition ability of weak signals such as weak speech, emotional breathing, and sobbing, and also prevents the waste of computing power caused by using low-precision strategies on high-performance platforms. If any real-time evaluation metric fails to meet the real-time evaluation conditions, the deviation of the corresponding real-time evaluation metric from its real-time judgment parameter is quantified to obtain the corresponding real-time performance tolerance value of the computer, which is then mapped to obtain the number of optimizable judgments. Based on the number of optimizable judgments, audio optimization judgments are called sequentially, realizing an adaptive scheduling mechanism that enables optimization in stages according to performance margins. This preserves the accuracy of emotion judgment as much as possible while ensuring real-time performance. If none of the real-time evaluation metrics meet the corresponding real-time evaluation conditions, the initial audio multi-judgment is used to analyze the speech emotion intensity, avoiding frame accumulation, buffer overflow, or system blocking caused by high-complexity calculations.

[0014] 3. The short-time energy change rate is calculated by differentiating the short-time energy of adjacent audio stream frames. By comparing the energy changes of adjacent frames rather than the absolute energy, even very low-amplitude weak sounds (such as inhalation, sobbing, and suppressed sounds) can still be detected, enhancing the weak sound detection capability. The obtained harmonic-to-noise ratio (HNR) - short-time energy influence is compensated with the set short-time energy change range to obtain the short-time energy optimization range. This helps reduce the probability of false positives and improves the stability of weak emotional signal detection, making emotion-related acoustic signals such as crying, vibrato, and slight sighs easier to identify. If the short-time energy change rate is higher than the maximum value of the short-time energy optimization range, it indicates that the audio stream frame is not silent, and further emotion intensity analysis is performed. This analysis prevents the traditional VAD from failing to effectively distinguish between "weak speech" and "light noise," thus avoiding the loss of weak emotions. If the short-time energy change rate is lower than the minimum value of the short-time energy optimization range, it indicates that the audio frame is silent, and the process is returned. This helps to quickly filter out most silent frames, reduce computational consumption, and enhance system stability. This avoids mistaking background noise and environmental disturbances for speech, thereby reducing the false trigger rate. If the short-time energy change rate is within the short-time energy optimization range, spectral entropy judgment is performed, which helps to quickly confirm the presence of speech, reduce latency, and avoid performing high-complexity calculations such as spectral entropy on obvious speech frames, reducing computational overhead. At the same time, it also improves the accuracy of weak emotion signal recognition.

[0015] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0016] The accompanying drawings are provided for a better understanding of this solution and do not constitute a limitation of this application. Wherein: Figure 1 This is a flowchart of a real-time psychological state recognition method based on voice emotion intensity analysis provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the process for optimizing real-time performance evaluation provided in an embodiment of the present invention; Figure 3 This is a flowchart of short-time energy optimization judgment provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a real-time psychological state recognition system based on voice emotion intensity analysis provided in an embodiment of the present invention. Detailed Implementation

[0017] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0018] Embodiment 1 of this invention provides a real-time psychological state recognition method based on voice emotion intensity analysis. For example... Figure 1 The diagram shown is a flowchart of a real-time psychological state recognition method based on voice emotion intensity analysis provided by an embodiment of the present invention. The processing flow of this method may include the following steps: Step 1: After performing front-end preprocessing on the acquired raw audio stream, enhanced streaming audio is obtained, and then frame-divided to obtain each frame audio stream. Front-end preprocessing is used to perform noise reduction, echo removal, and anti-saturation processing on the raw audio stream.

[0019] It should be noted that the front-end preprocessing cleans and enhances the original audio through algorithms such as noise reduction, echo removal, and anti-saturation, making the enhanced streaming audio more stable and clear, suitable for subsequent analysis. The main methods used include, but are not limited to, spectral subtraction, echo path model (such as AEC: Acoustic Echo Cancellation), and Dynamic Range Compression (DRC). Frame segmentation divides the enhanced audio into segments using fixed-length windows (such as 20–30ms) and uses overlap and windowing to generate a continuous frame-level audio stream for subsequent feature extraction and model input. Commonly used windows include the Hann window and the Hamming window.

[0020] Step two: During the frame segmentation process, multiple audio judgments are performed in real time to distinguish between silence and weak sounds, and a real-time optimization assessment is performed simultaneously to determine whether to call the corresponding audio optimization judgment, thereby improving the accuracy of audio processing.

[0021] It should be explained that the multiple determination of audio includes short-time energy determination, spectral entropy determination, and zero-crossing rate determination.

[0022] First, the specific content of short-time energy judgment is as follows: the signal of each frame of audio stream is input into the short-time energy calculation formula to obtain the corresponding short-time energy, and the corresponding fundamental frequency is obtained at the same time. If the fundamental frequency is obtained, the initial short-time energy judgment threshold is compensated according to the preset short-time energy assessment influence ratio to obtain the short-time energy optimization threshold; otherwise, no additional processing is performed. Among them, the short-time energy assessment influence ratio is data preset by the staff based on historical data and empirical rules, and is generally stored in a preset database for later use. The compensation processing means multiplying the short-time energy assessment influence ratio by the initial short-time energy judgment threshold.

[0023] Specifically, fundamental frequency identification is a common task in speech and audio analysis, which can be detected by time-domain, frequency-domain, or time-frequency-domain methods, including but not limited to autocorrelation, spectral peak detection, and CEPSTRUM cepstral method.

[0024] If the short-time energy is higher than the short-time energy optimization threshold, it means that the audio stream of the corresponding frame is not silent, and the spectral entropy judgment continues; if the short-time energy is not higher than the short-time energy optimization threshold, it means that the audio stream of the corresponding frame may be silent, and the process returns.

[0025] Specifically, the formula for calculating short-time energy is as follows: ; Where E represents the short-time energy of the audio stream in this frame, n represents the index of the sampling point, N represents the number of sampling points in the current frame, and x[n] represents the audio signal value of the nth sampling point of the audio signal; the physical meaning of this formula is to sum the squares of the amplitudes of all sampling points in the current frame to obtain the total energy of the frame.

[0026] Next, the specific content of spectral entropy judgment is as follows: the corresponding spectrum is obtained based on the audio stream signal of the current frame, and the corresponding probability distribution is obtained accordingly; the spectrum is obtained by performing a short-time Fourier transform on the audio signal of the current frame, and the spectrum or its features are input into a neural network acoustic model (such as DNN, CNN, RNN or Transformer), and the probability distribution of the corresponding category can be obtained by passing the Softmax function of the model output layer.

[0027] The probability distribution is substituted into the spectral entropy formula to obtain the corresponding spectral entropy, and then compared with the initial spectral entropy judgment threshold. If the spectral entropy is less than the spectral entropy judgment threshold, it indicates the existence of a speech structure, and the speech emotion intensity analysis continues. Otherwise, it indicates that it may be silence or noise, and the zero-crossing rate is judged. The spectral entropy judgment threshold is obtained in the same way as existing technologies, and is generally set by preset staff based on historical data and experience.

[0028] Specifically, the specific expression for the spectral entropy formula is as follows: ; In the formula, H represents spectral entropy, k represents frequency index, K represents the total number of spectral components, and P(k) represents the normalized power spectral density of the k-th frequency component. The default base of the logarithmic function is e.

[0029] Finally, the specific process for zero-crossing rate determination is as follows: import the audio stream signal of the current frame into the zero-crossing rate calculation expression, and output the zero-crossing rate of the corresponding frame. Specifically, the expression for calculating the zero-crossing rate takes the following form: ; ; In the formula, ZCR represents the zero-crossing rate, N represents the number of sampling points in the frame, x[n] represents the audio signal value of the nth sampling point, and x represents the signal amplitude.

[0030] If the zero-crossing rate of the audio stream frame falls within the initial silence judgment range set to limit the silence range (usually a range set by the staff that is close to and greater than 0), it indicates that the audio stream frame may be silent, and the process is returned; otherwise, the voice emotion intensity analysis of the audio stream frame continues.

[0031] Step 3: Based on the audio multi-determination, perform hierarchical feature extraction on the speech signal and output the corresponding time series feature matrix as input to the emotion intensity estimation model.

[0032] It should be added that hierarchical feature extraction can employ existing mature acoustic analysis methods, including low-level acoustic features based on short-time analysis (such as short-time energy, zero-crossing rate, spectral entropy, etc.), mid-level perceptual features based on Mel frequency and cepstral analysis (such as MFCC, Log-Mel filter bank energy, HNR, etc.), and high-level temporal features extracted based on machine learning models (such as deep acoustic embedding features such as CNN / LSTM / Transformer, etc.). These existing methods can characterize the energy structure, spectral characteristics, and time series patterns of speech at different levels, thereby outputting a time series feature matrix that can be used for sentiment analysis.

[0033] Specifically, the emotion intensity estimation model can employ existing mature speech emotion recognition models, such as LSTM, GRU, CNN-RNN hybrid networks, self-attention-based Transformer models, or self-supervised learning-based Wav2Vec2 / HuBERT / wavLM as feature extractors, outputting continuous emotion intensity values ​​through a regression head. Numerous existing technologies in this field support the implementation of these models.

[0034] Step four: Perform temporal smoothing and multi-scale fusion on the multidimensional emotion intensity vector output by the emotion intensity estimation model to obtain the emotion intensity time series, and input it into the psychological state inference model to map and output the psychological state.

[0035] It should be noted that temporal smoothing can employ existing mature methods such as moving average, exponential moving average, weighted filtering, Kalman filtering, and Gaussian smoothing to reduce short-term fluctuations in the emotion intensity vector; multi-scale fusion can employ existing techniques such as multi-window feature fusion, temporal pyramid, wavelet transform, multi-scale convolution, or attention-weighted fusion to integrate emotional information from different temporal receptive fields, thereby obtaining a more continuous, stable, and accurate time series of emotion intensity.

[0036] It's worth noting that existing technologies for inferring mental states primarily employ multiple approaches: First, language models based on dialogue text, such as the Seein model and the lightweight Menta model designed specifically for mental health, which infer emotions and beliefs by analyzing language; second, computational models based on physiological signals, such as the high-precision and interpretable PhysioFormer and brain-computer AI emotion assessment models, which quantify emotions using objective data such as heart rate and EEG; third, deep learning frameworks based on multimodal fusion, which improve the accuracy of comprehensive judgment by integrating various information such as text, speech, and facial expressions, and employing attention mechanisms; and fourth, basic models simulating human cognition, such as the Centaur model, which aims to learn and simulate deep human decision-making processes through massive behavioral data. These technologies approach the problem from different dimensions, and in practical applications, they are often selected or integrated based on the specific needs of the scenario (such as an emphasis on privacy, objectivity, interpretability, or comprehensiveness).

[0037] In this embodiment, by preprocessing and segmenting the original audio stream at the front end, combined with real-time audio multi-judgment and real-time evaluation, the accuracy and real-time performance of speech signal processing are improved. First, audio multi-judgment effectively distinguishes between silence and weak sounds. By optimizing the real-time evaluation to determine whether to call high-precision audio optimization judgment, stable operation is ensured under different system loads. Second, hierarchical feature extraction is performed on the speech signal obtained from audio multi-judgment, covering low-level acoustic features, mid-level spectral features, and high-level emotional features, which is beneficial for capturing emotional changes in speech. Based on the above features, the emotion intensity estimation model is further input, and high-quality emotion intensity time series is generated through temporal smoothing and multi-scale fusion, providing more accurate input for the psychological state inference model. Finally, the model can more accurately map the user's psychological state and provide real-time feedback on emotional changes. The method in this embodiment not only improves the accuracy of distinguishing weak emotional sounds and silence in speech recognition, but also improves the accuracy of emotion analysis. At the same time, it can adaptively adjust under different computing platforms and hardware conditions, ensuring the real-time performance and stability of the system when performing emotion intensity analysis and psychological state recognition.

[0038] like Figure 2 The diagram illustrates the process of optimizing real-time evaluation according to an embodiment of the present invention. The specific logic is as follows: Real-time evaluation indicators acquired in real-time are compared with extracted real-time judgment parameters to verify real-time evaluation conditions. If all real-time evaluation indicators meet the set real-time evaluation conditions, the corresponding audio optimization judgment is invoked to perform voice emotion intensity analysis. If any real-time evaluation indicator does not meet the real-time evaluation conditions, the deviation degree of the corresponding real-time evaluation indicator from its real-time judgment parameter is quantified to obtain the corresponding real-time performance tolerance value of the computer, which is then mapped to obtain the number of optimizable judgments. Based on the number of optimizable judgments, audio optimization judgments are invoked sequentially. If none of the real-time evaluation indicators meet the corresponding real-time evaluation conditions, the initial audio multiple judgments are used to perform voice emotion intensity analysis. Through the above process, not only is the accuracy of speech recognition improved, but it can also adaptively adjust in the application environment, ensuring the real-time processing capability of psychological emotion intensity analysis and psychological state recognition.

[0039] Example 2: Based on Example 1, in order to prevent the impact of multiple audio optimization judgments on the real-time performance of emotion intensity analysis, a real-time performance evaluation is performed simultaneously with the multiple audio judgments. The specific process of the real-time performance evaluation is as follows: The real-time evaluation metrics acquired in real time are verified against the extracted real-time judgment parameters. The real-time evaluation metrics include single-frame processing time, CPU utilization, single-core utilization ratio, and concurrency latency. The real-time judgment parameters include the maximum single-frame processing time, the maximum CPU utilization, the maximum single-core utilization ratio, and the maximum concurrency latency.

[0040] Specifically, the real-time evaluation conditions include: single frame processing time is less than the maximum single frame processing time, CPU utilization is less than the maximum CPU utilization, single core utilization is less than the maximum single core utilization, and concurrent latency is greater than the maximum concurrent latency.

[0041] If all real-time evaluation indicators meet the set real-time evaluation conditions, the corresponding audio optimization judgment is invoked to perform voice emotion intensity analysis. The audio optimization judgment includes short-time energy optimization judgment, spectral entropy optimization judgment, and zero-crossing rate optimization judgment.

[0042] If any real-time evaluation metric does not meet the real-time evaluation conditions, the deviation of the corresponding real-time evaluation metric from its real-time judgment parameter is quantified to obtain the corresponding computer real-time performance tolerance value, which is then mapped to obtain the number of optimizable judgments. Based on the number of optimizable judgments, audio optimization judgments are called sequentially.

[0043] Specifically, deviation quantification means that the difference between the real-time evaluation index that does not meet the real-time evaluation conditions and the corresponding real-time judgment parameter is calculated, and then the ratio is calculated to obtain the computer's real-time performance tolerance value. If there is more than one real-time evaluation index that does not meet the real-time evaluation conditions, the average of the deviation quantification results of each real-time evaluation index is calculated to obtain the computer's real-time performance tolerance value.

[0044] It should be added that the real-time performance tolerance value of the computer is input into the real-time assessment sequence, and the corresponding number of optimizable judgments is output. The real-time assessment sequence is used to fit the mapping relationship between the real-time performance tolerance value of the computer and the number of optimizable judgments. The real-time assessment sequence is obtained by training the initial data sequence constructed by the linear regression algorithm and relying on the least squares criterion and the statsmodels framework. The training data used are the real-time performance tolerance value of the computer obtained in the historical time period and the number of optimizable judgments set according to empirical rules.

[0045] If none of the real-time evaluation indicators meet the corresponding real-time evaluation conditions, then the initial audio multiple judgment will be used to analyze the intensity of speech emotion.

[0046] In this embodiment, by acquiring and verifying real-time evaluation metrics, including single-frame processing time, CPU utilization, single-core occupancy ratio, and concurrency latency, the current computing power and processing load of the system can be accurately determined. When all metrics meet the conditions, the system invokes a complete audio optimization judgment, including short-time energy, spectral entropy, and zero-crossing rate optimization analysis, thereby maximizing the accuracy of speech emotion intensity analysis while ensuring real-time performance. Furthermore, when some metrics do not meet the conditions, the number of judgments can be optimized by quantifying the degree of deviation, achieving adaptive invocation of optimization judgments, which ensures system stability while preserving the accuracy of emotion intensity analysis as much as possible. At the same time, when all metrics do not meet the conditions, an initial audio multiple judgment is adopted as a fallback solution to ensure the continuous availability of basic emotion analysis. This solution not only improves the robustness and accuracy of speech emotion recognition but also enables adaptive adjustment under different loads and hardware environments, balancing computing resource utilization efficiency and real-time processing capabilities, and ensuring the stable operation and continuity of the system in complex real-time scenarios.

[0047] like Figure 3 The diagram shows a flowchart of the short-time energy optimization judgment provided in this embodiment of the invention. The specific logic is as follows: The acquired harmonic noise ratio (HNR) is matched with each HNR interval extracted from a preset database, and the corresponding HNR influence ratio is obtained based on the proportion of each HNR in its corresponding HNR interval. If both the short-time energy assessment influence ratio and the HNR influence ratio exist, a difference calculation is performed between the two to obtain the difference in influence ratios. This difference is then compared with a preset difference. If the difference in influence ratios is greater than the preset difference, the larger of the two values ​​is recorded as HNR - short-time energy. The impact amount is calculated as follows: If the difference in the impact ratios is not greater than a preset difference, the average of the two is recorded as the harmonic noise ratio - short-time energy impact amount. If only one of the short-time energy assessment impact ratio and the harmonic noise ratio impact ratio exists, the obtained ratio is recorded as the harmonic noise ratio - short-time energy impact amount. If neither the short-time energy assessment impact ratio nor the harmonic noise ratio impact ratio exists, the short-time energy change threshold is not optimized. Based on the obtained harmonic noise ratio - short-time energy impact amount, it is determined whether to call the spectral entropy optimization judgment. Through the above process, the combination of harmonic noise ratio and short-time energy in the process of affecting the emotional intensity of speech recognition is ensured, which helps to improve the reliability of the psychological state recognition results.

[0048] Example 3: Based on Example 2, if a short-term energy optimization judgment is triggered during the short-term energy judgment process, the specific process is as follows: The obtained harmonic noise ratio is matched with each harmonic noise ratio interval extracted from the preset database, and the corresponding harmonic noise ratio influence ratio is obtained based on the proportion of each harmonic noise ratio in the corresponding harmonic noise ratio interval. That is, the absolute value of the difference between the harmonic noise ratio and the minimum value of the harmonic noise ratio interval is calculated by the ratio of the absolute value of the difference between the minimum and maximum values ​​of the harmonic noise ratio interval.

[0049] If the harmonic noise ratio (NNR) falls within the first NNR range, it indicates a possible state of silence; if it falls within the second NNR range, it indicates a possible state of breathing or sobbing; and if it falls within the third NNR range, it indicates a possible state of weak speech. The first, second, and third NNR ranges are all NNR differentiation ranges, which are preset data by the staff. They are pre-set and stored in a preset database based on historical data and empirical rules for later retrieval.

[0050] If both the short-time energy assessment impact ratio and the harmonic noise ratio impact ratio exist, then the difference between the short-time energy assessment impact ratio and the harmonic noise ratio impact ratio is calculated to obtain the difference in the corresponding impact ratio. The difference in the impact ratio is then compared with a preset difference value, where the preset difference value is the maximum value that measures the degree of difference between the short-time energy assessment impact ratio and the harmonic noise ratio impact ratio, and is preset by the staff in advance.

[0051] If the difference in the impact ratios is greater than the preset difference, the larger of the two, namely the short-time energy assessment impact ratio and the harmonic noise ratio impact ratio, will be recorded as the harmonic noise ratio - short-time energy impact amount.

[0052] If the difference in the impact ratio is not greater than the preset difference, the average of the two is recorded as the harmonic-to-noise ratio minus the short-time energy impact. If there is only one of the short-time energy assessment impact ratio and the harmonic noise ratio impact ratio, the obtained ratio will be recorded as harmonic noise ratio - short-time energy impact amount; If neither the short-time energy assessment impact ratio nor the harmonic noise ratio impact ratio exists, then the short-time energy change threshold will not be optimized.

[0053] The determination of whether to invoke spectral entropy optimization is based on the obtained harmonic-to-noise ratio-short-time energy influence.

[0054] Specifically, the process for determining whether to invoke spectral entropy optimization is as follows: The short-time energy change rate is obtained by calculating the difference between the short-time energy of adjacent audio stream frames. This means that after performing a difference operation on the short-time energy of two adjacent audio stream frames, the absolute value is taken. The resulting harmonic noise ratio-short-time energy impact is then compensated with a set short-time energy change range. Specifically, the harmonic noise ratio-short-time energy impact is multiplied by the minimum and maximum values ​​of the short-time energy change range to obtain the short-time energy optimization range. The short-time energy change range is the initial range, which is generally a static range set in advance by the staff.

[0055] If the short-time energy change rate is higher than the maximum value of the short-time energy optimization interval, it indicates that the audio stream of that frame is not silent, and emotion intensity analysis continues; if the short-time energy change rate is lower than the minimum value of the short-time energy optimization interval, it indicates that the audio stream of that frame is silent, and the process is returned; if the short-time energy change rate is within the short-time energy optimization interval, spectral entropy judgment is performed.

[0056] In this embodiment, by matching the acquired harmonic noise ratio (HNR) with various HNR intervals in a preset database, the influence ratio of the HNR in the audio signal can be calculated and quantified more accurately. By combining the influence ratio with short-time energy assessment, the system obtains the optimized influence of HNR and short-time energy through difference calculation, and judges the degree of optimization based on the preset difference. This optimization process ensures the appropriate combination of HNR and short-time energy in influencing emotion recognition, and provides the most accurate emotion intensity analysis within a controllable performance range. When the difference between the two is too large, the system automatically selects a larger value as the influence to ensure the effectiveness of optimization. When the difference between the two is small, the mean is used as the influence to avoid over-adjustment. At the same time, in the absence of an influence, the system does not perform short-time energy threshold optimization, thereby avoiding redundant calculations. In this way, the system can more flexibly and accurately call spectral entropy optimization judgment, further improving the accuracy and robustness of emotion recognition. Finally, the method provided in this embodiment enables the system to adaptively adjust the audio optimization strategy without exceeding real-time requirements, achieving higher accuracy in speech emotion analysis.

[0057] Example 4: Based on Example 2, if the spectral entropy optimization judgment is triggered during the short-time energy judgment process, the specific details are as follows: The spectral structure features of each frame of audio stream are extracted to obtain the corresponding spectral structure data, which includes spectral centroid, spectral flux, and spectral flatness. At the same time, after obtaining the spectral structure data, data normalization is performed uniformly.

[0058] Specifically, spectral structure feature extraction is based on short-time Fourier transform and amplitude spectrum calculation, as well as standard formulas to extract spectral centroid, spectral flux and spectral flatness. It is a mature technology widely used in the existing audio analysis and emotion recognition fields.

[0059] Based on the spectral structure data, spectral structure performance coefficients are obtained to quantify the possibility of speech structure, and then mapped to obtain the spectral entropy determination optimization quantity. Specifically, the spectral structure data after data normalization is multiplied by the corresponding weights and then added to obtain the spectral structure performance coefficients. The weights include spectral centroid weight, spectral flux weight, and spectral flatness weight, the sum of which is 1. These weights are preset by the staff based on historical data and empirical rules within a historical time period and stored in a preset database.

[0060] It should be added that the spectral structure performance coefficients are input into the trained spectral entropy determination optimization data table, and the corresponding spectral entropy determination optimization amount is obtained by comparison. The spectral entropy determination optimization data table is used to reflect the correlation between the spectral structure performance coefficients and the spectral entropy determination optimization amount. The spectral structure performance coefficients in the historical time period, as well as the spectral entropy determination optimization amount set by professionals based on empirical rules, are input into the initial data table constructed by the logistic regression algorithm. The data is then trained using the least squares criterion and the statsmodels framework to the corresponding spectral entropy determination optimization data table.

[0061] The spectrum and probability distribution of the current frame audio stream are obtained, and these are substituted together with the spectral entropy determination optimization amount into the spectral entropy formula to obtain the corresponding optimized spectral entropy.

[0062] The optimized spectral entropy is compared with the spectral entropy judgment threshold. If the optimized spectral entropy is less than the spectral entropy judgment threshold, it indicates that there may be a speech structure, and the speech emotion intensity analysis continues. Otherwise, it indicates that the audio stream of this frame may be silence and noise, and the zero-crossing rate judgment is performed.

[0063] In this embodiment, by extracting spectral structure features from each frame of audio stream, the spectral morphology and acoustic structure characteristics of the audio can be more accurately characterized. Based on these features, spectral structure performance coefficients are calculated, thereby quantifying the possibility of the existence of speech structure and mapping it to obtain the spectral entropy determination optimization amount, which helps to improve the determination accuracy of spectral entropy in weak speech and noise environments. Subsequently, the spectrum and probability distribution of the current frame and the spectral entropy determination optimization amount are substituted into the spectral entropy formula to obtain the optimized spectral entropy, thus realizing the dynamic adjustment and determination of the complex structure of the speech signal. Moreover, when the optimized spectral entropy is lower than a set threshold, the system can continue to perform speech emotion intensity analysis and further perform zero-crossing rate judgment, thereby enhancing the reliability of emotion feature extraction. This scheme not only improves the detection capability of weak speech and low-energy speech, but also ensures the stability of spectral entropy determination under noise interference, enhances the accuracy and robustness of overall speech emotion analysis, and maintains the real-time processing performance of the system.

[0064] Example 5: Based on Example 2, if the zero-crossing rate optimization judgment is triggered during the short-time energy judgment process, the specific process is as follows: The system detects whether there is an envelope fluctuation rate in the frame. If so, it maps the zero-crossing rate to obtain the zero-crossing rate determination influence value. Then, it performs a compensation operation on the zero-crossing rate and the zero-crossing rate determination influence value to obtain the optimized zero-crossing rate. That is, it performs a product operation on the zero-crossing rate and the zero-crossing rate determination influence value to obtain the optimized zero-crossing rate, and compares it with the silence determination proximity interval. If the optimized zero-crossing rate belongs to the silence determination proximity interval, it indicates that there may be silence, so it returns. Otherwise, it continues to perform voice emotion intensity analysis.

[0065] Envelope modulation rate is a common task in speech and audio analysis. Envelope modulation rate describes the slow oscillation of signal energy over time. It can be detected by time domain, frequency domain, or time-frequency domain methods, such as Hilbert envelope detection, frequency detection of short-time energy sequences, and moving average difference method.

[0066] Specifically, the envelope fluctuation rate is input into the zero-crossing rate optimization queue that has been trained, and the corresponding zero-crossing rate judgment influence value is obtained. The zero-crossing rate optimization queue is used to reflect the correlation between the envelope fluctuation rate and the zero-crossing rate judgment influence value. The envelope fluctuation rate in the historical time period and the zero-crossing rate judgment influence value set by professionals based on empirical rules are input into the initial data queue constructed by the logistic regression algorithm. The data is trained based on the least squares criterion and the statsmodels framework to obtain the corresponding zero-crossing rate optimization queue.

[0067] In this embodiment, the zero-crossing rate determination is dynamically adjusted by detecting the envelope fluctuation rate of the audio frame. When an envelope fluctuation rate is detected in the frame, the system generates a zero-crossing rate determination influence value based on the feature mapping and obtains an optimized zero-crossing rate through compensation calculation. This compensation mechanism can effectively correct the zero-crossing rate deviation caused by envelope fluctuation, thereby improving the accuracy of speech activity detection. In addition, comparing the optimized zero-crossing rate with the silence determination proximity interval helps to eliminate silence frames or low-energy interference frames more quickly, avoids invalid calculations, and improves the system processing efficiency. If the optimized zero-crossing rate does not belong to the silence interval, the speech emotion intensity analysis continues to ensure the integrity and continuity of emotion feature extraction. Therefore, while maintaining real-time processing capability, this method not only improves the recognition accuracy of weak speech and non-stationary signals, but also enhances the robustness and reliability of speech emotion analysis and effectively reduces the probability of misjudging silence frames.

[0068] Example 6: This invention also provides a real-time psychological state recognition system based on voice emotion intensity analysis, such as... Figure 4The diagram shown is a schematic representation of the structure of a real-time psychological state recognition system based on voice emotion intensity analysis provided in an embodiment of the present invention. It includes a raw audio preprocessing module, a silence / weak voice differentiation module, an emotion intensity estimation module, and a psychological state inference output module.

[0069] The original audio preprocessing module is used to perform front-end preprocessing on the acquired original audio stream to obtain enhanced streaming audio, and to divide the original audio stream into frames to obtain each frame audio stream. The front-end preprocessing is used to perform noise reduction, echo removal, and anti-saturation processing on the original audio stream.

[0070] The silence and weak sound differentiation module is used to perform multiple audio judgments in real time during frame division to distinguish between silence and weak sound, and simultaneously perform real-time optimization evaluation to determine whether to call the corresponding audio optimization judgment, thereby improving the accuracy of audio processing.

[0071] The emotion intensity estimation module is used to perform hierarchical feature extraction on the speech signal obtained after multiple audio judgments, and output the corresponding time series feature matrix as the input of the emotion intensity estimation model.

[0072] The psychological state inference output module is used to perform temporal smoothing and multi-scale fusion on the multidimensional emotion intensity vector output by the emotion intensity estimation model to obtain the emotion intensity time series, which is then input into the psychological state inference model to map and output the psychological state.

[0073] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0074] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device, equipment, and storage medium embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0075] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0076] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A real-time psychological state recognition method based on voice emotion intensity analysis, characterized in that, The method includes: After preprocessing the acquired raw audio stream, enhanced streaming audio is obtained, and then frame-divided to obtain each frame audio stream; During frame segmentation, multiple audio judgments are performed in real time to distinguish between silence and weak sound, and real-time optimization assessment is performed simultaneously to determine whether to call the corresponding audio optimization judgment. Based on the speech signal obtained after multiple audio determination, hierarchical feature extraction is performed, and the corresponding time series feature matrix is ​​output as the input of the emotion intensity estimation model. The multidimensional emotion intensity vector output by the emotion intensity estimation model is subjected to temporal smoothing and multi-scale fusion to obtain the emotion intensity time series, which is then input into the psychological state inference model to map and output the psychological state.

2. The real-time psychological state recognition method based on voice emotion intensity analysis as described in claim 1, characterized in that, The audio multi-determination includes short-time energy determination, spectral entropy determination, and zero-crossing rate determination, with the specific determinations as follows: The specific content of the short-time energy judgment is as follows: input the signal of each frame audio stream into the short-time energy calculation formula to obtain the corresponding short-time energy, and at the same time obtain the corresponding fundamental frequency. If the fundamental frequency is obtained, the initial short-time energy judgment threshold is compensated according to the preset short-time energy assessment influence ratio to obtain the short-time energy optimization threshold; otherwise, no additional processing is performed. If the short-time energy is higher than the short-time energy optimization threshold, it means that the audio stream of the corresponding frame is not silent, and the spectral entropy judgment continues. If the short-time energy is not higher than the short-time energy optimization threshold, then return.

3. The real-time psychological state recognition method based on voice emotion intensity analysis as described in claim 2, characterized in that, The specific details of the spectral entropy determination are as follows: The corresponding spectrum is obtained based on the audio stream signal of the current frame, and the corresponding probability distribution is obtained accordingly. Substitute the probability distribution into the spectral entropy formula to obtain the corresponding spectral entropy, and compare it with the initial spectral entropy judgment threshold. If the spectral entropy is less than the spectral entropy judgment threshold, it indicates that there is a speech structure, and the speech emotion intensity analysis continues; otherwise, the zero-crossing rate judgment is performed.

4. The real-time psychological state recognition method based on voice emotion intensity analysis as described in claim 2, characterized in that, The specific process for determining the zero-crossing rate is as follows: The audio stream signal of the current frame is imported into the zero-crossing rate calculation expression, and the zero-crossing rate of the corresponding frame is output. If the zero-crossing rate of the audio stream frame falls within the initial silence judgment range set to limit the silence range, then return; otherwise, continue to analyze the speech emotion intensity of the audio stream frame.

5. The real-time psychological state recognition method based on voice emotion intensity analysis as described in claim 1, characterized in that, The specific process for optimizing real-time performance evaluation is as follows: The real-time evaluation metrics acquired in real time are verified against the extracted real-time judgment parameters to meet the real-time evaluation conditions. The real-time evaluation metrics include single-frame processing time, CPU utilization, single-core utilization ratio, and concurrency latency. If all real-time evaluation indicators meet the set real-time evaluation conditions, the corresponding audio optimization judgment is called to perform voice emotion intensity analysis. The audio optimization judgment includes short-time energy optimization judgment, spectral entropy optimization judgment and zero-crossing rate optimization judgment. If a real-time evaluation indicator does not meet the real-time evaluation conditions, the deviation of the corresponding real-time evaluation indicator from its real-time judgment parameter is quantified to obtain the corresponding computer real-time performance tolerance value, so as to map the number of optimizable judgments, and the audio optimization judgment is called sequentially based on the number of optimizable judgments. If none of the real-time evaluation indicators meet the corresponding real-time evaluation conditions, then the initial audio multiple judgment will be used to analyze the intensity of speech emotion.

6. The real-time psychological state recognition method based on voice emotion intensity analysis as described in claim 5, characterized in that, The specific process of the short-time energy optimization judgment is as follows: The obtained harmonic noise ratio is matched with each harmonic noise ratio distinction interval extracted from the preset database, and the corresponding harmonic noise ratio influence ratio is obtained based on the proportion of each harmonic noise ratio in the corresponding harmonic noise ratio interval. If both the short-time energy assessment impact ratio and the harmonic noise ratio impact ratio exist, then the difference between the short-time energy assessment impact ratio and the harmonic noise ratio impact ratio is calculated to obtain the corresponding difference in impact ratios. This difference in impact ratios is then compared with a preset difference. If the difference in the impact ratio is greater than the preset difference, the larger of the two values ​​will be recorded as the harmonic noise ratio - short-time energy impact. If the difference in the impact ratio is not greater than the preset difference, the average of the two is recorded as the harmonic-to-noise ratio minus the short-time energy impact. If there is only one of the short-time energy assessment impact ratio and the harmonic noise ratio impact ratio, the obtained ratio will be recorded as harmonic noise ratio - short-time energy impact amount; If neither the short-time energy assessment impact ratio nor the harmonic noise ratio impact ratio exists, then the short-time energy change threshold will not be optimized. The determination of whether to invoke spectral entropy optimization is based on the obtained harmonic-to-noise ratio-short-time energy influence.

7. The real-time psychological state recognition method based on voice emotion intensity analysis as described in claim 6, characterized in that, The specific process for determining whether to invoke spectral entropy optimization is as follows: The short-time energy change rate is obtained by calculating the difference in short-time energy of adjacent audio stream frames. The resulting harmonic-to-noise ratio (HNR) - short-time energy impact is then compensated against the set short-time energy change range to obtain the short-time energy optimization range. If the short-time energy change rate is higher than the maximum value of the short-time energy optimization range, it indicates that the audio stream of that frame is not silent, and emotion intensity analysis continues. If the short-time energy change rate is lower than the minimum value of the short-time energy optimization range, it indicates that the audio frame is silent, and the process should be returned. If the short-time energy change rate falls within the short-time energy optimization range, then spectral entropy is used for judgment.

8. The real-time psychological state recognition method based on voice emotion intensity analysis as described in claim 5, characterized in that, The specific details of the spectral entropy optimization judgment are as follows: Spectral structure features are extracted from each frame of audio stream to obtain corresponding spectral structure data, which includes spectral centroid, spectral flux, and spectral flatness. Based on the spectral structure data, spectral structure performance coefficients are obtained to quantify the possibility of the existence of speech structure, and then the spectral entropy determination optimization quantity is obtained by mapping. The corresponding spectrum and probability distribution are obtained from the signal of the current frame audio stream, and then substituted together with the spectrum entropy determination optimization amount into the spectrum entropy formula to obtain the corresponding optimized spectrum entropy. The optimized spectral entropy is compared with the spectral entropy judgment threshold. If the optimized spectral entropy is less than the spectral entropy judgment threshold, the speech emotion intensity analysis continues; otherwise, the zero-crossing rate is judged.

9. The real-time psychological state recognition method based on voice emotion intensity analysis as described in claim 5, characterized in that, The specific process for zero-crossing rate optimization judgment is as follows: The system detects whether there is an envelope fluctuation rate in the frame. If it exists, it maps the zero-crossing rate to obtain the zero-crossing rate determination influence value. Then, it performs a compensation operation on the zero-crossing rate and the zero-crossing rate determination influence value to obtain an optimized zero-crossing rate and compares it with the silence determination proximity interval. If the optimized zero-crossing rate belongs to the silence determination proximity interval, it returns to the previous state; otherwise, it continues to perform voice emotion intensity analysis.

10. A real-time psychological state recognition system based on speech emotion intensity analysis, which applies the real-time psychological state recognition method based on speech emotion intensity analysis as described in any one of claims 1-9, including an original audio preprocessing module, a silence / weak voice differentiation module, an emotion intensity estimation module, and a psychological state inference output module; in, The original audio preprocessing module is used to perform front-end preprocessing on the acquired original audio stream to obtain enhanced streaming audio, and to perform frame division to obtain each frame audio stream. The mute and weak sound differentiation module is used to perform multiple audio judgments in real time during the frame division process to distinguish between mute and weak sound, and simultaneously perform real-time optimization evaluation to determine whether to call the corresponding audio optimization judgment, thereby improving the accuracy of audio processing. The emotion intensity estimation module is used to perform hierarchical feature extraction on the speech signal obtained after multiple audio judgments, and output the corresponding time series feature matrix as the input of the emotion intensity estimation model. The psychological state inference output module is used to perform temporal smoothing and multi-scale fusion on the multidimensional emotion intensity vector output by the emotion intensity estimation model to obtain the emotion intensity time series, which is then input into the psychological state inference model to map and output the psychological state.

Images