Psychological counseling real-time speech recognition method based on multi-modal data

Abstract

The invention relates to the technical field of speech recognition, and discloses a psychological counseling real-time speech recognition method based on multi-modal data, which comprises the following steps: analyzing and judging an emotion stability tendency degree by constructing a gender-exclusive speech psychological feature mapping baseline, and judging an expression tendency degree in combination with an amplitude peak value proportion; the fundamental frequency mean amplitude is dynamically corrected, and psychological feature misjudgment caused by individual pronunciation difference is solved; detecting accent positions, keywords, speech speed and pause features through fundamental frequency abrupt change and amplitude abrupt change, constructing a multi-dimensional emotion analysis model, and optimizing emotion focus positioning, emotion type judgment and change trend tracking problems; designing a comprehensive stable value calculation method, reflecting the emotional stability and the keyword influence degree at the same time, and providing quantitative indexes for mental health assessment; and constructing a three-level processing mechanism, and performing feature adaptation, cross validation and decision correction through exclusive baseline and multi-modal evidence fusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, in particular to a psychological counseling real-time speech recognition method based on multi-modal data. BACKGROUND

[0002] With the penetration of artificial intelligence technology in the field of mental health, speech recognition and emotion analysis technology has gradually become an important tool for psychological counseling decision-making. The existing technology extracts the acoustic features of the voice, combines a machine learning model to realize emotion recognition and preliminary judgment of the psychological state. However, the existing machine learning model still has significant limitations in the professional adaptation of the psychological counseling scene, and it is difficult to meet the needs of clinical counseling for accuracy, individual differences and quantitative evaluation. In the prior art, the existing speech psychological feature analysis technology generally adopts a general model approach and does not make targeted design for the physiological voice differences between men and women. The existing technology adopts a unified feature threshold and does not consider the influence of gender differences on feature distribution. The pronunciation pattern of the psychological counseling object is strongly related to the psychological state, but the existing technology does not quantitatively analyze the phoneme-level pronunciation features and only processes pronunciation errors through a speech recognition error correction mechanism, resulting in feature distortion. The existing technology does not fuse the real background information of the counseling object. Age growth will naturally cause the vibration frequency of the vocal cords to decrease, and there is a lack of a dynamic correction mechanism. It relies on a single feature dimension and identifies emotion types through keyword matching, ignoring the indication of stress position on emotional focus. It quantifies the timing features such as speech rate and pause, but cannot associate the semantics of keywords, resulting in keywords spoken by different emotions being classified into the same emotion level. In the balance between real-time performance and accuracy, enlarging the sliding window to ensure feature integrity increases the real-time response delay. Reducing the window size to improve real-time performance increases the emotion misjudgment rate due to incomplete semantic units, lacks coherent tracking of emotion change trends, only classifies emotions for a single window, cannot identify gradual emotion changes, and is difficult to assist consultants in grasping the counseling rhythm. The existing technology for evaluating the mental health state is mostly limited to qualitative description and lacks a quantitative index system. Consultants develop intervention frequency and programs based on the evaluation results, but qualitative description cannot provide specific references. Emotion intensity scores are introduced, but they do not combine keywords with long-term emotional impact and overall stability of emotional fluctuations, resulting in scores only reflecting instantaneous state and lacking comprehensive evaluation value. Therefore, it is necessary to provide a psychological counseling real-time speech recognition method based on multi-modal data. SUMMARY

[0003] The purpose of the present application is to provide a psychological counseling real-time speech recognition method based on multi-modal data. To solve the above-mentioned problems of the prior art, the present application realizes the following technical solutions: The psychological counseling real-time speech recognition method based on multi-modal data provided by the embodiment of the present application specifically comprises the following steps: Step 1: By constructing a gender-specific speech psychological feature mapping baseline, the degree of emotional stability tendency is judged by the initial analysis of the fundamental frequency mean and fundamental frequency variation coefficient, and the degree of expression tendency is judged by the proportion of amplitude peaks. Step 2: Combining pronunciation habit details with real data, dynamically correct the fundamental frequency mean amplitude to resolve misjudgments of psychological characteristics caused by individual pronunciation differences; Step 3: Detect stress position, keywords, speech rate and pause features by fundamental frequency mutation and amplitude mutation, construct a multi-dimensional sentiment analysis model, and optimize the issues of sentiment focus localization, sentiment type judgment and trend tracking. Step 4: Based on the correlation between keywords and emotional changes, design a comprehensive stability value calculation method that reflects both emotional stability and the degree of influence of keywords, and provides a quantitative indicator for mental health assessment. Step 5: To address language barriers and severe emotional disturbances, a three-tiered processing mechanism is constructed, which integrates a dedicated baseline with multimodal evidence to perform feature adaptation, cross-validation, and decision correction.

[0004] Furthermore, the method for constructing a gender-specific voice psychometric feature mapping baseline is as follows: A dual-microphone array was used to collect the initial dialogue voice of the consultation subject. The main microphone was placed directly in front of the visitor to focus on the target voice, and the reference microphone was placed to the side and rear to collect the ambient noise. The sampling rate and quantization bit depth were uniformly set. The LMS adaptive filtering algorithm is used to reduce noise in the dual-microphone signal, suppressing non-steady-state noise in the consultation room and preserving the pure voice signal; the mean fundamental frequency and the fundamental frequency variation coefficient are calculated for the fundamental frequency physiological range of men and women respectively, and the amplitude peak ratio is calculated in combination with the amplitude dynamic range of men and women. Based on the pure voice signals of male and female visitors of different ages and occupations, the random forest algorithm is used to train and optimize gender-specific mapping rules to achieve initial judgments on emotional stability tendency, emotional sensitivity tendency, and expression tendency. Furthermore, the method for dynamically correcting the mean amplitude of the fundamental frequency is as follows: The Kaldi speech recognition toolkit was used to decompose the preprocessed speech into phonemes, extract three types of pronunciation habit features, and calculate the deviation rate of the average duration of vowel pronunciation from the standard duration of Mandarin, the standard deviation of the pronunciation intensity of consonant initials, and the deviation rate of the four tones frequency from the standard tone of Mandarin. Structured information on the age, occupation, dialect background, and past psychological state of the counseling subjects is obtained, and correction factors are constructed. The corrected fundamental frequency mean is calculated by combining the original fundamental frequency mean with the age factor and occupation factor. The fundamental frequency variation coefficient and amplitude peak percentage are recalculated using the corrected fundamental frequency mean and amplitude, and the psychological characteristics are updated a second time by combining the pronunciation habit markers. To further clarify, the method for constructing the multi-dimensional sentiment analysis model is as follows: Set a 3-second sliding window to extract the maximum fundamental frequency, average fundamental frequency, maximum amplitude, and average amplitude; The stress position is determined by the joint distribution entropy of the fundamental frequency mutation value and the amplitude mutation value and the maximum value of the two; the words at the stress position are matched with a keyword library for psychological counseling selected by the TF-IDF algorithm; the real-time speech rate, speech rate change trend, single pause duration and pause frequency are calculated with a 3-second window and a 1-second step size; the emotion judgment rule is trained and optimized by the decision tree algorithm, and the model is constructed by combining keywords and multi-dimensional features. Based on multi-dimensional features, a model is built to analyze and judge the emotional classification of the counseling object, and the emotional change trend is calculated according to the emotional classification to optimize the problems of emotional focus positioning, emotional type judgment and change trend tracking. To further clarify, the method for determining the emotional classification of the consultation subject is as follows: Emotion determination rules are constructed by combining multi-dimensional features. The rules are trained and optimized by decision tree algorithm. By combining keywords, real-time speech rate, single pause duration and speech rate change trend, the emotion intensity value is analyzed, and the emotion category of the consultation object is determined based on the emotion intensity value. Furthermore, the method for obtaining the aforementioned trend of emotional change is as follows: Obtain the sum of the differences in emotion intensity values ​​from three consecutive windows, and analyze the trend of emotion changes based on the sum of the differences in emotion intensity values; If the trend of mood change is greater than 5, the mood will rapidly rise, either positive or negative; if the sum of the differences in the trend of mood change is less than -5, the mood will rapidly cool down, ease, or turn neutral; otherwise, the mood change will be stable. Furthermore, the method for obtaining the comprehensive stability value is as follows: The difference between the maximum emotional intensity within 3 seconds after the keyword appears and the average value within 3 seconds before its appearance is calculated, and the influence coefficient is defined. The standard deviation of emotional intensity and the number of emotional type jumps are calculated. The emotional fluctuation is obtained by combining the standard deviation of emotional intensity with the number of jumps. By combining mapping to establish a two-dimensional interval deduction item table, the sum of the keyword influence coefficients and the deduction value of emotional fluctuation are accumulated, and the total deduction value is obtained by subtracting the preset full score from the total deduction value; To further clarify, the method for quantifying the aforementioned mental health assessment is as follows: The emotional stability values ​​were standardized and categorized into stable, relatively stable, unstable, and high-risk types. If the number of emotion type jumps exceeds the frequency threshold or the absolute value of the keyword influence coefficient exceeds the conflict threshold, suspicious text containing the keyword, the time of occurrence, and the emotion change curve are marked; if the emotion stability value is <40 points, an early warning prompt is automatically triggered and synchronized to the counselor's terminal; the DS evidence theory is used to integrate the emotion stability value, the amount of stress position shift, the amount of pause interval change, and the rate of speech rate change to construct a correlation model, and the crisis correlation is obtained through the BPA function and the synthesis rule, and the crisis level is divided in combination with the change value of the emotion stability value; Furthermore, the method for feature adaptation is as follows: We collected voice samples from individuals seeking stuttering counseling, aligned abnormal voice sequences using a dynamic time warping algorithm, extracted pathological features to establish a baseline specific to each stuttering type, and categorized stuttering into developmental and acquired stuttering; we also analyzed emotion-related features. Pathological feature masking is used to process stuttering segments in speech, while preserving the emotional association features of non-stuttering segments. Fuzzy logic reasoning is then used to correct misjudgments. To further explain, the method for constructing the three-level processing mechanism is as follows: To address the distortions in vocalizations, vibrato, and breathy sounds during emotional outbursts, a layered enhancement algorithm is employed. Wavelet thresholding is used to decompose the speech signal and soft thresholding is applied to high-frequency plosive components. Adaptive Kalman filtering is used to smooth high-frequency fluctuations in vibrato. Spectral subtraction is used to improve the harmonic-to-noise ratio and extract the spectral centroid and Mel-frequency cepstral coefficients. If the speech distortion exceeds the preset limit, multimodal cross-validation is initiated, and the consistency between the dynamic change rate of 68 facial key points and the emotional features of the distorted speech is determined. The determination of negative emotions is strengthened when the degree of abnormality in heart rate variability and skin conductivity reaches the high arousal threshold.

[0005] This invention provides a real-time speech recognition system for psychological counseling based on multimodal data, which specifically includes the following modules: Analysis and Judgment Module: By constructing a gender-specific voice-psychological feature mapping baseline, the module uses the fundamental frequency mean and fundamental frequency variation coefficient for initial analysis to determine the degree of emotional stability tendency, and combines the amplitude peak ratio to determine the degree of expression tendency. Dynamic correction module: Combining pronunciation habit details with real data, it dynamically corrects the fundamental frequency mean amplitude to resolve misjudgments of psychological characteristics caused by individual pronunciation differences; The optimized analysis module detects stress position, keywords, speech rate, and pause features through fundamental frequency and amplitude mutations, constructs a multi-dimensional sentiment analysis model, and optimizes the issues of sentiment focus localization, sentiment type judgment, and trend tracking. Quantitative assessment module: Combining the correlation between keywords and emotional changes, a comprehensive stability value calculation method is designed to reflect both emotional stability and the degree of influence of keywords, providing quantitative indicators for mental health assessment; Adaptation and Correction Module: To address language barriers and severe emotional disorders, a three-tiered processing mechanism is constructed, which integrates a dedicated baseline with multimodal evidence fusion to perform feature adaptation, cross-validation, and decision correction.

[0006] The beneficial effects of this invention are: 1. Breaking through the limitations of traditional general speech feature analysis, this paper constructs a correction model by setting different physiological ranges of fundamental frequency and amplitude for males and females, training a unique mapping rule using random forest, and combining pronunciation habits with multi-dimensional real data to solve misjudgments caused by individual pronunciation differences. It uses the combined distribution entropy of fundamental frequency and amplitude mutation to determine the stress position, integrates keyword database, speech rate and pause features to construct a decision tree model, and designs a two-dimensional comprehensive stable value calculation method of keyword influence coefficient and emotion fluctuation degree to achieve emotion classification, trend tracking and quantitative grading of mental health. 2. For speech disorders such as stuttering, the system adapts pathological feature masking and emotional feature reconstruction, and combines multimodal cross-validation of facial key points and physiological indicators to solve the problems of distorted speech emotion extraction and crisis misjudgment, forming a complete technical system from basic recognition to special scene coverage. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 This is a flowchart of the steps of a real-time speech recognition method for psychological counseling based on multimodal data provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the structure of a real-time speech recognition system for psychological counseling based on multimodal data, provided in Embodiment 2 of the present invention. Detailed Implementation

[0009] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0010] Example 1: As Figure 1 As shown in the figure, the real-time speech recognition method for psychological counseling based on multimodal data provided by this invention specifically includes the following steps: Step 1: By constructing a gender-specific speech psychological feature mapping baseline, the degree of emotional stability tendency is judged by the initial analysis of the fundamental frequency mean and fundamental frequency variation coefficient, and the degree of expression tendency is judged by the proportion of amplitude peaks. In a specific embodiment, a dual-microphone array is used to collect the initial dialogue duration of the client. The main microphone is placed directly in front of the client to focus on collecting the target speech; the reference microphone is placed to the side and rear to collect ambient noise. The sampling rate is uniformly set, and the bit depth is quantized to ensure the integrity of speech details. The dual-microphone signal is denoised using the LMS adaptive filtering algorithm, which tracks changes in noise characteristics in real time, suppresses non-steady-state noise in the consultation room, preserves a clean speech signal with a signal-to-noise ratio of ≥25dB, and provides a basis for feature extraction. Core speech features were extracted for men and women separately, and the differences in features resulting from the differences in the physiological vocalization mechanisms of men and women were used to differentiate them. Specifically, the physiological range of male fundamental frequency is preset to [85Hz, 180Hz], and the mean fundamental frequency F0 is calculated based on the pure speech signal. m To reflect the fundamental vocal state, the fundamental frequency variation coefficient (CV) is calculated. F0 Reflects the degree of emotional fluctuation; amplitude volume dynamic range [40dB, 80dB], obtain the peak duration and divide by the total duration to calculate the amplitude peak percentage P. amp Among them, peak duration is defined as the period when the amplitude exceeds 75% of the preset maximum peak value. Peak duration reflects the initiative and emotional intensity of the expression. The physiological range of female fundamental frequency is [165Hz, 255Hz]. Calculate the mean fundamental frequency F0. m With fundamental frequency variation coefficient CV F0 ; It should be noted that the fundamental frequency of women is generally higher than that of men, and the variation range of women is generally greater than that of men. The amplitude dynamic range is [35dB, 75dB]. Considering that the average volume of women is lower than that of men, the amplitude peak threshold is adjusted to 70% of the preset maximum peak time period, and the amplitude peak percentage P is calculated simultaneously. amp index; Based on pure voice signals from psychological counseling, covering male and female clients of different ages and professions, gender-specific mapping rules are constructed. The rules are trained and optimized through random forest algorithm. The degree of emotional stability is judged by initial analysis of the fundamental frequency mean and fundamental frequency variation coefficient, and the degree of expression tendency is judged by combining the amplitude peak ratio. For example, for males, if the mean fundamental frequency is less than 110Hz and the coefficient of variation is less than 0.15, they are initially labeled as having a stable emotional tendency; if the mean fundamental frequency is greater than 150Hz or the coefficient of variation is greater than 0.25, they are initially labeled as having a sensitive emotional tendency; otherwise, they are initially labeled as having an unstable emotional tendency. For females, if the mean fundamental frequency is less than 190Hz and the coefficient of variation is less than 0.2, they are initially labeled as having a stable emotional tendency; if the mean fundamental frequency is greater than 220Hz or the coefficient of variation is greater than 0.3, they are initially labeled as having a sensitive emotional tendency; otherwise, they are initially labeled as having an unstable emotional tendency. For any counseling subject, if the percentage of peak amplitude is greater than 0.3, they are labeled as having an active tendency; if the percentage of peak amplitude is less than 0.15, they are labeled as having a passive tendency; otherwise, they are labeled as having a normal tendency. Step 2: Combining pronunciation habit details with real data, dynamically correct the fundamental frequency mean amplitude to resolve misjudgments of psychological characteristics caused by individual pronunciation differences and improve the accuracy of analysis; In a specific embodiment, the preprocessed speech is decomposed into phonemes at the phoneme level using the Kaldi speech recognition toolkit to achieve phoneme segmentation, extract three types of pronunciation habit features, and comprehensively characterize individual pronunciation patterns. To obtain the deviation rate between the average pronunciation duration of each vowel in the consultation subject and the standard pronunciation duration of vowels in Mandarin, the vowel duration deviation D is calculated by subtracting the standard pronunciation duration from the average pronunciation duration and then dividing by the standard pronunciation duration. vow If D vow If the duration deviation of a vowel is greater than the preset deviation ratio, the client is marked as having a habit of prolonging vowels. This habit of prolonging vowels is manifested in clients who are emotionally repressed or hesitant in expressing themselves. Analyze the intensity variation of initial consonants in the consultation subjects and obtain the standard deviation (SD) of the initial consonant pronunciation intensity of the consultation subjects. con If the strength standard deviation SD con If the value is greater than 5dB, it is marked as a habitual fluctuation in the strength of the initial consonant, reflecting the stability of airflow control during pronunciation. Large fluctuations may be related to anxiety. Otherwise, it is marked as normal pronunciation of consonant initials. Analyze the tone accuracy of the consultation subjects and obtain the tone deviation rate D between the frequencies of the four tones (high level, rising, falling-rising, and falling) of the consultation subjects and the frequencies of the standard Mandarin tones. tone If the tone deviation rate D tone If the value is greater than 0.25, it is marked as an unstable tone recording habit; otherwise, it is marked as a stable tone recording habit. We obtain structured information from the real data uploaded by the clients, including but not limited to: age, occupation, dialect background, and past psychological state. We then combine this information with clinical statistics to construct correction factors, adjust the fundamental frequency mean and amplitude, and achieve precise compensation for individual differences. For example, the mean value of the corrected fundamental frequency after compensation is analyzed and calculated. Analyze and calculate the corrected amplitude after compensation. ;in, This is the original fundamental frequency mean. The original amplitude, The age factor is used, for example, 0.05 for [20, 30] years old, 0.08 for [31, 40] years old, 0.12 for [41, 50] years old, and 0.15 for 51 years old and above. It is determined by combining the tracking analysis of the 5-year trend of the fundamental frequency of speech in different age groups, reflecting the natural influence of aging on the vibration frequency of the vocal cords. Occupational factors: Teachers or sales professionals tend to have higher vocal intensity due to long-term high-frequency vocalization, while programmers or researchers tend to have lower daily vocal intensity. The settings are based on statistical data of the average daily vocal intensity of each occupational group. The dialect factor is set at 0.04 for Northern dialects, 0.07 for Southern dialects, and 0.09 for Cantonese. It is determined based on the quantitative analysis results of the difference in vowel duration between dialects and Mandarin, to compensate for the influence of dialect pronunciation habits on amplitude. The historical psychological factors were used, with a history of anxiety of 0.09, a history of depression of 0.11, and no history of such a history of 0.02. These factors were derived from a comparative study of the speech characteristics of patients with a history of mental illness and healthy individuals. α, β, γ, and δ were all adjustment factors, determined by training from historical data using 10-fold cross-validation, with α=0.3, β=0.25, γ=0.2, and δ=0.35 to ensure that each factor contributed reasonably to the correction results. Use the corrected fundamental frequency mean F0 corr and correction amplitude A corr Recalculate the fundamental frequency variation coefficient CV F0 and the percentage of peak amplitude P amp By combining pronunciation habit markers such as prolonged vowels, fluctuating initials, and unstable tones, a comprehensive judgment is made, and the psychological characteristics are updated a second time. For example, if the original fundamental frequency mean of the consultation subject is F0 m =230Hz, fundamental frequency variation coefficient CV F0 =0.32, then it is marked as emotionally sensitive tendency; if the corrected fundamental frequency mean F0 corr =210Hz, fundamental frequency variation coefficient CV F0 =0.18, and there are no abnormal pronunciation habits, so the overall trend is updated to emotional stability; if the corrected fundamental frequency variation coefficient CV F0 =0.22, and there is a habit of prolonging vowels, then it is updated to be more emotionally sensitive, taking into account the influence of customer characteristics and behavioral habits; Step 3: Detect stress position, keywords, speech rate and pause features by fundamental frequency mutation and amplitude mutation, construct a multi-dimensional sentiment analysis model, and optimize the issues of sentiment focus localization, sentiment type judgment and trend tracking. In a specific embodiment, stress positions are detected by fundamental frequency and amplitude mutations, with stressed words corresponding to core words expressing emotions; a 3-second sliding window is set to extract the maximum fundamental frequency value F0. max and fundamental frequency average value F0 avg Maximum amplitude A max and the average amplitude A avg ; The fundamental frequency mutation value F0 is obtained by calculating the difference between the current window's maximum fundamental frequency value and the previous window's maximum fundamental frequency value, and then dividing by the previous window's maximum fundamental frequency value. change This reflects the differences in fundamental frequency characteristics; The amplitude abrupt change value A is obtained by calculating the difference between the maximum amplitude value of the current window and the maximum amplitude value of the previous window, and then dividing by the maximum amplitude value of the previous window. change This reflects the differences in amplitude characteristics; By calculating the joint distribution entropy H(F0) of fundamental frequency abrupt change and amplitude abrupt change... change A change If H(F0) change A change The maximum value (max(F0)) is found among the fundamental frequency abrupt change value and the amplitude abrupt change value, where F0 is greater than the preset distribution entropy threshold. change A change If the value is greater than the preset mutation threshold, it is determined to be an accent position. The lower the entropy value, the stronger the synergy between the fundamental frequency mutation and the amplitude mutation. The words corresponding to the stressed syllables were matched with a keyword database for psychological counseling. The "Scale of Commonly Used Emotional Vocabulary in Psychological Counseling" and clinical counseling data were collected. High-frequency emotional words such as sadness, fear, happiness, and ease were selected using the TF-IDF algorithm, as well as related thematic words such as stress, family, work, and childhood. These were manually annotated by psychological counselors to build a keyword database. The type of each keyword was determined as positive / negative and its emotional relevance. After being marked, it was used for emotion judgment. The sliding window is preset to a window size of 3 seconds and a step size of 1 second for real-time feature calculation. The real-time speech rate V is obtained by dividing the number of words recognized in the window by the total duration of the sliding window. speech ; The speech rate trend T is obtained by calculating the difference between the average speech rate of the current window and the speech rate of the previous window, and then dividing the difference by the average speech rate of the previous window. v ; The duration of pauses with speech energy less than -40dB within the statistical window is averaged to obtain the single pause duration T. pause ; Divide the number of pauses within the window by the total duration of the sliding window to obtain the pause frequency F. pause ; Emotion determination rules are constructed by combining multi-dimensional features. The rules are trained and optimized by decision tree algorithm. By combining keywords, real-time speech rate, single pause duration and speech rate change trend, the emotion intensity value is analyzed, and the emotion category of the consultation object is determined based on the emotion intensity value. For example, if there are positive emotion keywords, and the real-time speech rate is V... speech Greater than 2.5 words / second, single pause duration T pause Less than 0.2 seconds and the trend of speech rate change T v If the value is greater than 0, it is marked as a positive emotion, and the emotion intensity value is S. stress ×10, an emotional intensity score ≥8 corresponds to pleasure, and an emotional intensity score between [5, 7] corresponds to moderate pleasure; if negative emotional keywords are present, and the real-time speech rate is V... speech Less than 1.5 words / second, single pause duration T pause Greater than 0.5 seconds and the trend of speech rate change T v An emotion value less than -0.2 is marked as negative, with an emotion intensity value S. stress ×10, an emotion intensity score ≥8 points corresponds to extremely negative emotions, and an emotion intensity score between [5 points, 7 points] corresponds to moderately negative emotions; if there are no obvious keywords, and the real-time speech rate is V speech At a speed of [1.5 words / second, 2.5 words / second], with a single pause duration T... pause If the absolute value of the speech rate change trend is ≤0.2 seconds, it is marked as neutral emotion, with an intensity of 3 points, 5 points; Obtain the sum of the differences in emotion intensity values ​​from three consecutive windows, and analyze the trend of emotion changes based on the sum of the differences in emotion intensity values; If the trend of mood change is greater than 5, the mood will rapidly rise, either positive or negative; if the sum of the differences in the trend of mood change is less than -5, the mood will rapidly cool down, ease, or turn neutral; otherwise, the mood change will be stable. Step 4: Based on the correlation between keywords and emotional changes, design a comprehensive stability value calculation method that reflects both emotional stability and the degree of influence of keywords, and provides a quantitative indicator for mental health assessment. In a specific embodiment, for each tagged keyword, the maximum value of the emotional intensity within 3 seconds after the keyword appears is taken, minus the average value of the emotional intensity within 3 seconds before the keyword appears. The change in emotional intensity ΔE within 3 seconds before and after the keyword appears is calculated, and the influence coefficient K is defined. word Quantify the degree of influence of keywords on emotions, and K word =ΔE / T interval T interval The positive keyword influence coefficient K represents the time difference between the appearance of a keyword and the peak of sentiment change. word A negative keyword influence coefficient K greater than 0 wordThe value is less than 0, and the larger the absolute value, the stronger the influence. Calculate the standard deviation (SD) of emotional intensity throughout the entire conversation. E This reflects the dispersion of emotional fluctuations and calculates the number of emotional type transitions, N. switch Two consecutive windows with different sentiment types and a difference in intensity greater than 2 points are counted as one sentiment type jump. This avoids statistical errors caused by minor fluctuations, using the formula SD. E ×(1+0.2×N switch The analysis yielded the emotional fluctuation degree F. fluct Among them, 0.2 is the preset correction coefficient for the number of jumps, which amplifies the negative impact of frequent jumps on stability and comprehensively reflects the stability of emotions; Based on the obtained influence coefficients and combined with clinical interval mapping, the sum of keyword influence coefficients |ΣK is calculated through consultation cases. word |With emotional fluctuation F fluct The distribution range and corresponding degree of psychological health impact were used to establish a two-dimensional interval deduction table; Points are deducted based on the sum of the keyword influence coefficients and the range of emotional fluctuation. The deduction items corresponding to the sum of keyword influence coefficients are determined through clinical emotional stability correlation analysis, while the deduction items corresponding to emotional fluctuation are set with reference to counseling effect tracking data. The total emotional deduction is obtained by accumulating the two deduction values. The total emotional deduction is then subtracted from the preset full score to obtain the psychological health stability value S. calc ; For example, if the sum of the keyword influence coefficients is in the range of [0, 1], then 5 points are deducted; if the sum of the keyword influence coefficients is in the range of (1, 2], then 10 points are deducted; if the sum of the keyword influence coefficients is greater than 2, then 15 points are deducted; if the emotional fluctuation is in the range of [0, 0.5], then 3 points are deducted; if the emotional fluctuation is in the range of (0.5, 1], then 6 points are deducted; if the emotional fluctuation is greater than 1, then 9 points are deducted. Based on the obtained mental health stability values, the mental health stability of the counseling subjects is analyzed and judged. The mental health stability score was standardized, with a range of [0, 100]. [80, 100] was marked as stable, [60, 80] as relatively stable, [40, 60] as unstable, and [0, 40] as high risk. Based on the number of jumps in emotion type, suspicious cases are marked and warnings are issued; Keywords with a greater than the frequency threshold for emotional type jumps or a greater than the conflict threshold for influence coefficient are marked as suspicious. A greater than the frequency threshold for emotional type jumps indicates frequent changes in emotional state and poor psychological stability. A greater than the conflict threshold for influence coefficient indicates that the strong-influence keyword corresponds to the core psychological conflict. The marking content includes the keyword text, the time of appearance, and the corresponding emotional change curve. If the mental health stability score is less than 40, a mental health warning will be automatically triggered, and the warning message will be sent to the counselor's terminal to help the counselor adjust the counseling strategy in a timely manner. By integrating emotional stability values ​​and multimodal speech features, a mapping relationship between emotional change trajectories and crisis levels is constructed to achieve graded early warning and targeted intervention. The multimodal speech features include: stress position, pause interval, and speech rate variation. By setting the emotional stability value as the core indicator, and combining changes in stress position, pause interval, and speech rate, a correlation model is constructed using the DS fusion of evidence theory to quantify the contribution of each feature to the emotional crisis. The difference between the current window's accent position and the average accent position of the previous three windows is calculated to obtain the accent position offset. The difference between the average pause duration of the current window and the average pause duration of the previous 3 windows is calculated to obtain the change in pause interval. Subtract the average speech rate of the previous 3 windows from the current speech rate, and then divide by the average speech rate of the previous 3 windows to obtain the speech rate change rate. The emotional stability value, the amount of stress position shift, the amount of pause interval change, and the rate of speech rate change are respectively used as four evidence bodies E1, E2, E3, and E4. For each piece of evidence, the probability of supporting the crisis and the probability of not supporting the crisis are analyzed based on clinical data using the BPA function. The BPAs of the four pieces of evidence are fused using the DS synthesis rule to obtain the final fusion probability and the crisis relevance C. crisis ; Based on the correlation between changes in mental health stability values ​​and crises, emotional changes are assigned to different crisis levels. It should be noted that the change in mental health stability value is the difference between the current mental health stability value and the previous mental health stability value. For example, if the change in mental health stability value is less than -10, and the mental health stability value drops from ≥60 to <60, then the mood changes from stable to unstable; Crisis determination: if 0.8 > crisis correlation degree ≥0.6, then it is marked as a moderate crisis; if the crisis correlation degree ≥0.8, then it is marked as a severe crisis; if the change in mental health stability value is greater than 10, and the mental health stability value rises from <60 to ≥60, then the mood changes from unstable to stable; if the absolute value of the change in mental health stability value is less than or equal to 5, and 60 ≤ mental health stability value ≤ 80, then the mood remains stable; if the change in mental health stability value is less than or equal to 5, and the mental health stability value <40, then the mood remains unstable; otherwise, the mood is considered normal fluctuation. Step 5: To address the issues of language barriers and severe emotional disorders, a three-tiered processing mechanism is constructed. Through the fusion of a dedicated baseline and multimodal evidence, feature adaptation, cross-validation, and decision correction are performed to resolve the problems of distorted emotional feature extraction and misjudgment of crises in special populations. In a specific embodiment, voice samples of stuttering counselors are collected, and abnormal voice sequences are aligned using a dynamic time warping algorithm to extract pathological features and emotional association features. For pathological characteristics, obtain the syllable repetition rate, the proportion of prolonged sounds, and the frequency of blocking pauses of the counseling subjects, establish a stuttering type-specific baseline, and distinguish between developmental stuttering and acquired stuttering based on the syllable repetition rate and the frequency of blocking pauses. It should be noted that the syllable repetition rate is the ratio of the number of times a single syllable is repeated to the total number of syllables, the extended syllable ratio is the ratio of the duration of extension between syllables to the total duration of pronunciation, and the occlusion frequency is the number of silent occlusions per unit time. For emotion-related characteristics, we analyzed the ratio of net speech rate after excluding pathological repetition and prolongation, the ratio of non-blocking pause duration to total pause duration, and the fundamental frequency variation coefficient after excluding stuttering segments. The pathological feature masking and emotional feature reconstruction method is used to mask the stuttering segments identified in the speech signal based on the dedicated baseline judgment, retain the emotional correlation features of the non-stuttering segments, and correct the misjudgment of speech rate or pauses caused solely by stuttering. For example, by combining fuzzy logic reasoning, the combination of net speech rate < 1.2 words / second, emotional pauses > 60%, and fundamental frequency fluctuation coefficient > 0.25 is judged as an emotional anxiety tendency; The distorted speech is preprocessed and its features are restored. For the distortion types of popping, vibrato and breathy sounds during emotional outbursts, a hierarchical enhancement algorithm is adopted. The speech signal is decomposed by wavelet threshold denoising algorithm, and the high-frequency popping components are processed by soft thresholding to preserve the trend features of fundamental frequency and amplitude. An adaptive Kalman filter is used to construct a state equation based on the fundamental frequency mean of the first three windows, which smooths the high-frequency fluctuations caused by vibrato. The harmonic-to-noise ratio is improved by spectral subtraction, and the enhanced spectral centroid and Mel-frequency cepstral coefficients are extracted. If the speech distortion exceeds the preset distortion limit, multimodal cross-validation is initiated. The dynamic change rate of 68 facial key points is extracted and compared with the emotional features of the distorted speech for consistency judgment. If the consistency rate reaches the consistency standard, the emotion type is confirmed. The abnormality of heart rate variability and skin conductivity is calculated. When both indicators reach the preset high arousal threshold, the judgment of negative emotions in the distorted speech segment is strengthened to avoid missed emotion judgment due to distortion.

[0011] Example 2: Figure 2 As shown in the figure, the real-time speech recognition system for psychological counseling based on multimodal data provided in this embodiment of the invention specifically includes the following modules: Analysis and Judgment Module: By constructing a gender-specific voice-psychological feature mapping baseline, the module uses the fundamental frequency mean and fundamental frequency variation coefficient for initial analysis to determine the degree of emotional stability tendency, and combines the amplitude peak ratio to determine the degree of expression tendency. Dynamic correction module: Combining pronunciation habit details with real data, it dynamically corrects the fundamental frequency mean amplitude to resolve misjudgments of psychological characteristics caused by individual pronunciation differences; The analysis module has been optimized: stress position, keywords, speech rate and pause features have been tested to build a multi-dimensional emotion analysis model, while the issues of emotion focus positioning, emotion type judgment and change trend tracking have been optimized. Quantitative assessment module: Combining the correlation between keywords and emotional changes, a comprehensive stability value calculation method is designed to reflect both emotional stability and the degree of influence of keywords, providing quantitative indicators for mental health assessment; Adaptation and Correction Module: To address language barriers and severe emotional disorders, a three-tiered processing mechanism is constructed, which integrates a dedicated baseline with multimodal evidence fusion to perform feature adaptation, cross-validation, and decision correction.

[0012] The above provides a detailed description of one embodiment of the present invention, but the content described is only a preferred embodiment of the present invention and should not be considered as limiting the scope of the present invention. The above formulas are all dimensionless numerical calculations, and the formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world situation. The preset parameters in the formulas are set by those skilled in the art based on actual conditions and historical experience, and can be adjusted according to actual conditions. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. All equivalent changes and improvements made in accordance with the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A psychological counseling real-time speech recognition method based on multi-modal data, characterized in that, The method comprises the following steps: By constructing a gender-specific voice psychological feature mapping baseline, the initial analysis of the mean fundamental frequency and the coefficient of variation of the fundamental frequency is used to judge the emotional stability tendency, and the expression tendency is judged by combining the amplitude peak value ratio; Combining the details of pronunciation habits and real data, dynamically correcting the mean amplitude of the fundamental frequency, solving the psychological feature misjudgment caused by individual pronunciation differences; Through the detection of accent position, key words, speech rate and pause features by the fundamental frequency mutation and amplitude mutation, a multi-dimensional emotion analysis model is constructed, and the problems of emotion focus positioning, emotion type judgment and change trend tracking are optimized; Combined with the correlation between keywords and emotional changes, a comprehensive stability value calculation method is designed, which reflects the emotional stability and the influence degree of keywords, and provides quantitative indicators for mental health assessment; For language barriers and severe emotional disorders, a three-level processing mechanism is constructed to adapt features, cross-verify and correct decisions through the fusion of exclusive baseline and multi-modal evidence.

2. The real-time voice recognition method for psychological counseling based on multi-modal data according to claim 1, characterized in that, The method for constructing a gender-specific voice psychological feature mapping baseline is: A double-microphone array is used to collect the initial dialogue voice of the consultation object, the main microphone is placed in front of the visitor to focus on the target voice, and the reference microphone is placed behind to collect environmental noise, the sampling rate is uniformly set and the quantization bit is quantified; The LMS adaptive filtering algorithm is used to reduce noise of the double-microphone signal, suppress the non-stationary noise in the consultation room, and retain the pure voice signal; For the physiological range of the fundamental frequency of men and women, the mean fundamental frequency and the coefficient of variation of the fundamental frequency are calculated, and the amplitude peak value ratio is calculated combined with the amplitude dynamic range of men and women; Based on the pure voice signals of male and female visitors of different ages and occupations, the gender-specific mapping rules are trained and optimized by the random forest algorithm to realize the initial judgment of emotional stability tendency, emotional sensitivity tendency and expression tendency. 3.The real-time voice recognition method for psychological counseling based on multi-modal data according to claim 1, wherein, The method for dynamically correcting the mean amplitude of the fundamental frequency is: The Kaldi speech recognition toolkit is used to disassemble the preprocessed voice at the phoneme level, extract three types of pronunciation habit features, calculate the deviation rate of the average duration of vowel pronunciation and the standard duration of Mandarin, the standard deviation of the intensity of consonant initial sound, and the deviation rate of the frequency of four tones and the standard tone of Mandarin; The structured information of the age, occupation, dialect background and previous psychological state of the consultation object is obtained to construct a correction factor, and the corrected mean fundamental frequency is calculated by combining the original mean fundamental frequency with the age factor and the occupation factor; The corrected mean fundamental frequency and amplitude are used to recalculate the coefficient of variation of the fundamental frequency and the amplitude peak value ratio, and the psychological features are updated again combined with the pronunciation habit markers.

4. The real-time voice recognition method for psychological counseling based on multi-modal data according to claim 1, characterized in that, The method for constructing a multi-dimensional emotion analysis model is: Set a 3-second sliding window to extract the maximum fundamental frequency, mean fundamental frequency, maximum amplitude and mean amplitude; The joint distribution entropy of the fundamental frequency mutation value and the amplitude mutation value and the maximum value of the two are used to determine the accent position; the accent position vocabulary is matched with the psychological counseling special keyword library filtered by the TF-IDF algorithm; the real-time speech rate, speech rate change trend, single pause duration and pause frequency are calculated with a 3-second window and a 1-second step; The decision tree algorithm is used to train and optimize the emotion judgment rules, and the model is constructed combined with keywords and multi-dimensional features; The method for judging the emotion classification of the consultation object is:

5. The real-time voice recognition method for psychological counseling based on multi-modal data according to claim 4, characterized in that, The emotion judgment rules are constructed in combination with multi-dimensional features, the rules are optimized through decision tree algorithm training, the emotion intensity value is analyzed in combination with keywords, real-time speech speed, single pause duration and speech speed change trend, and the emotion classification of the consultation object is judged according to the emotion intensity value. The method for obtaining the emotion change trend is:

6. The real-time voice recognition method for psychological counseling based on multi-modal data according to claim 4, characterized in that, The sum of the emotion intensity value differences of three consecutive windows is obtained, and the emotion change trend is analyzed according to the sum of the emotion intensity value differences; If the emotion change trend is greater than 5, the emotion is rapidly warming up positively or the negative emotion is intensified; If the sum of the emotion change trend differences is less than -5, the emotion is rapidly cooling down, the emotion is relieved or turned to neutral; otherwise, the emotion change is stable. The method for obtaining the comprehensive stability value is:

7. The real-time voice recognition method for psychological counseling based on multi-modal data according to claim 1, characterized in that, The difference between the maximum emotion intensity value within 3 seconds after the appearance of the keyword and the average value within 3 seconds before the appearance of the keyword is calculated to define the influence coefficient; the standard deviation of the emotion intensity and the emotion type jump frequency are calculated, and the emotion fluctuation degree is obtained by correcting the emotion intensity standard deviation in combination with the jump frequency; A double-dimensional interval deduction item table is established in combination with mapping, the sum of the keyword influence coefficients and the deduction value of the emotion fluctuation degree are accumulated, the psychological health stability value is obtained by subtracting the total deduction value from the preset full score. The method for quantifying the psychological health evaluation is: 8.The real-time voice recognition method for psychological counseling based on multi-modal data according to claim 1, wherein, The emotion stability value is standardized and divided into stable, relatively stable, unstable and high-risk types; If the emotion type jump frequency is greater than the frequent threshold or the absolute value of the keyword influence coefficient is greater than the conflict threshold, the suspicious mark of the text containing the keyword, the appearance time and the emotion change curve is performed; if the emotion stability value is less than 40 points, an early warning prompt is automatically triggered and synchronized to the counselor terminal; a correlation degree model is constructed by using the D-S evidence theory to fuse the emotion stability value, the pitch position offset, the pause interval change and the speech speed change rate, the danger correlation degree is obtained through the BPA function and the synthesis rule, and the danger level is divided in combination with the emotion stability value change value. The method for feature adaptation is: 9.The real-time voice recognition method for psychological counseling based on multi-modal data according to claim 1, wherein, The speech samples of the stuttering consultation object are collected, the abnormal speech sequence is aligned through the dynamic time warping algorithm, the pathological features are extracted to establish the stuttering type exclusive baseline, and the developmental and acquired stuttering is divided; the emotion correlation features are analyzed; The stuttering segments in the speech are processed by using the pathological feature mask to retain the emotion correlation features of the non-stuttering segments, and the misjudgment is corrected by using the fuzzy logic reasoning. The method for constructing a three-level processing mechanism is:

10. The real-time voice recognition method for psychological counseling based on multi-modal data according to claim 1, characterized in that, For the explosion sound, tremolo and air sound distortion in the emotion burst, a hierarchical enhancement algorithm is used to decompose the speech signal by wavelet threshold denoising and process the high-frequency explosion components by soft threshold, to smooth the high-frequency fluctuations of tremolo by adaptive Kalman filtering, to improve the harmonic-to-noise ratio by spectral subtraction and to extract the spectral centroid and mel cepstral coefficients; If the speech distortion degree is greater than a preset limit value, a multi-modal cross-validation is started, the dynamic change rate of 68 facial key points and the emotion features of distorted speech are determined for consistency; The heart rate variability and the abnormality degree of skin conductance are calculated, and the negative emotion judgment is intensified when both reach a high awakening threshold. ​

Images