Intelligent mental health monitoring system

Abstract

The invention relates to the technical field of health monitoring, in particular to an intelligent psychological health monitoring system which comprises a baseline construction module, a heat map analysis module, a collaborative evaluation module, an anomaly recognition module and a label output module. According to the method, the psychological assessment sensitivity is adjusted by combining the voice tail end features, the adaptability to individual behavior differences is enhanced, the dynamic heat map disturbance index is constructed by using the face temperature difference direction change and the inversion frequency, the timeliness and accuracy of anomaly detection are improved, and the accuracy of anomaly detection is improved. A time offset of an intonation and expression boundary is adopted to measure and express a coordination state, the recognition capability of asynchronous behaviors is enhanced, time cross matching is performed on a scoring trend and a behavior offset region, focusing intensity judgment of an abnormal event is optimized, and a label path is constructed in combination with multichannel features, so that the expression integrity of a psychological state label is improved, and the recognition capability of the asynchronous behaviors is improved. And the perception capability of expression delay, rhythm disorder and various recessive psychological abnormalities is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of health monitoring, in particular to an intelligent mental health monitoring system. BACKGROUND

[0002] The technical field of health monitoring includes related technologies for obtaining user physiological or behavioral data through sensor devices, continuously monitoring individual health conditions and analyzing data in combination with information processing systems, and the core content includes the construction of physiological parameter acquisition devices, biological signal recognition and acquisition, communication methods of data acquisition modules, logical structures of data processing modules, and collaborative methods of storage systems and prompting mechanisms, covering hardware device composition, biological signal processing flow, data acquisition and conversion, data filtering mechanism, dynamic tracking analysis process, and being applied to chronic disease management, personalized health services, remote medical management, and mental health monitoring.

[0003] Among them, the intelligent mental health monitoring system refers to the combination of individual psychological state characteristics, physiological parameter monitoring and behavioral data analysis to realize the evaluation and change trend recording of mental health level, covering continuous collection and analysis of heart rate variability parameters, emotion recognition based on voice signal feature extraction, psychological state trend determination through facial expression changes, data classification and labeling based on a preset psychological feature model, including heart rate monitoring by an optical plethysmography sensor, extracting the fundamental frequency and formant features of the voice signal using short-time Fourier transform, extracting emotion feature patterns combining the gray level gradient distribution of the face image, and performing emotion classification and psychological state scoring through a support vector machine algorithm, and constructing a psychological state analysis process based on multi-modal information cross-validation.

[0004] In the traditional mental health monitoring technology, the recognition of psychological state focuses on the extraction and classification of static features of single physiological or behavioral signals, relies on single-dimensional parameters such as fundamental frequency, gray level, and pulse change to construct a scoring model, ignores the importance of individual expression behavior differences and time sequence information in the evolution of behavioral state, and the static features are easily hidden by average processing in the stage where the expression rhythm deviation is significant and the behavior is not coordinated yet transformed into physiological abnormality. The scoring result is delayed or even misjudged. Modeling multiple channels of voice, expression, and heart rate separately cannot determine whether they constitute a linked abnormality in a unified time structure, resulting in a disconnection between psychological state scoring and micro-behavioral abnormalities, making it difficult to identify early signal deviations such as expression suppression, tone overshoot, and facial freezing, affecting the timeliness and accuracy of psychological intervention. SUMMARY

[0005] The purpose of the present application is to solve the shortcomings in the prior art and to provide an intelligent mental health monitoring system.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: The intelligent mental health monitoring system includes: The baseline construction module collects users' speech data in a resting state, analyzes the duration and proportion of the last syllable in the user's speech, obtains the speech end delay feature data based on the changing trend of the proportion data in multiple speech segments, adjusts the psychological assessment sensitivity, and obtains the baseline adjustment results. The thermal image analysis module calls the baseline adjustment results, uses a thermal imaging sensor to acquire the temperature data of the user's face during psychological assessment, calculates the temperature difference value at multiple locations, and identifies facial temperature disturbance events and obtains abnormal disturbance information based on the change in the directional angle between consecutive temperature difference values ​​and the number of reversals. The collaborative assessment module calls the abnormal disturbance information and baseline adjustment results, collects the user's voice and facial image data during psychological assessment, identifies tone change and facial expression change events, calculates the degree of difference in tone change and facial expression change over time, identifies synchronization anomaly events, and generates synchronization deviation records. The anomaly identification module calls upon the abnormal disturbance information and synchronization deviation records, calculates the user's psychological state score by collecting multiple physiological parameters in real time, identifies abnormal state events, and obtains abnormal focusing intensity information by combining facial temperature disturbances and synchronization abnormal events.

[0007] As a further aspect of the present invention, the baseline adjustment result specifically includes the speech end delay ratio sequence, delay fluctuation range, and assessment sensitivity parameter; the abnormal perturbation information includes the number of temperature reversals, the direction of thermal gradient change, and the amplitude of temperature difference fluctuation; the synchronization deviation record includes the position of tone fluctuation, the start and end time of facial expression dynamics, and the temporal difference parameter; and the abnormal focus intensity information specifically refers to the score jump density, the frequency of behavioral abnormal clustering, and the duration of score fluctuation.

[0008] As a further aspect of the present invention, the baseline construction module includes: The speech acquisition and processing submodule collects the user's speech data in a resting state, analyzes the duration of the last syllable in each speech segment, identifies the proportion of the last syllable in the speech, and obtains the duration parameter of the speech ending structure. The speech tail feature recognition submodule calls the speech tail structure ratio parameter, calculates the sum of squares of the differences between adjacent speech tail ratios, superimposes the absolute difference of each ratio deviation from the average value in the whole sequence and squares it, and obtains the ratio with the degree of change of the trend direction mark of the adjacent segment, calculates the speech tail delay trend deviation, and generates speech tail delay feature data. The evaluation baseline adjustment submodule calls the speech delay feature data to analyze the delay characteristics of the user's expression rhythm, evaluate the stability of language output, adjust the sensitivity of psychological evaluation, including adjusting the facial temperature difference judgment boundary and the behavior coordination judgment interval, and obtain the baseline adjustment results.

[0009] As a further aspect of the present invention, the heatmap analysis module includes: The thermal image data acquisition submodule calls the baseline adjustment results and uses a thermal imaging sensor to collect temperature data from multiple locations on the user's face during the psychological assessment process, including the tip of the nose, cheeks, forehead, and chin. It calculates the temperature difference values ​​at multiple locations and generates a continuous thermal image data set. The disturbance trend calculation submodule calls the continuous heat map data group, analyzes the change amplitude of the angle between the temperature difference direction vectors between adjacent frames, records the rate of change of the angle between each frame, extracts the normalized value of the temperature difference amplitude and the normalized value of the temperature range for each frame, and calculates the disturbance trend score. The boundary comparison and recognition submodule calls the disturbance trend score, combines it with the temperature difference disturbance to determine the boundary, identify facial temperature disturbance events, and obtain abnormal disturbance information.

[0010] As a further aspect of the present invention, the collaborative evaluation module includes: The speech graph data acquisition submodule calls the abnormal perturbation information and baseline adjustment results to collect the user's voice signal and facial image frame sequence in real time during the psychological assessment process, identify tone change and facial expression change events, and generate speech graph behavior sampling dataset; The emotion boundary assessment submodule calls the speech graph behavior sampling dataset, extracts the start timestamps corresponding to each tone change and facial expression change event, analyzes the degree of difference between the user's tone change and facial expression change in time, calculates the expression coordination index, and generates speech boundary difference information. The timing deviation determination submodule calls the speech boundary difference information to evaluate the user's emotional expression status in real time, compares the expression coordination degree index with the behavior coordination judgment interval, identifies synchronization abnormal events, and generates synchronization deviation records.

[0011] As a further aspect of the present invention, the anomaly identification module includes: The physiological parameter acquisition submodule calls the abnormal disturbance information and synchronization deviation record to collect multiple physiological parameters of the user in real time during the psychological assessment process, including heart rate, respiratory rhythm and voice volume, and generates a standardized physiological dataset. The psychological score calculation submodule calls the standardized physiological dataset, uses heart rate, respiratory rhythm and voice volume to calculate the user's psychological state score in real time, establishes a psychological state score sequence and analyzes the score change trend, and obtains psychological score change trend indicators. The anomaly focusing analysis submodule calls the psychological score change trend index to identify abnormal state events. Combining the start timestamp information of the already identified facial temperature disturbance events and synchronous abnormal events, it identifies time overlap segments and analyzes the event density and coupling strength of the overlap segments to obtain anomaly focusing intensity information.

[0012] As a further aspect of the present invention, the system further includes: The tag output module calls the abnormal focus intensity information, extracts the timestamp information of facial temperature disturbance events, synchronization abnormal events, and state abnormal events, constructs the event occurrence path according to the time sequence and duration of each event, classifies the user's psychological state and matches psychological state tags, and outputs psychological state monitoring records. The psychological state monitoring records include abnormal event timestamps, event path sequence, and psychological state tags.

[0013] As a further aspect of the present invention, the tag output module includes: The event time extraction submodule calls the abnormal focus intensity information to extract the timestamp data of facial temperature disturbance events, synchronization abnormal events and state abnormal events, establishes a time structure table of abnormal events, and generates a set of abnormal event time nodes. The path construction and classification submodule calls the set of abnormal event time nodes, constructs the event occurrence path according to the temporal order and duration of each event, and obtains the abnormal event path sequence; The tag matching output submodule calls the abnormal event path sequence, analyzes the combination characteristics and evolution patterns of the event occurrence path, classifies the user's psychological state, matches psychological state tags, and obtains psychological state monitoring records.

[0014] Compared with the prior art, the advantages and positive effects of the present invention are as follows: In this invention, the sensitivity of psychological assessment is adjusted by combining speech tail features, thereby enhancing the adaptability to individual behavioral differences. A dynamic heatmap perturbation index is constructed using changes in facial temperature direction and reversal frequency, improving the timeliness and accuracy of anomaly detection. The time offset of tone and facial expression boundaries is used to measure the state of expression coordination, enhancing the ability to recognize asynchronous behavior. By performing time cross-matching of scoring trends and behavioral offset regions, the focus intensity judgment of abnormal events is optimized. A label path is constructed by combining multi-channel features, improving the expression completeness of psychological state labels and realizing the ability to perceive expression delays, rhythm disorders, and various latent psychological abnormalities. Attached Figure Description

[0015] Figure 1 This is a system flowchart of the present invention; Figure 2 This is a flowchart of the baseline construction module of the present invention; Figure 3 This is a flowchart of the heat map analysis module of the present invention; Figure 4 This is a flowchart of the collaborative evaluation module of the present invention; Figure 5This is a flowchart of the anomaly identification module of the present invention; Figure 6 This is a flowchart of the label output module of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0017] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0018] Please see Figure 1 The intelligent mental health monitoring system includes: The baseline construction module collects users' speech data in a resting state, analyzes the duration and proportion of the last syllable in the user's speech, obtains the speech end delay feature data based on the changing trend of the proportion data in multiple speech segments, adjusts the psychological assessment sensitivity, and obtains the baseline adjustment results. The thermal image analysis module calls the baseline adjustment results, uses thermal imaging sensors to acquire the temperature data of the user's face during psychological assessment, calculates the temperature difference value at multiple locations, and identifies facial temperature disturbance events and obtains abnormal disturbance information based on the change in the directional angle between continuous temperature difference values ​​and the number of reversals. The collaborative assessment module calls up abnormal disturbance information and baseline adjustment results, collects voice and facial image data of users when they are conducting psychological assessments, identifies tone and facial expression change events, calculates the degree of difference in tone and facial expression changes over time, identifies synchronization anomalies, and generates synchronization deviation records. The anomaly identification module calls up abnormal disturbance information and synchronization deviation records, calculates the user's psychological state score by collecting multiple physiological parameters in real time, identifies abnormal state events, and obtains abnormal focusing intensity information by combining facial temperature disturbance and synchronization abnormal events. The tag output module calls the abnormal focus intensity information, extracts the timestamp information of facial temperature disturbance events, synchronization abnormal events, and state abnormal events, constructs the event occurrence path according to the time sequence and duration of each event, classifies the user's psychological state and matches psychological state tags, and outputs psychological state monitoring records.

[0019] The baseline adjustment results specifically include the speech delay ratio sequence, delay fluctuation range, and assessment sensitivity parameters. Abnormal perturbation information includes the number of temperature reversals, the direction of thermal gradient changes, and the amplitude of temperature difference fluctuations. Synchronous deviation records include the position of tone fluctuations, the start and end times of facial expression dynamics, and temporal difference parameters. Abnormal focus intensity information specifically refers to the score jump density, the frequency of abnormal behavior clustering, and the duration of score fluctuations. Psychological state monitoring records include abnormal event timestamps, event path sequence, and psychological state labels.

[0020] Please see Figure 2 The baseline building block includes: The speech acquisition and processing submodule collects the user's speech data in a resting state, analyzes the duration of the last syllable in each speech segment, identifies the proportion of the last syllable in the speech, and obtains the duration parameter of the speech ending structure. With the user's authorization and consent, several segments of voice data were collected from the user in a resting state, each segment lasting 10 seconds. Taking a typical scenario as an example, the user continuously emitted 5 segments of voice data. The last syllable of each segment was extracted independently, and the start and end points of each last syllable were determined. Specifically, the end point of the last syllable was determined by the point where the voice amplitude steadily decayed to the background noise amplitude. The starting point of the syllable was defined as the moment when the initial amplitude of the last syllable was significantly greater than the background noise amplitude by more than 20%. The timestamps of the start and end points of the last syllables were extracted from the time-domain curve of the voice waveform, thereby determining the duration of each last syllable. For example, the duration parameters of each last syllable were measured and obtained as follows (unit: seconds): By calculating the ratio of the duration of the last syllable of each segment to the total duration of the corresponding entire speech segment, for example, dividing the duration of the last syllable of the first segment (0.32 seconds) by the segment duration (10 seconds), the ratio parameter for the last syllable is obtained as 0.032. Similarly, the ratios for the last syllables of other segments are calculated as follows: The calculation process is precise, down to the calculation of the specific duration and ratio of each speech segment, and finally obtains a set of speech end structure duration parameters accurate to three decimal places.

[0021] The suffix feature recognition submodule calls the suffix structure proportion parameter, calculates the sum of squared differences between adjacent suffix proportions, adds the absolute difference of each proportion's deviation from the average in the entire sequence and squares it, then ratios it to the degree of change of the trend direction markers of adjacent segments, using the formula: ; Calculate the deviation of the speech end delay trend to obtain speech end delay feature data; in, The degree of deviation of the linguistic delay trend. For the first The proportion of sustained syllables at the end of a speech segment. For the first The proportion of sustained syllables at the end of a speech segment. The arithmetic mean of the proportion of sustained word endings across all segments. For the first Section and the The direction indicator for the change in the proportion of the paragraph ending is 1 when the change is upward and -1 when the change is downward. This represents the total number of speech sample segments. Calling the sentence ending structure ratio parameter obtained from the first paragraph First, calculate the squared difference between adjacent proportions for each segment: for example, the difference between segments 1 and 2 is... Its square value is The total squared difference is calculated segment by segment and summed to obtain: Then, the average value of all scaling parameters is calculated. The calculation process is as follows: Then calculate the absolute value of the difference between the proportion and the average value for each segment and sum them up to get: Squared further, we get: Meanwhile, for trend direction marking The value is assigned as 1 if the proportion increases and -1 if the proportion decreases. For the first two segments, the proportion decreases by -1, and for the second and third segments, the proportion increases by 1, thus deriving the label sequence: Multiply each trend direction marker by the absolute value of its corresponding ratio difference, take the absolute value again, and sum them up to get: Then, the formula is used to calculate the deviation of the delayed trend at the end of the sentence. : ; in, This indicates the duration of the final continuation of the i-th speech segment. The arithmetic mean of the proportion of sustained word endings across all segments. The direction marker indicates the trend of speech tail proportion changes, and m represents the total number of speech sample segments. Speech tail delay trend deviation refers to the comprehensive measure of the intensity and direction consistency of the change in the speech tail persistence proportion sequence relative to the average speech tail rhythm across multiple speech segments. It assesses whether the speech tail delay phenomenon has a continuous trend of deviating from the resting speech structure and whether it possesses irregular offset characteristics on the change path. The parameter is used to dynamically adjust the sensitivity of behavioral feature recognition in the subsequent system, adjusting the scoring critical ranges such as the facial temperature difference perturbation judgment boundary and the behavioral coordination synchronization interval. A larger deviation indicates a more unstable speech tail change, and the system will correspondingly lower the trigger threshold for score jumps; conversely, a smaller deviation maintains or tightens the judgment boundary, allowing the psychological state scoring system to adapt to individual speech characteristics and reduce the risk of misjudgment. The formula, through differentiated measurement of the speech tail persistence proportion, clearly reveals the degree of trend deviation in the user's language output under resting conditions, thereby quantifying the delay characteristics.

[0022] Calculate using the above values ​​in the formula: ; The result of 0.0011522 is the final speech delay feature data, indicating the degree of deviation of the speech delay trend in the user's resting state.

[0023] The baseline adjustment submodule calls the speech delay feature data to analyze the delay characteristics of the user's expression rhythm, evaluate the stability of language output, adjust the sensitivity of psychological assessment, including adjusting the facial temperature difference judgment boundary and the behavior coordination judgment interval, and obtain the baseline adjustment results. The sentence delay feature data of 0.0011522 was retrieved and compared with a preset threshold range. The threshold was set with reference to typical experimental data, as shown in Table 1. Table 1. Threshold Range for Sentence End Delay Trend Deviation: ; As shown in Table 1, when the deviation When the value is within the range of 0 to 0.0015, it indicates that the user's expression rhythm delay characteristics are stable. In this example, 0.0011522 is within the range of 0 to 0.0015, so it is determined that the user's current speech delay trend is in a stable state. Subsequently, the sensitivity of psychological assessment is adjusted based on this judgment. Specifically, the boundary setting of subsequent facial temperature difference judgment is adjusted. For example, when the deviation is in the stable state range, the threshold of facial temperature difference judgment is set to 0.3℃~0.5℃. The behavioral coordination judgment range is adjusted accordingly, and the coordination range is defined as the allowable time difference range of 0~0.4 seconds, thereby obtaining the specific baseline adjustment result.

[0024] Please see Figure 3 The heatmap analysis module includes: The thermal image data acquisition submodule calls the baseline adjustment results and uses a thermal imaging sensor to collect temperature data from multiple locations on the user's face during the psychological assessment process, including the tip of the nose, cheeks, forehead, and chin. It calculates the temperature difference values ​​at multiple locations and generates a continuous thermal image data set. Using baseline adjustment results, a high-resolution thermal imaging sensor was employed to continuously monitor real-time temperature changes at different facial locations during the psychological assessment. Specific monitoring locations included the tip of the nose, left cheek, right cheek, center of the forehead, and directly below the chin. Taking a typical scenario where five consecutive frames of thermal image data were collected during the assessment, facial feature recognition technology was used to accurately pinpoint the coordinates of each of these facial locations. Temperature values ​​for each location were extracted in real-time through coordinate mapping. For example, in the first frame, the temperature of the tip of the nose was 34.8℃, the left cheek 34.5℃, the right cheek 34.7℃, the forehead 35.0℃, and the chin 34.4℃. Temperature data for the subsequent four frames were obtained sequentially, as shown in Table 2. Table 2. Continuous Facial Temperature Monitoring During Psychological Assessment: ; As shown in Table 2, the temperature difference between multiple locations is calculated by the temperature difference within the frame. For example, in the first frame, the temperature difference between the tip of the nose and the forehead is 0.2℃, and the temperature difference between the chin and the forehead is 0.6℃. The temperature difference between each location within each frame is calculated in this way, and finally, the temperature difference values ​​of 5 consecutive frames are obtained to form a continuous heat map data set.

[0025] The disturbance trend calculation submodule calls the continuous heatmap data set, analyzes the change amplitude of the angle between the temperature difference direction vectors of adjacent frames, records the rate of change of the angle between each frame, and extracts the normalized value of the temperature difference amplitude and the normalized value of the temperature range for each frame, using the formula: ; Calculate the disturbance trend score; in, The perturbation trend score, For the first The angle between the direction of the temperature difference vector in the frame and the horizontal baseline. For the first The included angle corresponding to the frame, For the first Normalized value of frame temperature difference amplitude. For the first Normalized values ​​of the temperature range at five points on the face in the frame. For all The arithmetic mean of the normalized range of facial temperature differences in the frame. The total number of consecutive image frames. For frame sequence index number; The continuous heatmap data set being called calculates the directional change vector between adjacent frames based on facial temperature differences. Specifically, a reference vector is formed in each frame with the forehead as the origin and the tip of the nose as the endpoint, and the angle between this vector and the horizontal baseline is calculated. Taking the first frame as an example, the angle between the forehead position (0,0) and the nose tip position (0,-0.2℃) is 90°. The absolute value of the angle difference is calculated by the angle between adjacent vectors in consecutive frames. For example, the angle between frame 1 and frame 2 changes from 90° to 85°, with a difference of 5°. Simultaneously, the temperature difference amplitude of each frame is normalized using the ratio of the current frame's temperature difference to the maximum temperature difference across all frames. For instance, if the temperature difference in frame 1 is 0.6°C and the maximum temperature difference across all frames is 0.8°C, then the normalized temperature difference value for frame 1 is... As shown in Table 3, the data is calculated sequentially to obtain... The array is: [0.75, 0.875, 1.0, 0.875, 0.875]; then extract the five-point temperature range of the face in each frame (maximum value minus minimum value). For example, the range of the first frame is 0.6℃. The normalization calculation method is the same as above, to obtain... The array is: [0.75, 1.0, 1.0, 0.875, 1.0]; calculate its mean. .

[0026] Formula used: ; In the above formula, The perturbation trend score, For the first The angle between the frame temperature difference vector and the baseline horizontal line, For the first Frame temperature difference normalized value, For the first Frame temperature range normalized value, The normalized mean of the temperature range across all frames is given by n, where n is the number of frames. The formula, through a combination of angle variation and temperature difference amplitude, accurately reveals the trend of facial temperature changes. The disturbance trend score is a comprehensive measure of the temperature change trend of the facial thermogram within a certain time window. It combines the directional intensity of heat fluctuations, the level of abrupt changes in intensity, and the stability of spatial distribution. A higher score indicates more unstable facial temperature changes, more frequent directional fluctuations, and more significant regional thermal differences within the current time period, making it more likely to be identified as an abnormal disturbance. This score is used to identify abnormal disturbance events and serves as a crucial intermediate driving signal for the linkage between behavioral state perception and temperature channel judgment.

[0027] Substituting the above data into the formula, we get: ; ; The perturbation trend score is 0.1055, reflecting the degree of current facial temperature perturbation trend.

[0028] The boundary comparison and recognition submodule calls the disturbance trend score, combines it with temperature difference disturbance to determine the boundary, identify facial temperature disturbance events, and obtain abnormal disturbance information; The disturbance trend score of 0.1055 was used to compare specific values ​​within the temperature disturbance judgment boundary range determined based on the baseline adjustment results. The process of setting the disturbance judgment boundary was obtained through statistical analysis of long-term monitoring experimental data. Based on the temperature disturbance trend data of different user groups in the standard psychological assessment scenario, the interval range of the disturbance trend score was divided. The specific classification and threshold interval settings are shown in Table 3. Table 3. Fractional Boundary Table for Facial Temperature Difference Disturbance Trends: ; As shown in Table 3, after clarifying the perturbation trend score range, the actual perturbation trend score of 0.1055 is compared. The process is as follows: 0.1055 is compared with the upper and lower limits of the stable state range (0~0.08). It is found that 0.1055 is greater than the upper limit of the stable state value of 0.08, so the stable state is excluded. Then, the upper and lower limits of the range of the slightly abnormal state (0.08~0.12) are compared. Specifically, 0.1055 is compared with the upper and lower limits of the range (0.08 and 0.12). Since 0.1055 is greater than 0.08 and less than 0.12, it is clearly determined that the perturbation trend score of 0.1055 is completely within the slightly abnormal state range. Therefore, the current user's facial temperature difference perturbation is identified as a slightly abnormal event, and then the abnormal perturbation information is extracted. The abnormal perturbation information includes the starting frame position, ending frame position, and the numerical range of the temperature difference perturbation intensity during the corresponding time period, etc., to complete the identification and processing of temperature perturbation events in the user's psychological assessment process.

[0029] Please see Figure 4 The collaborative evaluation module includes: The speech graph data acquisition submodule calls abnormal perturbation information and baseline adjustment results to collect users' voice signals and facial image frame sequences in real time during the psychological assessment process, identify tone change and facial expression change events, and generate speech graph behavior sampling dataset; Using abnormal perturbation information and baseline adjustment results, taking a typical scenario during a user's psychological assessment as an example, the system continuously collects speech signals and corresponding facial image data sequences during the user's emotional expression. The speech signals are recorded with high precision via a microphone array at a sampling frequency of 44.1 kHz, and the image data is captured in real-time by a high-definition camera at a frame rate of 30 frames per second. The specific process includes: using temporal analysis to identify the occurrence point of each intonation change event; for example, the starting frame of the first intonation change is numbered as frame 180, continuing until frame 240, and the starting frame of the second intonation change is numbered... The frame number is 320, continuing to 370, and so on. At the same time, image processing technology is used to analyze facial images and capture expression change events. For example, the first expression change event lasts from frame 182 to frame 239, and the second expression change starts from frame 323 to frame 371. The start and end frame numbers of multiple sets of change events are extracted in sequence. Finally, the speech and image sequences are matched in a temporal manner to generate a speech-image behavior sampling dataset. This ensures that each frame number corresponds to a clear relationship between speech and facial image data, thereby accurately capturing the speech-image behavior details in the process of emotional expression.

[0030] The emotion boundary assessment submodule calls the speech graph behavior sampling dataset to extract the start timestamps corresponding to each tone change and facial expression change event, and analyzes the degree of temporal difference between user tone changes and facial expression changes using the following formula: ; Calculate the expression coordination index and generate information on the differences in word / table boundaries; in, To express the degree of coordination, For the first The starting time frame number of the segment of intonation change behavior. For the first The starting time frame number of the group of facial expression change behavior segments. For the first Normalized duration of the intonation segment, The normalized mean duration of all intonation segments. The number of tone and expression matching segments included in the comparison. The index number of the matching segment in the overall sequence; The speech graph behavior sampling dataset is used to extract the specific starting frame numbers of tone change and facial expression change events, forming temporal matching data. Taking a typical data scenario as an example, three sets of matching segment sequence data are extracted, as shown in Table 4: Table 4. Starting frame matching data for intonation and facial expression changes: ; The normalized intonation duration frame length The calculation method is as follows: Using the maximum duration of all intonation segments (70 frames) as a baseline, calculate the normalized value of the duration of other segment durations. For example, if the first group of intonation segments has a duration of 60 frames, the normalized length is... And so on. Given the array [0.857, 0.714, 1.000], calculate its mean. .

[0031] Subsequently, the expression coordination index is calculated by substituting it into the following formula. : ; In the above formula, To express the degree of coordination, This indicates the starting frame number of the k-th tone change group. This indicates the starting frame number of the k-th group of facial expression changes. The length of the normalized intonation duration of the k-th group. The normalized mean length of all intonation segments is given, and M represents the total number of matching segments included in the comparison. The formula quantifies the coordination of user emotional expression by calculating the temporal differences between intonation and facial expression changes. The expression coordination index is a comprehensive value used to measure whether there is structural mismatch between intonation changes and facial expression responses in the temporal dimension. The larger the index, the greater the difference in the starting point of intonation and facial expression, the more unstable the intonation rhythm, or the more severe the mismatch, indicating a stronger tendency for synchronization discrepancies in the expression path. The parameters form the quantitative basis for inter-behavioral coordination and mainly serve to determine whether the current behavior exhibits unnatural psychological manifestations such as delayed response or asynchronous expression, acting as a basis for the temporal adaptation and linkage between the emotional response channel and the voice behavior channel.

[0032] Substitute the data from Table 4 into the formula: ; The following indicators were obtained regarding the degree of coordination in expression: ; The result of 0.7002 represents the degree of coordination between the user's current emotional expression, which in turn generates speech boundary difference information, reflecting the degree of coordination between the user's voice and facial expressions in emotional expression.

[0033] The timing deviation determination submodule calls the speech boundary difference information to evaluate the user's emotional expression status in real time, compares the expression coordination degree index with the behavior coordination judgment interval, identifies synchronization abnormal events, and generates synchronization deviation records. The system retrieves the speech boundary difference information and uses the expression coordination index of 0.7002 as a basis. It then compares the behavior coordination judgment interval from the aforementioned baseline adjustment results in real time. An example of setting the behavior coordination judgment interval is: [0~0.5] is coordinated, [0.5~1.0] is slightly abnormal, and [greater than 1.0] is severely abnormal. In this example, the expression coordination index of 0.7002 is within the range of 0.5~1.0. Therefore, the current user's emotional expression coordination is determined to be slightly abnormal. Based on this, the system further identifies and determines the synchronous abnormal events that occur during the psychological assessment process. The occurrence time and degree of the corresponding synchronous abnormal events are recorded in real time, ultimately forming a clear synchronous deviation record. This completes the identification and recording of the temporal deviation of emotional expression during the user's psychological assessment process.

[0034] Please see Figure 5 The anomaly detection module includes: The physiological parameter acquisition submodule calls abnormal disturbance information and synchronization deviation records to collect multiple physiological parameters of users in real time during the psychological assessment process, including heart rate, respiratory rhythm and voice volume, and generates a standardized physiological dataset. By invoking abnormal disturbance information and synchronization deviation records, multiple physiological parameters of the user are continuously monitored during the psychological assessment process using physiological sensing devices, including heart rate, respiratory rhythm, and voice volume. The monitoring devices include a patch-mounted heart rate sensor, a chest strap respiratory rhythm sensor, and a microphone for real-time voice volume measurement. Taking a typical scenario as an example, physiological parameters are recorded in real time at five consecutive time points during the assessment process, as shown in Table 5: Table 5. Monitoring data of physiological parameters during the psychological assessment process: ; The data collected in real time by the aforementioned sensors undergoes standardization processing. Specifically, heart rate, respiratory rhythm, and voice volume are normalized by dividing each by its maximum value within its respective normal range (upper limit for heart rate is 120 beats / min, upper limit for respiratory rhythm is 30 beats / min, and upper limit for voice volume is 70dB). Taking time T1 as an example, the normalized heart rate value is... Normalized value of respiratory rhythm The normalized value of voice volume is Data from other time points were processed in the same way to form a standardized physiological dataset.

[0035] The psychological score calculation submodule calls a standardized physiological dataset and uses heart rate, respiratory rhythm and voice volume to calculate the user's psychological state score in real time, establish a psychological state score sequence and analyze the score change trend to obtain psychological score change trend indicators. The obtained standardized physiological dataset is called, and the psychological state score is used as the output target. The standardized heart rate, respiratory rhythm and voice volume are called in real time and assigned weights of 0.4, 0.3 and 0.3 respectively (the weights were obtained through previous experiments). The user's psychological state score is calculated. The specific calculation method is as follows: at each time point, the standardized values ​​of heart rate, respiratory rhythm and voice volume are multiplied by their respective weight values, and then the three products are added together to obtain the psychological state score at that time point. For example, taking the data at time point T1 as an example, the standardized heart rate is 0.600, the respiratory rhythm is 0.600, and the voice volume is 0.643. The calculation process is as follows: First, multiply the standardized heart rate value of 0.600 by a heart rate weight of 0.4 to obtain a partial score of 0.240; second, multiply the standardized respiratory rhythm value of 0.600 by a respiratory rhythm weight of 0.3 to obtain a partial score of 0.180; then, multiply the standardized voice volume value of 0.643 by a voice volume weight of 0.3 to obtain a partial score of 0.193; finally, add these three partial scores together, i.e., 0.240 + 0.180 + 0.193 = 0.613. Therefore, the psychological state score at time point T1 is 0.613. Similarly, calculate the psychological state scores at times T2 to T5, with results as follows: T2: 0.250 + 0.200 + 0.210 = 0.660, T3: 0.260. +0.220 + 0.223 = 0.703, T4: 0.283 + 0.250 + 0.244 = 0.777, T5: 0.267 + 0.210 + 0.227 = 0.704, finally obtaining the psychological state score sequence [0.613, 0.660, 0.703, 0.777, 0.704]. Further analysis of the sequence change trend and calculation of the difference between adjacent scores are performed. For example, the difference between T2 and T1 is 0.047, the difference between T3 and T2 is 0.043, the difference between T4 and T3 is 0.074, and the difference between T5 and T4 is -0.073. This trend data is used as an indicator of the psychological score change trend, representing the degree of user emotional fluctuation.

[0036] The anomaly focusing analysis submodule calls the psychological score change trend index to identify abnormal state events. It combines the start timestamp information of the already identified facial temperature disturbance events and synchronous abnormal events to identify time overlap segments and analyze the event density and coupling strength of the overlap segments to obtain anomaly focusing intensity information. The aforementioned psychological score change trend indicators were used to further identify abnormal state events. Abnormal event identification was based on the rate of change of psychological scores, with an abnormal score change threshold defined as 0.05. This threshold was set as the median of previous clinical trial data. Through measurements of 50 user samples, a score change exceeding the threshold of 0.05 was determined to be considered an abnormal state. Specifically, the aforementioned score change trend sequence was analyzed, where the rate of change from T3 to T4 was 0.074, exceeding the threshold of 0.05. Therefore, this period was identified as the interval where abnormal state events occurred. The previously identified facial temperature disturbance events (e.g., starting at T3) and synchronization anomalies were then used. The start timestamp of an event (e.g., starting at the end of T3) is used to identify time overlap segments between the three types of events. Taking the overlap between T3 and T4 (e.g., 185 seconds at the end of T3 and 240 seconds at the end of T4) as an example, the density and coupling strength of the overlap segments are further analyzed. The specific method is as follows: taking the start and end time of each event as a node, the proportion of time overlap to the total duration of that period is calculated (overlap duration of 40 seconds, accounting for 72.7% of the total duration of 55 seconds), which is then defined as the abnormal focus intensity. By obtaining the abnormal focus intensity information, the precise quantitative analysis and recording of the focus degree of abnormal psychological state events can be completed.

[0037] Please see Figure 6 The label output module includes: The event time extraction submodule calls the abnormal focus intensity information to extract the timestamp data of facial temperature disturbance events, synchronization abnormal events and state abnormal events, establishes a time structure table of abnormal events, and generates a set of abnormal event time nodes. By invoking the abnormal focus intensity information, the specific start and end timestamp data of facial temperature disturbance events, synchronous abnormal events, and state abnormal events are accurately extracted. Taking a typical psychological assessment process as an example, if a facial temperature disturbance event is identified during the assessment, its start time is 120 seconds after the assessment begins and its end time is 175 seconds after the assessment begins; the start time of the synchronous abnormal event is 160 seconds after the assessment begins and its end time is 210 seconds after the assessment begins; and the state abnormal event starts 165 seconds after the assessment begins and ends 220 seconds after the assessment begins. These event timestamp data are further standardized to establish an abnormal event time structure table, as shown in Table 6. Table 6. Time Structure of Abnormal Events: ; As shown in Table 6, by organizing and recording the start and end timestamps and duration of abnormal events, an accurate set of abnormal event time nodes containing all key abnormal event nodes is generated, serving as the data foundation for subsequent event path analysis.

[0038] The path construction and classification submodule calls the abnormal event time node set, constructs the event occurrence path based on the temporal order and duration of each event, and obtains the abnormal event path sequence; The obtained set of abnormal event time nodes is retrieved, and an event path is established based on the temporal sequence and duration of each abnormal event during the psychological assessment. The specific process includes: sorting each event by its start and end timestamps, using the earliest occurring facial temperature disturbance event as the starting point of the path (120 seconds), and then recording subsequent events sequentially, such as synchronization abnormal events (starting at 160 seconds) and state abnormal events (starting at 165 seconds). The duration of each event is also recorded, such as the facial temperature disturbance event lasting 55 seconds, the synchronization abnormal event lasting 50 seconds, and the state abnormal event lasting 55 seconds, and so on. Based on the duration and time overlap, the overlapping segments between events are clearly defined. For example, the overlap between the facial temperature disturbance event and the synchronous abnormal event is 160 to 175 seconds, with a duration of 15 seconds. The overlap between the synchronous abnormal event and the state abnormal event is 165 to 210 seconds, with a duration of 45 seconds. This forms a clear and quantifiable sequence of event occurrence paths, specifically represented as: "facial temperature disturbance event (120s→175s) → synchronous abnormal event (160s→210s) → state abnormal event (165s→220s)", thus obtaining a complete and specific sequence of abnormal event paths.

[0039] The tag matching output submodule calls the abnormal event path sequence, analyzes the combination characteristics and evolution patterns of the event occurrence path, classifies the user's psychological state and matches psychological state tags, and obtains psychological state monitoring records; The system retrieves the abnormal event path sequence and analyzes its specific combination characteristics and evolution patterns based on the combination features and evolution patterns between events. A psychological state classification model is then established, with the following classification method: typical path patterns are defined as stable, mildly abnormal, and severely abnormal. Stable patterns are defined as events with an overlap duration of less than 20%, mildly abnormal patterns as events with an overlap duration of 20% to 60%, and severely abnormal patterns as events with an overlap duration exceeding 60%. In the current example, the cumulative overlap duration between abnormal events in the path is 15 seconds (facial temperature and synchronization events) plus 45 seconds (synchronization and state abnormal events), totaling 60 seconds. The overall duration of the events is between 120 and 220 seconds, totaling 100 seconds. The overlap ratio is calculated as 60 seconds divided by 100 seconds, resulting in 60%, corresponding to the mildly abnormal pattern. Based on this classification, the specific psychological state label is matched as "mild anxiety," thus generating the final psychological state monitoring record, clearly identified as "mild anxiety," completing the entire psychological state monitoring system's assessment and classification of the user's psychological state.

[0040] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An intelligent mental health monitoring system, characterized in that, The system comprises: The baseline construction module collects voice data of the user in a resting state, analyzes the duration and proportion of the last syllable in the user's voice, obtains tail delay characteristic data according to the change trend of the proportion data in multiple voice segments, adjusts the sensitivity of psychological assessment, and obtains baseline adjustment results; The thermal map analysis module calls the baseline adjustment results, obtains temperature data of the user's face during psychological assessment by using a thermal imaging sensor, calculates temperature difference values of multiple positions, identifies facial temperature disturbance events according to the change amount and reversal times of the direction included angle between consecutive temperature difference values, and obtains abnormal disturbance information; The collaborative assessment module calls the abnormal disturbance information and baseline adjustment results, collects voice and facial image data of the user during psychological assessment, identifies tone change and expression change events, calculates the difference degree of tone change and expression change in time, identifies synchronous abnormal events, and generates synchronous deviation records; The abnormality identification module calls the abnormal disturbance information and synchronous deviation records, calculates the psychological state score of the user by collecting multiple physiological parameters in real time, identifies state abnormal events, combines facial temperature disturbance and synchronous abnormal events, and obtains abnormal focus intensity information.

2. The intelligent mental health monitoring system of claim 1, wherein, The baseline adjustment results are specifically a tail delay proportion sequence, a delay fluctuation interval, and an assessment sensitivity parameter, the abnormal disturbance information includes temperature reversal times, thermal gradient change directions, and temperature difference fluctuation amplitudes, the synchronous deviation records include tone fluctuation positions, expression dynamic start and end times, and time sequence difference parameters, and the abnormal focus intensity information specifically refers to score jump density, behavior abnormality aggregation frequency, and score fluctuation duration.

3. The intelligent mental health monitoring system of claim 2, wherein, The baseline construction module comprises: The voice acquisition and processing submodule collects voice data of the user in a resting state, analyzes the duration of the last syllable in each voice segment, identifies the proportion of the last syllable in the voice, and obtains a tail structure duration parameter; The tail feature identification submodule calls the tail structure proportion parameter, calculates the sum of squares of the difference between adjacent tail proportions, squares the absolute difference of each proportion in the full sequence from the average value, and obtains the change degree of the trend direction marker of adjacent segments, calculates the tail delay trend deviation degree, and generates tail delay characteristic data; The evaluation benchmark adjustment submodule calls the tail delay characteristic data, analyzes the delay characteristics of the user's expression rhythm, evaluates the language output stability, adjusts the sensitivity of psychological assessment, including adjusting the face temperature difference judgment boundary and the behavior coordination judgment interval, and obtains baseline adjustment results.

4. The intelligent mental health monitoring system of claim 3, wherein, The thermal map analysis module comprises: The thermal map data acquisition submodule calls the baseline adjustment results, uses a thermal imaging sensor to collect temperature data of multiple positions of the user's face during psychological assessment, including the tip of the nose, both cheeks, the forehead, and the lower jaw, calculates temperature difference values of multiple positions, and generates a continuous thermal map data set; The disturbance trend calculation submodule calls the continuous thermal map data set, analyzes the included angle change amplitude of the temperature difference direction vector between adjacent frames, records the included angle change rate between frames, extracts the normalized value of each frame temperature difference amplitude and the normalized value of the temperature range, and calculates the disturbance trend score; The boundary comparison identification submodule calls the disturbance trend score, combines temperature disturbance judgment boundary, identifies facial temperature disturbance event, and obtains abnormal disturbance information.

5. The intelligent mental health monitoring system of claim 4, wherein, The cooperative evaluation module includes: The speech graph data acquisition submodule calls the abnormal disturbance information and baseline adjustment result, acquires the speech signal and facial image frame sequence of the user in real time in the psychological evaluation process, identifies the tone change and expression change event, and generates the speech graph behavior sampling dataset; The emotion boundary evaluation submodule calls the speech graph behavior sampling dataset, extracts the start timestamp corresponding to each tone change and expression change event, analyzes the difference degree of the user tone change and expression change in time, calculates the expression coordination degree index, and generates the speech table boundary difference information; The timing deviation judgment submodule calls the speech table boundary difference information, evaluates the expression state of the user in real time, compares the expression coordination degree index with the behavior coordination judgment interval, identifies the synchronization abnormal event, and generates the synchronization deviation record.

6. The intelligent mental health monitoring system of claim 5, wherein, The abnormality identification module includes: The physiological parameter acquisition submodule calls the abnormal disturbance information and synchronization deviation record, acquires multiple physiological parameters of the user in the psychological evaluation process in real time, including heart rate, breathing rhythm, and speech volume, and generates the standardized physiological dataset; The psychological score calculation submodule calls the standardized physiological dataset, calculates the psychological state score of the user in real time by using the heart rate, breathing rhythm, and speech volume, establishes the psychological state score sequence and analyzes the change trend of the score, and obtains the psychological score change trend index; The abnormal focus analysis submodule calls the psychological score change trend index, identifies the state abnormal event, combines the start timestamp information of the facial temperature disturbance event and the synchronization abnormal event, identifies the time overlap section, analyzes the event density and coupling strength of the overlap section, and obtains the abnormal focus strength information.

7. The intelligent mental health monitoring system of claim 1, wherein, The system further includes: The label output module calls the abnormal focus strength information, extracts the timestamp information of the facial temperature disturbance event, the synchronization abnormal event, and the state abnormal event, constructs the event occurrence path according to the time sequence and duration of each event, classifies and matches the psychological state label of the user, and outputs the psychological state monitoring record; The psychological state monitoring record includes the abnormal event timestamp, the event path sequence, and the psychological state label.

8. The intelligent mental health monitoring system of claim 7, wherein, The label output module includes: The event time extraction submodule calls the abnormal focus strength information, extracts the timestamp data of the facial temperature disturbance event, the synchronization abnormal event, and the state abnormal event, establishes the time structure table of the abnormal event, and generates the abnormal event time node set; The path construction and classification submodule calls the abnormal event time node set, constructs the event occurrence path according to the sequence relationship and duration of each event in time, and obtains the abnormal event path sequence; The label matching output submodule calls the abnormal event path sequence, classifies and matches the psychological state label of the user by analyzing the combination characteristics and evolution mode of the event occurrence path, and obtains the psychological state monitoring record.

Images