Psychological risk early warning method and system based on multi-dimensional emotion data of user

Abstract

The invention relates to the technical field of health management, in particular to a psychological risk early warning method and system based on multi-dimensional emotion data of a user. The method comprises the following steps: acquiring user multi-dimensional emotion data corresponding to a user emotion expression image, a user emotion voice signal and a user emotion physiological index, and performing multi-source time sequence alignment processing on the user multi-dimensional emotion data to obtain user multi-dimensional time sequence synchronous emotion data; an emotion three-level feature extraction framework is constructed, modal emotion feature extraction and cross-modal fusion are carried out, and cross-modal fusion psychological features of the user are obtained; and obtaining user environment data and user social network data, and performing emotional psychological dynamic graph reasoning and psychological risk early warning on the user cross-modal fusion psychological features based on the user environment data and the user social network data to generate corresponding user emotional psychological risk early warning response measures. According to the invention, early recognition and intervention of psychological health problems can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of health management, and particularly to a psychological risk warning method and system based on multi-dimensional emotion data of users. Background Art

[0002] Mental health problems have attracted increasing global attention. Especially with the acceleration of the pace of life and the increase of social pressure, the incidence of mental problems such as mood disorders, anxiety, and depression has been rising year by year. Mental health problems not only affect the physical and mental health of individuals, but also have a wide impact on society and the economy. Therefore, early detection and warning of mental risks have important clinical and social significance. By integrating multi-dimensional emotion data of users (such as physiological signals, behavioral data, speech analysis, facial expressions, social interactions, etc.), continuous monitoring of the individual's emotional state can be achieved, so as to more accurately evaluate their mental health status. Currently, existing research has used physiological data such as heart rate and skin conductance response, as well as social media data, etc. for emotion analysis, but most technologies are still in the initial stage, with problems such as insufficient data processing accuracy and difficulty in information fusion. More importantly, existing methods often cannot provide real-time and accurate psychological risk warnings, nor are there effective mechanisms to comprehensively evaluate mental health risks based on multi-dimensional emotion data. Summary of the Invention

[0003] Based on this, it is necessary for the present invention to provide a psychological risk warning method and system based on multi-dimensional emotion data of users to solve at least one of the above technical problems.

[0004] To achieve the above object, a psychological risk warning method based on multi-dimensional emotion data of users includes the following steps: Step S1: Obtain multi-dimensional emotion data of users corresponding to user emotion expression images, user emotion voice signals, and user emotion physiological indicators, and perform multi-source time series alignment processing on the multi-dimensional emotion data of users to obtain multi-dimensional time series synchronized emotion data of users; Step S2: Construct an emotion three-level feature extraction architecture, and perform modal emotion feature extraction on the multi-dimensional time series synchronized emotion data of users based on the emotion three-level feature extraction architecture to obtain user emotion psychological features corresponding to each level of modality; by designing an inter-modal attention matrix to cross-modal fuse the user emotion psychological features corresponding to each level of modality, obtain user cross-modal fusion psychological features; Step S3: Obtain user environmental data and user social network data, and perform emotion psychological dynamic graph reasoning on the user cross-modal fusion psychological features based on the user environmental data and user social network data to generate an emotion psychological causal dynamic relationship graph between the user's corresponding environmental factors, behavior patterns, and mental states; Step S4: Obtain the corresponding user emotion evolution risk inflection point through the emotional psychological causal dynamic relationship diagram among the user's corresponding environmental factors, behavior patterns, and psychological states, and conduct psychological risk early warning based on the user emotion evolution risk inflection point to generate corresponding user emotional psychological risk early warning response measures.

[0005] Further, step S1 includes the following steps: Step S11: Obtain the user's emotion expression image; Step S12: Obtain the user's emotion voice signal; Step S13: Obtain the user's emotion physiological indicators, including the heart rate variability index HRV and the galvanic skin response index GSR corresponding to the user's emotion; Step S14: Perform interference elimination preprocessing on the user's multi-dimensional emotion data corresponding to the user's emotion expression image, user's emotion voice signal, and user's emotion physiological indicators, so as to uniformly eliminate the environmental interference corresponding to the user's emotion expression image under different lighting conditions by using a generative adversarial network, and separate the corresponding human voice and environmental noise in the user's emotion voice signal based on deep clustering. At the same time, use the Temporal Convolutional Network (TCN) to predict and compensate the missing data segments in the user's emotion physiological indicators to obtain the user's multi-dimensional interference elimination emotion data; Step S15: Perform multi-source time series alignment processing on the user's multi-dimensional interference elimination emotion data to eliminate the time series sampling frequency differences between different data sources and obtain the user's multi-dimensional time series synchronous emotion data.

[0006] Further, step S2 includes the following steps: Step S21: Construct an emotion three-level feature extraction architecture, including emotion feature extraction modules corresponding to the first, second, and third levels. The first level is used to extract the emotion visual features corresponding to the user's emotion expression image, the second level is used to extract the emotion voice features corresponding to the user's emotion voice signal, and the third level is used to extract the emotion time series features corresponding to the user's emotion physiological indicators; Step S22: Perform image modal feature extraction on the user's emotion expression image corresponding to the user's multi-dimensional time series synchronous emotion data based on the first-level emotion feature extraction module in the emotion three-level feature extraction architecture to obtain the user's emotional psychological features corresponding to the first modal, including the degree of frown between the eyebrows and the deviation angle of the drooping corner of the mouth corresponding to the user's facial movements in the image; Step S23: Perform voice modal feature extraction on the user's emotion voice signal corresponding to the user's multi-dimensional time series synchronous emotion data based on the second-level emotion feature extraction module in the emotion three-level feature extraction architecture, so as to extract the prosodic features and semantic features through the pre-trained Wav2Vec 2.0 model in this module. The prosodic features include the fundamental frequency F0 and the speech rate, and the semantic features include the frequency of emotional words in the wind direction, so as to obtain the user's emotional psychological features corresponding to the second modal; Step S24: The three-level emotion feature extraction module in the emotion three-level feature extraction architecture extracts physiological modal features from the corresponding user emotion physiological indicators in the user's multi-dimensional time-series synchronized emotion data to obtain the user emotion psychological features corresponding to the three levels of modalities, including the LF / HF ratio of HRV and the rising slope of the GSR index; Step S25: By designing an inter-modal attention matrix, the user emotion psychological features corresponding to the first, second, and third levels of modalities are fused across modalities to obtain the user cross-modal fused psychological features.

[0007] Furthermore, step S22 includes the following steps: Input the corresponding user emotion expression image in the user's multi-dimensional time-series synchronized emotion data into the first-level emotion feature extraction module in the emotion three-level feature extraction architecture to use the EfficientNet-B4 network to extract the facial action unit time periods corresponding to the user image sequence, including the user's eyebrow movement unit time period and the user's mouth corner movement unit time period; Obtain the action duration and action amplitude corresponding to the eyebrow movement unit through the user's eyebrow movement unit time period, and quantify the muscle contraction tension corresponding to the eyebrow area within the user's eyebrow movement unit time period based on the action duration and action amplitude corresponding to the eyebrow movement unit to obtain the eyebrow muscle contraction frequency corresponding to the eyebrow movement unit; Obtain the number of wrinkles, wrinkle depth, and wrinkle direction change corresponding to the eyebrow movement unit through the user's eyebrow movement unit time period, and evaluate the attenuation of the wrinkling contribution of the muscles in the eyebrow area during the action based on the number of wrinkles, wrinkle depth, and wrinkle direction change corresponding to the eyebrow movement unit to obtain the eyebrow wrinkling contribution attenuation index corresponding to the eyebrow movement unit; Based on the eyebrow wrinkling contribution attenuation index corresponding to the eyebrow movement unit and combined with the eyebrow muscle contraction frequency, action duration, and action amplitude corresponding to the eyebrow movement unit, dynamically estimate the eyebrow wrinkling degree of the user's image facial action during this time period to obtain the eyebrow wrinkling degree corresponding to the user's image facial action at this time period; Statistically analyze the downward deviation amplitude of the action of the user's mouth corner movement unit time period to obtain the downward deviation angle of the mouth corner corresponding to the user's image facial action at this time period; Combine the eyebrow wrinkling degree and the downward deviation angle of the mouth corner corresponding to the user's image facial action into a vector as the user emotion psychological feature corresponding to the first-level emotion feature extraction module, so as to obtain the user emotion psychological feature corresponding to the first-level modality.

[0008] Furthermore, the statistical analysis of the downward deviation amplitude of the action of the user's mouth corner movement unit time period includes the following steps: Analyze the movement trajectory of the user's mouth corners during the action unit period of the mouth corners to generate the corresponding movement trajectory of the user's mouth corners during this period; Obtain the drooping time series fluctuation corresponding to the user's mouth corner action through the corresponding movement trajectory of the user's mouth corner during this period, and fit the dynamic change of the mouth corner action within the action unit period of the user's mouth corner based on the drooping time series fluctuation corresponding to the user's mouth corner action to generate the fitting function relationship between time and the change of the user's mouth corner drooping position during this period; Measure the deviation of the mouth corner drooping amplitude according to the fitting function relationship between time and the change of the user's mouth corner drooping position during this period to obtain the corresponding fitting amplitude deviation of the user's mouth corner drooping during this period; Obtain the corresponding mouth corner length of the user, and statistically analyze the action drooping offset amplitude of the corresponding fitting amplitude deviation of the user's mouth corner drooping during this period based on the corresponding mouth corner length of the user to obtain the mouth corner drooping offset angle corresponding to the facial action of the user image during this period.

[0009] Further, step S24 includes the following steps: Step S241: Input the user's emotional physiological indicators corresponding to the user's multi-dimensional time series synchronous emotional data, including the heart rate variability index HRV and the galvanic skin response index GSR, into the three-level emotional feature extraction module in the three-level emotional feature extraction architecture to extract the corresponding LF value and HF value on HRV and the rising slope section interval corresponding to GSR; Step S242: Calculate the ratio according to the corresponding LF value and HF value on HRV to obtain the LF / HF ratio of HRV; Step S243: Obtain the corresponding rising change amplitude of the GSR index and the rising time deviation of the GSR index through the rising slope section interval corresponding to GSR; Step S244: Perform a rising slope fitting calculation on the rising slope section interval corresponding to GSR based on the rising change amplitude of the GSR index and the rising time deviation of the GSR index to obtain the rising slope of the GSR index.

[0010] Further, step S25 includes the following steps: Step S251: Design the corresponding inter-modal attention matrices between the first-level, second-level, and third-level emotional feature extraction modules in the three-level emotional feature extraction architecture, and obtain the feature query vectors and key vectors between different modalities based on this inter-modal attention matrix; Step S252: Calculate the corresponding feature vector distance value according to the feature query vectors and key vectors between different modalities; Step S253: Cross-modally weighted fuse the user's emotional psychological features corresponding to the first-level, second-level, and third-level modalities based on this feature vector distance value to obtain the user's cross-modal fusion psychological features.

[0011] Further, step S3 includes the following steps: Step S31: Obtain user environmental data and user social network data; Step S32: Extract environmental factors from the user environmental data to obtain each user environmental factor; Step S33: Analyze the user behavior patterns in the user social network data to obtain the corresponding user emotional and psychological behavior patterns of the user on the social network; Step S34: Based on each user environmental factor and the corresponding user emotional and psychological behavior patterns of the user on the social network, perform emotional and psychological dynamic graph reasoning on the emotional and psychological characteristic states corresponding to the user's cross-modal fusion psychological characteristics within the same time, so as to generate an emotional and psychological causal dynamic relationship graph between the corresponding environmental factors, behavior patterns and psychological states of the user.

[0012] Further, step S4 includes the following steps: Step S41: Obtain the emotional and psychological fluctuation influence path of the user under the corresponding environmental factors and behavior patterns through the emotional and psychological causal dynamic relationship graph between the user's corresponding environmental factors, behavior patterns and psychological states; Step S42: Quantify the emotional fluctuation level at each time point in the emotional and psychological fluctuation influence path of the user under the corresponding environmental factors and behavior patterns to obtain the emotional and psychological fluctuation level at each time point on the emotional and psychological fluctuation influence path of the user; Step S43: Predict the extreme inflection point of emotional evolution according to the emotional and psychological fluctuation level at each time point on the emotional and psychological fluctuation influence path of the user, so as to obtain the corresponding user emotional evolution risk inflection point on the emotional and psychological fluctuation influence path of the user; Step S44: Calculate the corresponding relative risk index through the corresponding user emotional evolution risk inflection point on the emotional and psychological fluctuation influence path of the user, and dynamically divide the risk level of the relative risk index according to the preset five-stage risk assessment system, where the preset five-stage risk assessment system includes normal when the relative risk index ≤ 10, subclinical when 10 < relative risk index ≤ 30, mild risk when 30 < relative risk index ≤ 40, moderate and mild risk when 40 < relative risk index ≤ 60, and high risk when the relative risk index > 60, so as to divide the corresponding user emotional and psychological risk level. At the same time, according to the corresponding user emotional and psychological risk level, perform a risk warning response on the user emotional evolution risk inflection point to generate corresponding user emotional and psychological risk warning response measures, including automatically triggering self-regulation suggestions for risk levels of mild risk and below, and automatically notifying emergency contacts for risk levels above mild risk.

[0013] Furthermore, the present invention also provides a psychological risk early warning system based on multi-dimensional emotion data of users, which is used to execute the psychological risk early warning method based on multi-dimensional emotion data of users as described above. The psychological risk early warning system based on multi-dimensional emotion data of users includes: A multi-dimensional emotion time series alignment module, configured to obtain multi-dimensional emotion data of a user corresponding to a user emotion expression image, a user emotion voice signal, and a user emotion physiological index, and perform multi-source time series alignment processing on the multi-dimensional emotion data of the user, so as to obtain multi-dimensional time series synchronized emotion data of the user; An emotion psychological feature fusion module, configured to construct an emotion three-level feature extraction architecture, and perform modal emotion feature extraction on the multi-dimensional time series synchronized emotion data of the user based on the emotion three-level feature extraction architecture, so as to obtain user emotion psychological features corresponding to each level of modality; by designing an inter-modal attention matrix to cross-modally fuse the user emotion psychological features corresponding to each level of modality, so as to obtain user cross-modal fusion psychological features; An emotion psychological dynamic reasoning module, configured to obtain user environment data and user social network data, and perform emotion psychological dynamic graph reasoning on the user cross-modal fusion psychological features based on the user environment data and the user social network data, so as to generate an emotion psychological causal dynamic relationship graph between the corresponding environmental factors, behavior patterns, and psychological states of the user; An emotion psychological risk early warning module, configured to obtain corresponding user emotion evolution risk inflection points through the emotion psychological causal dynamic relationship graph between the corresponding environmental factors, behavior patterns, and psychological states of the user, and perform psychological risk early warning according to the user emotion evolution risk inflection points, so as to generate corresponding user emotion psychological risk early warning response measures.

[0014] Advantages of the present invention: 1. The psychological risk early warning method based on multi-dimensional emotion data of users proposed by the present invention, compared with the prior art, the beneficial effect of the present application is that by combining multiple sensors and data sources, multi-dimensional data about the emotions of users is obtained, including emotion expression images, voice signals, and physiological index data. The multi-source nature of this data provides rich information for in-depth analysis of users' emotions and can capture the diversity and complexity of emotions. For example, emotion expression images can reflect the micro-expression changes on the user's face, voice signals reveal the semantic information of the user's emotions through characteristics such as tone and pitch, and physiological indicators, such as heart rate and skin conductance response, provide physiological signals reflecting emotional changes at the physical level. Aligning these multi-source data in time series is the core of this step. It can make various emotion data match with time, ensure data synchronization. Through time series alignment, it is possible to accurately know the user's emotion state at each moment, reduce errors caused by data deviation, and thus obtain the user's true emotion response, which provides reliable data support for subsequent extraction, reasoning, and early warning of emotional psychological characteristics. Secondly, an emotion three-level feature extraction architecture is first constructed, which is to comprehensively analyze and extract the psychological characteristics of users' emotions at multiple levels. This hierarchical feature extraction architecture not only improves the accuracy of emotion analysis but also provides a structured framework for the fusion of emotional psychological characteristics. Next, cross-modal feature fusion is performed using an inter-modal attention matrix. This operation can effectively fuse information from different modalities and highlight the complementarity between modalities. For example, facial expressions and voice signals may have similar emotional information, and physiological signals can provide auxiliary emotional clues at the physical level. By designing the attention matrix, these information can be intelligently weighted and fused according to the weights and importance of different modalities, and finally a more comprehensive and accurate cross-modal emotional psychological characteristic of the user is obtained. These fused characteristics help to reveal the internal connections of the user's emotion state, thereby improving the processing accuracy of multi-source emotion data and solving the problem of difficult fusion of emotion feature information. Then, by introducing environmental data and social network data, important external influencing factors are provided for the emotion psychological model. Environmental data can include factors such as weather, scene, and noise, which will directly or indirectly affect the emotional fluctuations of users. For example, too high or too low temperature, and the noise of the environment will cause negative changes in the emotions of users. At the same time, social network data reveals social phenomena such as users' social activities, interaction content, and emotional contagion, which can further affect the user's emotion state. After combining these environmental and social data with the cross-modal fused psychological characteristics, reasoning can be performed based on the emotion psychological dynamic graph to reveal the evolution law and dynamic changes of emotions. By inferring the evolution process of emotions, a causal relationship graph between the user's emotions, environment, and behavior patterns can be constructed, which helps to understand how different environmental and behavioral factors affect the user's emotions.Finally, by analyzing the emotional psychological causal dynamic relationship diagram, the risk inflection points of users' emotional changes can be identified, especially the early warnings at moments of large emotional fluctuations or sudden changes. These inflection points usually indicate that the user's emotions will deteriorate sharply, thereby triggering mental health problems. By identifying these risk inflection points of emotional evolution, real-time and accurate mental risk early warnings can be provided, offering a time window for intervention and mitigation measures. For example, if it is detected that a user has large emotional fluctuations at a certain moment and the analysis shows that the fluctuations are caused by environmental stress or negative interactions in the social network, an alarm will be issued and corresponding intervention measures will be recommended. These measures include emotional regulation techniques, social interaction suggestions, environmental adjustment suggestions, etc., which help users take actions before the occurrence of emotional crises, prevent the exacerbation of emotional problems, and effectively reduce the negative impacts of emotional problems, thus enabling a better comprehensive assessment of mental health risks.

[0015] 2. The mental risk early warning system based on users' multi-dimensional emotional data proposed by the present invention is generally composed of a multi-dimensional emotional time series alignment module, an emotional psychological feature fusion module, an emotional psychological dynamic reasoning module, and an emotional psychological risk early warning module. It can implement any of the mental risk early warning methods based on users' multi-dimensional emotional data described in the present invention. For the operation between computer programs running on each module to implement the mental risk early warning method based on users' multi-dimensional emotional data, the internal structure of the system collaborates with each other, which can greatly reduce repetitive work and manpower input, quickly and effectively provide a more accurate and efficient mental risk early warning process based on users' multi-dimensional emotional data, thus simplifying the operation process of the mental risk early warning system based on users' multi-dimensional emotional data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Other features, objectives, and advantages of the present invention will become more obvious by reading the detailed description of the non-limiting embodiments with reference to the following drawings: Figure 1 It is a schematic flowchart of the steps of the mental risk early warning method based on users' multi-dimensional emotional data of the present invention; Figure 2 For Figure 1 the detailed flowchart of step S1 in Figure 3 For Figure 1 the detailed flowchart of step S2 in DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] The technical method of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0018] In addition, the accompanying drawings are only schematic diagrams of the present invention and are not necessarily drawn to scale. The same reference numerals in the drawings represent the same or similar parts, and thus repeated descriptions thereof will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor methods and / or microcontroller methods.

[0019] It should be understood that although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the associated listed items.

[0020] To achieve the above object, please refer to Figures 1 to 3 , the present invention provides a psychological risk early warning method based on user multi-dimensional emotion data, and the method includes the following steps: Step S1: Obtain user multi-dimensional emotion data corresponding to user emotion expression images, user emotion voice signals, and user emotion physiological indicators, and perform multi-source time series alignment processing on the user multi-dimensional emotion data to obtain user multi-dimensional time series synchronous emotion data; Step S2: Construct an emotion three-level feature extraction architecture, and perform modal emotion feature extraction on the user multi-dimensional time series synchronous emotion data based on the emotion three-level feature extraction architecture to obtain user emotion psychological features corresponding to each level of modality; by designing an inter-modal attention matrix to cross-modal fuse the user emotion psychological features corresponding to each level of modality, obtain user cross-modal fusion psychological features; Step S3: Obtain user environment data and user social network data, and perform emotion psychological dynamic graph reasoning on the user cross-modal fusion psychological features based on the user environment data and user social network data to generate an emotion psychological causal dynamic relationship graph between the user's corresponding environmental factors, behavior patterns, and psychological states; Step S4: Obtain the corresponding user emotion evolution risk inflection point through the emotional psychological causal dynamic relationship diagram among the user's environmental factors, behavior patterns, and psychological states, and perform psychological risk early warning based on the user emotion evolution risk inflection point to generate corresponding user emotional psychological risk early warning response measures.

[0021] In the embodiment of the present invention, please refer to Figure 1 As shown, it is a schematic diagram of the step flow of the psychological risk early warning method based on user multi-dimensional emotion data of the present invention. In this example, the psychological risk early warning method based on user multi-dimensional emotion data includes the following steps: Step S1: Obtain the user multi-dimensional emotion data corresponding to the user's emotion expression image, user emotion voice signal, and user emotion physiological index, and perform multi-source time series alignment processing on the user multi-dimensional emotion data to obtain the user multi-dimensional time series synchronized emotion data; In the embodiment of the present invention, in the intelligent mental health monitoring scenario, the user's emotion expression image is collected by a fixedly deployed high-definition camera at a frame rate of 30 frames per second, the user's emotion voice signal is recorded by a microphone at a sampling rate of 44.1 kHz and a quantization precision of 16 bits. At the same time, the wearable physiological monitoring device collects the user's emotion physiological index, including heart rate, skin conductance response, and respiratory rate, at a frequency of 100 times per second. After collecting the user multi-dimensional emotion data, multi-source time series alignment processing is performed. Taking the heart rate data and the voice signal as an example, assuming that the timestamp of the heart rate data starts from 10:00:00.000 and the timestamp of the voice signal starts from 10:00:00.100, the timestamp mapping algorithm is used to subtract 0.1 second from the timestamp of the voice signal to align it with the timestamp of the heart rate data. For the expression image, according to the collection frame rate and the key frame detection algorithm, each frame of the image is corresponding to an accurate time point. In this way, the data of all dimensions are unified to the same time basis to obtain the user multi-dimensional time series synchronized emotion data. For example, at the time point of 10:00:01, there are corresponding expression image data, voice signal segments, and physiological index data at the same time.

[0022] Step S2: Construct an emotion three-level feature extraction architecture, and perform modal emotion feature extraction on the user multi-dimensional time series synchronized emotion data based on the emotion three-level feature extraction architecture to obtain the user emotion psychological features corresponding to each level of modality; design an inter-modal attention matrix to cross-modal fuse the user emotion psychological features corresponding to each level of modality to obtain the user cross-modal fusion psychological features; In the embodiments of the present invention, when constructing the three-level emotion feature extraction architecture, based on the processing requirements of multi-dimensional emotion data of users in a certain psychological counseling scenario, the first-level emotion feature extraction module adopts a convolutional neural network (CNN) architecture, and selects EfficientNet-B4 as the backbone network. This network contains 66 convolutional layers, and the input image resolution is fixed at 224×224 pixels, which is used to extract the emotional visual features of the user's emotional expression images, including the degree of frown between the eyebrows and the deviation angle of the drooping corners of the mouth corresponding to the facial movements of the user's image. The second-level emotion feature extraction module adopts a pre-trained Wav2Vec2.0 model, which is based on the Transformer architecture and has 12 encoder layers. The sampling rate of the input voice signal is set at 16 kHz, which is used to extract the emotional voice features of the user's emotional voice signals, including prosodic features and semantic features. Among them, the prosodic features include the fundamental frequency F0 and the speech rate, and the semantic features include the frequency of windward emotional words. The third-level emotion feature extraction module adopts a temporal convolutional network (TCN) architecture, with a network depth of 5 layers and a convolutional kernel size of 3, which is used to extract the emotional temporal features of the user's emotional physiological indicators, including the LF / HF ratio of HRV and the slope of the rising slope of the GSR index. Each module is orderly connected through a data interface to form a complete three-level emotion feature extraction architecture. At the same time, by designing a 3×3 inter-modal attention matrix , which is used to cross-modal fuse the emotional psychological features of the user corresponding to the first, second, and third modalities. Assume the first-modal feature vector , the second-modal feature vector , the third-modal feature vector . First, calculate the attention weights between each modality. For the attention weight of the first modality to the second modality , use the formula (where is a trainable weight matrix. Assume is the identity matrix). Similarly, calculate other weights to obtain the complete attention matrix . Then, use the attention matrix to perform weighted fusion on the features of each modality. The user's cross-modal fusion psychological feature vector . After calculation, the fused feature vector is obtained, realizing the effective fusion of multi-modal emotion features and providing comprehensive feature data for subsequent psychological risk warning.

[0023] Step S3: Obtain the user's environmental data and the user's social network data, and perform emotional psychological dynamic graph reasoning on the user's cross-modal fusion psychological features based on the user's environmental data and the user's social network data to generate an emotional psychological causal dynamic relationship graph between the user's corresponding environmental factors, behavior patterns, and psychological states; In the embodiments of the present invention, user environmental data and social network data are obtained through an environmental sensor and a social network data interface. The environmental sensor collects environmental temperature, humidity, light intensity, and noise decibel values once per minute. The social network data interface synchronizes the text content published by the user, like and comment records, and friend relationship data once per hour. The user environmental data is processed. Taking the temperature data as an example, the average value of 24 data within a day is calculated. , standard deviation . If the standard deviation is greater than 3°C, it is determined as an environmental factor with large temperature fluctuations. For social network data, natural language processing technology is used to analyze the emotional tendency of the text content, and the frequency of negative words appearing is calculated. If the frequency of negative words exceeds 30% within a certain period of time, and the interaction frequency with friends having a negative emotional tendency accounts for more than 40% of the total interaction frequency, it is determined that there is a negative social behavior pattern. Based on the user environmental factors and social behavior patterns, combined with the user's cross-modal fusion psychological characteristics, emotional and psychological dynamic graph reasoning is performed. A Bayesian network is used to construct a dynamic graph, with environmental factors, behavior patterns, and psychological characteristics as nodes. The weights of the edges are determined by analyzing the joint probability distribution between these nodes in historical data. For example, it is known that in historical data, when the temperature fluctuates greatly and there is a negative social behavior, the probability of the user showing an anxious mental state is 0.7. Based on this, the weights of the corresponding edges are set in the Bayesian network, and finally, an emotional and psychological causal dynamic relationship graph between the user's corresponding environmental factors, behavior patterns, and mental states is generated.

[0024] Step S4: Obtain the corresponding user emotion evolution risk inflection point through the emotional and psychological causal dynamic relationship graph between the user's corresponding environmental factors, behavior patterns, and mental states, and perform psychological risk early warning according to the user emotion evolution risk inflection point to generate corresponding user emotional and psychological risk early warning response measures.

[0025] In an embodiment of the present invention, by obtaining the inflection point of the user's emotional evolution risk from the emotional psychological causal dynamic relationship diagram, using the depth-first search algorithm, starting from the environmental factor and behavior pattern nodes, exploring the path along the edge with a larger probability weight, and recording the change of the emotional psychological fluctuation level during the exploration process. The emotional psychological fluctuation level is calculated by weighted summation. The weight of environmental factors is set to 0.4, and the weight of behavior patterns is set to 0.6. Taking the example of the emotional change path caused by "large temperature fluctuation" and "negative social behavior", the probability weight between "large temperature fluctuation" and the "mild anxiety" node is 0.6, and the probability weight between "negative social behavior" and the "moderate anxiety" node is 0.7. Then the emotional fluctuation level under the influence of "large temperature fluctuation" is 0.4×0.6 = 0.24, and the emotional fluctuation level under the influence of "negative social behavior" is 0.6×0.7 = 0.42. The total fluctuation level is 0.24 + 0.42 = 0.66. The quadratic difference algorithm is used to predict the extreme inflection point. When the quadratic difference changes from positive to negative, it is the maximum inflection point, that is, the inflection point of the emotional evolution risk. For the determined risk inflection point, calculate the relative risk index. The formula is relative risk index = emotional fluctuation level × 10 × (1 + adjacent risk inflection point interval time / average interval time). Suppose the emotional fluctuation level of a certain risk inflection point is 1.5, the adjacent risk inflection point interval time is 3 hours, and the average interval time is is 5 hours. Then the relative risk index is 1.5×10×(1 + 3 / 5) = 24. According to the preset five-stage risk assessment system, divide the risk level. 10 < 24 ≤ 30, and the risk level corresponding to this risk inflection point is subclinical. For mild risks and below levels, automatically push self-regulation suggestions including deep breathing exercises and meditation guidance to the user; for levels above mild risks, automatically send a warning notice including the risk level, time, and key influencing factors to the user's preset emergency contacts to complete the early warning response to the user's emotional psychological risk.

[0026] Further, step S1 includes the following steps: Step S11: Obtain the user's emotional expression image; Step S12: Obtain the user's emotional voice signal; Step S13: Obtain the user's emotional physiological indicators, including the heart rate variability index HRV and the galvanic skin response index GSR corresponding to the user's emotions; Step S14: Perform interference elimination preprocessing on the user's multi-dimensional emotional data including the user's emotional expression image, the user's emotional voice signal, and the user's emotional physiological indicators, so as to uniformly eliminate the environmental interference corresponding to the user's emotional expression image under different lighting conditions by using a generative adversarial network, and separate the corresponding human voice and environmental noise in the user's emotional voice signal based on deep clustering. At the same time, use the TCN temporal convolutional network to predict and compensate the missing data segments in the user's emotional physiological indicators to obtain the user's multi-dimensional interference elimination emotional data; Step S15: Perform multi-source time-series alignment processing on the user's multi-dimensional interference-eliminated emotion data to eliminate the time-series sampling frequency differences between different data sources, and obtain the user's multi-dimensional time-series synchronized emotion data.

[0027] As an embodiment of the present invention, refer to Figure 2 shown in Figure 1 the detailed step flow diagram of step S1 in Step S11: Obtain the user's emotional expression image; In the embodiment of the present invention, in an actual application scenario, such as a psychological counseling room, a high-definition camera is deployed to obtain the user's emotional expression image. The camera resolution is 4K (3840×2160 pixels), and the frame rate is set to 30 frames per second, which can clearly capture the subtle facial expression changes of the user. For example, when the user enters the counseling room and sits down, the camera starts to continuously collect image data from 1.5 meters in front. During the next 10-minute counseling process, a total of 18,000 images are collected. Each image contains the complete information of the user's key facial parts such as eyebrows, eyes, and mouth, providing the original visual data for subsequent emotion analysis.

[0028] Step S12: Obtain the user's emotional voice signal; In the embodiment of the present invention, in the same psychological counseling room environment, a professional directional microphone is used to obtain the user's emotional voice signal. The microphone sensitivity is -38dBV / Pa, and the frequency response range is 20Hz - 20kHz, which can effectively collect the user's voice. During the above 10-minute counseling process, the microphone records the user's voice at a sampling frequency of 44.1kHz and a quantization precision of 16 bits, generating an audio file in PCM format. For example, when the user expresses anxiety, the microphone accurately records the voice characteristics of the user's accelerated speech rate and increased intonation, as well as the details such as pauses and sighs when elaborating on the troubles, completely retaining the emotional information in the user's emotional voice signal.

[0029] Step S13: Obtain the user's emotional physiological indicators, including the heart rate variability index HRV and the galvanic skin response index GSR corresponding to the user's emotion; In an embodiment of the present invention, a wearable physiological monitoring device is used to obtain the user's emotional physiological indicators. The device is equipped with a high-precision heart rate sensor and a skin galvanic response sensor. The heart rate sensor adopts PPG photoelectric volume pulse wave technology with a sampling frequency of 100Hz, which can monitor the user's heart rate variability index HRV in real time. The skin galvanic response sensor obtains the skin galvanic response index GSR by measuring the tiny current changes on the skin surface. The sampling frequency is 50Hz. During the 10-minute consultation process, the HRV data records the millisecond-level changes in each heartbeat interval of the user, such as the fluctuations of adjacent heartbeat intervals under different emotional states; the GSR data reflects the changes in the user's skin conductivity. For example, when the user is nervous, the skin sweats, resulting in increased conductivity. The device accurately captures the change and generates a corresponding data sequence, providing a physiological basis for emotional analysis.

[0030] Step S14: performing interference elimination preprocessing on the user's multi-dimensional emotional data including the user's emotional expression image, the user's emotional voice signal, and the user's emotional physiological indicators, so as to uniformly eliminate the environmental interference corresponding to the user's emotional expression image under different lighting conditions using a generative adversarial network, and separate the human voice and environmental noise corresponding to the user's emotional voice signal based on deep clustering, while using a TCN temporal convolutional network to predict and compensate for the missing data segments corresponding to the user's emotional physiological indicators, so as to obtain the user's multi-dimensional interference eliminated emotional data; In an embodiment of the present invention, interference elimination preprocessing is performed on the collected multi-dimensional emotional data of the user, and a generative adversarial network (GAN) is used to eliminate environmental interference for the user's emotional expression images. Both the generator and the discriminator use a convolutional neural network structure. The generator input includes original images and random noise vectors under different lighting conditions, and a denoised image is generated through multi-layer convolution and deconvolution operations; the discriminator distinguishes between real noise-free images and generated images. During the training process, the learning rate is set to 0.0002 and the batch size is 32. After 50 training cycles, the 18,000 collected images are processed, such as uniformly improving the brightness of images in dark environments to eliminate the interference of shadows on expression recognition. For user emotional voice signals, a deep clustering algorithm is used to separate human voices from environmental noise. The density-based DBSCAN clustering algorithm is used to set the neighborhood radius. = 0.5, the minimum sample number MinPts = 5. Cluster the time-frequency domain features of the audio signal to divide human voices and environmental noises into different categories, thereby removing environmental noises such as the running sound of the air conditioner and the noise of vehicles outside the window. And through the user's emotional physiological indicators, use the TCN (Temporal Convolutional Network) to predict and compensate for the missing data segments. Assume that there are 100 missing data points in the HRV data from the 3rd to the 5th minute. The TCN network inputs the complete data of the 5 minutes before and after, captures the temporal dependence relationship through causal convolution and dilated convolution, and outputs the predicted missing data. The network contains 3 convolutional layers, the size of the convolutional kernel in each layer is 3, and the dilation factors are 1, 2, and 4 respectively. After training, the mean square error between the predicted value and the actual value is 0.05, effectively filling the missing data, and finally obtaining the user's multi-dimensional interference-eliminated emotional data.

[0031] Step S15: Perform multi-source temporal alignment processing on the user's multi-dimensional interference-eliminated emotional data to eliminate the temporal sampling frequency differences between different data sources, and obtain the user's multi-dimensional temporally synchronized emotional data.

[0032] In the embodiment of the present invention, through multi-source temporal alignment processing of the user's multi-dimensional interference-eliminated emotional data, the sampling frequency of the user's emotional expression images is 30 frames per second, the sampling frequency of the user's emotional voice signal is 44.1 kHz, the sampling frequency of the user's emotional physiological indicator HRV is 100 Hz, and the sampling frequency of GSR is 50 Hz. The dynamic time warping (DTW) algorithm is used to eliminate the temporal sampling frequency differences between different data sources. Taking the alignment of the emotional voice signal and the HRV data as an example, the voice signal is segmented by frames, the Euclidean distance between each frame and each time point of the HRV data is calculated to construct a distance matrix, the maximum slope of the bending path is set to 2, and the cumulative distance between the two sequences is minimized by finding the optimal path. After processing, all the data are uniformly aligned to the time axis based on 100 Hz to ensure the temporal consistency of different-dimensional data, and finally obtain the user's multi-dimensional temporally synchronized emotional data, providing a reliable data basis for subsequent psychological risk early warning analysis.

[0033] Further, step S2 includes the following steps: Step S21: Construct an emotional three-level feature extraction architecture, which includes emotional feature extraction modules corresponding to the first, second, and third levels. The first level is used to extract the emotional visual features corresponding to the user's emotional expression images, the second level is used to extract the emotional voice features corresponding to the user's emotional voice signals, and the third level is used to extract the emotional temporal features corresponding to the user's emotional physiological indicators; Step S22: Based on the primary emotion feature extraction module within the three-level emotion feature extraction architecture, perform image modal feature extraction on the corresponding user emotion expression images in the user's multi-dimensional time-series synchronous emotion data to obtain the user's emotion psychological features corresponding to the primary modality, including the degree of eyebrow wrinkling corresponding to the user's facial movements in the image and the deviation angle of the downturned mouth corner; Step S23: Based on the secondary emotion feature extraction module within the three-level emotion feature extraction architecture, perform speech modal feature extraction on the corresponding user emotion speech signals in the user's multi-dimensional time-series synchronous emotion data to extract prosodic features and semantic features through the pre-trained Wav2Vec 2.0 model within this module. The prosodic features include fundamental frequency F0 and speech rate, and the semantic features include the frequency of windward emotional words, to obtain the user's emotion psychological features corresponding to the secondary modality; Step S24: Based on the tertiary emotion feature extraction module within the three-level emotion feature extraction architecture, perform physiological modal feature extraction on the corresponding user emotion physiological indicators in the user's multi-dimensional time-series synchronous emotion data to obtain the user's emotion psychological features corresponding to the tertiary modality, including the LF / HF ratio of HRV and the rising slope of the GSR index; Step S25: Design an inter-modal attention matrix to cross-modally fuse the user's emotion psychological features corresponding to the primary, secondary, and tertiary modalities to obtain the user's cross-modal fused psychological features.

[0034] As an embodiment of the present invention, refer to Figure 3 shown in Figure 1 for the detailed step flow diagram of step S2 in Step S21: Construct a three-level emotion feature extraction architecture, which includes emotion feature extraction modules corresponding to the primary, secondary, and tertiary levels. The primary level is used to extract the emotional visual features corresponding to the user's emotion expression images, the secondary level is used to extract the emotional speech features corresponding to the user's emotion speech signals, and the tertiary level is used to extract the emotional time-series features corresponding to the user's emotion physiological indicators; In an embodiment of the present invention, when constructing the three-level emotion feature extraction architecture, based on the processing requirements of multi-dimensional emotion data of users in a certain psychological counseling scenario, the first-level emotion feature extraction module adopts a convolutional neural network (CNN) architecture, and selects EfficientNet-B4 as the backbone network. This network contains 66 convolutional layers, and the input image resolution is fixed at 224×224 pixels, which is used to extract the emotional visual features of the user's emotional expression images. The second-level emotion feature extraction module adopts a pre-trained Wav2Vec 2.0 model, which is based on the Transformer architecture and has 12 encoder layers. The sampling rate of the input voice signal is set to 16 kHz, which is used to extract the emotional voice features of the user's emotional voice signal. The third-level emotion feature extraction module adopts a temporal convolutional network (TCN) architecture, with a network depth of 5 layers and a convolutional kernel size of 3, which is used to extract the emotional temporal features of the user's emotional physiological indicators. Each module is orderly connected through a data interface to form a complete three-level emotion feature extraction architecture, laying a foundation for the feature extraction of subsequent multi-dimensional emotion data.

[0035] Step S22: Based on the first-level emotion feature extraction module in the three-level emotion feature extraction architecture, perform image modal feature extraction on the corresponding user emotional expression images in the user's multi-dimensional time-series synchronous emotion data to obtain the user's emotional psychological features corresponding to the first modality, including the degree of eyebrow wrinkling and the deviation angle of the mouth corner drooping corresponding to the user's image facial movements; In an embodiment of the present invention, taking the user's multi-dimensional time-series synchronous emotion data collected during psychological counseling as an example, processing the user's emotional expression images therein. Suppose a sequence of 300 frames with a resolution of 4K (3840×2160 pixels) is selected and input into the EfficientNet-B4 network of the first-level emotion feature extraction module. The network first performs feature extraction on the image through the initial convolutional layer, with a convolutional kernel size of 3×3 and a stride of 2, to obtain a preliminary feature map. Then, through multiple compound scaling blocks, each block contains structures such as depthwise separable convolution and squeeze-and-excitation modules, gradually extracting the features of facial movements in the image. By analyzing the features output by the network, the user's eyebrow action units and mouth corner action units are extracted. For the eyebrow action units, calculate the action duration: Suppose the starting frame is the 50th frame, the ending frame is the 150th frame, and the frame rate is 30 frames per second, then the action duration seconds. Determine the action amplitude by calculating the coordinate change of the eyebrow key points. Suppose the initial distance between two key points is 8 pixels, and the distance after the action is 12 pixels, then the action amplitude =12 - 8 = 4 pixels. Use the formula (Suppose = 3 times) to calculate the contraction frequency of the eyebrow muscles = 0.9 times per second, and then by analyzing the changes in the number, depth, and direction of wrinkles, using the formula (assuming = 2, average depth = 0.4 mm, direction change = 10°) to calculate the attenuation index of glabellar contraction contribution =(2 × 0.4 × 10) / = 2.4, and then according to the formula to obtain the glabellar contraction degree = 2.4 × 0.9 × 4 × = 28.8. For the mouth corner action unit, by calculating the relationship between the mouth corner downward amplitude and the mouth corner length, using the formula (assuming the downward amplitude deviation = 2 pixels, mouth corner length = 25 pixels) to obtain the mouth corner downward deviation angle ≈ 4.6°, and finally obtain the user's emotional and psychological characteristics corresponding to the first-level mode including the glabellar contraction degree of 28.8 and the mouth corner downward deviation angle of 4.6°.

[0036] Step S23: Based on the secondary emotion feature extraction module within the emotion three-level feature extraction architecture, perform speech modality feature extraction on the user's emotional speech signal corresponding to the user's multi-dimensional time-series synchronized emotion data, so as to extract prosodic features and semantic features through the pre-trained Wav2Vec 2.0 model within this module, where the prosodic features include fundamental frequency F0 and speech rate, and the semantic features include the frequency of windward emotional words, so as to obtain the user's emotional and psychological characteristics corresponding to the secondary mode; In an embodiment of the present invention, taking a user's emotional speech signal with a duration of 5 minutes, a sampling rate of 16 kHz, and 16-bit quantization as an example, it is input into the Wav2Vec 2.0 model of the secondary emotion feature extraction module. The model first converts the speech signal into a logarithmic mel spectrogram. The spectrogram contains 80 mel filter banks. The Transformer encoder is used to extract features from the spectrogram. Each encoder layer contains a multi-head attention mechanism and a feed-forward neural network. When extracting prosodic features, for the fundamental frequency F0, the autocorrelation method is used for calculation. In the 5-minute speech signal, the average fundamental frequency F0 = 120 Hz; the speech rate is calculated by statistically analyzing the ratio of the effective speech duration to the total duration in the speech signal. Assuming the effective speech duration is 240 seconds and the total duration is 300 seconds, then the speech rate v = 240 / 300 = 0.8. When extracting semantic features, a pre-trained word vector model is used to convert the speech into text, and then the frequency of windward emotional words is statistically analyzed. Assuming that the text contains 15 negative emotional words such as "anxiety" and "tension", and the total number of words is 100, then the frequency of windward emotional words p = 15 / 100 = 0.15. Finally, the user's emotional psychological features corresponding to the secondary modality including the fundamental frequency of 120 Hz, the speech rate of 0.8, and the frequency of windward emotional words of 0.15 are obtained.

[0037] Step S24: Based on the tertiary emotion feature extraction module within the emotion tertiary feature extraction architecture, physiological modality feature extraction is performed on the user's emotional physiological indicators corresponding to the user's multi-dimensional time-series synchronous emotion data to obtain the user's emotional psychological features corresponding to the tertiary modality, including the LF / HF ratio of HRV and the rising slope of the GSR index. In an embodiment of the present invention, taking the heart rate variability index HRV and the galvanic skin response index GSR data collected from the user within 10 minutes as an example, the sampling frequency of HRV data is 100 Hz, and the sampling frequency of GSR data is 50 Hz. These data are input into the TCN architecture of the tertiary emotion feature extraction module. For HRV data, first perform band-pass filtering (passband frequency is 0.04 - 0.4 Hz), and then calculate the power of the low-frequency (LF, 0.04 - 0.15 Hz) and high-frequency (HF, 0.15 - 0.4 Hz) components. Using the formula (assuming = 12, = 8) to calculate and obtain the ratio is 12 / 8 = 1.5. For GSR data, the least squares method is used to perform linear fitting on the data. Assuming that in a certain rising interval, the starting time = 100 s, the GSR value = 2 μS; the ending time = 120 s, the GSR value = 5 μS, then the rising slope of the index is calculated through the formula ​ =(5 - 2) / (120 - 100)=0.15 μS / second, and finally obtain the user's emotional and psychological characteristics corresponding to the three - level mode with a ratio of 1.5 and a rising slope of 0.15 μS / second of the GSR index.

[0038] Step S25: Design an inter - modal attention matrix to cross - modal fuse the user's emotional and psychological characteristics corresponding to the first, second, and third levels to obtain the user's cross - modal fusion psychological characteristics.

[0039] In the embodiment of the present invention, by designing a 3×3 inter - modal attention matrix , which is used to cross - modal fuse the user's emotional and psychological characteristics corresponding to the first, second, and third levels. Assume the first - level modal feature vector , the second - level modal feature vector , the third - level modal feature vector . First, calculate the attention weights between each mode. For the attention weight of the first - level mode to the second - level mode , use the formula (where is a trainable weight matrix. Assume is the identity matrix). Similarly, calculate other weights to obtain the complete attention matrix . Then, use the attention matrix to perform weighted fusion on the modal features. The user's cross - modal fusion psychological feature vector . After calculation, the fused feature vector is obtained, realizing the effective fusion of multi - modal emotion features and providing comprehensive feature data for subsequent psychological risk warning.

[0040] Further, step S22 includes the following steps: Input the user's emotional expression image corresponding to the user's multi - dimensional time - series synchronous emotion data into the first - level emotion feature extraction module in the emotion three - level feature extraction architecture to use the EfficientNet - B4 network to extract the facial action unit time periods corresponding to the user's image sequence, including the user's eyebrow - movement action unit time period and the user's mouth - corner action unit time period; In an embodiment of the present invention, in the environment of a psychological counseling room, based on the facial expression images in the obtained multi-dimensional time-series synchronized emotion data of the user, in-depth analysis is carried out. Assuming that the image sequence of the user in the 3rd - 5th minute during the consultation process is selected as the research object, this sequence contains a total of 3600 images with a 4K resolution (3840×2160 pixels). These images are input into the first-level emotion feature extraction module within the three-level emotion feature extraction architecture, and the EfficientNet-B4 network is used to extract the time periods of facial action units. The EfficientNet-B4 network has efficient feature extraction capabilities, and its network structure includes multiple convolutional layers, pooling layers, and fully connected layers. During the training process, a dataset containing 10,000 different facial expression images is used to train the network so that it can accurately identify facial action units. The user's emotion facial expression images are input into the network one by one. The network extracts local features in the images through convolutional operations, then reduces the features through pooling operations, and finally outputs the time periods of facial action units through the fully connected layer. In this process, the network can accurately identify the time periods of the user's glabella action units and the time periods of the user's mouth corner action units, providing important basic data for subsequent emotion analysis.

[0041] Preferably, the action duration and action amplitude corresponding to the glabella action unit are obtained through the time period of the user's glabella action unit, and the muscle contraction tension corresponding to the glabella region within the time period of the user's glabella action unit is quantified for the glabella contraction frequency based on the action duration and action amplitude corresponding to the glabella action unit, so as to obtain the glabella muscle contraction frequency corresponding to the glabella action unit; In an embodiment of the present invention, based on the previously extracted time period of the user's glabella action unit, the key parameters corresponding to the glabella action unit are further obtained. Assuming that within this time period, through the analysis of the image sequence, it is determined that the start time of the glabella action is seconds, and the end time is seconds, then the action duration seconds. At the same time, through the analysis of the change in the key point coordinates of the glabella region, the action amplitude is calculated. For example, by comparing the distance change between two specific key points in the glabella region before and after the action, assuming the initial distance is = 10 pixels, and the distance after the action is = 15 pixels, then the action amplitude = 5 pixels. Based on the action duration and action amplitude, the muscle contraction tension corresponding to the glabella region within the time period of the user's glabella action unit is quantified for the glabella contraction frequency, and the formula is used, where is the number of muscle contractions within the action duration . Assuming that the muscle contracts 5 times within 10 seconds, then the glabella muscle contraction frequency = 5 / 10 = 0.5 times per second, and finally the contraction frequency of the glabellar muscles corresponding to the glabellar action unit is obtained.

[0042] Preferably, the number of wrinkles, the depth of wrinkles, and the change in wrinkle direction corresponding to the glabellar action unit are obtained through the user's glabellar action unit time period, and an evaluation of the attenuation of the contraction contribution of the muscles in the glabellar area during the action is performed based on the number of wrinkles, the depth of wrinkles, and the change in wrinkle direction corresponding to the glabellar action unit, so as to obtain the glabellar contraction contribution attenuation index corresponding to the glabellar action unit; In the embodiment of the present invention, by continuing to analyze the user's glabellar action unit time period, the wrinkle-related parameters corresponding to the glabellar action unit are obtained. Through image analysis technology, the number of wrinkles within the glabellar action unit time period is identified pieces, and using an image processing algorithm, such as a method based on gray-scale change, the depth of each wrinkle is measured , assuming that the depths of three measured wrinkles are respectively = 0.5 mm, [[ID=1'st]] = 0.6 mm, = 0.4 mm, then the average wrinkle depth mm. By tracking the position and direction changes of the wrinkles in different frame images, the change in wrinkle direction is determined , based on these parameters, an evaluation of the attenuation of the contraction contribution of the muscles in the glabellar area during the action is performed for the user's glabellar action unit time period, and using the formula , the glabellar contraction contribution attenuation index corresponding to the glabellar action unit is obtained = 2.25.

[0043] Preferably, based on the glabellar contraction contribution attenuation index corresponding to the glabellar action unit and in combination with the glabellar muscle contraction frequency, action duration, and action amplitude corresponding to the glabellar action unit, a dynamic estimation of the glabellar contraction degree is performed for the user's glabellar action unit time period, so as to obtain the glabellar contraction degree corresponding to the facial action of the user's image in this time period; In the embodiment of the present invention, through the previously obtained glabellar contraction contribution attenuation index, in combination with the glabellar muscle contraction frequency, action duration, and action amplitude, a dynamic estimation of the glabellar contraction degree is performed for the user's glabellar action unit time period, so as to adopt the formula , substituting = 2.25, = 0.5, = 5, = 10 into the formula, and finally the glabellar contraction degree corresponding to the facial action of the user's image in this time period is obtained = 56.25.

[0044] Preferably, perform a statistical analysis on the downward deviation amplitude of the user's mouth corner movement unit during a period to obtain the downward deviation amplitude angle of the mouth corner corresponding to the facial movement in the user image during this period; In an embodiment of the present invention, by analyzing the user's mouth corner movement unit period in detail to obtain the downward deviation amplitude angle of the mouth corner. Assume that during this period, by tracking and analyzing the mouth corner movement, the starting position and ending position of the mouth corner downward movement are determined, and by calculating the change in the Euclidean distance between the two endpoints of the mouth corner, the amplitude deviation of the mouth corner downward movement is determined = 0.3 pixels. At the same time, the length of the user's mouth corner is known 30 pixels. According to the trigonometric function relationship, calculate the downward deviation amplitude angle of the mouth corner , and substitute = 3 pixels and = 30 pixels into the formula to obtain ≈ 5.73°.

[0045] Preferably, combine the frown degree between the eyebrows and the downward deviation amplitude angle of the mouth corner corresponding to the facial movement of the user image to form a vector as the user's emotional and psychological feature corresponding to this first-level emotion feature extraction module, so as to obtain the user's emotional and psychological feature corresponding to the first-level modality.

[0046] In an embodiment of the present invention, by combining the frown degree between the eyebrows and the downward deviation amplitude angle of the mouth corner corresponding to the facial movement of the user image to form a vector as the user's emotional and psychological feature corresponding to this first-level emotion feature extraction module. Assume the frown degree between the eyebrows , and the downward deviation amplitude angle of the mouth corner is , then the formed vector is [56.25, 5.73], and this vector is the user's emotional and psychological feature corresponding to the first-level modality.

[0047] Furthermore, the statistical analysis on the downward deviation amplitude of the user's mouth corner movement unit period includes the following steps: Analyze the movement trajectory of the user's mouth corner during the mouth corner movement unit period to generate the corresponding movement trajectory of the user's mouth corner during this period; In an embodiment of the present invention, in a psychological counseling room scenario, based on the facial expression images in the user's multi-dimensional time-series synchronous emotional data previously acquired, the user's mouth corner action unit period is analyzed. It is assumed that the image sequence of the user in the 3rd to 5th minute of the consultation process is selected as the research period. This period contains a total of 3,600 images with 4K resolution (3840×2160 pixels). The mouth corner key points are located using a method based on the active shape model (ASM). The ASM model is first trained using a data set containing 10,000 images with different facial expressions. Each mouth corner contour in the model is described by 68 key points. Each image in the period is input into the model. Through iterative search, the shape of the model is matched with the actual shape of the mouth corner in the image, thereby accurately locating the 68 key point coordinates of the mouth corner in each image. Based on the located key point coordinates, a three-dimensional space interpolation algorithm (such as cubic spline interpolation) is used to connect the key points of adjacent images, and the coordinates of the left end point of the mouth corner in the two adjacent images are used. and For example, the coordinates of the intermediate time points are calculated through the cubic spline interpolation formula to construct a continuous motion trajectory of the corners of the mouth. Finally, the corresponding motion trajectory of the corners of the mouth of the user in this period is generated, which fully presents the movement changes of the corners of the mouth of the user in this period and provides basic data for subsequent analysis.

[0048] Preferably, the drooping time sequence fluctuation corresponding to the user's mouth corner movement is obtained through the corresponding motion trajectory of the user's mouth corner movement in the time period, and the dynamic change of the user's mouth corner movement within the user's mouth corner movement unit time period is fitted based on the drooping time sequence fluctuation corresponding to the user's mouth corner movement, so as to generate a fitting function relationship between time and the change in the drooping position of the user's mouth corner in the time period; In the embodiment of the present invention, the coordinate change of the drooping direction of the mouth corner in the trajectory is used as the research object through the previously generated user's mouth corner movement trajectory to obtain the drooping time sequence fluctuation corresponding to the user's mouth corner movement. Assuming that in this period, the coordinate sequence of the drooping direction of the mouth corner is , the time series is , time interval Minutes, the least squares method is used to fit the dynamic changes of the user's mouth corner movement within the unit period of the mouth corner movement. The fitting function is set as , the goal is to make Reach the minimum and solve the coefficients through matrix operations , ,…, , assuming that the fitting function can be obtained This function accurately describes the relationship between time and the change in the position of the user's mouth corner droop during this period, providing a quantitative basis for the subsequent analysis of the amplitude of the mouth corner droop.

[0049] Preferably, according to the fitting function relationship between time and the change in the position of the user's downturned mouth corner during this period, the deviation of the downturn amplitude of the mouth corner is measured to obtain the corresponding fitting amplitude deviation of the user's downturned mouth corner during this period; In the embodiment of the present invention, by measuring the deviation of the downturn amplitude of the mouth corner according to the previously obtained fitting function relationship, for each time point within this period , calculate the actual downturn position of the mouth corner and the predicted position of the fitting function , and the difference is the deviation . Calculate the corresponding fitting amplitude deviation of the user's downturned mouth corner during this period, and use the root mean square error (RMSE) formula: . Assume that after calculation, = 3 pixels. This value reflects the deviation degree between the fitting function and the actual downturn position of the mouth corner. The smaller the value, the better the fitting effect. Through this deviation measurement, the stability and regularity of the user's downturned mouth corner movement can be effectively evaluated, providing an important reference for psychological risk warning.

[0050] Preferably, obtain the corresponding mouth corner length of the user, and based on the corresponding mouth corner length of the user, conduct a statistical analysis of the downturn amplitude deviation of the corresponding fitting amplitude deviation of the user's downturned mouth corner during this period to obtain the downturn amplitude angle corresponding to the facial movement of the user's image during this period.

[0051] In the embodiment of the present invention, by obtaining the corresponding mouth corner length of the user, and through the previously located key points of the mouth corner, calculate the Euclidean distance between the two endpoints of the mouth corner . Assume that the calculated mouth corner length of a certain user = 30 pixels. Based on the corresponding mouth corner length of the user, conduct a statistical analysis of the downturn amplitude deviation of the corresponding fitting amplitude deviation of the user's downturned mouth corner during this period, and according to the trigonometric function relationship, calculate the downturn amplitude angle . Substitute = 3 pixels and = 30 pixels into the formula to obtain ≈5.73°. This angle accurately quantifies the downturn amplitude degree corresponding to the facial movement of the user in the image during this period. Combining with other emotion data, the user's emotional state can be judged more accurately, providing a key quantitative index for psychological risk warning based on the user's multi-dimensional emotion data.

[0052] Further, step S24 includes the following steps: Step S241: By inputting the user's emotional physiological indicators corresponding to the user's multi-dimensional time-series synchronized emotional data, including the heart rate variability index HRV and the galvanic skin response index GSR, into the three-level emotional feature extraction module within the three-level emotional feature extraction architecture, the corresponding LF value and HF value on the HRV and the index rising slope section corresponding to the GSR are extracted. In the embodiment of the present invention, in a certain psychological counseling scenario, physiological indicators in the user's 15-minute continuous multi-dimensional time-series synchronized emotional data are obtained. The heart rate variability index HRV is recorded at a sampling frequency of 100 Hz, and the galvanic skin response index GSR is recorded at a sampling frequency of 50 Hz. These data are input into the three-level emotional feature extraction module within the three-level emotional feature extraction architecture. This module is constructed based on a 5-layer deep temporal convolutional network (TCN) architecture with a convolutional kernel size of 3. For the HRV data, the module first passes it through a band-pass filter with a passband frequency set to 0.04 - 0.4 Hz to remove noise interference. Then, using the power spectral density estimation method, the Welch method, with a Hann window function selected and a window length set to 256 sampling points and an overlap rate of 50%, the power values of the HRV data in the low-frequency band (LF, 0.04 - 0.15 Hz) and the high-frequency band (HF, 0.15 - 0.4 Hz) are calculated, thereby extracting the corresponding LF value and HF value on the HRV. For the GSR data, the module uses the sliding window algorithm with a window size set to 100 sampling points and a step size of 10 sampling points to traverse the entire GSR data sequence. When it detects that the data shows continuous increase and the increase amplitude exceeds 0.5 μS, this section of data is marked as the index rising slope section, thereby extracting the index rising slope section corresponding to the GSR. For example, in the 15-minute GSR data, a rising slope section from the 300th second to the 350th second is successfully identified.

[0053] Step S242: Calculate the ratio based on the corresponding LF value and HF value on the HRV to obtain the LF / HF ratio of the HRV. In the embodiment of the present invention, by calculating the ratio based on the previously extracted corresponding LF value and HF value on the HRV, assuming that the LF value calculated by the power spectral density estimation is 18 and the HF value is 12, substituting LF = 18 and HF = 12 into the formula LF / HF according to the formula LF / HF, we can get LF / HF = 1.5. This ratio can reflect the balance state of the sympathetic and parasympathetic nerves of the autonomic nervous system and has important reference value in psychological risk warning. The change in the value can assist in judging the user's emotional stress level and the stability of the psychological state.

[0054] Step S243: Obtain the corresponding GSR index rising change amplitude and GSR index rising time deviation through the index rising slope section corresponding to the GSR. In an embodiment of the present invention, by taking the rising slope section interval corresponding to the previously extracted GSR as an example, the rising slope section interval from the 300th second to the 350th second is analyzed. The sampling frequency of the GSR data in this interval is 50 Hz, and a total of (350 - 300) × 50 = 2500 data points are included. The corresponding rising change amplitude of the GSR index is obtained by calculating the difference between the maximum value and the minimum value of the GSR data in this interval. Assuming that in this interval, the minimum value of the GSR data is 3 μS and the maximum value is 8 μS, then the rising change amplitude of the GSR index is 8 - 3 = 5 μS. Calculate the rising time deviation of the GSR index. First, determine the difference between the theoretically linearly rising time and the actual rising time. Assuming that according to the linear model prediction, it theoretically takes 48 seconds to rise from the minimum value to the maximum value, but the actual rising time is 50 seconds. Then the rising time deviation of the GSR index is 50 - 48 = 2 seconds. These parameters can quantify the characteristics of the GSR index rising process and provide data support for subsequent analysis.

[0055] Step S244: Based on the rising change amplitude of the GSR index and the rising time deviation of the GSR index, perform a rising slope fitting calculation on the rising slope section interval corresponding to the GSR to obtain the rising slope of the GSR index.

[0056] In an embodiment of the present invention, by based on the previously obtained rising change amplitude of the GSR index and the rising time deviation of the GSR index, perform a rising slope fitting calculation on the rising slope section interval corresponding to the GSR. Taking the rising slope section interval from the 300th second to the 350th second as an example, it is known that the rising change amplitude of the GSR index is 5 μS, the rising time deviation is 2 seconds, and the actual rising time is 50 seconds. The least squares method is used for linear fitting. Let the fitting straight line equation be , where is the GSR value, is the time, is the slope, is the intercept. Substitute the GSR data points ([[]] ) in this interval into the formula, and by minimizing is the number of data points in this interval, here = 2500) to solve and , after calculation, the slope = 5 / 50 = 0.1 μS / second, that is, the rising slope of the GSR index is 0.1 μS / second. This slope can reflect the change rate of the GSR during the rising process. Combining other physiological index characteristics such as the LF / HF ratio of HRV, it provides an accurate physiological modal feature basis for psychological risk warning based on the user's multi-dimensional emotion data.

[0057] Further, step S25 includes the following steps: Step S251: Design the inter-modal attention matrices corresponding to each level between the first-level, second-level, and third-level emotion feature extraction modules within the three-level emotion feature extraction architecture, and obtain the feature query vectors and key vectors between different modalities based on the inter-modal attention matrices; In the embodiment of the present invention, within the three-level emotion feature extraction architecture, the first-level feature extraction module has extracted the emotion visual features including the degree of eyebrow frown and the deviation angle of the drooping mouth corner from the user's emotion expression image, the second-level module has extracted the emotion speech features such as the fundamental frequency, speech rate, and frequency of emotional words in the wind direction from the emotion speech signal, and the third-level module has extracted the emotion time-series features such as the LF / HF ratio of HRV and the rising slope of the GSR index from the emotion physiological indicators. On this basis, inter-modal attention matrices are respectively designed between the three modules. Taking the matrix between the first-level and second-level modules as an example, this matrix is a two-dimensional structure, with rows corresponding to the first-level modal feature dimensions and columns corresponding to the second-level modal feature dimensions. Assuming that the first-level modal feature dimension is 2 (the degree of eyebrow frown and the deviation angle of the drooping mouth corner), and the second-level modal feature dimension is 3 (the fundamental frequency, speech rate, frequency of emotional words in the wind direction), then a 2×3 matrix is constructed. Each element in the matrix is obtained through training with a large amount of labeled multi-modal emotion data. During the training process, the model is allowed to learn the correlation degree of different modal features in reflecting the emotion state, so that the matrix can reflect the attention relationship between modalities. Based on this matrix, the feature vector of the first-level modality is used as the query vector, and the feature vector of the second-level modality is used as the key vector. Specifically, each element of the first-level modal feature vector is multiplied by the corresponding row elements of the matrix and accumulated to obtain a new query vector; each element of the second-level modal feature vector is multiplied by the corresponding column elements of the matrix and accumulated to obtain a new key vector. In this way, the feature query vectors and key vectors between the first-level and second-level modalities are obtained. Similarly, the corresponding vectors between other modal combinations are obtained.

[0058] Step S252: Calculate the corresponding feature vector distance values according to the feature query vectors and key vectors between different modalities; In an embodiment of the present invention, taking the feature query vector and the key vector between the first-level and second-level modalities obtained as an example, the calculation idea of the Euclidean distance is adopted to measure the difference between them. The query vector and the key vector are regarded as two points in space, and each vector element corresponds to the coordinates on the spatial dimension. Suppose the feature query vectors of the first-level and second-level modalities are [1.2, 0.8], and the key vector is [3.5, 2.1, 0.9]. First, calculate the squares of the differences of the corresponding elements of the two vectors, that is, (1.2 - 3.5)² and (0.8 - 2.1)². Since the dimension of the query vector is 2 and the dimension of the key vector is 3, for the missing dimension, after padding the query vector with 0, calculate (0 - 0.9)². Then add these squared values and take the square root of the sum. The obtained value is the distance value between these two vectors. According to the same method, calculate the distance values between the feature query vectors and the key vectors of the first-level and third-level modalities, and the second-level and third-level modalities respectively. These distance values reflect the degree of difference between the feature vectors of different modalities. The smaller the distance value, the more similar the features of the two modalities are in the current dimension; the larger the distance value, the greater the difference, providing a quantitative basis for subsequent cross-modal fusion.

[0059] Step S253: Based on the feature vector distance value, cross-modally weighted fuse the user's emotional and psychological characteristics corresponding to the first, second, and third-level modalities to obtain the user's cross-modal fused psychological characteristics.

[0060] In an embodiment of the present invention, by based on the previously calculated feature vector distance values, cross-modally weighted fuse the user's emotional and psychological characteristics corresponding to the first, second, and third-level modalities. For the pair of modalities with a small distance value, a higher weight is given during fusion, which means they have stronger relevance and complementarity in reflecting the user's emotional and psychological characteristics; for the pair of modalities with a large distance value, a lower weight is given. For example, it is known that the user's emotional and psychological characteristic vector corresponding to the first-level modality is [28.8, 4.6], the vector corresponding to the second-level modality is [120, 0.8, 0.15], and the vector corresponding to the third-level modality is [1.5, 0.15]. Suppose the distance value between the first-level and second-level modality feature vectors is 2.79, the distance value between the first-level and third-level modalities is 1.36, and the distance value between the second-level and third-level modalities is 1.16. According to the magnitude of the distance values, it is determined that the weights of the first-level and third-level modalities are relatively high, the second-level and third-level modalities are the second, and the weights of the first-level and second-level modalities are relatively low. Weighted sum the feature vectors of each modality according to the corresponding weights. The specific operation is to multiply each element of each modality feature vector by the corresponding weight, and then add the elements at the corresponding positions of the three modalities to obtain new elements. The vector composed of these new elements is the user's cross-modal fused psychological characteristic. Through this weighted fusion method, the emotional information of multiple modalities is integrated, and a feature vector that more comprehensively and accurately reflects the user's emotional and psychological state is obtained, providing strong data support for subsequent psychological risk early warning.

[0061] Further, step S3 includes the following steps: Step S31: Obtain user environmental data and user social network data; In the embodiment of the present invention, in a certain online mental health monitoring platform, data is obtained through fixedly deployed sensors and social network interfaces. The user environmental data is collected by sensors installed in the user's activity area. For example, a temperature and humidity sensor collects environmental temperature and humidity data at a frequency of once per minute, a light sensor records the light intensity value every 30 seconds, a noise sensor monitors the environmental sound decibel value in real time and stores the data in seconds. At the same time, by docking with the official data interface of the social network platform, user social network data is pulled at a frequency of once per hour, including text dynamics, comments, like records, friend interaction information, etc. published by the user. For example, within a day, an environmental data set containing 24 groups of temperature and humidity data, 48 groups of light intensity data, and 86,400 groups of noise decibel data, as well as a social network data set of 15 dynamics, 50 comments, 80 likes, and interaction records with 20 friends published by the user are obtained, providing basic data for subsequent analysis.

[0062] Step S32: Extract environmental factors from the user environmental data to obtain each user environmental factor; In the embodiment of the present invention, by processing the obtained user environmental data, taking the environmental temperature data as an example, the 24 groups of temperature data within a day are sorted, and their average value (assuming = 22°C, = 23°C... = 25°C, after calculation °C), standard deviation , to obtain the temperature fluctuation situation. If the standard deviation is greater than 2°C, it is marked as an environmental factor with large temperature fluctuations. For the light intensity data, the 48 groups of data are divided into two time periods: daytime (6:00 - 18:00) and night (18:00 - 6:00), and the average values are calculated respectively. If the average light intensity during the day is lower than 300 lux, it is marked as insufficient light; if the light intensity at night is higher than 50 lux, it is marked as light pollution. In this way, specific user environmental factors such as "large temperature fluctuations", "insufficient light", and "noise decibel exceeding 60 (assuming the environmental quiet threshold is 60 decibels)" are extracted from the environmental data.

[0063] Step S33: Analyze the user behavior patterns of the user social network data to obtain the corresponding user emotional and psychological behavior patterns of the user on the social network; In the embodiments of the present invention, by analyzing the user's social network data. Taking the text dynamics published by the user as an example, using the bag-of-words model combined with the sentiment analysis algorithm, the 15 dynamic texts are segmented, the occurrence frequency of each word is counted. At the same time, using the pre-trained sentiment analysis model, the sentiment polarity of each dynamic is judged, which is divided into three categories: positive, neutral, and negative. Assuming that 5 of them are negative emotions, calculate the proportion of negative emotion dynamics ≈0.33. Analyze the user's comment and like behaviors, count the number of comments and likes of the user on different types of content. If the proportion of the number of comments on negative topic content in the total number of comments exceeds 40%, and the proportion of likes for negative content exceeds 30%, it is determined that the user has a behavioral tendency to pay attention to negative information. By analyzing the friend interaction information, calculate the interaction frequency between the user and different friends. If the interaction frequency with several friends with negative emotion tendencies accounts for more than 50% of the total interaction frequency, it is marked as a behavioral pattern affected by negative friends. Combining these analyses, user emotional and psychological behavioral patterns corresponding to the user on the social network, such as "frequently posting negative dynamics", "paying attention to negative information", "being affected by negative friends", etc., are obtained

[0064] Step S34: Based on each user environmental factor and the corresponding user emotional and psychological behavioral pattern of the user on the social network, perform emotional and psychological dynamic graph reasoning between the emotional and psychological characteristic states corresponding to the user's cross-modal fusion psychological characteristics within the same time, so as to generate an emotional and psychological causal dynamic relationship graph between the corresponding environmental factors, behavioral patterns and psychological states of the user

[0065] In the embodiments of the present invention, by reasoning based on the previously extracted user environmental factors, the obtained user emotional and psychological behavioral patterns, and the previously obtained user cross-modal fusion psychological characteristics, taking the environmental factor of "large temperature fluctuations", the behavioral pattern of "frequently posting negative dynamics", and the cross-modal fusion psychological characteristic vector [126.6, 5.32] of the user at a certain moment as an example, use the Bayesian network reasoning method to construct an emotional and psychological dynamic graph, taking the environmental factor, behavioral pattern and psychological characteristic state as nodes, and determine the weight of the edge by analyzing the association probability between these nodes in the historical data. Assuming that in the historical data, when "large temperature fluctuations" occur, the probability that the user generates a negative psychological state is =0.6; when the user "frequently posts negative dynamics", the probability of generating a negative psychological state is =0.7. Through Bayes' formula (where , , Calculate the probability that the user is in a negative psychological state under the combined action of environmental factors and behavioral patterns (which can be obtained by statistical analysis of historical data). According to the calculation results, in the emotional psychological causal dynamic relationship diagram, use arrows to connect the nodes of "large temperature fluctuations" and "frequently posting negative dynamics" with the node representing the negative psychological state, and mark the corresponding probability weights. Finally, generate a complete emotional psychological causal dynamic relationship diagram that shows the causal relationship between the corresponding environmental factors, behavioral patterns, and psychological states of the user, providing an intuitive and logically related analysis basis for psychological risk warning.

[0066] Further, step S4 includes the following steps: Step S41: Obtain the emotional and psychological fluctuation influence path of the user under the corresponding environmental factors and behavioral patterns through the emotional psychological causal dynamic relationship diagram between the user's corresponding environmental factors, behavioral patterns, and psychological states. In the embodiment of the present invention, based on the previously generated emotional psychological causal dynamic relationship diagram, which includes environmental factor and behavioral pattern nodes such as "large temperature fluctuations", "frequently posting negative dynamics", and "influenced by negative friends", as well as nodes representing different psychological states, and the nodes are connected by arrows marked with probability weights. Select the starting nodes related to the user's current state from the diagram, such as "large temperature fluctuations" and "frequently posting negative dynamics", and use the depth-first search algorithm. Starting from the starting nodes, along the direction indicated by the arrows, explore possible paths in the order of decreasing probability weights. If there are two paths pointing to the psychological state node starting from the "large temperature fluctuations" node, with probabilities of 0.6 and 0.4 respectively, preferentially select the path with a probability of 0.6 to continue the exploration. During the exploration process, record all the nodes and edges passed through to form a complete path from environmental factors and behavioral patterns to changes in psychological states. In this way, obtain multiple emotional and psychological fluctuation influence paths of the user under the corresponding environmental factors and behavioral patterns, such as "large temperature fluctuations → the user generates anxiety emotions → frequently posting negative dynamics exacerbates anxiety → psychological state deteriorates", clearly presenting the correlation context between the environment, behavior, and emotional changes.

[0067] Step S42: Quantify the emotional fluctuation level at each time point within the emotional and psychological fluctuation influence path of the user under the corresponding environmental factors and behavioral patterns to obtain the emotional and psychological fluctuation level of the user at each time point on the emotional and psychological fluctuation influence path. In an embodiment of the present invention, for the previously obtained emotional and psychological fluctuation influence path, the time within the path is divided into time points at one-hour intervals. Taking the path of "substantial temperature fluctuation → user generates anxiety → frequent posting of negative dynamics exacerbates anxiety → psychological state deteriorates" as an example, assume that it takes 3 hours from the start of temperature fluctuation to the appearance of anxiety, and another 2 hours from the appearance of anxiety to frequent posting of negative dynamics. To quantify the emotional fluctuation level at each time point, a weighted summation method is adopted. Set the influence weight of environmental factors on emotional fluctuation to 0.4 and the influence weight of behavior patterns to 0.6. At the first time point when the temperature starts to fluctuate, according to the probability weight of 0.6 between the "substantial temperature fluctuation" node and the anxiety node, calculate the emotional fluctuation level at this time point as 0.4 × 0.6 = 0.24; at the first time point after the appearance of anxiety, combine the probability weight of 0.7 between the "frequent posting of negative dynamics" node and the exacerbation of anxiety node, as well as the behavior pattern weight, calculate the emotional fluctuation level as 0.6 × 0.7 = 0.42, and then add the fluctuation level of the previous time point to obtain the total fluctuation level at this time point as 0.24 + 0.42 = 0.66. And so on, calculate the emotional and psychological fluctuation levels at each time point on this path to form a quantization sequence reflecting the change of emotional fluctuation.

[0068] Step S43: Predict the extreme inflection points of emotional evolution based on the emotional and psychological fluctuation levels of the user at each time point on the emotional and psychological fluctuation influence path, so as to obtain the corresponding user emotional evolution risk inflection points of the user on the emotional and psychological fluctuation influence path; In an embodiment of the present invention, based on the previously obtained sequence of emotional and psychological fluctuation levels of the user at each time point on the emotional and psychological fluctuation influence path, a second-order difference algorithm is used to predict the extreme inflection points of emotional evolution. Taking the fluctuation level sequence [0.24, 0.66, 0.9, 1.2, 1.1, 0.9, 0.7] as an example, first calculate the first-order difference to obtain the difference sequence [0.42, 0.24, 0.3, -0.1, -0.2, -0.2] of the fluctuation levels between adjacent time points; then calculate the second-order difference to obtain the difference sequence of differences [-0.18, 0.06, -0.4, -0.1, 0]. When the second-order difference changes from positive to negative, that is, a maximum inflection point appears; when the second-order difference changes from negative to positive, that is, a minimum inflection point appears. In the above sequence, the second-order difference changes from 0.06 to -0.4, indicating that a maximum inflection point of emotional fluctuation, that is, an emotional evolution risk inflection point, appears near the corresponding time point (assumed to be the 4th time point). Through this algorithm, analyze the fluctuation level sequences on all emotional and psychological fluctuation influence paths to determine multiple user emotional evolution risk inflection points corresponding to the user on the emotional and psychological fluctuation influence path, providing key time nodes for risk assessment.

[0069] Step S44: Calculate the corresponding relative risk index based on the user's emotional evolution risk inflection point corresponding to the emotional and psychological fluctuation influence path, and dynamically divide the risk level of the relative risk index according to the preset five-stage risk assessment system. The preset five-stage risk assessment system includes that when the relative risk index ≤ 10, it is normal; when 10 < relative risk index ≤ 30, it is subclinical; when 30 < relative risk index ≤ 40, it is mild risk; when 40 < relative risk index ≤ 60, it is moderate and mild risk; when the relative risk index > 60, it is high risk, so as to divide the corresponding user's emotional and psychological risk level. At the same time, according to the corresponding user's emotional and psychological risk level, carry out a risk warning response to the user's emotional evolution risk inflection point to generate corresponding user's emotional and psychological risk warning response measures, including automatically triggering self-regulation suggestions for risk levels of mild risk and below, while automatically notifying emergency contacts for risk levels above mild risk.

[0070] In the embodiment of the present invention, for the previously determined user's emotional evolution risk inflection point, calculate the corresponding relative risk index. Taking a certain risk inflection point as an example, assuming that the emotional and psychological fluctuation level at this inflection point is 1.2, using the formula relative risk index = emotional fluctuation level × 10 × (1 + adjacent risk inflection point interval time / average interval time) (assuming that the average interval time is 5 time points, and the interval between this risk inflection point and the previous risk inflection point is 3 time points), then the relative risk index is 1.2 × 10 × (1 + 3 / 5) = 1.2 × 10 × 1.6 = 19.2. Divide the relative risk index according to the preset five-stage risk assessment system. Since 10 < 19.2 ≤ 30, the user's emotional and psychological risk level corresponding to this risk inflection point is subclinical. Respond according to the risk level. Because subclinical belongs to the mild risk level and below, the system automatically triggers self-regulation suggestions, such as pushing relaxation meditation audio to the user, providing tips on emotion management, etc. If the calculated relative risk index is 45, because 40 < 45 ≤ 60, the risk level is moderate and mild risk, then automatically notify the user's preset emergency contact to inform them of the user's current emotional risk status, so as to realize the dynamic warning and effective intervention of the user's emotional and psychological risk.

[0071] Furthermore, the present invention also provides a psychological risk warning system based on multi-dimensional user emotion data for executing the psychological risk warning method based on multi-dimensional user emotion data as described above. The psychological risk warning system based on multi-dimensional user emotion data includes: A multi-dimensional emotion time series alignment module, configured to obtain multi-dimensional user emotion data corresponding to the user's emotional expression images, user's emotional voice signals, and user's emotional physiological indicators, and perform multi-source time series alignment processing on the multi-dimensional user emotion data to obtain multi-dimensional time series synchronized emotion data of the user; The emotional psychological feature fusion module is used to construct an emotional three-level feature extraction architecture, and based on the emotional three-level feature extraction architecture, extract modal emotional features from the user's multi-dimensional time-series synchronous emotional data to obtain the user's emotional psychological features corresponding to each level of modality; by designing an inter-modal attention matrix to cross-modally fuse the user's emotional psychological features corresponding to each level of modality, so as to obtain the user's cross-modal fusion psychological features; The emotional psychological dynamic reasoning module is used to obtain the user's environmental data and the user's social network data, and based on the user's environmental data and the user's social network data, perform emotional psychological dynamic graph reasoning on the user's cross-modal fusion psychological features to generate an emotional psychological causal dynamic relationship graph between the user's corresponding environmental factors, behavior patterns and psychological states; The emotional psychological risk warning module is used to obtain the corresponding user emotional evolution risk inflection point through the emotional psychological causal dynamic relationship graph between the user's corresponding environmental factors, behavior patterns and psychological states, and perform psychological risk warning according to the user emotional evolution risk inflection point to generate the corresponding user emotional psychological risk warning response measures.

[0072] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, it is intended to cover all changes falling within the meaning and scope of the equivalent elements of the application documents within the present invention.

[0073] The above are only the specific embodiments of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A psychological risk warning method based on multi-dimensional emotion data of users, characterized in that, Including the following steps: Step S1: Obtain the user's multi-dimensional emotion data corresponding to the user's emotion expression image, user's emotion voice signal, and user's emotion physiological indicators, and perform multi-source time series alignment processing on the user's multi-dimensional emotion data to obtain the user's multi-dimensional time series synchronized emotion data; Step S2: Construct an emotion three-level feature extraction architecture, and perform modal emotion feature extraction on the user's multi-dimensional time series synchronized emotion data based on the emotion three-level feature extraction architecture to obtain the user's emotion psychological features corresponding to each level of modality; By designing an inter-modal attention matrix to cross-modal fuse the user's emotion psychological features corresponding to each level of modality, obtain the user's cross-modal fused psychological features; Step S3: Obtain the user's environmental data and the user's social network data, and perform emotion psychological dynamic graph reasoning on the user's cross-modal fused psychological features based on the user's environmental data and the user's social network data to generate an emotion psychological causal dynamic relationship graph between the user's corresponding environmental factors, behavior patterns, and psychological states; Step S4: Obtain the corresponding user emotion evolution risk inflection point through the emotion psychological causal dynamic relationship graph between the user's corresponding environmental factors, behavior patterns, and psychological states, and perform psychological risk early warning according to the user emotion evolution risk inflection point to generate the corresponding user emotion psychological risk early warning response measure.

2. The psychological risk early warning method based on multi-dimensional user emotion data according to claim 1, wherein Step S1 includes the following steps: Step S11: Obtain the user's emotion expression image; Step S12: Obtain the user's emotion voice signal; Step S13: Obtain the user's emotion physiological indicators, including the heart rate variability index HRV and the galvanic skin response index GSR corresponding to the user's emotion; Step S14: Perform interference elimination preprocessing on the user's multi-dimensional emotion data including the user's emotion expression image, user's emotion voice signal, and user's emotion physiological indicators, so as to uniformly eliminate the environmental interference corresponding to the user's emotion expression image under different lighting conditions by using a generative adversarial network, and separate the corresponding human voice and environmental noise in the user's emotion voice signal based on deep clustering, and at the same time use a TCN temporal convolutional network to predict and compensate the missing data segments in the user's emotion physiological indicators to obtain the user's multi-dimensional interference elimination emotion data; Step S15: Perform multi-source time series alignment processing on the user's multi-dimensional interference elimination emotion data to eliminate the time series sampling frequency difference between different data sources and obtain the user's multi-dimensional time series synchronized emotion data.

3. The psychological risk warning method based on multi-dimensional user emotion data according to claim 2, characterized in that Step S2 includes the following steps: Step S21: Construct an emotion three-level feature extraction architecture, including emotion feature extraction modules corresponding to the first, second, and third levels. The first level is used to extract the emotion visual features corresponding to the user's emotion expression image, the second level is used to extract the emotion voice features corresponding to the user's emotion voice signal, and the third level is used to extract the emotion time series features corresponding to the user's emotion physiological indicators; Step S22: Perform image modal feature extraction on the user's emotion expression image corresponding to the user's multi-dimensional time series synchronized emotion data based on the first-level emotion feature extraction module in the emotion three-level feature extraction architecture to obtain the user's emotion psychological features corresponding to the first level of modality, including the inter-brow wrinkling degree and the angle of deviation of the mouth corner drooping corresponding to the user's facial movements in the image; Step S23: Based on the secondary emotion feature extraction module within the emotion three-level feature extraction architecture, perform speech modality feature extraction on the corresponding user emotion speech signal in the user multi-dimensional time-series synchronous emotion data, so as to extract prosodic features and semantic features through the pre-trained Wav2Vec 2.0 model within this module. The prosodic features include fundamental frequency F0 and speech rate, and the semantic features include windward sentiment word frequency, so as to obtain the user emotion psychological features corresponding to the secondary modality; Step S24: Based on the tertiary emotion feature extraction module within the emotion three-level feature extraction architecture, perform physiological modality feature extraction on the corresponding user emotion physiological indicators in the user multi-dimensional time-series synchronous emotion data, so as to obtain the user emotion psychological features corresponding to the tertiary modality, including the LF / HF ratio of HRV and the rising slope of the GSR indicator; Step S25: Design an inter-modal attention matrix to cross-modally fuse the user emotion psychological features corresponding to the first, second, and third modalities to obtain the user cross-modal fusion psychological features.

4. The psychological risk early warning method based on multi-dimensional user emotion data according to claim 3, wherein Step S22 includes the following steps: Input the corresponding user emotion expression image in the user multi-dimensional time-series synchronous emotion data into the primary emotion feature extraction module within the emotion three-level feature extraction architecture to use the EfficientNet-B4 network to extract the facial action unit time periods corresponding to the user image sequence, including the user's eyebrow movement unit time period and the user's mouth corner movement unit time period; Obtain the action duration and action amplitude corresponding to the eyebrow movement unit through the user's eyebrow movement unit time period, and quantify the muscle contraction tension corresponding to the eyebrow area within the user's eyebrow movement unit time period based on the action duration and action amplitude corresponding to the eyebrow movement unit to obtain the eyebrow muscle contraction frequency corresponding to the eyebrow movement unit; Obtain the number of wrinkles, wrinkle depth, and wrinkle direction change corresponding to the eyebrow movement unit through the user's eyebrow movement unit time period, and evaluate the attenuation of the wrinkling contribution of the muscles in the eyebrow area during the movement within the user's eyebrow movement unit time period based on the number of wrinkles, wrinkle depth, and wrinkle direction change corresponding to the eyebrow movement unit, so as to obtain the eyebrow wrinkling contribution attenuation index corresponding to the eyebrow movement unit; Based on the eyebrow wrinkling contribution attenuation index corresponding to the eyebrow movement unit and in combination with the eyebrow muscle contraction frequency, action duration, and action amplitude corresponding to the eyebrow movement unit, dynamically estimate the eyebrow wrinkling degree of the user's eyebrow movement unit time period to obtain the eyebrow wrinkling degree corresponding to the facial action of the user image at this time period; Perform a statistical analysis of the downward deviation amplitude of the action on the user's mouth corner movement unit time period to obtain the downward deviation amplitude angle of the mouth corner corresponding to the facial action of the user image at this time period; Combine the eyebrow wrinkling degree and the downward deviation amplitude angle of the mouth corner corresponding to the facial action of the user image into a vector as the user emotion psychological feature corresponding to the primary emotion feature extraction module, so as to obtain the user emotion psychological feature corresponding to the primary modality.

5. The psychological risk warning method based on multi-dimensional user emotion data according to claim 4, characterized in that, The statistical analysis of the downward deviation amplitude of the action on the user's mouth corner movement unit time period includes the following steps: Analyze the movement trajectory of the user's mouth corner during the user's mouth corner movement unit time period to generate the corresponding user mouth corner movement trajectory at this time period; Obtain the drooping time series fluctuation corresponding to the user's mouth corner movement based on the movement trajectory of the user's mouth corner at this time period, and fit the dynamic changes of the mouth corner movement within the time period of the user's mouth corner action unit based on the drooping time series fluctuation corresponding to the user's mouth corner movement, so as to generate the fitting function relationship between time and the change of the user's mouth corner drooping position at this time period; Measure the deviation of the mouth corner drooping amplitude according to the fitting function relationship between time and the change of the user's mouth corner drooping position at this time period, so as to obtain the fitting amplitude deviation of the user's mouth corner drooping corresponding to this time period; Obtain the corresponding mouth corner length of the user, and perform action drooping amplitude statistics on the fitting amplitude deviation of the user's mouth corner drooping corresponding to this time period based on the corresponding mouth corner length of the user, so as to obtain the mouth corner drooping amplitude angle corresponding to the facial action of the user image at this time period.

6. The psychological risk early warning method based on multi-dimensional user emotion data according to claim 3, characterized in that Step S24 includes the following steps: Step S241: Input the user's emotional physiological indicators corresponding in the user's multi-dimensional time series synchronous emotional data, including the heart rate variability index HRV and the galvanic skin response index GSR, into the three-level emotional feature extraction module in the emotional three-level feature extraction architecture to extract the corresponding LF value and HF value on HRV and the rising slope section interval corresponding to GSR; Step S242: Calculate the ratio according to the corresponding LF value and HF value on HRV to obtain the LF / HF ratio of HRV; Step S243: Obtain the corresponding rising change amplitude of the GSR index and the rising time deviation of the GSR index through the rising slope section interval corresponding to GSR; Step S244: Perform a rising slope fitting calculation on the rising slope section interval corresponding to GSR based on the rising change amplitude of the GSR index and the rising time deviation of the GSR index to obtain the rising slope of the GSR index.

7. The psychological risk warning method based on multi-dimensional user emotion data according to claim 3, wherein Step S25 includes the following steps: Step S251: Design the corresponding inter-modal attention matrices at each level between the first-level, second-level, and third-level emotional feature extraction modules in the emotional three-level feature extraction architecture, and obtain the feature query vectors and key vectors between different modalities based on this inter-modal attention matrix; Step S252: Calculate the corresponding feature vector distance value according to the feature query vectors and key vectors between different modalities; Step S253: Cross-modally weight and fuse the user's emotional psychological features corresponding to the first-level, second-level, and third-level modalities based on this feature vector distance value to obtain the user's cross-modal fusion psychological features.

8. The psychological risk early warning method based on multi-dimensional user emotion data according to claim 1, characterized in that Step S3 includes the following steps: Step S31: Obtain the user's environmental data and the user's social network data; Step S32: Extract environmental factors from the user's environmental data to obtain each user environmental factor; Step S33: Analyze the user's behavior patterns in the user's social network data to obtain the user's emotional psychological behavior patterns corresponding to the user on the social network; Step S34: Perform emotional psychological dynamic graph reasoning on the emotional psychological feature states corresponding to the user's cross-modal fusion psychological features within the same time based on each user environmental factor and the user's emotional psychological behavior patterns corresponding to the user on the social network, so as to generate an emotional psychological causal dynamic relationship graph between the user's corresponding environmental factors, behavior patterns, and psychological states.

9. The psychological risk warning method based on multi-dimensional user emotion data according to claim 1, characterized in that, Step S4 includes the following steps: Step S41: Obtain the emotional and psychological fluctuation influence path of the user under the corresponding environmental factors and behavior patterns through the emotional and psychological causal dynamic relationship diagram among the user's corresponding environmental factors, behavior patterns and psychological states; Step S42: Quantify the emotional fluctuation level at each time point within the emotional and psychological fluctuation influence path of the user under the corresponding environmental factors and behavior patterns to obtain the emotional and psychological fluctuation level of the user at each time point on the emotional and psychological fluctuation influence path; Step S43: Predict the extreme inflection point of emotional evolution based on the emotional and psychological fluctuation level of the user at each time point on the emotional and psychological fluctuation influence path to obtain the corresponding user emotional evolution risk inflection point on the emotional and psychological fluctuation influence path; Step S44: Calculate the corresponding relative risk index through the user emotional evolution risk inflection point on the emotional and psychological fluctuation influence path, and dynamically divide the risk level of the relative risk index according to the preset five-stage risk assessment system. The preset five-stage risk assessment system includes normal when the relative risk index ≤ 10, subclinical when 10 < relative risk index ≤ 30, mild risk when 30 < relative risk index ≤ 40, moderate and mild risk when 40 < relative risk index ≤ 60, and high risk when the relative risk index > 60, so as to divide the corresponding user emotional and psychological risk level. At the same time, conduct a risk warning response to the user emotional evolution risk inflection point according to the corresponding user emotional and psychological risk level to generate the corresponding user emotional and psychological risk warning response measures, including automatically triggering self-regulation suggestions for risk levels of mild risk and below, and automatically notifying emergency contacts for risk levels above mild risk.

10. A psychological risk early warning system based on multi-dimensional emotional data of users, characterized in that, For implementing the psychological risk warning method based on user multi-dimensional emotional data as described in claim 1, the psychological risk warning system based on user multi-dimensional emotional data includes: A multi-dimensional emotional time series alignment module, configured to obtain the user's multi-dimensional emotional data corresponding to the user's emotional expression images, user's emotional voice signals, and user's emotional physiological indicators, and perform multi-source time series alignment processing on the user's multi-dimensional emotional data to obtain the user's multi-dimensional time series synchronized emotional data; An emotional and psychological feature fusion module, configured to construct an emotional three-level feature extraction architecture, and perform modal emotional feature extraction on the user's multi-dimensional time series synchronized emotional data based on the emotional three-level feature extraction architecture to obtain the user's emotional and psychological features corresponding to each level of the modality; design an attention matrix between modalities to cross-modally fuse the user's emotional and psychological features corresponding to each level of the modality, so as to obtain the user's cross-modal fusion psychological features; An emotional and psychological dynamic reasoning module, configured to obtain the user's environmental data and the user's social network data, and perform emotional and psychological dynamic graph reasoning on the user's cross-modal fusion psychological features based on the user's environmental data and the user's social network data to generate an emotional and psychological causal dynamic relationship diagram among the user's corresponding environmental factors, behavior patterns and psychological states; An emotional and psychological risk warning module is used to obtain the corresponding inflection point of the user's emotional evolution risk through the emotional and psychological causal dynamic relationship diagram among the user's corresponding environmental factors, behavior patterns and psychological states, and conduct psychological risk warning according to the inflection point of the user's emotional evolution risk, so as to generate corresponding user emotional and psychological risk warning response measures.

Images