Self-injury behavior early warning and intervention method and system based on multi-modal physiological data and AI

Abstract

The invention relates to the technical field of psychological health monitoring, in particular to a self-injury behavior early warning and intervention method and system based on multi-modal physiological data and AI.The method comprises the steps that physiological signals, behavior data, text data and environment parameters of a user are collected in real time based on a contact sensor and non-contact sensing equipment, and the physiological signals, the behavior data, the text data and the environment parameters of the user are obtained; generating a multi-source original data set; performing time synchronization and spatial correlation analysis on the multi-source original data set to form a fusion feature vector; inputting the fusion feature vector into a preset multi-modal deep learning model, obtaining a self-injury behavior risk score value, and dividing risk levels based on the self-injury behavior risk score value; and triggering a hierarchical intervention strategy of AI psychological counseling, emergency contact notification and medical resource linkage according to the risk level. The method has the effect of improving the precision and quality of teenager mental health assessment and self-injury behavior early warning results.

Description

Technical Field

[0001] The present invention relates to the technical field of mental health monitoring, and in particular to a method and system for early warning and intervention of self-harm behavior based on multimodal physiological data and AI. Background Art

[0002] In recent years, with the rapid development of social economy and the continuous changes in education models, adolescent mental health problems have become increasingly serious, and the incidence of self-harm behavior has continued to rise. Traditional mental health assessment methods mainly rely on single physiological indicator monitoring and standardized psychological test questionnaires. However, such methods have problems such as strong subjectivity, long time consumption, and poor evaluation effect of single modality data. Therefore, it has become an important task to quickly and accurately assess the mental health status of adolescents.

[0003] At present, the development of big data and artificial intelligence technology has provided new technical means for early warning of adolescent mental health. Existing multimodal perception and machine learning technologies mostly use static fusion methods such as feature splicing or weighted averaging. However, they ignore the spatiotemporal correlation between adolescents' physiological signals, behavioral movements and the environmental parameters in which they are located, resulting in limited accuracy of early warning results. Therefore, there is room for improvement. Summary of the Invention

[0004] In order to solve the above technical problems and improve the accuracy and quality of adolescent mental health assessment and self-harm behavior warning results, this application provides a self-harm behavior warning and intervention method and system based on multimodal physiological data and AI.

[0005] In the first aspect, the above-mentioned invention object of the present application is achieved through the following technical solutions:

[0006] A self-injury behavior early warning and intervention method based on multimodal physiological data and AI, comprising the following steps:

[0007] Based on contact sensors and contactless sensing devices, users' physiological signals, behavioral data, text data and environmental parameters are collected in real time to generate multi-source raw data sets;

[0008] Performing time synchronization and spatial correlation analysis on the multi-source original data sets to form a fusion feature vector;

[0009] Inputting the fused feature vector into a preset multimodal deep learning model to obtain a self-injury behavior risk score value, and dividing the risk level based on the self-injury behavior risk score value;

[0010] A hierarchical intervention strategy is triggered based on the risk level, including AI psychological counseling, emergency contact notification, and medical resource linkage.

[0011] By adopting the above technical solutions, in the process of evaluating and warning the mental health of adolescent users, through the multi-source collaboration of contact sensors (such as flexible electronic skin patches to monitor physiological heart rate variability and EDA skin conductance response in real time) and non-contact sensing devices (such as infrared-RGB cameras to capture micro-expressions and microphones to analyze voice emotions), the perception limitations of single-modality data are broken through, and full-dimensional coverage of physiological signals (heart rate, skin electricity), behavioral data (frequency of weapon movements, facial muscle tension), environmental parameters (light intensity, noise decibels) and text information (suicidal tendency keywords) is achieved. Multi-source original data sets are collected, providing comprehensive data for subsequent analysis, which is conducive to a more in-depth analysis of the user's psychological state and behavioral patterns, and time synchronization and spatial correlation analysis are performed on the multi-source original data sets, integrating Information from different channels builds the inherent connections and potential laws between different source data, forming a fused feature vector, which provides a high-information-density input basis for subsequent mental health analysis. The obtained fused feature vector is input into the preset multimodal deep learning model, and the physiological signals, behavioral data, and text data of adolescents are analyzed in different dimensions. The self-harm behavior risk score of the adolescent user is calculated based on the analysis results, and the accuracy and quality of the adolescent mental health assessment and early warning results are improved. The risk level is divided according to the self-harm behavior risk score. According to the risk level, the adolescent user is triggered to take a graded intervention strategy of AI psychological counseling, emergency contact notification and medical resource linkage, so that corresponding intervention measures can be taken according to different risk levels, thereby improving the effectiveness of the intervention and user experience, and reducing the burden on the medical system.

[0012] In a preferred example, the present application can be further configured as follows: based on the contact sensor and the contactless sensing device, the user's physiological signals, behavioral data, text data and environmental parameters are collected in real time to generate an original data set, specifically including:

[0013] Based on the preset sampling frequency, the flexible electronic skin patch continuously collects wide-area capacitance pulse wave, skin electrical activity and skin temperature signals, and integrates them into the user's physiological signals;

[0014] capturing the user's facial micro-expressions and body movements in real time based on a preset frame rate, extracting facial action units based on the user's facial micro-expressions and body movements, and identifying self-injury precursor movements based on the facial action units to form behavioral data;

[0015] Collect user voice data in real time, extract Mel-frequency cepstral coefficients and fundamental frequency standard deviation features based on the user voice data to form user text data;

[0016] Environmental parameters are acquired in real time through environmental sensors, and the physiological signals, behavioral data, text data and environmental parameters are integrated to form a multi-source original data set.

[0017] By adopting the above technical solution, the flexible electronic skin patch continuously captures photoplethysmography (physiological), electrodermal activity (EDA), and skin temperature signals at a high-frequency sampling rate of 100-200Hz, breaking through the limitations of traditional single-point measurement and achieving real-time tracking of deep physiological indicators such as heart rate variability (HRV) and sympathetic nerve activation level (LF / HF ratio). A 30fps high-frame-rate infrared-RGB dual-spectrum camera captures facial micro-expressions and body movements. Facial action units are extracted based on the user's facial micro-expressions and body movements. Based on the precise analysis of facial action units, self-injury precursor movements are identified to form behavioral data. Simultaneously, voice data is collected and Mel-frequency cepstral coefficients and fundamental frequency standard deviation (reflecting vocal cord tension) are extracted to identify high-risk text data such as "I don't want to live anymore." Environmental sensors are combined to obtain real-time stress parameters such as light and noise. A multi-source raw data set is constructed to achieve collaborative verification of the four-dimensional data of physiology, behavior, environment, and language. This provides a multi-dimensional data foundation for subsequent mental health analysis and significantly improves the accuracy and timeliness of self-injury behavior warnings.

[0018] In a preferred example, the present application may be further configured as follows: performing time synchronization and spatial correlation analysis on the multi-source original data sets to form a fusion feature vector specifically includes:

[0019] Performing hardware clock synchronization on the multi-source raw data sets based on a time protocol, and aligning the temporal relationship between physiological signals and behavioral data using a dynamic time warping algorithm;

[0020] Mapping the physiological signals, behavioral data, text data, and environmental parameters into nodes, constructing a heterogeneous multimodal knowledge graph, and calculating the association weights between each node;

[0021] The weight of each modal data is dynamically allocated according to the association weight between each node to generate a fusion feature vector.

[0022] By adopting the above technical solution, hardware-level clock synchronization of multiple source devices is performed based on the precise time protocol, solving the problem of cumulative drift caused by timestamp synchronization. For example, this ensures millisecond-level alignment of skin conductance spike events with weapon-wielding action video frames. Furthermore, a dynamic time warping algorithm is used to flexibly match non-uniformly sampled data streams (such as 100Hz physiological signals and 30fps behavioral videos), accurately capturing the cross-modal event chain of "onset of voice tremor → heart rate acceleration → onset of scratching action." At the spatial correlation level, physiological signal heart rate variability, EDA skin conductance response slope, facial action unit activation frequency, and ambient light-noise coupling coefficient are mapped into multidimensional nodes. A graph attention network is used to calculate the causal and temporal dual correlation weights between nodes. A fused feature vector is generated through a dynamic weight allocation mechanism (for example, increasing the weight of the visual modality when a camera failure is detected). Through spatiotemporal alignment and cross-modal correlation modeling, the accuracy and robustness of self-injury behavior warning data are significantly improved.

[0023] In a preferred example, the present application may be further configured as follows: inputting the fused feature vector into a preset multimodal deep learning model to obtain a self-harm behavior risk score, specifically including:

[0024] Inputting the fused feature vector into a preset multimodal deep learning model, extracting a physiological vector based on the user's physiological signal features, extracting a behavioral vector based on the user's behavioral data features, and extracting a semantic vector based on the user's text data;

[0025] Based on the vector information of each dimension and its corresponding weight value, the self-harm behavior risk score is calculated.

[0026] By adopting the above technical solution, the multimodal deep learning model adopts a sub-modal feature extraction architecture. The physiological branch uses a 1D CNN to deeply analyze the nonlinear characteristics of heart rate variability in physiological signals and the slope of skin conductance response in EDA signals to generate a 32-dimensional high-semantic physiological vector. The behavioral branch uses a 3D CNN to extract the spatiotemporal patterns of weapon-holding movements in video clips (such as the rate of change of grip angle and the entropy of movement trajectory). Combined with the conflict index of facial action units (such as the frequency of frowning and smirking), it constructs a 64-dimensional behavioral vector. The text branch uses a Transformer encoder to explore emotional polarity (such as the frequency of the word "despair") and semantic contradictions (such as the coexistence of "I'm fine" and sobbing) in speech and text to generate a 32-dimensional semantic vector. During the feature fusion stage, the weights of each modality are dynamically calculated. For example, when speech data is missing, the weight of the text branch is automatically reduced, and the modal weights of the physiological and behavioral branches are enhanced. This is used to calculate the self-injury behavior risk score. Through the multimodal deep learning model and dynamic weight optimization mechanism, the self-injury behavior risk score is accurately quantified.

[0027] In a preferred example, the present application can be further configured as follows: the hierarchical intervention strategy of triggering AI psychological counseling, emergency contact notification, and medical resource linkage according to the risk level specifically includes:

[0028] When the risk level is low, AI dialogue tasks and mindfulness training videos based on cognitive behavioral therapy are pushed;

[0029] When the risk level is medium, a virtual character is activated to guide the user to complete the emotional log and notify the preset emergency contact;

[0030] When the risk level is high and lasts for 10 minutes, the system will automatically call the bound psychologist and send the user's real-time location information;

[0031] Optimize the interaction strategy of virtual characters based on reinforcement learning, and adjust the tone of conversation and task difficulty according to the user's historical response data.

[0032] By adopting the above technical solutions, in the low-risk stage, the AI ​​dialogue engine based on cognitive behavioral therapy generates personalized counseling words, combined with mindfulness training videos, to enable users to self-regulate; in the medium-risk stage, the driven 3D virtual character dynamically optimizes the interaction strategy through reinforcement learning, such as switching to encouragement or listening mode according to the user's historical feedback, guiding the user to complete emotional log recording, and sending encrypted early warning information to emergency contacts to achieve human-computer collaborative intervention; when the risk level is high and lasts for 10 minutes, the medical linkage agreement is automatically triggered, and the bound psychologist is automatically called and the user's real-time location information is sent. Through the dynamic hierarchical intervention mechanism and reinforcement learning optimization, a closed-loop management system for the prevention and control of self-harm behavior is constructed, which significantly improves crisis response efficiency and user compliance.

[0033] In a preferred example, the present application can be further configured as follows: mapping the physiological signals, behavioral data, and environmental parameters into nodes, constructing a heterogeneous multimodal knowledge graph, and calculating the association weights between the nodes, specifically including:

[0034] Define node types, including physiological indicator nodes, behavioral nodes, and environmental nodes, and build edge relationships between nodes;

[0035] The attention coefficients between nodes are calculated according to the graph attention network, and based on the attention coefficients between nodes, a heterogeneous multimodal knowledge graph is constructed.

[0036] By adopting the above technical solution, high-dimensional heterogeneous data such as heart rate variability of photoplethysmography signals, response slope of skin electrodermal activity, conflict frequency of facial action units, and ambient light-noise coupling index are mapped into three types of nodes: physiological, behavioral, and environmental. This realizes semantic-level integration of multi-source information and constructs edge relationships between nodes, such as causal association edges (such as "nighttime light intensity <50 lux leads to a 12% increase in depression scores") and temporal accompanying edges (such as "EDA amplitude surges by 200% within 5 seconds after the weapon-holding action occurs"). The dynamic attention coefficient between nodes is calculated through a graph attention network, and based on the attention coefficient between nodes, a heterogeneous multimodal knowledge graph is constructed. Through deep association modeling of the heterogeneous multimodal knowledge graph, the decision-making interpretability and cross-modal reasoning ability of the self-injury behavior warning system are significantly improved.

[0037] Secondly, the above-mentioned invention objectives of this application are achieved through the following technical solutions:

[0038] A self-injury behavior warning and intervention system based on multimodal physiological data and AI, comprising:

[0039] Multi-source data acquisition module, which is used to collect users' physiological signals, behavioral data, text data and environmental parameters in real time based on contact sensors and non-contact sensing devices to generate multi-source raw data sets;

[0040] A data processing module is used to perform time synchronization and spatial correlation analysis on the multi-source original data sets to form a fusion feature vector;

[0041] A self-injury behavior analysis module is configured to input the fused feature vector into a preset multimodal deep learning model to obtain a self-injury behavior risk score value, and to classify the risk level based on the self-injury behavior risk score value;

[0042] An intervention measure generation module is used to trigger a hierarchical intervention strategy of AI psychological counseling, emergency contact notification, and medical resource linkage based on the risk level.

[0043] By adopting the above technical solutions, in the process of evaluating and warning the mental health of adolescent users, through the multi-source collaboration of contact sensors (such as flexible electronic skin patches to monitor physiological heart rate variability and EDA skin conductance response in real time) and non-contact sensing devices (such as infrared-RGB cameras to capture micro-expressions and microphones to analyze voice emotions), the perception limitations of single-modality data are broken through, and full-dimensional coverage of physiological signals (heart rate, skin electricity), behavioral data (frequency of weapon movements, facial muscle tension), environmental parameters (light intensity, noise decibels) and text information (suicidal tendency keywords) is achieved. Multi-source original data sets are collected, providing comprehensive data for subsequent analysis, which is conducive to a more in-depth analysis of the user's psychological state and behavioral patterns, and time synchronization and spatial correlation analysis are performed on the multi-source original data sets, integrating Information from different channels builds the inherent connections and potential laws between different source data, forming a fused feature vector, which provides a high-information-density input basis for subsequent mental health analysis. The obtained fused feature vector is input into the preset multimodal deep learning model, and the physiological signals, behavioral data, and text data of adolescents are analyzed in different dimensions. The self-harm behavior risk score of the adolescent user is calculated based on the analysis results, and the accuracy and quality of the adolescent mental health assessment and early warning results are improved. The risk level is divided according to the self-harm behavior risk score. According to the risk level, the adolescent user is triggered to take a graded intervention strategy of AI psychological counseling, emergency contact notification and medical resource linkage, so that corresponding intervention measures can be taken according to different risk levels, thereby improving the effectiveness of the intervention and user experience, and reducing the burden on the medical system.

[0044] Thirdly, the above-mentioned purpose of this application is achieved through the following technical solutions:

[0045] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned self-injury behavior warning and intervention method based on multimodal physiological data and AI are implemented.

[0046] Fourthly, the above-mentioned purpose of the present application is achieved through the following technical solutions:

[0047] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned self-injury behavior warning and intervention method based on multimodal physiological data and AI.

[0048] In summary, this application includes at least one of the following beneficial technical effects:

[0049] 1. In the process of assessing and warning the mental health of adolescent users, through the multi-source collaboration of contact sensors (such as flexible electronic skin patches to monitor physiological heart rate variability and EDA skin conductance response in real time) and non-contact sensing devices (such as infrared-RGB cameras to capture micro-expressions and microphones to analyze voice emotions), we can break through the perception limitations of single-modal data and achieve full-dimensional coverage of physiological signals (heart rate, skin electricity), behavioral data (frequency of weapon movements, facial muscle tension), environmental parameters (light intensity, noise decibels) and text information (keywords for suicidal tendencies). We collect multi-source original data sets, provide comprehensive data for subsequent analysis, and contribute to a deeper analysis of the user's psychological state and behavioral patterns. We perform time synchronization and spatial correlation analysis on multi-source original data sets, integrating different channels. The information is used to build the inherent connections and potential laws between different source data, forming a fusion feature vector, which provides a high-information-density input basis for subsequent mental health analysis. The obtained fusion feature vector is input into the preset multimodal deep learning model to analyze the physiological signals, behavioral data, and text data of adolescents in different dimensions. Based on the analysis results, the self-harm behavior risk score of the adolescent user is calculated to improve the accuracy and quality of the adolescent mental health assessment and early warning results. The risk level is divided according to the self-harm behavior risk score. According to the risk level, a graded intervention strategy of AI psychological counseling, emergency contact notification, and medical resource linkage is triggered for the adolescent user, so that corresponding intervention measures can be taken according to different risk levels, improving the effectiveness of the intervention and user experience, while reducing the burden on the medical system;

[0050] 2. The flexible electronic skin patch continuously captures photoplethysmography (physiological), electrodermal activity (EDA), and skin temperature signals at a high-frequency sampling rate of 100-200Hz, breaking through the limitations of traditional single-point measurement and enabling real-time tracking of deep physiological indicators such as heart rate variability (HRV) and sympathetic nerve activation (LF / HF ratio). A 30fps high-frame-rate infrared-RGB dual-spectrum camera captures facial micro-expressions and body movements. Based on the user's facial micro-expressions and body movements, facial action units are extracted. Based on the precise analysis of facial action units, premonitory movements of self-injury are identified to generate behavioral data. Simultaneously, voice data is collected and Mel-frequency cepstral coefficients and fundamental frequency standard deviation (reflecting vocal cord tension) are extracted to identify high-risk textual data such as "I don't want to live anymore." Environmental sensors are used to obtain real-time stress parameters such as light and noise, constructing a multi-source raw data set and achieving collaborative verification of physiological, behavioral, environmental, and language data. This provides a multi-dimensional data foundation for subsequent mental health analysis and significantly improves the accuracy and timeliness of self-injury behavior warnings.

[0051] 3. Hardware-level clock synchronization of multiple source devices based on a precise time protocol addresses the cumulative drift caused by timestamp synchronization. For example, this ensures millisecond-level alignment of skin conductance spikes with frames of video of weapon-wielding actions. Furthermore, a dynamic time warping algorithm is employed to flexibly match non-uniformly sampled data streams (e.g., 100Hz physiological signals with 30fps behavioral videos), accurately capturing the cross-modal event chain of "onset of voice tremor → heart rate acceleration → onset of scratching." At the spatial correlation level, physiological signal heart rate variability, EDA skin conductance response slope, facial action unit activation frequency, and ambient light-noise coupling coefficient are mapped into multidimensional nodes. A graph attention network is used to calculate the causal-temporal dual correlation weights between nodes. A dynamic weight allocation mechanism (e.g., increasing the weight of the visual modality upon detecting camera failure) is used to generate a fused feature vector. Through spatiotemporal alignment and cross-modal correlation modeling, the accuracy and robustness of self-injury behavior warning data are significantly improved.

[0052] 4. The multimodal deep learning model adopts a multimodal feature extraction architecture. The physiological branch uses a 1D CNN to deeply analyze the nonlinear characteristics of heart rate variability in physiological signals and the slope of skin conductance response in EDA signals to generate a 32-dimensional high-semantic physiological vector. The behavioral branch uses a 3D CNN to extract the spatiotemporal patterns of weapon-holding movements in video clips (such as the rate of change of grip angle and the entropy of movement trajectories). Combined with the conflict index of facial action units (such as the frequency of frowning and smirking), this constructs a 64-dimensional behavioral vector. The text branch uses a Transformer encoder to mine emotional polarity (such as the frequency of the word "despair") and semantic contradictions (such as the coexistence of "I'm fine" and sobbing) in speech and text to generate a 32-dimensional semantic vector. During the feature fusion stage, the weights of each modality are dynamically calculated. For example, when speech data is missing, the weight of the text branch is automatically reduced, and the modal weights of the physiological and behavioral branches are strengthened. This is used to calculate the self-injury behavior risk score. Through the multimodal deep learning model and dynamic weight optimization mechanism, a precise quantification of the self-injury behavior risk score is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of a method for early warning and intervention of self-injurious behavior based on multimodal physiological data and AI in one embodiment of the present application;

[0054] Figure 2 This is a flowchart for implementing step S10 in the self-injury behavior warning and intervention method based on multimodal physiological data and AI in one embodiment of the present application;

[0055] Figure 3 This is a flowchart for implementing step S20 in the self-injury behavior warning and intervention method based on multimodal physiological data and AI in one embodiment of the present application;

[0056] Figure 4This is a flowchart for implementing step S30 in the self-injury behavior warning and intervention method based on multimodal physiological data and AI in one embodiment of the present application;

[0057] Figure 5 This is a flowchart for implementing step S40 in the self-injury behavior warning and intervention method based on multimodal physiological data and AI in one embodiment of the present application;

[0058] Figure 6 This is a flowchart of the implementation of step S22 in the self-injury behavior warning and intervention method based on multimodal physiological data and AI in one embodiment of the present application.

[0059] Figure 7 This is a principle block diagram of a self-injury behavior warning and intervention system based on multimodal physiological data and AI in one embodiment of the present application;

[0060] Figure 8 It is a schematic diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The present application is further described in detail below with reference to the accompanying drawings.

[0062] In one embodiment, if Figure 1 As shown, this application discloses a self-injury behavior warning and intervention method based on multimodal physiological data and AI, which specifically includes the following steps:

[0063] S10: Based on contact sensors and contactless sensing devices, the user's physiological signals, behavioral data, text data and environmental parameters are collected in real time to generate a multi-source raw data set.

[0064] In one embodiment, if Figure 2 As shown, in step S10, based on the contact sensor and the contactless sensing device, the user's physiological signals, behavioral data, text data and environmental parameters are collected in real time to generate a multi-source original data set, which specifically includes:

[0065] S11: Based on the preset sampling frequency, the flexible electronic skin patch continuously collects the wide-area capacitance pulse wave, skin electrical activity and skin temperature signals, and integrates them to form the user's physiological signals.

[0066] Specifically, in the process of assessing and warning the mental health of adolescent users, contact sensors such as flexible electronic skin patches are used to continuously capture photoplethysmography (physiological), electrodermal activity (EDA) and skin temperature signals at a high frequency of 100-200Hz, breaking through the limitations of traditional single-point measurement and achieving real-time tracking of deep physiological indicators such as heart rate variability (HRV) and sympathetic nerve activation level (LF / HF ratio).

[0067] S12: Capturing the user's facial micro-expressions and body movements in real time based on a preset frame rate, extracting facial action units based on the user's facial micro-expressions and body movements, and identifying precursor movements of self-injury based on the facial action units to form behavioral data.

[0068] S13: Collect user voice data in real time, extract Mel-frequency cepstral coefficients and fundamental frequency standard deviation features based on the user voice data, and form user text data.

[0069] Specifically, a 30fps high-frame-rate infrared-RGB dual-spectrum camera is used to capture facial micro-expressions and body movements. Facial action units are extracted based on the user's facial micro-expressions and body movements. Based on the precise analysis of facial action units, precursor movements of self-injury are identified to form behavioral data. Simultaneously, voice data is collected and Mel-frequency cepstral coefficients and fundamental frequency standard deviation (reflecting vocal cord tension) are extracted to identify high-risk text data such as "I don't want to live anymore."

[0070] S14: Acquire environmental parameters in real time through environmental sensors, and integrate the physiological signals, behavioral data, text data, and environmental parameters to form a multi-source original data set.

[0071] Specifically, by combining environmental sensors to obtain real-time pressure parameters such as light and noise, a multi-source original data set is constructed to achieve collaborative verification of four-dimensional data of physiology, behavior, environment and language, providing a multi-dimensional data foundation for subsequent mental health analysis, and significantly improving the accuracy and timeliness of self-harm behavior warnings.

[0072] S20: Performing time synchronization and spatial correlation analysis on the multi-source original data sets to form a fusion feature vector.

[0073] In one embodiment, if Figure 3 As shown, in step S20, time synchronization and spatial correlation analysis are performed on the multi-source original data sets to form a fusion feature vector, which specifically includes:

[0074] S21: Performing hardware clock synchronization on the multi-source original data sets based on a time protocol, and using a dynamic time warping algorithm to align the temporal relationship between physiological signals and behavioral data.

[0075] Specifically, hardware-level clock synchronization of multi-source devices is performed based on the precise time protocol to solve the problem of cumulative drift caused by timestamp synchronization. For example, it ensures the millisecond-level alignment of skin conductance surge events with weapon-holding action video frames. Furthermore, a dynamic time warping algorithm is used to flexibly match non-uniformly sampled data streams (such as 100Hz physiological signals and 30fps behavioral videos) to accurately capture the cross-modal event chain of "start of voice tremor → acceleration of heart rate → appearance of scratching action".

[0076] S22: Map the physiological signals, behavioral data, text data and environmental parameters into nodes, construct a heterogeneous multimodal knowledge graph, and calculate the association weights between each node.

[0077] Specifically, at the spatial correlation level, the heart rate variability of physiological signals, the skin conductance response slope of EDA, the activation frequency of facial action units, the ambient light-noise coupling coefficient, etc. are mapped into multi-dimensional nodes, and the causal-temporal dual correlation weights between nodes are calculated through the graph attention network.

[0078] In one embodiment, if Figure 6 As shown, in step S22, physiological signals, behavioral data, and environmental parameters are mapped into nodes to construct a heterogeneous multimodal knowledge graph, and the association weights between each node are calculated, specifically including:

[0079] S221: Define node types, including physiological indicator nodes, behavior nodes, and environment nodes, and construct edge relationships between nodes.

[0080] S222: Calculate the attention coefficients between nodes according to the graph attention network, and construct a heterogeneous multimodal knowledge graph based on the attention coefficients between nodes.

[0081] Specifically, high-dimensional heterogeneous data such as heart rate variability of photoplethysmography signals, response slope of electrodermal activity, conflict frequency of facial action units, and ambient light-noise coupling index are mapped into three types of nodes: physiological, behavioral, and environmental. This enables semantic-level integration of multi-source information, and constructs edge relationships between nodes, such as causal edges (e.g., "nighttime light intensity <50 lux leads to a 12% increase in depression scores") and temporal accompanying edges (e.g., "EDA amplitude surges by 200% within 5 seconds of weapon-holding action"). The dynamic attention coefficient between nodes is calculated through a graph attention network, and based on the attention coefficient between nodes, a heterogeneous multimodal knowledge graph is constructed. Through deep association modeling of the heterogeneous multimodal knowledge graph, the decision-making interpretability and cross-modal reasoning ability of the self-injury behavior warning system are significantly improved.

[0082] S23: Dynamically assign weights to each modal data according to the association weights between nodes to generate a fusion feature vector.

[0083] Specifically, a fusion feature vector is generated through a dynamic weight allocation mechanism (for example, when a camera failure is detected, the visual modality weight is increased), and time synchronization and spatial correlation analysis are performed on multi-source original data sets. Information from different channels is integrated, and the inherent connections and potential laws between different source data are constructed to form a fusion feature vector, providing a high-information-density input basis for subsequent mental health analysis.

[0084] S30: Inputting the fused feature vector into a preset multimodal deep learning model to obtain a self-injury behavior risk score value, and dividing the risk level based on the self-injury behavior risk score value.

[0085] In one embodiment, if Figure 4 As shown, in step S30, the fused feature vector is input into a preset multimodal deep learning model to obtain a self-harm behavior risk score, which specifically includes:

[0086] S31: Input the fused feature vector into a preset multimodal deep learning model, extract a physiological vector based on the user's physiological signal features, extract a behavioral vector based on the user's behavioral data features, and extract a semantic vector based on the user's text data.

[0087] Specifically, the multimodal deep learning model adopts a sub-modal feature extraction architecture. The physiological branch uses 1D CNN to deeply analyze the nonlinear characteristics of heart rate variability of physiological signals and the slope of skin conductance response of EDA signals to generate a 32-dimensional high-semantic physiological vector. The behavioral branch uses 3D CNN to extract the spatiotemporal patterns of weapon-holding movements in video clips (such as the rate of change of grip angle and the entropy of movement trajectory), and combines it with the conflict index of facial action units (such as the concurrent frequency of frowning and smirking) to construct a 64-dimensional behavioral vector. The text branch uses the Transformer encoder to mine the emotional polarity (such as the frequency of the word "despair") and semantic contradictions (such as the coexistence of "I'm fine" and sobbing) in speech / text to generate a 32-dimensional semantic vector.

[0088] S32: Calculate a self-harm behavior risk score based on the vector information of each dimension and its corresponding weight value.

[0089] Specifically, in the feature fusion stage, the weights of each modality are dynamically calculated. For example, when voice data is missing, the weight of the text branch is automatically lowered, and the modal weights of the physiological branch and behavioral branch are enhanced. The self-injury behavior risk score is calculated. Through the multimodal deep learning model and dynamic weight optimization mechanism, the accurate quantification of the self-injury behavior risk score is achieved.

[0090] S40: triggering a hierarchical intervention strategy of AI psychological counseling, emergency contact notification, and medical resource linkage according to the risk level.

[0091] In one embodiment, if Figure 5 As shown, in step S40, a hierarchical intervention strategy of AI psychological counseling, emergency contact notification, and medical resource linkage is triggered according to the risk level, specifically including:

[0092] S41: When the risk level is low, push AI dialogue tasks and mindfulness training videos based on cognitive behavioral therapy.

[0093] Specifically, in the low-risk stage, an AI dialogue engine based on cognitive behavioral therapy generates personalized counseling scripts, combined with mindfulness training videos, to enable users to self-regulate.

[0094] S42: When the risk level is medium risk, the virtual character is activated to guide the user to complete the emotional log recording and notify the preset emergency contact.

[0095] Specifically, in the medium-risk stage, the driven 3D virtual character dynamically optimizes the interaction strategy through reinforcement learning, such as switching to encouragement or listening mode based on user historical feedback, guiding users to complete emotional log recording, and sending encrypted warning information to emergency contacts to achieve human-machine collaborative intervention.

[0096] S43: When the risk level is high and lasts for 10 minutes, the bound psychologist is automatically called and the user's real-time location information is sent.

[0097] Specifically, when the risk level is high and lasts for 10 minutes, the medical linkage agreement will be automatically triggered, the bound psychologist will be automatically called and the user's real-time location information will be sent. Through the dynamic hierarchical intervention mechanism and reinforcement learning optimization, a closed-loop management system for the prevention and control of self-harm behavior is constructed, which significantly improves crisis response efficiency and user compliance.

[0098] S44: Optimize the interaction strategy of virtual characters based on reinforcement learning, and adjust the conversation tone and task difficulty according to the user's historical response data.

[0099] In this embodiment, in the process of evaluating and warning the mental health of adolescent users, through the multi-source collaboration of contact sensors (such as flexible electronic skin patches to monitor physiological heart rate variability and EDA skin conductance response in real time) and non-contact sensing devices (such as infrared-RGB cameras to capture micro-expressions and microphones to analyze voice emotions), the perception limitations of single-modal data are broken through, and full-dimensional coverage of physiological signals (heart rate, skin electricity), behavioral data (frequency of weapon movements, facial muscle tension), environmental parameters (light intensity, noise decibels) and text information (suicidal tendency keywords) is achieved. Multi-source original data sets are collected, which provides comprehensive data for subsequent analysis, helps to more deeply analyze the user's psychological state and behavioral patterns, performs time synchronization and spatial correlation analysis on multi-source original data sets, and integrates different The information from different channels is used to build the intrinsic connections and potential laws between different source data, forming a fusion feature vector, which provides a high-information-density input basis for subsequent mental health analysis. The obtained fusion feature vector is input into the preset multimodal deep learning model, and the physiological signals, behavioral data, and text data of adolescents are analyzed in different dimensions. The self-harm behavior risk score of the adolescent user is calculated according to the analysis results, and the accuracy and quality of the adolescent mental health assessment and early warning results are improved. The risk level is divided according to the self-harm behavior risk score. According to the risk level, a graded intervention strategy of AI psychological counseling, emergency contact notification and medical resource linkage is triggered for the adolescent user, so that corresponding intervention measures can be taken according to different risk levels, thereby improving the effectiveness of the intervention and user experience, and reducing the burden on the medical system.

[0100] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0101] In one embodiment, a self-injury behavior warning and intervention system based on multimodal physiological data and AI is provided. The self-injury behavior warning and intervention system based on multimodal physiological data and AI corresponds one-to-one with the self-injury behavior warning and intervention method based on multimodal physiological data and AI in the above embodiment. Figure 7 As shown in Figure 1, the self-injury behavior warning and intervention system based on multimodal physiological data and AI includes a multi-source data acquisition module, a data processing module, a self-injury behavior analysis module, and an intervention measure generation module. The functional modules are described in detail below:

[0102] Multi-source data acquisition module, which is used to collect users' physiological signals, behavioral data, text data and environmental parameters in real time based on contact sensors and non-contact sensing devices to generate multi-source raw data sets;

[0103] A data processing module is used to perform time synchronization and spatial correlation analysis on the multi-source original data sets to form a fusion feature vector;

[0104] A self-injury behavior analysis module is configured to input the fused feature vector into a preset multimodal deep learning model to obtain a self-injury behavior risk score value, and to classify the risk level based on the self-injury behavior risk score value;

[0105] An intervention measure generation module is used to trigger a hierarchical intervention strategy of AI psychological counseling, emergency contact notification, and medical resource linkage based on the risk level.

[0106] Preferably, the data processing module includes:

[0107] A timing alignment processing submodule is used to synchronize the hardware clocks of the multi-source raw data sets based on a time protocol, and to align the timing relationship between physiological signals and behavioral data using a dynamic time warping algorithm;

[0108] A multimodal knowledge graph construction submodule is used to map the physiological signals, behavioral data, text data and environmental parameters into nodes, construct a heterogeneous multimodal knowledge graph, and calculate the association weights between each node;

[0109] The fusion feature vector generation submodule is used to dynamically allocate the weights of each modal data according to the association weights between each node to generate a fusion feature vector.

[0110] Regarding the specific limitations of the self-injury behavior warning and intervention system based on multimodal physiological data and AI, please refer to the limitations of the self-injury behavior warning and intervention method based on multimodal physiological data and AI above, which will not be repeated here. Each module in the above-mentioned self-injury behavior warning and intervention system based on multimodal physiological data and AI can be implemented in whole or in part through software, hardware, and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0111] In one embodiment, an electronic device is provided. The electronic device may be a server, and its internal structure diagram may be as follows: Figure 8As shown. The electronic device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store a database. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a self-injury behavior warning and intervention method based on multimodal physiological data and AI is implemented.

[0112] In one embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0113] Based on contact sensors and contactless sensing devices, users' physiological signals, behavioral data, text data and environmental parameters are collected in real time to generate multi-source raw data sets;

[0114] Performing time synchronization and spatial correlation analysis on the multi-source original data sets to form a fusion feature vector;

[0115] Inputting the fused feature vector into a preset multimodal deep learning model to obtain a self-injury behavior risk score value, and dividing the risk level based on the self-injury behavior risk score value;

[0116] A hierarchical intervention strategy is triggered based on the risk level, including AI psychological counseling, emergency contact notification, and medical resource linkage.

[0117] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0118] Based on contact sensors and contactless sensing devices, users' physiological signals, behavioral data, text data and environmental parameters are collected in real time to generate multi-source raw data sets;

[0119] Performing time synchronization and spatial correlation analysis on the multi-source original data sets to form a fusion feature vector;

[0120] Inputting the fused feature vector into a preset multimodal deep learning model to obtain a self-injury behavior risk score value, and dividing the risk level based on the self-injury behavior risk score value;

[0121] A hierarchical intervention strategy is triggered based on the risk level, including AI psychological counseling, emergency contact notification, and medical resource linkage.

[0122] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0123] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0124] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A self-injury behavior warning and intervention method based on multimodal physiological data and AI, characterized by: The self-injury behavior warning and intervention method based on multimodal physiological data and AI includes the following steps: Based on contact sensors and contactless sensing devices, users' physiological signals, behavioral data, text data and environmental parameters are collected in real time to generate multi-source raw data sets; Performing time synchronization and spatial correlation analysis on the multi-source original data sets to form a fusion feature vector; Inputting the fused feature vector into a preset multimodal deep learning model to obtain a self-injury behavior risk score value, and dividing the risk level based on the self-injury behavior risk score value; A hierarchical intervention strategy is triggered based on the risk level, including AI psychological counseling, emergency contact notification, and medical resource linkage.

2. The self-injury behavior early warning and intervention method based on multimodal physiological data and AI according to claim 1 is characterized in that: The contact sensor and the contactless sensing device are used to collect the user's physiological signals, behavioral data, text data and environmental parameters in real time to generate the original data set, which specifically includes: Based on the preset sampling frequency, the flexible electronic skin patch continuously collects wide-area capacitance pulse wave, skin electrical activity and skin temperature signals, and integrates them into the user's physiological signals; capturing the user's facial micro-expressions and body movements in real time based on a preset frame rate, extracting facial action units based on the user's facial micro-expressions and body movements, and identifying self-injury precursor movements based on the facial action units to form behavioral data; Collect user voice data in real time, extract Mel-frequency cepstral coefficients and fundamental frequency standard deviation features based on the user voice data to form user text data; Environmental parameters are acquired in real time through environmental sensors, and the physiological signals, behavioral data, text data and environmental parameters are integrated to form a multi-source original data set.

3. The self-injury behavior warning and intervention method based on multimodal physiological data and AI according to claim 1 is characterized in that: The performing time synchronization and spatial correlation analysis on the multi-source original data sets to form a fusion feature vector specifically includes: Performing hardware clock synchronization on the multi-source raw data sets based on a time protocol, and aligning the temporal relationship between physiological signals and behavioral data using a dynamic time warping algorithm; Mapping the physiological signals, behavioral data, text data, and environmental parameters into nodes, constructing a heterogeneous multimodal knowledge graph, and calculating the association weights between each node; The weight of each modal data is dynamically allocated according to the association weight between each node to generate a fusion feature vector.

4. The self-injury behavior warning and intervention method based on multimodal physiological data and AI according to claim 1 is characterized in that: Inputting the fused feature vector into a preset multimodal deep learning model to obtain a self-harm behavior risk score specifically includes: Inputting the fused feature vector into a preset multimodal deep learning model, extracting a physiological vector based on the user's physiological signal features, extracting a behavioral vector based on the user's behavioral data features, and extracting a semantic vector based on the user's text data; Based on the vector information of each dimension and its corresponding weight value, the self-harm behavior risk score is calculated.

5. The self-injury behavior warning and intervention method based on multimodal physiological data and AI according to claim 1 is characterized in that: The hierarchical intervention strategy of triggering AI psychological counseling, emergency contact notification, and medical resource linkage based on the risk level specifically includes: When the risk level is low, AI dialogue tasks and mindfulness training videos based on cognitive behavioral therapy are pushed; When the risk level is medium, a virtual character is activated to guide the user to complete the emotional log and notify the preset emergency contact; When the risk level is high and lasts for 10 minutes, the system will automatically call the bound psychologist and send the user's real-time location information; Optimize the interaction strategy of virtual characters based on reinforcement learning, and adjust the tone of conversation and task difficulty according to the user's historical response data.

6. The self-injury behavior warning and intervention method based on multimodal physiological data and AI according to claim 3 is characterized in that: Mapping the physiological signals, behavioral data, and environmental parameters into nodes, constructing a heterogeneous multimodal knowledge graph, and calculating the association weights between the nodes specifically includes: Define node types, including physiological indicator nodes, behavioral nodes, and environmental nodes, and build edge relationships between nodes; The attention coefficients between nodes are calculated according to the graph attention network, and based on the attention coefficients between nodes, a heterogeneous multimodal knowledge graph is constructed.

7. A self-injury behavior warning and intervention system based on multimodal physiological data and AI, characterized by: The self-injury behavior warning and intervention system based on multimodal physiological data and AI includes: Multi-source data acquisition module, which is used to collect users' physiological signals, behavioral data, text data and environmental parameters in real time based on contact sensors and non-contact sensing devices to generate multi-source raw data sets; A data processing module is used to perform time synchronization and spatial correlation analysis on the multi-source original data sets to form a fusion feature vector; A self-injury behavior analysis module is configured to input the fused feature vector into a preset multimodal deep learning model to obtain a self-injury behavior risk score value, and to classify the risk level based on the self-injury behavior risk score value; An intervention measure generation module is used to trigger a hierarchical intervention strategy of AI psychological counseling, emergency contact notification, and medical resource linkage based on the risk level.

8. The self-injury behavior warning and intervention system based on multimodal physiological data and AI according to claim 7 is characterized in that: The data processing module includes: A timing alignment processing submodule is used to synchronize the hardware clocks of the multi-source raw data sets based on a time protocol, and to align the timing relationship between physiological signals and behavioral data using a dynamic time warping algorithm; A multimodal knowledge graph construction submodule is used to map the physiological signals, behavioral data, text data and environmental parameters into nodes, construct a heterogeneous multimodal knowledge graph, and calculate the association weights between each node; The fusion feature vector generation submodule is used to dynamically allocate the weights of each modal data according to the association weights between each node to generate a fusion feature vector.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the self-injury behavior warning and intervention method based on multimodal physiological data and AI are implemented as described in any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a self-injury behavior warning and intervention method based on multimodal physiological data and AI are implemented as described in any one of claims 1 to 6.

Images