Intelligent emotional pacification system and method based on multi-modal perception and environmental intervention

Abstract

The application discloses an intelligent emotion pacification system and method based on multi-modal perception and environmental intervention, comprising a perception module for collecting multi-modal physiological and behavioral data of a patient, an edge processing unit for data desensitization, a decision module for calculating an emotion agitation index, and an environmental intervention module for controlling a ward environment. The application quantitatively calculates the emotion agitation index and triggers environmental intervention through multi-modal sensor joint perception, edge-side feature-level desensitization, individualized baseline modeling and dynamic weight fusion, so as to realize automatic and intelligent emotion pacification and risk warning, reduce false positives and invalid interventions, and improve patient comfort and ward safety.

Description

Technical Field

[0001] This invention relates to the field of medical devices and intelligent nursing technology, and in particular to an intelligent emotion soothing system and method based on multimodal perception and environmental intervention. Background Technology

[0002] In healthcare settings, patients may experience emotional agitation, restlessness, shouting, or self-harm due to pain, anxiety, delirium, mental disorders, or stress responses. Existing monitoring equipment often focuses on vital signs or a single modality (e.g., video or audio-only) monitoring, which has the following limitations:

[0003] 1. Single-modal data is greatly affected by occlusion, noise, and changes in lighting, making it difficult to comprehensively and stably reflect changes in patients' emotions;

[0004] 2. Emotional assessment often relies on manual observation or experience scales, which lacks real-time accuracy and consistency, and also places a heavy workload on medical staff.

[0005] 3. Existing intervention methods are often passive or singular, lacking automated environmental adjustment linked to monitoring results, making it difficult to provide adaptive treatment for different patients and different levels of agitation;

[0006] 4. Directly uploading or centrally storing sensitive data such as audio and video may pose privacy and compliance risks.

[0007] Therefore, there is an urgent need for an intelligent system that can accurately assess the level of emotional agitation of patients in real time within the ward and automatically link environmental equipment to implement calming interventions while ensuring privacy. Summary of the Invention

[0008] This invention aims to provide an intelligent emotion-soothing system and method based on multimodal perception and environmental intervention. By using multimodal sensors for joint perception, edge-side feature-level desensitization, individualized baseline modeling, and dynamic weight fusion, it quantitatively calculates the emotional agitation index and triggers environmental intervention, thereby achieving automated and intelligent emotion soothing and risk warning, reducing false alarms and ineffective interventions, and improving patient comfort and ward safety.

[0009] To achieve the above objectives, the present invention employs the following technical solution:

[0010] This invention discloses an intelligent emotion-soothing system based on multimodal perception and environmental intervention, comprising the following modules:

[0011] Sensing module: used to collect multimodal physiological and behavioral data of patients, including two or more devices such as radar, thermal imaging sensors, depth cameras, and microphone arrays;

[0012] Edge processing unit: used for time synchronization, preprocessing, denoising, feature extraction and irreversible desensitization of the collected multimodal raw data. The desensitized feature vectors are transmitted to the decision module for emotion assessment.

[0013] Decision module: Includes an AI risk assessment engine, used to build a personalized baseline model for patients, calculate the deviation of each modality feature from the baseline, and dynamically adjust the fusion weights based on the reliability assessment results of each modality's data quality, thereby fusing the data to obtain the emotional agitation index. Based on this, intervention instructions are output to the environmental intervention module;

[0014] Environmental intervention module: used to automatically adjust the environmental control devices in the ward to alleviate the patient's emotions after being triggered.

[0015] Interactive module: Provides a visual interface for medical staff to display the emotional agitation index in real time. Modal contribution information, alarm events and historical backtracking, and annotation feedback.

[0016] Preferably, the environmental control device includes a central controller and environmental peripheral devices connected thereto, including an intelligent lighting system, a network speaker, an aroma diffuser, and a vibration motor array installed inside the mattress.

[0017] The present invention also discloses a method using the aforementioned system, characterized by comprising the following steps:

[0018] S1. Non-contact acquisition of patients' multimodal physiological and behavioral data;

[0019] S2. In the ward, perform time synchronization, preprocessing, feature extraction on multimodal physiological and behavioral data, and perform irreversible desensitization on sensitive original data to obtain desensitized feature vectors.

[0020] S3. Based on the desensitized feature vector, establish a personalized baseline model for the patient, determine the deviation of each modality feature from the baseline, and conduct a reliability assessment of the data quality of each modality.

[0021] S4. Based on the reliability assessment results, the fusion weights of each modality are dynamically adjusted, and the deviations of each modality are fused according to the fusion weights to obtain the emotional agitation index. ;

[0022] S5, When the emotional intensity index When the preset threshold is exceeded continuously for a preset time, the environmental control device adjusts the ward environment to alleviate the patient's emotions.

[0023] S6. Receive manual annotations from medical staff and perform incremental training based on these annotations;

[0024] S7. Roll back to a historical stable version model when performance degrades.

[0025] Preferably, the methods for alleviating the patient's emotions in step S5 include interventions such as dimming the lights, playing soothing music, releasing calming scents, and initiating gentle vibrations of the bed.

[0026] Preferably, the information collected in step S1 includes breathing / heart rate and body movement information, temperature distribution information, three-dimensional skeletal joint point or posture trajectory information, pitch / volume / speech rate and acoustic characteristics of crying or shouting.

[0027] Preferably, the desensitization feature vector in step S2 includes Mel frequency cepstral coefficients, skeleton key points, and depth feature vectors.

[0028] Preferably, the reliability assessment method in step S3 is as follows:

[0029]

[0030] in Rate usability For short-term consistency scoring, .

[0031] Preferably, the fusion weights in step S4 The evaluation method is as follows:

[0032]

[0033] in .

[0034] Preferably, the emotional agitation index in step S4 The calculation method is as follows:

[0035] ;

[0036] The deviation is calculated as follows:

[0037]

[0038] in, For a moment modal eigenvalues, For personalized baseline mean, For individualized standard deviation, It is a very small positive number.

[0039] Preferably, the ward environment in step S5 includes lighting, volume, odor, or bed vibration.

[0040] The beneficial effects of this invention are:

[0041] 1. This invention overcomes the problems of occlusion, noise and light sensitivity of single mode by multimodal fusion, thereby improving the stability and robustness of the evaluation;

[0042] 2. This invention improves assessment accuracy by using individualized baselines and dynamic weight updates to adapt to differences among patients and fluctuations in sensor quality.

[0043] 3. This invention performs feature-level desensitization processing at the edge, reducing the risk of sensitive audio and video being transmitted outside the device and improving privacy compliance;

[0044] 4. This invention achieves real-time, automated, and tiered reassurance intervention by linking environmental intervention with decision-making results, thereby reducing the burden on medical staff;

[0045] 5. This invention continuously improves model performance and ensures system stability through incremental learning and rollback mechanism based on medical and nursing annotations. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the overall system architecture of the present invention;

[0047] Figure 2 This is a schematic diagram illustrating the process of calculating the exhilaration index and updating dynamic weights.

[0048] Figure 3 A schematic diagram of the sensor layout in the ward;

[0049] Figure 4 This is a schematic diagram of the user interface for medical staff. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings.

[0051] like Figure 1-4 As shown, this invention discloses an intelligent emotion-soothing system based on multimodal perception and environmental intervention, comprising the following modules:

[0052] Sensing module: used to collect multimodal physiological and behavioral data of patients, including two or more devices such as radar, thermal imaging sensors, depth cameras, and microphone arrays;

[0053] Edge processing unit: used for time synchronization, preprocessing, denoising, feature extraction and irreversible desensitization of the collected multimodal raw data. The desensitized feature vectors are transmitted to the decision module for emotion assessment.

[0054] Decision module: Includes an AI risk assessment engine, used to build a personalized baseline model for patients, calculate the deviation of each modality feature from the baseline, and dynamically adjust the fusion weights based on the reliability assessment results of each modality's data quality, thereby fusing the data to obtain the emotional agitation index. Based on this, intervention instructions are output to the environmental intervention module;

[0055] Environmental intervention module: used to automatically adjust the environmental control devices in the ward to alleviate the patient's emotions after being triggered.

[0056] Interactive module: Provides a visual interface for medical staff to display the emotional agitation index in real time. Modal contribution information, alarm events and historical backtracking, and annotation feedback.

[0057] Preferably, the environmental control device includes a central controller and environmental peripheral devices connected thereto, including an intelligent lighting system, a network speaker, an aroma diffuser, and a vibration motor array installed inside the mattress.

[0058] Preferably, the edge processing unit further includes multimodal timestamp alignment and filtering denoising processing to synchronously align different sensor data before feature extraction and desensitization.

[0059] Preferably, the edge processing unit transmits the desensitized feature vector to the decision module for emotion assessment and restricts sensitive raw data from being stored or processed locally in the ward.

[0060] Preferably, the visualization interface of the interactive module displays at least the real-time curve of the emotional agitation index, the multimodal contribution graph, and the desensitized data summary, and can provide trend warning, event annotation and recording functions, and allow medical staff to review historical data.

[0061] Preferably, the intervention program is a tiered program, based on the emotional agitation index. The range selects different combinations of environmental parameters and their intensity.

[0062] The present invention also discloses a method for using the aforementioned system, comprising the following steps:

[0063] S1. Non-contact acquisition of patients' multimodal physiological and behavioral data;

[0064] S2. In the ward, perform time synchronization, preprocessing, feature extraction on multimodal physiological and behavioral data, and perform irreversible desensitization on sensitive original data to obtain desensitized feature vectors.

[0065] S3. Based on the desensitized feature vector, establish a personalized baseline model for the patient, determine the deviation of each modality feature from the baseline, and conduct a reliability assessment of the data quality of each modality.

[0066] S4. Based on the reliability assessment results, the fusion weights of each modality are dynamically adjusted, and the deviations of each modality are fused according to the fusion weights to obtain the emotional agitation index. ;

[0067] S5, When the emotional intensity index When the preset threshold is exceeded continuously for a preset time, the environmental control device adjusts the ward environment to alleviate the patient's emotions.

[0068] S6. Receive manual annotations from medical staff and perform incremental training based on these annotations;

[0069] S7. Roll back to a historical stable version model when performance degrades.

[0070] Step S5 involves measures to alleviate the patient's emotional state, including interventions such as dimming the lights, playing soothing music, releasing calming scents, and initiating gentle vibrations of the bed.

[0071] Preferably, the information collected in step S1 includes breathing / heart rate and body movement information, temperature distribution information, three-dimensional skeletal joint point or posture trajectory information, pitch / volume / speech rate and acoustic characteristics of crying or shouting.

[0072] Preferably, the desensitization feature vector in step S2 includes Mel frequency cepstral coefficients, skeleton key points, and depth feature vectors.

[0073] Preferably, the reliability assessment method in step S3 is as follows:

[0074]

[0075] in Rate usability For short-term consistency scoring, ,

[0076] The determination is based on at least one of the following: occlusion rate, frame drop rate, signal-to-noise ratio, echo quality, or effective sampling percentage of the modal data.

[0077] The modal characteristics are determined based on at least one of the following: variance, confidence fluctuation, correlation, or self-consistency index within the sliding time window.

[0078] Preferably, the fusion weights in step S4 The evaluation method is as follows:

[0079]

[0080] in .

[0081] Preferably, the emotional agitation index in step S4 The calculation method is as follows:

[0082] ;

[0083] The deviation is calculated as follows:

[0084]

[0085] in, For a moment modal eigenvalues, For personalized baseline mean, For individualized standard deviation, It is a very small positive number.

[0086] Preferably, the ward environment in step S5 includes lighting, volume, odor, or bed vibration.

[0087] Preferably, the emotional agitation index in step S5 The environmental intervention module is only triggered after the threshold is exceeded continuously for a duration not less than a preset duration, in order to avoid false alarms caused by instantaneous noise.

[0088] In actual use, the system hardware and modules of this invention are divided as follows:

[0089] An embedded edge processing unit, millimeter-wave radar, depth camera, thermal imaging sensor, microphone array, and environmental control device are installed in the ward. The environmental control device includes a central controller and peripheral environmental devices, including at least one of the following: a dimmable and color-adjustable intelligent lighting system, a network speaker, an automatic aroma diffuser, and a vibrating motor array inside the mattress.

[0090] The edge processing unit runs data acquisition and processing software, performing time synchronization, filtering and noise reduction, feature extraction and irreversible desensitization: for example, for audio, it only outputs acoustic features such as Mel frequency cepstral coefficients (MFCC); for video, it only outputs three-dimensional skeleton key points or deep feature vectors extracted by deep learning models.

[0091] The calculation and triggering intervention of the emotional agitation index in this invention are as follows:

[0092] The edge processing unit outputs modal features to the decision module at a fixed sampling period. The decision module establishes an individualized baseline during periods of relative patient stability and performs dynamic weighted fusion based on deviation and reliability assessments to obtain the emotional agitation index. .when The environmental intervention module is triggered when the threshold is exceeded continuously for a duration not less than a preset time.

[0093] The environmental intervention module adjusts at least one of the following based on a preset intervention plan: light, volume, odor, or bed vibration. For example, it can reduce light brightness or adjust color temperature, play soothing music, release calming odors, or activate gentle mattress vibration.

[0094] The interactive annotation, incremental learning, and rollback in this invention are as follows:

[0095] The interactive module provides medical staff with a visual interface that displays the real-time curve of the emotional agitation index, multimodal contribution information and desensitized data summary, and provides trend warning, event annotation and historical backtracking functions.

[0096] Medical staff can confirm or correct system alarms, creating a labeled dataset; the decision-making module uses this labeled data to incrementally train the risk assessment model. To ensure system stability, the system saves the parameters of historical stable versions of the model, and when incremental training leads to performance degradation, it restores to a historical stable version through a rollback mechanism.

[0097] The quality scoring, hysteresis, and graded intervention methods used in this invention to enhance robustness are as follows:

[0098] In scenarios with multiple users, occlusion, or strong noise, quality can be reflected and weight updates can be driven by modal availability and consistency metrics: for example, when the occlusion rate of a depth camera increases or the frame drop rate increases, its availability score is reduced, thereby reducing the fusion weight to reduce false alarms.

[0099] To reduce frequent triggers, a delay strategy can be adopted: when the emotional agitation index... Exceeding the trigger threshold continued Triggering intervention; when Below the release threshold continued The intervention is lifted when the time is right, and an intervention cooldown period is set. .

[0100] Interventions can employ tiered prescriptions: based on By selecting different combinations and intensities of environmental parameters within a range, a tiered response from mild reassurance to high-priority alarms can be achieved, and medical staff can take over and disable certain peripherals with a single click.

[0101] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. An intelligent emotion-soothing system based on multimodal perception and environmental intervention, characterized in that... Includes the following modules: Sensing module: used to collect multimodal physiological and behavioral data of patients, including two or more devices such as radar, thermal imaging sensors, depth cameras, and microphone arrays; Edge processing unit: used for time synchronization, preprocessing, denoising, feature extraction and irreversible desensitization of the collected multimodal raw data. The desensitized feature vectors are transmitted to the decision module for emotion assessment. Decision module: Includes an AI risk assessment engine, used to build a personalized baseline model for patients, calculate the deviation of each modality feature from the baseline, and dynamically adjust the fusion weights based on the reliability assessment results of each modality's data quality, thereby fusing the data to obtain the emotional agitation index. Based on this, intervention instructions are output to the environmental intervention module; Environmental intervention module: used to automatically adjust the environmental control devices in the ward to alleviate the patient's emotions after being triggered. Interactive module: Provides a visual interface for medical staff to display the emotional agitation index in real time. Modal contribution information, alarm events and historical backtracking, and annotation feedback.

2. The intelligent emotion-soothing system according to claim 1, characterized in that: The environmental control device includes a central controller and connected peripheral environmental devices, including an intelligent lighting system, network speakers, aroma diffusers, and a vibrating motor array installed inside the mattress.

3. A method using the intelligent emotion-soothing system according to claim 1 or 2, characterized in that... Includes the following steps: S1. Non-contact acquisition of patients' multimodal physiological and behavioral data; S2. In the ward, perform time synchronization, preprocessing, feature extraction on multimodal physiological and behavioral data, and perform irreversible desensitization on sensitive original data to obtain desensitized feature vectors. S3. Based on the desensitized feature vector, establish a personalized baseline model for the patient, determine the deviation of each modality feature from the baseline, and conduct a reliability assessment of the data quality of each modality. S4. Based on the reliability assessment results, the fusion weights of each modality are dynamically adjusted, and the deviations of each modality are fused according to the fusion weights to obtain the emotional agitation index. ; S5, When the emotional intensity index When the preset threshold is exceeded continuously for a preset time, the environmental control device adjusts the ward environment to alleviate the patient's emotions. S6. Receive manual annotations from medical staff and perform incremental training based on these annotations; S7. Roll back to a historical stable version model when performance degrades.

4. The method according to claim 3, characterized in that: Step S5 involves measures to alleviate the patient's emotional state, including interventions such as dimming the lights, playing soothing music, releasing calming scents, and initiating gentle vibrations of the bed.

5. The method according to claim 3, characterized in that: The information collected in step S1 includes breathing / heart rate and body movement information, temperature distribution information, three-dimensional skeletal joint point or posture trajectory information, pitch / volume / speech rate and acoustic characteristics of crying or shouting.

6. The method according to claim 3, characterized in that... In step S2, the desensitization feature vector includes Mel frequency cepstral coefficients, skeleton key points, and depth feature vectors.

7. The method according to claim 3, characterized in that... The reliability assessment method in step S3 is as follows: in Rate usability For short-term consistency scoring, .

8. The method according to claim 7, characterized in that... In step S4, the fusion weights The evaluation method is as follows: in .

9. The method according to claim 8, characterized in that... Emotional agitation index in step S4 The calculation method is as follows: ； The deviation is calculated as follows: in, For a moment modal eigenvalues, For personalized baseline mean, For individualized standard deviation, It is a very small positive number.

10. The method according to claim 3, characterized in that: In step S5, the ward environment includes lighting, volume, odor, or bed vibration.

Images