An interactive system based on emotion recognition

Abstract

The application relates to the technical field of intelligent interaction, in particular to an interaction system based on emotion recognition, which comprises the following steps: collecting multi-modal sensing data and environmental context of a user; when a first situation is matched, constructing a first situation feature by using each single-dimension feature deviation and corresponding coupling feature deviation calculated from the current multi-modal sensing data of the user; when the first situation is not matched, constructing a second situation feature by using each single-dimension feature deviation calculated from the current multi-modal sensing data of the user; an emotion risk index calculation module is used for fusing the first situation feature or the second situation feature according to a pre-trained fusion model to obtain an emotion risk index; and an interaction module is used for triggering an interaction strategy of a corresponding level according to the relationship between the emotion risk index and a latest emotion risk threshold. In the emotion recognition process, the situation category in which the user is currently located is fused, and the accuracy of emotion recognition is greatly improved.

Description

Technical Field

[0001] This application relates to the field of intelligent interaction technology, and in particular to an interaction system based on emotion recognition. Background Technology

[0002] Some individual users often exhibit poor emotional management skills, and existing technologies for user emotion management have at least the following limitations:

[0003] Relying on subjective observation and lacking objective, quantitative assessment methods, current systems struggle to accurately measure the intensity, frequency, and triggering factors of negative emotions, resulting in a lack of targeted coping strategies. Furthermore, current emotion recognition and interaction strategies cannot be linked to users' real-time multimodal sensor data, failing to dynamically understand the user's state, leading to low accuracy and high false alarm rates. Therefore, there is an urgent need in this field for a system capable of objectively quantifying and interacting with users' emotions, thereby achieving accurate emotion state recognition and natural real-time interaction to support long-term personalized adaptation and functional evolution. Summary of the Invention

[0004] The purpose of this application is to provide an interaction system based on emotion recognition. In the initial stage of operation, the system performs emotion recognition based on the user's personal baseline settings. During the emotion recognition process, the system continuously collects user interaction data and analyzes this data to identify high-risk situations associated with the user. This allows for the selection of appropriate emotion recognition schemes based on whether the user's current situation falls under the category of high-risk situations, thereby improving the accuracy of emotion recognition.

[0005] In some embodiments, this application provides an interaction system based on emotion recognition, the system comprising:

[0006] The multimodal sensor data acquisition module is used to collect multimodal sensor data from users.

[0007] The environment awareness module is used to obtain the environmental context during the interaction process of emotion recognition;

[0008] The context matching module is used to match a first context based on the environmental context;

[0009] The contextualized feature calculation module is used to construct first context features when a first context is matched, using the deviation of each single-dimensional feature calculated from the user's current multimodal sensor data and the corresponding coupling feature deviation. When a first context is not matched, it constructs second context features using the deviation of each single-dimensional feature calculated from the user's current multimodal sensor data. The risk of the user experiencing negative emotions in the first context is greater than the risk of the user experiencing negative emotions in the second context.

[0010] The emotional risk index calculation module is used to fuse the first situation features or the second situation features according to the pre-trained fusion model to obtain the emotional risk index.

[0011] The interaction module is used to trigger the corresponding level of interaction strategy based on the relationship between the emotional risk index and the latest emotional risk threshold, as well as the duration of the current emotional risk index.

[0012] In some embodiments, the contextualized feature calculation module further includes a first contextual single-dimensional feature calculation unit, a first contextual coupled feature calculation unit, and a second contextual single-dimensional feature calculation unit. The first contextual single-dimensional feature calculation unit is used to calculate multimodal sensing features based on the user's current multimodal sensing data, and to calculate a single-dimensional feature deviation based on the deviation of each modal sensing feature from the current first feature baseline of the corresponding modality. The first contextual coupled feature calculation unit is used to calculate multimodal sensing features based on the user's current multimodal sensor data, and to calculate a coupled feature deviation based on the offset between the multimodal sensing features and the current multimodal first feature baseline, as well as the correlation between the modal features. The second contextual single-dimensional feature calculation unit is used to calculate multimodal sensing features based on the user's current multimodal sensing data, and to calculate a single-dimensional feature deviation based on the deviation of each modal sensing feature from the second feature baseline of the corresponding modality.

[0013] In some embodiments, the contextualized feature calculation module further includes a first contextual feature baseline calculation unit and a second contextual feature baseline calculation unit; the first contextual feature baseline calculation unit is used to calculate the mean and standard deviation of each modal sensor data and the covariance between different modal sensor data based on the user's multimodal sensor data before emotion recognition interaction, and obtain a first feature baseline model of the user in the current first context based on the mean, standard deviation and covariance; the second contextual feature baseline calculation unit is used to acquire the user's multimodal sensor data before emotion recognition interaction, and obtain a second feature baseline model of the user in the current second context based on the mean and standard deviation of each modal sensor data.

[0014] In some embodiments, the first context feature baseline calculation unit further includes a first context feature baseline dynamic update subunit, and the second context feature baseline calculation unit further includes a second context feature baseline dynamic update subunit; the first context feature baseline dynamic update subunit is used to perform an exponentially weighted moving average on the mean of the multimodal first feature baseline using a phased learning rate factor, the latest obtained multimodal features, and the current multimodal first feature baseline to obtain a dynamic mean update, and to perform an exponentially weighted moving average on the standard deviation of the multimodal first feature baseline using a phased learning rate factor, the latest obtained multimodal features, and the current multimodal first feature baseline to obtain a dynamic standard deviation. The first feature baseline of the multimodal features is updated by using dynamic mean updates and dynamic standard deviation updates. The second feature baseline dynamic update subunit is used to perform an exponentially weighted moving average on the mean of the multimodal second feature baseline using a phased learning rate factor, the latest multimodal features, and the current multimodal second feature baseline to obtain a dynamic mean update. The second feature baseline standard deviation is also performed an exponentially weighted moving average on the standard deviation of the multimodal second feature baseline using a phased learning rate factor, the latest multimodal features, and the current multimodal second feature baseline to obtain a dynamic standard deviation update. The updated multimodal second feature baseline is obtained by using dynamic mean updates and dynamic standard deviation updates.

[0015] In some embodiments, the contextualized feature calculation module further includes a dynamic update judgment unit; the dynamic update judgment unit is used to determine whether the user is in the interaction strategy execution state, and if the user is not in the interaction strategy execution state, it triggers the update steps of the first feature baseline and the second feature baseline.

[0016] In some embodiments, the interaction module further includes a second context threshold calculation unit and a first context threshold calculation unit; the second context threshold calculation unit is used to calculate a first level threshold based on the latest second feature baseline and a first parameter, and to calculate a second level threshold based on the latest second feature baseline and a second parameter, wherein the first parameter is less than the second parameter, and the first level threshold is less than the second level threshold; the first context threshold calculation unit is used to calculate a third level threshold based on the latest first feature baseline and a third parameter, wherein the third parameter is less than the first parameter and the second parameter, and the third level threshold is less than the first level threshold and the second level threshold.

[0017] In some embodiments, the interaction module further includes a second contextual interaction strategy decision unit and a first contextual interaction strategy decision unit; the second contextual interaction strategy decision unit is used to trigger a first-level warning prompt interaction strategy when the emotional risk index in the second context is greater than a first-level threshold and continues for a preset duration, and to trigger a second-level actual interaction strategy when the emotional risk index in the second context is greater than a second-level threshold and continues for a preset duration; the first contextual interaction strategy decision unit is used to trigger the optimal interaction strategy in the first context when the emotional risk index in the first context is greater than a third-level threshold.

[0018] In some embodiments, the second contextual interaction strategy decision unit further includes a dynamic decision subunit; the dynamic decision subunit is used to obtain the user's environmental context, historical interaction effects, physiological state and preferences, and determine whether to enable the immersive interaction strategy based on the environmental context. If the immersive interaction strategy is enabled, the interaction strategy type is matched based on the historical interaction effects and preferences. If the immersive interaction strategy is not enabled, the cognitive task is reset.

[0019] In some embodiments, the system further includes a first context filtering module; the first context filtering module is used to obtain the environmental context within a preset time period before triggering the target-level interaction strategy, extract structured data from the environmental context, construct a transaction database based on each structured data, perform correlation mining on the transaction database to obtain the correlation between condition combinations and emotion warning events, calculate the support risk ratio corresponding to the correlation, and determine the condition combinations that meet the preset conditions as the first context.

[0020] In some embodiments, the context matching module further includes a condition combination matching unit; the condition combination matching unit is used to extract structured data from the environmental context, construct a transaction database based on each structured data, and match the transaction database with a preset first context condition combination. When the match is successful, it is determined that the user is currently in the first context; otherwise, it is determined that the user is currently in the second context.

[0021] In the above embodiments, during the initial operation of the interactive system, user emotion recognition is performed using initial settings adapted to the user's personal baseline. During the emotion recognition process, user interaction data is continuously collected, and analysis of this data identifies high-risk situations associated with the user. This allows for more precise selection of appropriate emotion recognition schemes based on whether the user's situation falls under the category of high-risk situations, thereby improving the accuracy of emotion recognition. Furthermore, highly personalized evaluation and judgment criteria are established. Personal dynamic baselines under different situations provide each user with a unique normal state benchmark, and contextualized sub-baselines further refine the evaluation criteria for different situations, improving the accuracy and reliability of emotion state assessment and anomaly detection. Attached Figure Description

[0022] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein:

[0023] Figure 1 This is a schematic diagram of an interaction system module based on emotion recognition provided in one embodiment of this application;

[0024] Figure 2 This is a schematic diagram showing an overview of the interaction relationships of a hardware system provided in one embodiment of this application;

[0025] Figure 3 This is a schematic diagram of an interaction system based on emotion recognition provided in another embodiment of this application;

[0026] Figure 4 This is a schematic diagram of an interaction system based on emotion recognition provided in another embodiment of this application;

[0027] Figure 5 This is a schematic diagram of a knowledge graph and adaptive interactive decision-making provided in one embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the interactive decision-making process in the secondary early warning mode provided in one embodiment of this application;

[0029] Figure 7 This is a schematic diagram of an interaction system based on emotion recognition provided in another embodiment of this application;

[0030] Figure 8 This is a logical schematic diagram of a training feature fusion model provided in another embodiment of this application;

[0031] Figure 9 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0033] The technical solutions of the various embodiments of this application can be combined with each other, but only if they are based on the ability of a person skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by this application.

[0034] This solution does not aim to obtain disease diagnosis results or health status, but is a method for processing users' multimodal sensor data and thereby achieving adaptive interaction. All steps are information processing methods implemented by devices such as computers.

[0035] It should be fully understood that the user information involved in this application (including but not limited to users' physiological data, action data, and personal information) is all information and data authorized by the user or fully authorized by all parties. The use of user information shall comply with the privacy policies and practices of the industry that are generally considered to meet or exceed the requirements for maintaining user privacy. The collection, use and processing of related data shall comply with relevant laws, regulations and standards, and provide corresponding operation access points for users to choose to authorize or refuse.

[0036] It should be noted that the emotions in this application can be negative emotions, specifically impulsive emotions. The impulsive emotions in the following embodiments are only one example for illustration, but in other embodiments, other types of emotions may also be present, which are not limited here.

[0037] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, specific embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0038] In some embodiments, see Figure 1 This application provides an interaction system 100 based on emotion recognition. The system includes a multimodal sensing data acquisition module 101, an environmental perception module 102, a context matching module 103, a contextual feature calculation module 104, an emotion risk index calculation module 105, and an interaction module 106, as detailed below.

[0039] The multimodal sensor data acquisition module 101 is used to acquire multimodal sensor data from users.

[0040] The environment perception module 102 is used to obtain the environmental context during the interaction process of emotion recognition.

[0041] The context matching module 103 is used to match the first context based on the environmental context.

[0042] The contextualized feature calculation module 104 is used to construct a first context feature when a first context is matched, by using the deviation of each single-dimensional feature calculated from the user's current multimodal sensor data and the corresponding coupling feature deviation. When a first context is not matched, a second context feature is constructed by using the deviation of each single-dimensional feature calculated from the user's current multimodal sensor data. The risk of the user experiencing negative emotions in the first context is greater than the risk of the user experiencing negative emotions in the second context.

[0043] The emotional risk index calculation module 105 is used to fuse the first situational features or the second situational features according to the pre-trained fusion model to obtain the emotional risk index.

[0044] The interaction module 106 is used to trigger the corresponding level of interaction strategy based on the relationship between the emotional risk index and the latest emotional risk threshold, as well as the duration of the current emotional risk index.

[0045] Specifically, the interactive system provided in this application includes six core functional modules, which form a complete optimized closed loop of monitoring, early warning and interaction through data flow and control flow.

[0046] The multimodal sensing data acquisition module 101 is the system's perception layer, responsible for synchronously acquiring neurophysiological, autonomic physiological, and behavioral data related to user emotion recognition, and extracting quantitative feature indicators. Specifically, the multimodal sensing data includes neural sensing data, physiological sensing data, and behavioral data.

[0047] See Figure 2 In actual implementation, the user-worn hardware collects relevant data. Core components include a headband integrating EEG (Electroencephalogram) and fNIRS (functional Near Infrared Spectroscopy), a multi-parameter smart wristband, a vibration motor, and an optional portable respiratory sensor. Its main functions are to achieve non-intrusive acquisition of neurophysiological, autonomic physiological, and motor data, and to receive and execute tactile alerts such as vibration.

[0048] The neural sensing data includes electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) data. Specifically, wearable headband devices integrating EEG and fNIRS technologies can be used to collect prefrontal cortex activity signals, enabling the monitoring of neural activity. Further processing of the EEG signals is also included to acquire raw EEG voltage signals. After preprocessing such as filtering and denoising, the power spectral density is calculated using a fast Fourier transform. Extracting the theta band power β-band power And calculate EEG indicators This EEG index reflects the real-time regulatory efficiency of the prefrontal cortex in attention and behavioral inhibition. An abnormally elevated ratio is a sensitive electrophysiological signal of insufficient cognitive control resources, inattention, and impulsive tendencies. The entropy of nonlinear feature samples is calculated; for an EEG signal segment of length N, a template length is set. With tolerance threshold Its sample entropy is calculated using the following formula: in, and The signals are respectively in peacekeeping The average similarity probability in the 1D vector space. This feature is used to quantify the complexity and regularity of EEG signals. Further processing includes data processing of the fNIRS signal to calculate the amplitude and temporal characteristics of changes in oxyhemoglobin in the prefrontal cortex.

[0049] Monitoring of physiological and behavioral data includes collecting physiological and behavioral data via a smart wristband. R-wave to R-wave interval sequences are extracted from electrocardiogram and pulse wave signals. Calculate the root mean square of successive differences (RMSSD) of the differences between adjacent RR intervals. N represents the total number of consecutive RR intervals within the selected ECG recording segment. This represents the RR interval between the i-th heartbeats. This represents the difference between two consecutive heartbeats, reflecting the instantaneous change in heart rate. LF / HF power ratio (low-frequency / high-frequency power ratio). The LF (Low Frequency) component is mainly regulated by both the sympathetic and parasympathetic systems, while the HF (High Frequency) component mainly reflects parasympathetic (vagus nerve) activity. The ratio of the two is often used to measure the balance between the sympathetic and parasympathetic systems. This represents the power in the 0.04 to 0.15 Hz frequency band obtained through power spectrum analysis. This represents the power in the 0.15 to 0.4 Hz frequency band obtained from power spectrum analysis. The skin conductance response frequency is calculated from the skin conductance signal. Calculate kinetic energy from a triaxial accelerometer. , , It refers to the instantaneous acceleration on the three spatial coordinate axes at time t.

[0050] This application also includes an environment perception module 102, used to obtain environmental context during the emotion recognition interaction process. See also Figure 2 The environmental perception module's environmental perception and interaction terminal can specifically be a smart terminal, such as a smart tablet or smartphone, which collects environmental context data. Environmental context data includes ambient sound pressure level, ambient light intensity, and task metadata. The ambient sound pressure level can be obtained by collecting ambient sound through a microphone. Ambient light intensity (Lux value) can be collected via a light sensor. Task metadata includes task type, difficulty level, and duration. Deeply integrated with educational and gaming applications, it can obtain structured task data in real-time through predefined API interfaces, including task identifier, task type, preset difficulty level, start and end timestamps, and real-time performance data such as accuracy and reaction time. Screen images are captured using the device's accessibility services or screen recording (user authorization required). A lightweight convolutional neural network is used for real-time image classification to identify common task scenarios. Combined with recognition technology, text keywords in the interface are identified, and a quick annotation interface is provided. Parents or teachers can manually select or input the current activity type on the system terminal.

[0051] It's important to note that environmental context data not only serves as monitoring input but is also deeply integrated into every aspect of the system. In the early warning logic, environmental context data is used to identify high-risk scenarios (the first scenario) and trigger dynamic threshold adjustments under these scenarios. In interactive decision-making, environmental context data is one of the bases for adapting interaction schemes, such as whether the current situation allows for immersive interaction. In causal analysis, it participates in the mining as a key dimension to locate external factors leading to impulsive actions. In system optimization, it drives the establishment of contextualized sub-baselines, enabling more refined and personalized management.

[0052] See Figure 2The collected multimodal sensing data and environmental context data serve as the main data link, continuously uploading raw or pre-processed feature data to the edge computing device via Bluetooth Low Energy. The multimodal sensing data acquisition module and the environmental context data perception module can also serve as backup data links; when the edge computing device is unavailable, data can be relayed to a tablet or mobile phone via Bluetooth. Furthermore, it can receive tactile warning commands from the edge computing device, driving the wristband motor to generate specific vibration patterns.

[0053] The core components of the environmental perception and interaction terminal include smart tablets or smartphones, which are mainly used to collect environmental data such as noise and light, run learning or game applications, provide task metadata, present interactive content such as games, animations, cognitive tasks, etc., and provide some local computing resources.

[0054] As the main interaction object and control terminal, the environmental perception and interaction terminal provides environmental sensor data, application logs, task types, difficulty, duration, and user interaction events to the edge computing device. It also receives instructions from the edge unit, updates the user interface, displays warning icons, or launches specific interactive programs such as neurofeedback games and mindfulness breathing guidance in full screen.

[0055] Under a specific architecture, the environmental perception and interaction terminal can act as a Bluetooth central node, aggregating sensor data and forwarding it uniformly to the edge unit (edge ​​computing device).

[0056] The data processing center, deployed locally (e.g., integrated into a tablet or a standalone home gateway), handles real-time computations, providing robust storage and computing power, and is responsible for non-real-time large-scale data analysis and model iteration. Its main functions include receiving and synchronizing multimodal sensor data; running personal dynamic baseline models, impulsive risk index (IRI) calculation models, and tiered early warning logic; making interactive decisions and issuing control commands; storing historical data; performing offline data mining; and training and optimizing algorithm models such as attention fusion models. Edge units periodically synchronize feature data, early warning events, and post-interaction results to the cloud; the cloud distributes updated algorithm models, high-risk scenario definitions, and configuration parameters to the edge units, enabling continuous evolution. The edge units act as the brain of the real-time closed loop, directly commanding the wearable devices and interactive terminals; the cloud serves as the intelligent hub, supporting long-term system optimization and collaborative services.

[0057] The core components of the user-end hardware include smartphones, computers, and smart devices for teachers, such as smartwatches and tablets. Its main functions include receiving system-generated reports, alerts, and personalized suggestions; remotely setting and viewing information; connecting to a cloud-based home-school collaboration service platform via the internet; and regularly receiving visualized user sentiment reports, weekly summaries, trigger analysis results, and data-driven action suggestions. Parents can provide feedback or adjust system preferences, and authorized teachers can confirm classroom reminders or access anonymized group data. Additionally, the system maintains a hardware interoperability matrix for selecting the decision engine.

[0058] Specifically, the interaction methods can include neurofeedback interaction, mindfulness breathing interaction, and cognitive task reset. Neurofeedback interaction requires an EEG headband and a display device; optional hardware includes a large-screen monitor and a game controller. If the required hardware is missing, it cannot be executed, and an alternative interaction method will be automatically selected. Mindfulness breathing interaction requires a display device and audio output; optional hardware includes a breathing sensor, bone conduction headphones, and a wristband (haptic). If optional hardware is missing, such as a breathing sensor, standard breathing rhythms can be used for guidance. Cognitive task reset requires a display device and a touchscreen; if these are missing, there are no alternative hardware options.

[0059] This application also includes hardware status monitoring and decision-making. It determines which devices are currently available based on device connectivity status, avoids interactions dependent on critical devices (such as headbands) when the battery level is below 20% by monitoring battery power, and avoids neural feedback interactions when sensor data quality monitoring, such as when the EEG signal quality index is below a threshold.

[0060] See Figure 3 This application provides an interaction system based on emotion recognition, the core of which lies in constructing a complete intelligent closed loop of perception, cognition, decision-making, execution, and optimization. The system synchronously collects the user's neural sensor data, physiological sensor data, and behavioral data through a multimodal sensor data acquisition module, in conjunction with the collected environmental context. The data processing and emotion risk assessment module integrates a context matching module, a contextualized feature calculation module, and an emotion risk index calculation module. It utilizes the user's dynamic baseline combined with contextualized sub-baselines for personalized state assessment and employs a multimodal fusion model to calculate the emotion risk index, i.e., the impulse risk index (IRI), in real time under the corresponding context. Furthermore, a tiered early warning module implements tiered early warnings for the impulse risk index IRI and drives context-aware adaptive interaction. Through data-driven causal analysis and effect evaluation, the system feeds back the analysis results to the preceding modules, dynamically optimizing the baseline, threshold, and interaction strategies, thereby achieving system self-evolution.

[0061] See Figure 4 This document presents a flowchart of an interaction system based on emotion recognition, as provided in this application. Specifically, a multimodal sensor data acquisition module collects the user's multimodal sensor data and extracts features from it to obtain a feature vector F_real. Based on the collected environmental context data, it further matches the scenario type to determine if the user is currently in a first scenario (high-risk scenario). If the user is in the first scenario, the corresponding parameter indicators are used to assess the emotional risk in that scenario. If the user is not in the first scenario, it is determined that the user is currently in a second scenario, and the corresponding parameter indicators are used to assess the emotional risk in that scenario. The parameter indicators for the first scenario include a first feature baseline, and the parameter indicators for the second scenario include a second feature baseline. The deviation of the user's current multimodal sensor features from the first or second feature baseline is used to calculate the first or second scenario features. Then, an attention-based fusion model is used to calculate the emotional risk index IRI (specifically, an impulsive risk index), and the corresponding interaction strategy is triggered based on the emotional risk index.

[0062] The pre-trained fusion model (multimodal fusion model) includes a regular fusion model and an enhanced fusion model. The regular fusion model is used for emotion recognition in the second context, while the enhanced fusion model is used for emotion recognition in the first context. The pre-trained fusion model needs to be trained offline, utilizing a labeled training dataset containing a large amount of data from users with emotional issues. Optimal internal parameters are learned through an optimization algorithm. After training, the fixed model parameters are deployed to the system for real-time impulsive risk assessment. The specific training steps are as follows:

[0063] The training data preparation phase includes data collection and labeling. A cohort of users with specific emotion types is recruited and have them wear the system's devices to collect data in simulated classroom and homework scenarios. During data collection, trained experts, such as child psychologists and behavior therapists, use one-way mirrors or video recordings to label the level of impulsive emotion (negative emotion) for each time segment in real-time or retrospectively. The labeling can employ one or more of the following methods: discrete labels, dividing impulsive intensity into multiple levels, e.g., 0 for normal, 1 for mild irritability, 2 for moderate impulsivity, and 3 for severe outburst; or continuous labels, where experts assign an impulsive intensity score on a continuous scale, such as 0 to 100, based on observation. These expert-labeled labels will serve as the gold standard for model learning.

[0064] In the data preprocessing and feature engineering stage, the collected raw multimodal data undergoes time alignment, noise reduction, and cleaning. Following the previous description of the data acquisition module, all feature metrics within each time window are calculated. For conventional fusion models, a feature vector is constructed for each sample point. and corresponding expert annotation labels .

[0065] In the model building and training phase, a neural network model containing embedding layers and attention layers is constructed, and all trainable parameters of the model are randomly initialized. The impulsive risk index predicted by the model is then used. Labeled by real experts The comparisons are made, and the model parameters are iteratively optimized using the backpropagation algorithm.

[0066] In another embodiment, in the augmented training model provided in this application, we randomly select a subset of samples from the training data. Besides constructing a feature vector F for these samples, we also calculate the Mahalanobis distance based on the global baseline and use this distance as an additional feature. This trained augmented model can learn "how to more accurately assess emotional risk when there exists a comprehensive indicator representing the overall degree of deviation." Although the Mahalanobis distance used here is calculated based on the global baseline, it has the same physical meaning as the Mahalanobis distance calculated based on sub-baselines in high-risk scenarios—both measure the overall deviation across multiple dimensions.

[0067] See details Figure 8 After obtaining the training dataset, if it is used to train a regular model, feature vectors need to be constructed based on the training dataset and then used to train the regular model. In this case, the model input does not include Mahalanobis distance features. If it is used to train an augmented model, not only do feature vectors need to be constructed based on the training dataset, but a subset of samples also needs to be selected from the training data to calculate the Mahalanobis distance based on the global baseline, and then the augmented model is trained.

[0068] The above describes the training of the fusion model. During the model deployment and application phase, the parameters of the trained and successfully evaluated fusion model will be fixed and integrated into the data processing and impulsive risk assessment module. In practical applications, the system automatically selects the corresponding model based on whether the current situation matches the first scenario. The system inputs the real-time calculated multimodal features (first scenario features or second scenario features) into the fixed fusion model, which can then output a reliable emotional risk index (i.e., impulsive risk index) based on the learned knowledge.

[0069] It's important to note that the risk of a user experiencing negative emotions in the first scenario is greater than in the second scenario. In other words, the first scenario is statistically analyzed based on the user's historical emotional data to identify situations more likely to induce negative emotions. Therefore, when a user is determined to be in the first scenario, specific and stringent parameters specific to that scenario are needed to calculate the user's impulsive risk index, in order to trigger corresponding coping strategies more promptly.

[0070] The interactive system provided in the above embodiments achieves a fundamental transformation from subjective experience-based judgment to objective data-driven approaches, from general interaction to personalized adaptation, and from isolated responses to a continuous optimization loop, providing a precise and intelligent digital therapy solution for managing impulsive emotions in specific users. Furthermore, by differentiating different situations and adopting corresponding calculation and judgment methods, it ensures that users receive timely attention in high-risk situations, reducing the risk of generating negative emotions.

[0071] In some embodiments, the contextualized feature calculation module further includes a first contextual single-dimensional feature calculation unit, a first contextual coupled feature calculation unit, and a second contextual single-dimensional feature calculation unit. The first contextual single-dimensional feature calculation unit is used to calculate multimodal sensing features based on the user's current multimodal sensing data, and to calculate a single-dimensional feature deviation based on the deviation of each modal sensing feature from the current first feature baseline of the corresponding modality. The first contextual coupled feature calculation unit is used to calculate multimodal sensing features based on the user's current multimodal sensor data, and to calculate a coupled feature deviation based on the offset between the multimodal sensing features and the current multimodal first feature baseline and the correlation between the modal features. The second contextual single-dimensional feature calculation unit is used to calculate multimodal sensing features based on the user's current multimodal sensing data, and to calculate a single-dimensional feature deviation based on the deviation of each modal sensing feature from the second feature baseline of the corresponding modality.

[0072] Specifically, the contextualized feature calculation module further includes a first contextual single-dimensional feature calculation unit, a first contextual coupled feature calculation unit, and a second contextual single-dimensional feature calculation unit.

[0073] When the user's current situation is determined to be the second situation based on the acquired multimodal sensing data and environmental context, the risk level of the user's current situation is judged to be low. In this case, only the single-dimensional feature deviation needs to be calculated. Specifically, the second situation single-dimensional feature calculation unit calculates multimodal sensing features based on the user's current multimodal sensing data, and calculates the single-dimensional feature deviation based on the deviation of each modality's sensing feature from the corresponding modality's second feature baseline. Specifically, the corresponding feature deviation is calculated for the i-th feature. When the total number of features is M, the feature deviations of all features constitute a feature deviation vector. By fusing different features through the model, a feature deviation fusion vector is obtained. , ,in, This is a pre-trained model with parameters θ. Further, based on the calculated feature deviation fusion vector, the emotional risk index (IRI) corresponding to the second scenario is calculated. ,in, This is a scaling parameter for the feature deviation, used to adjust the mapping relationship between the feature deviation and the impulsive risk index.

[0074] When the acquired multimodal sensing data and environmental context determine that the user's current situation is the first situation, the risk level of the user's current situation is judged to be high, and the user is more likely to experience negative emotions. In this case, it is necessary not only to calculate the single-dimensional feature deviation but also to further calculate the coupling feature deviation between multimodal data. Specifically, when the corresponding situation is the first situation, the first situation single-dimensional feature calculation unit is used to calculate multimodal sensing features based on the user's current multimodal sensing data, and to calculate the single-dimensional feature deviation based on the deviation of each modality's sensing feature from the corresponding modality's current first feature baseline. The first situation coupling feature calculation unit is used to calculate multimodal sensing features based on the user's current multimodal sensor data, and to calculate the coupling feature deviation based on the offset between the multimodal sensing features and the current multimodal first feature baseline, as well as the correlation between each modality's features.

[0075] Compared to the second scenario, in the first scenario, in addition to calculating the deviation of single-dimensional features... It also calculates the coupling feature deviation, that is, it uses the covariance matrix of the sub-baselines to calculate the multidimensional Mahalanobis distance. This measures the degree to which the current physiological state deviates from the normal clustering in this context. ,in This represents the real-time multimodal physiological feature vector of the user within the current time window. It is a vector containing core feature values ​​extracted from all physiological sensors. As an additional comprehensive feature, it is input into the subsequent fusion model along with the original single-dimensional feature deviation to obtain the emotional risk index, i.e., the impulsive risk index IRI, corresponding to the first scenario, making the risk assessment more accurate.

[0076] In the above embodiments, if the user is identified as being in a first scenario, the feature baseline of the first scenario is switched. That is, the system monitors the current environment and task metadata in real time. If a high-risk scenario is matched with a first scenario, the system automatically switches to the corresponding contextualized sub-baseline for feature calculation. By identifying the corresponding scenario type and switching the corresponding calculation method in different scenarios, the assessment of emotional risk becomes more accurate and effective.

[0077] In some embodiments, the contextualized feature calculation module further includes a first contextual feature baseline calculation unit and a second contextual feature baseline calculation unit; the first contextual feature baseline calculation unit is used to calculate the mean and standard deviation of each modal sensor data and the covariance between different modal sensor data based on the user's multimodal sensor data before emotion recognition interaction, and obtain a first feature baseline model of the user in the current first context based on the mean, standard deviation and covariance; the second contextual feature baseline calculation unit is used to acquire the user's multimodal sensor data before emotion recognition interaction, and obtain a second feature baseline model of the user in the current second context based on the mean and standard deviation of each modal sensor data.

[0078] First, multimodal sensor data of the user is collected before the emotion recognition interaction, i.e., when the user is in a calm state, and the feature data corresponding to the multimodal sensor data is calculated. For each feature Calculate the statistical distribution, establish the Gaussian line model, and set the baseline as follows: Thus, the first feature baseline model under the first scenario and the second feature baseline model under the second scenario are obtained.

[0079] Specifically, in the second scenario, in the initial stage of using the interactive system, such as before the system is started, a reliable multimodal feature baseline can be established for each modality in the user's multimodal sensing data to represent the individual's baseline starting point. This avoids misjudgments caused by using a uniform threshold. Specifically, this includes first collecting and calculating data. For example, the interactive system can set a calibration period, such as a preset initial calibration period of 3 to 7 days. The multimodal data sensing module continuously collects the user's multimodal sensing data in a natural state and obtains multimodal features, i.e., various feature data. After the calibration period, the system performs feature analysis on each mode. Perform statistical analysis on the entire dataset and calculate the initial mean. and initial standard deviation This forms a multimodal feature baseline, i.e., the initial baseline model. .

[0080] Specifically, in the first scenario, constructing a contextualized sub-baseline for a high-risk scenario includes: firstly, data filtering, i.e., extracting all data segments from the historical database that match the conditions of the high-risk scenario and where the user is in a calm state, such as an IRI consistently below a certain set threshold; and then calculating the mean vector of each physiological characteristic within this subset. and standard deviation vector Calculate the covariance matrix between features By storing and updating the correlation patterns of quantitative feature changes, the system can effectively transform and quantify these patterns. The model is stored as a sub-baseline model for this scenario and dynamically updated using an update algorithm.

[0081] This module is the core of the system's intelligent analysis, responsible for establishing an individual's dynamic baseline and transforming multi-source heterogeneous data into a unified impulsive risk index through a multimodal feature fusion model. It should be noted that in the above embodiments, different baseline construction methods are used for different scenarios to improve baseline accuracy.

[0082] In some embodiments, this application further includes dynamically updating the established contextual feature baseline to adapt and match it with the user's latest current state. Specifically, the first contextual feature baseline calculation unit further includes a first contextual feature baseline dynamic update subunit, and the second contextual feature baseline calculation unit further includes a second contextual feature baseline dynamic update subunit; the first contextual feature baseline dynamic update subunit is used to perform an exponentially weighted moving average on the mean of the multimodal first feature baseline using a phased learning rate factor, the latest obtained multimodal features, and the current multimodal first feature baseline to obtain a dynamic mean update; to perform an exponentially weighted moving average on the standard deviation of the multimodal first feature baseline using a phased learning rate factor, the latest obtained multimodal features, and the current multimodal first feature baseline to obtain a dynamic standard deviation update; and to obtain the updated multimodal first feature baseline using the dynamic mean update and the dynamic standard deviation update. The second contextual feature baseline dynamic update subunit is used to perform an exponentially weighted moving average on the mean of the multimodal second feature baseline using a phased learning rate factor, the latest obtained multimodal features, and the current multimodal second feature baseline to obtain a dynamic mean update. It also performs an exponentially weighted moving average on the standard deviation of the multimodal second feature baseline using a phased learning rate factor, the latest obtained multimodal features, and the current multimodal second feature baseline to obtain a dynamic standard deviation update. Finally, it uses the dynamic mean update and the dynamic standard deviation update to obtain the updated multimodal second feature baseline.

[0083] In some embodiments, the contextualized feature calculation module further includes a dynamic update judgment unit; the dynamic update judgment unit is used to determine whether the user is in an interaction strategy execution state, and if the user is not in an interaction strategy execution state, triggering an update step for the first feature baseline and the second feature baseline. The dynamic update judgment unit includes a judgment unit for dynamically updating the first contextual feature baseline and a judgment unit for dynamically updating the second contextual feature baseline.

[0084] In the above embodiments, the baseline is updated only when the user is in a stable state, thus ensuring the stability of the system.

[0085] In some embodiments, the interaction module further includes a second context threshold calculation unit and a first context threshold calculation unit; the second context threshold calculation unit is used to calculate a first level threshold based on the latest second feature baseline and a first parameter, and to calculate a second level threshold based on the latest second feature baseline and a second parameter, wherein the first parameter is less than the second parameter, and the first level threshold is less than the second level threshold; the first context threshold calculation unit is used to calculate a third level threshold based on the latest first feature baseline and a third parameter, wherein the third parameter is less than the first parameter and the second parameter, and the third level threshold is less than the first level threshold and the second level threshold.

[0086] Specifically, a tiered threshold can be set based on an individual's dynamic baseline, i.e., the latest feature baseline, and the corresponding interaction strategy can be triggered based on the tiered threshold.

[0087] In the second scenario, based on the latest second feature baseline and the first parameter The first-level threshold was calculated. According to the latest second characteristic baseline and the second parameter The second-level threshold was calculated. , where the first parameter Less than the second parameter First-level threshold Less than the second-level threshold Specifically, the first-level threshold This could be a threshold used for prompting, a second-level threshold. It is a threshold higher than the first-level threshold used to trigger interaction. Details are as follows. ,in , ,in .

[0088] In the first scenario, i.e., the high-risk scenario, the system uses the sub-baseline parameters of this scenario to dynamically calculate a more sensitive early warning update threshold. , ,in, Update parameters in the first scenario and , and These are the mean and standard deviation of the resting-state IRI in this scenario. Because and In other words, updating parameters in the first scenario The update parameter is less than the second scenario. as well as Therefore, the interactive system can issue an alarm earlier in the dangerous environment corresponding to the first scenario.

[0089] In practice, a minimum duration criterion can be set. For example, it can be set to 5 to 10 seconds to prevent false alarms due to momentary interference.

[0090] In some embodiments, the interaction module further includes a second contextual interaction strategy decision unit and a first contextual interaction strategy decision unit; the second contextual interaction strategy decision unit is used to trigger a first-level warning prompt interaction strategy when the emotional risk index in the second context is greater than a first-level threshold and continues for a preset duration, and to trigger a second-level actual interaction strategy when the emotional risk index in the second context is greater than a second-level threshold and continues for a preset duration; the first contextual interaction strategy decision unit is used to trigger the optimal interaction strategy in the first context when the emotional risk index in the first context is greater than a third-level threshold.

[0091] In practice, the tiered early warning module implements a tiered early warning strategy based on the level and duration of the emotional risk index, also known as the impulsive risk index (IRI), to achieve optimal interaction with minimal interference. (See also...) Figure 5 In the second scenario, the second scenario interaction strategy decision unit is used when the emotional risk index in the second scenario... Greater than the first level threshold And continue for the preset duration When the first-level warning and alert interaction strategy is triggered, the emotional risk index in the second scenario will be adjusted accordingly. Greater than the second-level threshold And continue for the preset duration At that time, the actual interaction strategy at the second level is triggered.

[0092] The tiered response strategy is as follows: Level 1 alert is the notification level, and the triggering condition is... And continue The first-level warning interaction strategy involves a short, gentle vibration from the wristband, an amber-colored warning icon displayed on the smart terminal, and the system entering a ready state to preload the interaction module. The second-level warning involves the actual execution of the interaction, triggered by the following conditions: And continue The second level of actual interaction strategies includes continuous strong vibration of the wristband, automatic triggering of the adaptive interaction module, and optional notification to the caregiver (with authorization).

[0093] In the above embodiments, when the system detects that a user has entered the first scenario, i.e., a high-risk scenario, such as starting a math assignment, the decision engine will preload the most effective interaction strategy historically used in that scenario. Simultaneously, the priority of this effective strategy will be increased in the decision weighting for that scenario. Preventative prompts, combined with metadata such as the duration of the scenario, allow the system to proactively provide preventative prompts before the risk accumulates to a threshold, such as suggesting a break after 40 minutes of studying, thereby avoiding potential subsequent emotional risks.

[0094] In some embodiments, the second contextual interaction strategy decision unit further includes a dynamic decision subunit; the dynamic decision subunit is used to obtain the user's environmental context, historical interaction effects, physiological state and preferences, and determine whether to enable the immersive interaction strategy based on the environmental context. If the immersive interaction strategy is enabled, the interaction strategy type is matched based on the historical interaction effects and preferences. If the immersive interaction strategy is not enabled, the cognitive task is reset.

[0095] The environmental context includes information such as location and current activity. Location can be determined through device positioning or network connection, such as whether the user is at home or at school. If at home or in a clinic, the environment is controllable, and neurofeedback or mindful breathing, which require a certain degree of immersion, can be prioritized. If in a classroom or public place, then interference should be minimized, and cognitive task interruption and reset are the most discreet and quickest options. Current activity includes whether the user is engaged in a cognitively demanding task such as doing homework, in which case cognitive task reset is most appropriate; if during a break between classes, mindful breathing can be tried.

[0096] Historical interaction effectiveness can be obtained through quantitative evaluation, aiming to objectify and measurable the interaction effect. The efficiency of a single interaction is calculated as follows: ,in This indicates an impulsive risk index before interaction. It represents the impulse risk index after interaction, and stores each interaction record, including interaction type, context, and effectiveness rate, in the database for subsequent efficacy analysis and decision optimization.

[0097] User profiles and historical response data include preferences, historical therapeutic data, and fatigue and compliance levels. Preferences include pre-set preferences by the user or parent, such as a preference for games or breathing exercises. Historical interaction data is maintained by the system backend as an interaction effectiveness matrix, recording the rate and magnitude of decrease in the emotional risk index after each use of a particular interaction. For example, if data shows that a user's response to neurofeedback is consistently better than mindful breathing, the system will prioritize recommending the former. Fatigue and compliance are managed to prevent boredom from continuous use of the same interaction method. The system will rotate interactions appropriately to maintain novelty.

[0098] The assessment of physiological status includes evaluating parameters such as physiological arousal and galvanic skin response. Specifically, if real-time data indicates extremely high physiological arousal, such as extremely low heart rate variability (HRV) and extremely high galvanic skin response (GSR), mindful breathing may be recommended first to quickly lower body temperature. If data shows that distraction is the main cause (the ratio of the theta band power to the beta band power, TBR, is elevated), then neural feedback may be more targeted.

[0099] See details Figure 6 This is a schematic diagram of the interaction decision-making process in a level-two warning mode provided in one embodiment of this application. The system does not randomly select interaction strategies, but rather uses a decision engine that comprehensively considers real-time and historical information. Its selection logic can be simulated as follows: Specifically, when a level-two warning interaction is determined, an actual interaction strategy needs to be implemented. This requires inputting environmental context information, historical interaction data, user preferences, and real-time physiological state data. Based on this data, a decision is made to determine whether an immersive interaction strategy can be executed. If it is determined that an immersive interaction strategy can be executed, then either neurofeedback or mindful breathing can be selected. The final interaction strategy is determined through historical interaction strategies. For example, if neurofeedback is determined to be the best, then neurofeedback is executed; if mindful breathing is determined to be the best, then mindful breathing is executed. If there is no difference between different interaction strategies, then different interaction strategies can be selected in rotation. Conversely, if it is determined that an immersive interaction is not suitable, then cognitive task reset is prioritized.

[0100] Figure 6The adaptive interaction module provided can intelligently select and execute the most suitable personalized interaction strategy based on the warning level, environmental context, and the user's historical data. Specifically, it includes a neurofeedback interaction unit that maps real-time EEG features to game control parameters. When TBR (Total Brain Pressure) increases, the game character (such as a spaceship) becomes unstable. The user stabilizes the character by calming emotions and concentrating to lower TBR, dynamically adjusting the control threshold based on historical performance. The adaptive interaction module also includes an immersive mindfulness breathing interaction unit, a multi-sensory guidance unit. Visually, it presents animations synchronized with breathing, such as flowers opening and closing; auditorily, it plays rhythmic guiding voices and soothing music; tactilely, the wristband vibrates in sync with breathing; and biofeedback monitors actual breathing through a breathing sensor, providing real-time corrective feedback. The adaptive interaction module also includes a cognitive task interruption and reset unit. The task design principles are: simplicity, short duration (<60 seconds), and the need for cold cognition. Typical tasks include 3×3 jigsaw puzzles, short sequence memory, and simple classification. The process involves interrupting the current task, executing the cognitive task, and returning to the original task upon successful completion.

[0101] Through this dynamic and intelligent decision-making mechanism, the adaptive interaction module ensures that every interaction is not mechanical or one-size-fits-all, but truly tailored to the user at the current moment, thereby significantly improving the acceptance and long-term effectiveness of the interaction.

[0102] In some embodiments, the system further includes a first context filtering module; the first context filtering module is used to obtain the environmental context within a preset time period before triggering the target-level interaction strategy, extract structured data from the environmental context, construct a transaction database based on each structured data, perform correlation mining on the transaction database to obtain the correlation between condition combinations and emotion warning events, calculate the support risk ratio corresponding to the correlation, and determine the condition combinations that meet the preset conditions as the first context.

[0103] Specifically, in the initial stage of operation, the interactive system of this application performs user emotion recognition with initial settings adapted to the user's personal baseline. During the process of emotion recognition, the system continuously collects user interaction data and analyzes this data to identify high-risk situations associated with the user. Thus, in subsequent emotion recognition processes, the system further determines the user's current situation category based on the acquired user data and selects the appropriate emotion recognition algorithm accordingly.

[0104] Specifically, the situational category identification in this application is a potential trigger analysis. Specifically, high-risk situations are defined as the first situation. This step is crucial for the system's intelligence, aiming to automatically and accurately discover regular triggers that lead to negative emotions, such as impulsiveness, from massive amounts of data. Specifically, this is achieved by aggregating multi-dimensional contextual data within a time window preceding each negative emotional event (defined as an event triggering a secondary warning), such as task attributes (type, difficulty, duration), environmental conditions (noise, lighting), and time information (time period), forming structured records, integrating them into a transaction, and constructing a transaction database ID, as shown in the example below.

[0105] Table 1 Transaction Database

[0106]

[0107] Further analysis of the data using association rule mining algorithms was conducted to identify specific combinations of conditions that frequently co-occur with negative emotional impulsive events (denoted as T). For example, the itemset {mathematical task, duration > 20 minutes, impulsive event} was found to be frequently occurring. Further, causal rules were generated and risks quantified. Specifically, for each frequently occurring itemset, association rules in the form of condition combinations and impulsive events were generated, and the support-risk ratio was calculated. The support-risk ratio includes quantitative indicators such as support, confidence, and relative risk ratio. Support is the frequency with which all conditions in the rule co-occur with the impulsive event. Let the condition combination be... Impulsive events are The formula for calculating support is: ,in To contain simultaneously and The number of transactions, Let be the total number of transactions, and let be the confidence level, which is the conditional probability of an impulsive event occurring given a certain combination of conditions. , , It is a combination of conditions in the database. The number of transactions, the relative risk ratio is the ratio of the risk of an impulsive event occurring under this combination of conditions to the risk of not being exposed. ,in For conditional combinations The probability of an impulsive event occurring when it does not occur.

[0108] The first scenario, or high-risk scenario, is based on a preset quantitative threshold (such as support level). 5% Confidence level 60%, relative risk ratio Version 2.0 automatically defines the combination of conditions that meets the criteria as the first scenario, i.e., the high-risk scenario.

[0109] In some embodiments, the context matching module further includes a condition combination matching unit; the condition combination matching unit is used to extract structured data from the environmental context, construct a transaction database based on each structured data, and match the transaction database with a preset first context condition combination. When the match is successful, it is determined that the user is currently in the first context; otherwise, it is determined that the user is currently in the second context.

[0110] In the above embodiments, high-risk scenarios corresponding to users are obtained by analyzing users' historical data. When users' current data is collected, the category of the current scenario can be evaluated. When it is determined that the user is currently in the first scenario corresponding to the high-risk scenario, the corresponding emotion recognition is performed according to the high-risk scenario method.

[0111] In some embodiments, the method provided in this application further includes long-term trend analysis, with core indicators including daily impulsive event frequency, such as the number of times the IRI exceeds the interaction threshold; average impulsive intensity, such as the average IRI during the impulsive event period; self-regulation success rate, such as the proportion of cases that do not escalate to a level two warning after a level one warning; and visualization, such as the system generating daily, weekly, and monthly trend charts to visually demonstrate the user's progress. These charts can serve as digital biomarkers to assist doctors in assessing the overall effectiveness of medication or behavioral therapy.

[0112] See Figure 7 In one specific embodiment, the user's multimodal sensor data is used and feature extraction is performed to obtain multimodal features. Based on the obtained environmental context, the current situation type of the user is determined. If it is determined that the user is currently in a high-risk situation corresponding to the first situation, then the dynamic baseline belonging to the high-risk situation is selected. Otherwise, the second situation, i.e. the ordinary dynamic baseline, is selected, and the emotional risk index under the corresponding situation is calculated. The final interaction strategy is determined based on the emotional risk index.

[0113] Specifically, this includes extracting real-time feature vectors from raw data. The system determines whether the current environmental task context matches a predefined high-risk scenario. Based on the matching result, it selects the corresponding baseline parameters. Specifically, if a high-risk scenario is matched, a contextualized sub-baseline and covariance matrix are used. If there is no match, the global personal baseline, i.e., the baseline parameters in the second scenario, is used. Then, when a high-risk scenario is matched, contextualized feature calculation is performed, calculating the single-dimensional feature deviation. Calculate the multidimensional Mahalanobis distance Further construct an enhanced feature vector by combining the single-dimensional feature deviation vector S with a multi-dimensional distance scalar. Combined, forming an enhanced feature vector. Continue calculating the emotional risk index, also known as the impulsive risk index (IRI), and enhance the feature vector. The input is fed into a pre-trained multimodal fusion model, whose parameters have learned during training how to balance different features. All information, including the independent deviation of each feature and the overall comprehensive deviation, is fused and judged. Finally, the model outputs a scalar between 0 and 1, namely the emotional risk index (impulse risk index IRI).

[0114] In the above embodiments, under high-risk scenarios, the system not only switches the mean and variance references, but also adds correlation. References make risk assessments more accurate in this context.

[0115] This application also includes a home-school collaboration data platform with in-depth reports for parents, data visualization capabilities to provide daily or weekly trend charts of children's impulse frequency and interaction success rates; core insight push notifications to clearly display the latest high-risk situation analysis results, such as this week's math homework being the main challenge; and personalized action suggestion functions to provide specific suggestions based on data, such as data showing that children are more easily distracted in noisy environments, suggesting the creation of a quiet learning environment.

[0116] This application also includes a teacher-side collaboration interface (authorized), featuring security and privacy protection functions. It displays the overall status trends of students requiring attention in a de-identified manner, rather than specific diagnostic information. It provides classroom support suggestions, offering data-driven adjustments to teaching strategies; for example, breaking down long tasks into smaller ones might help student A maintain focus. Finally, it serves as a communication bridge, providing secure communication capabilities that allow parents and teachers to communicate based on shared data facts, ensuring consistent interaction strategies between home and school.

[0117] Through this module, the system completes a full value loop from data collection to intelligent analysis, and then to action optimization and collaborative management, making it not only a monitoring tool, but also an intelligent partner that can continuously learn, grow, and empower users, families, and schools.

[0118] In a specific context, Attention Deficit Hyperactivity Disorder (ADHD) is one of the most common neurodevelopmental disorders in childhood, with impulsivity being one of its three core symptoms. Current management of impulsive emotions in ADHD has the following limitations: Subjectivity and lag, relying on subjective observations and scale assessments by parents and teachers, making it impossible to identify emotional fluctuations in real time and objectively. Lack of objective quantification, making it difficult to accurately measure the intensity, frequency, and triggering factors of impulsive emotions, resulting in a lack of targeted intervention measures. Limited interaction methods; traditional behavioral interactions and drug treatments cannot be linked to the user's real-time physiological state, failing to achieve precise interaction.

[0119] Management is fragmented, and management strategies in different scenarios such as families and schools lack coordination, making it difficult to form a consistent support environment. Although technologies such as electroencephalography (EEG) and near-infrared spectroscopy (fNIRS) have been used in ADHD research, existing technologies are mostly limited to single detection in laboratory environments, lacking long-term, dynamic monitoring capabilities in natural scenarios, and failing to form a complete, self-optimizing closed-loop system with real-time interaction.

[0120] On the one hand, the embodiments of this application can overcome a series of shortcomings in the prior art, such as reliance on subjective observation, lack of objective quantitative data, disconnect between interaction methods and physiological state, and fragmented management scenarios, in managing impulsive emotions in children with ADHD. It provides a closed-loop system and method that can monitor the impulsive emotions of children with ADHD in real time, without disturbance, and objectively, provide accurate early warnings and personalized interactions before emotional outbursts, and achieve system self-optimization through data analysis. On the other hand, the embodiments of this application can also overcome the problem in the prior art that users who need to manage emotions (such as drivers working in closed environments for extended periods) lack monitoring and interaction methods. It provides a closed-loop system and method that can identify emotions and provide personalized interactions for such users, enabling real-time, without disturbance, and objective monitoring of user emotions, providing accurate early warnings and personalized interactions before emotional outbursts, and achieving system self-optimization through data analysis.

[0121] To address the aforementioned technical challenges, this application proposes an objective monitoring system based on multimodal data fusion. By integrating simultaneous acquisition of EEG, near-infrared spectroscopy (fNIRS), heart rate variability (HRV), skin conductance response (GSR), motion sensing, and environmental perception (noise, light, task metadata), a multi-dimensional and complementary objective data monitoring system is constructed, resolving the issue of monitoring subjectivity. A modeling method combining a personal dynamic baseline with contextualized sub-baselines is proposed. A personal dynamic physiological baseline for each user in a calm state is established and updated using an exponentially weighted moving average algorithm. Furthermore, high-risk situations are identified through data mining, and contextualized sub-baselines containing mean, standard deviation, and covariance matrices are established for these high-risk situations, providing a personalized and contextualized dual benchmark for accurate anomaly detection. A multimodal fusion and hierarchical early warning model based on an attention mechanism is provided. A neural network model based on an attention mechanism is constructed, which adaptively weighs the contribution of different modal features to the current impulsive state, outputting a unified impulsive risk index (IRI). Based on the IRI and the personalized contextualized baseline, a hierarchical early warning strategy with dynamically adjustable thresholds is implemented. This paper proposes a context-aware adaptive interaction decision engine. The engine comprehensively considers real-time physiological state, environmental context (location, noise, task), historical interaction efficacy, hardware availability, and user preferences. It dynamically selects and executes the optimal interaction from various options, including neurofeedback, mindful breathing, and cognitive reset. A data-driven trigger analysis and closed-loop optimization mechanism is also proposed. Using association rule mining and other methods, the engine automatically analyzes potential triggers (environment, task, etc.) for impulsive emotions and feeds the analysis results back to preceding modules. This drives the system to perform adaptive optimization, such as creating contextualized sub-baselines, dynamically adjusting warning thresholds, and preloading interaction strategies, forming a self-improving closed loop.

[0122] By implementing the aforementioned technical protection points, this application has achieved the following significant technical effects: It realizes objective, real-time, and forward-looking monitoring of impulsive emotions. Through the synchronous collection and fusion analysis of multimodal physiological and environmental data, emotion monitoring is transformed from subjective and delayed to objective and real-time. Furthermore, by comparing personal baselines, it can identify abnormal precursors before emotional outbursts, achieving forward-looking early warning. It establishes highly personalized assessment and judgment standards. Personal dynamic baselines provide each user with a unique normal state benchmark, and contextualized sub-baselines further refine the judgment standards for different scenarios, improving the accuracy and reliability of emotional state assessment and abnormality detection. It provides precise and adaptive personalized interaction. Interaction decisions are no longer static but can be dynamically adjusted according to the specific state of the person, place, and time, ensuring the relevance and timeliness of the interaction, thereby significantly improving the acceptability and immediate effect of the interaction. It forms an intelligent and sustainably optimized management closed loop. By automatically learning the triggers from data and feeding back to optimize system parameters, the system possesses the ability to self-evolve. Meanwhile, the home-school collaboration platform generates reports and suggestions based on objective data, breaking down barriers in management scenarios, realizing evidence-based collaborative management, and ultimately improving long-term interaction results.

[0123] It is understood that the computer device used to implement the user interaction system solution provided in this application can be a server, and its internal structure diagram can be as follows: Figure 9 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The database stores relevant data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the method provided in this application.

[0124] Those skilled in the art will understand that Figure 9The structures shown are merely block diagrams of a portion of the structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. The computer device may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program may include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0125] It should be understood that the processor mentioned in the embodiments of this application can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0126] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An emotion recognition based interactive system, characterized in that, The system includes: The multimodal sensor data acquisition module is used to collect multimodal sensor data from users. The environment awareness module is used to obtain the environmental context during the interaction process of emotion recognition; The context matching module is used to match a first context based on the environmental context; The contextualized feature calculation module is used to construct first-context features when a first context is matched, using the deviation of each single-dimensional feature calculated from the user's current multimodal sensor data and the corresponding coupling feature deviation. When no first context is matched, it constructs second-context features using the deviation of each single-dimensional feature calculated from the user's current multimodal sensor data, wherein the risk of the user experiencing negative emotions in the first context is greater than the risk of the user experiencing negative emotions in the second context. The contextualized feature calculation module further includes a first-context single-dimensional feature calculation unit, a first-context coupling feature calculation unit, and a second-context single-dimensional feature calculation unit. The first-context single-dimensional feature calculation unit is used to calculate features based on the user's current... Multimodal sensing data is used to calculate multimodal sensing features, and a single-dimensional feature deviation is calculated based on the deviation of each modal sensing feature from the current first feature baseline of the corresponding modality. The first context coupling feature calculation unit is used to calculate multimodal sensing features based on the user's current multimodal sensor data, and to calculate the coupling feature deviation based on the offset between the multimodal sensing features and the current first multimodal feature baseline, as well as the correlation between the features of each modality. The second context single-dimensional feature calculation unit is used to calculate multimodal sensing features based on the user's current multimodal sensing data, and to calculate the single-dimensional feature deviation based on the deviation of each modal sensing feature from the second feature baseline of the corresponding modality. The emotional risk index calculation module is used to fuse the first situation features or the second situation features according to the pre-trained fusion model to obtain the emotional risk index. The interaction module is used to trigger the corresponding level of interaction strategy based on the relationship between the emotional risk index and the latest emotional risk threshold, as well as the duration of the current emotional risk index.

2. The system of claim 1, wherein, The contextualized feature calculation module further includes a first contextual feature baseline calculation unit and a second contextual feature baseline calculation unit; The first contextual feature baseline calculation unit is used to calculate the mean and standard deviation of each modal sensor data and the covariance between different modal sensor data based on the user's multimodal sensor data before emotion recognition interaction, and to obtain the first feature baseline model of the user in the current first context based on the mean, standard deviation and covariance. The second contextual feature baseline calculation unit is used to acquire the user's multimodal sensor data before emotion recognition interaction, and to obtain the user's second feature baseline model in the current second context based on the mean and standard deviation of each modal sensor data.

3. The system of claim 2, wherein, The first context feature baseline calculation unit further includes a first context feature baseline dynamic update subunit, and the second context feature baseline calculation unit further includes a second context feature baseline dynamic update subunit; The first context feature baseline dynamic update subunit is used to perform an exponentially weighted moving average on the mean of the multimodal first feature baseline using a phased learning rate factor, the latest obtained multimodal features, and the current multimodal first feature baseline to obtain a dynamic mean update; and to perform an exponentially weighted moving average on the standard deviation of the multimodal first feature baseline using a phased learning rate factor, the latest obtained multimodal features, and the current multimodal first feature baseline to obtain a dynamic standard deviation update; and to obtain the updated multimodal first feature baseline using the dynamic mean update and the dynamic standard deviation update. The second contextual feature baseline dynamic update subunit is used to perform an exponentially weighted moving average on the mean of the multimodal second feature baseline using a phased learning rate factor, the latest obtained multimodal features, and the current multimodal second feature baseline to obtain a dynamic mean update. It also performs an exponentially weighted moving average on the standard deviation of the multimodal second feature baseline using a phased learning rate factor, the latest obtained multimodal features, and the current multimodal second feature baseline to obtain a dynamic standard deviation update. Finally, it uses the dynamic mean update and the dynamic standard deviation update to obtain the updated multimodal second feature baseline.

4. The system of claim 3, wherein, The contextualized feature calculation module also includes a dynamic update judgment unit; The dynamic update judgment unit is used to determine whether the user is in the interaction strategy execution state. If the user is not in the interaction strategy execution state, the update steps of the first feature baseline and the second feature baseline are triggered.

5. The system of claim 1, wherein, The interaction module further includes a second context threshold calculation unit and a first context threshold calculation unit; The second context threshold calculation unit is used to calculate a first level threshold based on the latest second feature baseline and the first parameter, and to calculate a second level threshold based on the latest second feature baseline and the second parameter, wherein the first parameter is less than the second parameter, and the first level threshold is less than the second level threshold; The first context threshold calculation unit is used to calculate a third-level threshold based on the latest first feature baseline and the third parameter, wherein the third parameter is less than the first parameter and the second parameter, and the third-level threshold is less than the first-level threshold and the second-level threshold.

6. The system of claim 5, wherein, The interaction module also includes a second contextual interaction strategy decision unit and a first contextual interaction strategy decision unit. The second situational interaction strategy decision unit is used to trigger a first-level warning prompt interaction strategy when the emotional risk index in the second situation is greater than the first-level threshold and continues for a preset duration, and to trigger a second-level actual interaction strategy when the emotional risk index in the second situation is greater than the second-level threshold and continues for a preset duration. The first contextual interaction strategy decision unit is used to trigger the best interaction strategy in the first context when the emotional risk index in the first context is greater than the third level threshold.

7. The system of claim 6, wherein, The second contextual interaction strategy decision-making unit also includes a dynamic decision-making subunit; The dynamic decision-making subunit is used to obtain the user's environmental context, historical interaction effects, physiological state and preferences, and determine whether to enable the immersive interaction strategy based on the environmental context. If the immersive interaction strategy is enabled, the interaction strategy type is matched according to the historical interaction effects and preferences. If the immersive interaction strategy is not enabled, the cognitive task is reset.

8. The system according to claim 1, characterized in that, The system also includes a first context filtering module; The first scenario filtering module is used to obtain the environmental context within a preset time period before triggering the target-level interaction strategy, extract structured data from the environmental context, construct a transaction database based on each structured data, perform correlation mining on the transaction database to obtain the correlation between the condition combination and the emotion warning event, calculate the support risk ratio corresponding to the correlation, and determine the condition combination that meets the preset conditions as the first scenario.

9. The system according to claim 8, characterized in that, The context matching module also includes a condition combination matching unit; The condition combination matching unit is used to extract structured data from the environmental context, construct a transaction database based on each structured data, and match the transaction database with a preset first scenario condition combination. When the match is successful, it is determined that the user is currently in the first scenario; otherwise, it is determined that the user is currently in the second scenario.

Images