Method for driving emotion interaction of intelligent device based on multi-modal understanding

Abstract

The invention relates to the technical field of data processing, in particular to a method for driving emotion interaction of an intelligent device based on multi-modal understanding, and aims to eliminate illumination and noise interference and output a standardized face video stream, an effective voice segment and a touch thermodynamic diagram through an environment adaptive acquisition module. The feature extraction module extracts facial action optical flow features, voice Mel-frequency cepstral coefficient vectors and tactile pressure gradient parameters. The cross-modal correlation model adopts a tensor decomposition algorithm to calculate a space-time correlation matrix of visual and voice features, and the tactile feature weight is dynamically adjusted in combination with environmental parameters. According to the response strategy, an intervention scheme is retrieved based on a graph database, emotion confirmation statements, guide statements and behavior suggestions are fused to generate multi-mode response, and PID adjustment of the temperature control device and tactile pulse output of the vibration device are synchronously driven. And the feedback evaluation module verifies the emotion recognition consistency through a Pearson's correlation coefficient, triggers conflict sample separation storage and model increment training, and realizes closed-loop optimization.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for driving emotional interaction in intelligent devices based on multimodal understanding. Background Technology

[0002] Smart devices driven by multimodal understanding integrate technologies such as computer vision, speech recognition, and tactile sensing to collaboratively analyze visual, auditory, and tactile input information in the environment, forming a comprehensive semantic inference of user behavior. Using machine learning frameworks to train models for multi-source data association and intent prediction, the system can adaptively generate response strategies, support natural interaction processes, promote children's cognitive and social skills development, optimize the gaming experience, and enhance the effectiveness of entertainment-based education. In the field of affective computing, existing systems struggle to effectively integrate multimodal data for accurate emotion recognition. This challenge stems from the diverse characteristics of heterogeneous data from multiple sources and the high complexity of emotional states. Existing algorithms fail to achieve collaborative feature extraction and comprehensive semantic understanding when processing different modalities such as visual, speech, and physiological signals, resulting in insufficient robustness of emotion classification models. For instance, in electronic digital data processing systems, when a user's facial expressions and speech input exhibit contradictory characteristics, the system may be unable to reliably infer the true emotional state, such as misinterpreting a forced smile as a positive emotion, thereby affecting the quality of subsequent personalized emotion support service decisions. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a method for emotional interaction in intelligent devices based on multimodal understanding. This invention solves the problem of low accuracy in user emotion recognition caused by insufficient collaborative feature extraction and comprehensive semantic understanding of different modal data in intelligent device systems.

[0004] To solve the above-mentioned technical problems, the specific details of the present invention are as follows: This invention provides a method for emotional interaction in intelligent devices based on multimodal understanding, comprising: Step 1: The smart device collects the user's facial video stream, voice signal and tactile interaction data. The feature extraction module adds timestamps to the facial video stream, voice signal and tactile interaction data and performs format standardization. The feature extraction module extracts facial action features of the facial video stream, prosodic feature vectors of the voice signal and spatial distribution parameters of the tactile interaction data. Step 2: The model building module establishes a cross-modal association model. The cross-modal association model calculates the association matrix between facial motion features and prosodic feature vectors. The environmental noise intensity and light sensor readings are input into the weight dynamic allocator to generate the adjustment coefficient of spatial distribution parameters in real time. Step 3: Input the fused features into the hierarchical classifier. The first level outputs the basic emotion category and intensity value, and the second level outputs the probability distribution of complex psychological states. When the anomaly detection module detects that the KL divergence value of the facial action feature and the prosodic feature vector is >0.3, it sets it to the compensation activation state and activates the emotion compensation channel in the user's historical interaction database. Combined with real-time heart rate variability data, it generates the emotion classification offset used to correct the output of the hierarchical classifier. Step 4: The offset correction module uses the emotion classification offset to correct the output of the hierarchical classifier. Using the output of the hierarchical classifier corrected by the offset correction module, it retrieves intervention plans from the psychological knowledge base. Based on the historical value of the execution success rate of the behavioral suggestions in the intervention plan, it adjusts the weight of the empathy expression template, generates a response strategy, converts the response strategy into a voice signal, and synchronously controls the PID gain parameter of the temperature control device to dynamically adjust with the emotion intensity value. It also drives the vibration device to generate tactile pulses that match the user's skin conductance response curve collected by the physiological sensor. Step 5: The feedback evaluation module records the rate of change of the user's facial micro-expressions and the delay time of the voice response. It calculates the consistency metric between the corrected hierarchical classifier output and the emotional state indicated by the user's response behavior using the Pearson correlation coefficient. When the consistency metric is lower than the preset threshold, the feedback evaluation module updates the user's emotion expression library and feeds the consistency metric back to the cross-modal association model for parameter tuning.

[0005] Furthermore, in the multimodal understanding-driven emotional interaction method for intelligent devices of the present invention, step 1 includes: When the ambient light intensity is below the threshold, the intelligent device activates the infrared supplementary light mode, uses a face detection algorithm to process the facial video stream, and outputs the processed facial area video stream to the feature extraction module. The intelligent device uses dual-microphone beamforming to suppress ambient noise, segments the speech signal through an endpoint detection algorithm, and outputs the segmented effective speech segments to the feature extraction module. The intelligent device collects tactile interaction data through a matrix of capacitive pressure sensors, converts the tactile interaction data into a pressure distribution heatmap with a sampling frequency of not less than 10Hz, and outputs it to the feature extraction module.

[0006] Furthermore, in the multimodal understanding-driven emotional interaction method for intelligent devices of the present invention, step 1 further includes: Calculate the Mel frequency cepstral coefficients of the speech signal and generate a Mel feature vector of a specified dimension; The Mel feature vector is input into the semantic understanding model deployed in the feature extraction module, and the output sentiment keyword confidence vector is fed into the hierarchical classifier as the input feature of the speech modality.

[0007] Furthermore, in the multimodal understanding-driven emotional interaction method for intelligent devices of the present invention, step 3 includes: Calculate the KL divergence value based on facial movement features and prosodic feature vectors; When the KL divergence value exceeds 0.3, the emotion compensation channel in the user's historical interaction database is activated to retrieve the facial movement amplitude threshold, speech rate preference value and stress response sensitivity parameters recorded by the user in historical interaction scenarios. The parameters are weighted and averaged by combining the ambient noise intensity and the light sensor readings to generate the emotion classification offset, which is then input into the offset correction module.

[0008] Furthermore, in the multimodal understanding-driven emotional interaction method for intelligent devices of the present invention, step 4 includes: Based on the probability distribution of complex psychological states in the output of the hierarchical classifier after the offset correction module, the empathy expression template library stored in the psychological knowledge base is called to generate emotion confirmation statements. By combining user emotional interaction response records stored in the user's historical interaction database, progressive guidance statements with three levels of guidance intensity are generated; Based on the node similarity-based intervention matching mechanism in the psychological knowledge base, cognitive restructuring exercises or interactive games are output as behavioral suggestions. The system dynamically integrates emotional affirmation statements, progressive guidance statements, and behavioral suggestions, and generates a final response strategy based on the adjusted empathy expression template weights from step 4.

[0009] Furthermore, in the multimodal understanding-driven emotional interaction method for intelligent devices of the present invention, step 4 further includes: Extract the basic emotional intensity value from the output of the hierarchical classifier as the pleasure parameter, set a linear function of the basic temperature value, and generate a PID gain adjustment command for the temperature control device. Based on the activation value in the basic emotional intensity value in the output of the hierarchical classifier and the skin conductance response curve collected by the physiological sensor, the vibration device is controlled to generate a sinusoidal pulse wave of the corresponding frequency as a tactile pulse. The PID gain adjustment command and the tactile pulse signal are synchronously output to the temperature control device and the vibration device for execution.

[0010] Furthermore, in the multimodal understanding-driven emotional interaction method for intelligent devices of the present invention, step 5 includes: When the consistency metric is lower than the preset threshold, the feature samples that conflict with the user response behavior after the hierarchical classifier output and the environmental noise intensity and light sensor reading data recorded in step 2 during the conflict are separated and stored to generate a fine-tuned training dataset. Based on the fine-tuned training dataset, the parameters of the feature extraction module are frozen, and only the weights of the fully connected layers of the cross-modal association model are updated. The updated cross-modal correlation model parameters are fed back to the model building module, replacing the cross-modal correlation model already deployed in the model building module to complete parameter tuning.

[0011] Furthermore, in the multimodal understanding-driven emotional interaction method for intelligent devices of the present invention, step 4 further includes: Extract continuous values ​​of pleasure and activation from the basic emotion intensity values ​​in the output of the hierarchical classifier; When a smart device is configured with a screen, it generates and displays particle animations with color-coded continuous values ​​of pleasure. When the smart device is equipped with a screenless device, the LED light ring outputs a light signal whose hue is inversely proportional to the continuous value of pleasure and whose brightness is directly proportional to the continuous value of activation.

[0012] Furthermore, in the multimodal understanding-driven emotional interaction method for intelligent devices of the present invention, step 3 includes: The first level outputs continuous values ​​of pleasure and activation as the basic emotional intensity value; The second level maps the basic emotional intensity value to the annular emotional space through polar coordinate transformation, and outputs a joint probability distribution composed of the probability values ​​of anxiety, loneliness and ambivalence as the probability distribution of complex psychological states to step 4.

[0013] Furthermore, in the multimodal understanding-driven emotional interaction method for intelligent devices of the present invention, step 4 includes: Emotional state type, cognitive behavioral therapy strategy, and user profile features are used as nodes, with the user profile feature nodes derived from the user's historical interaction database. The correlation strength between emotion state type nodes and user profile feature nodes is calculated using the Pearson correlation coefficient, and the correlation strength is stored as the relation edge weight attribute value in the graph database. A knowledge graph structure is generated based on the relationship edges between nodes and including weight attribute values, and the structured topological relationship of the knowledge graph is invoked when retrieving intervention plans.

[0014] Beneficial effects of this invention; This invention addresses the heterogeneity of multi-source data by eliminating light and noise interference through an environment-adaptive acquisition module, outputting standardized facial video streams, effective speech segments, and tactile heatmaps. The feature extraction module extracts facial motion optical flow features, Mel-frequency cepstral coefficient vectors, and tactile pressure gradient parameters. Using a cross-modal association model, it calculates the spatiotemporal association matrix using a tensor decomposition algorithm, dynamically adjusting tactile feature weights based on environmental parameters to enhance the robustness of multi-modal feature fusion. A hierarchical classifier, based on a circular emotion model, outputs basic emotion intensity values ​​and probability distributions of complex psychological states. Combined with an emotion compensation channel triggered by an anomaly detection module, it generates offset corrections, significantly improving the accuracy of emotion recognition under contradictory modalities. The response strategy generation module calls upon a knowledge graph constructed from a graph database to retrieve intervention schemes, integrating emotion confirmation statements, hierarchical guidance statements, and behavioral suggestions. This synchronously drives the PID adjustment of the temperature control device and the tactile pulse output of the vibration device, achieving precise responses to multi-modal emotional interactions. The feedback evaluation module, through conflict sample separation and storage and a fully connected layer directional update mechanism, dynamically replaces parameters of the cross-modal association model, forming a closed-loop optimization system that continuously enhances environmental adaptability. The collaborative effect of the modules in each step of this invention ultimately achieves the iterative evolution of multimodal semantic understanding capabilities, maintaining the stability of emotion recognition in complex environments. Attached Figure Description

[0015] To more clearly illustrate the present invention, the accompanying drawings used in the embodiments will be briefly described below. Obviously, those skilled in the art can obtain other drawings based on the accompanying drawings without any creative effort.

[0016] Figure 1 A flowchart of a method for driving emotional interaction in intelligent devices based on multimodal understanding, provided for an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, advantages, and benefits of this invention clearer, the invention will be described clearly and completely below with reference to specific embodiments and accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. The invention provided by various embodiments will be described in detail below with reference to the accompanying drawings. To better understand the objectives of this invention, it will be described in further detail below.

[0018] Please see Figure 1 The present invention provides a method for emotional interaction of intelligent devices based on multimodal understanding, comprising: Step 1: The smart device collects the user's facial video stream, voice signal and tactile interaction data. The feature extraction module adds timestamps to the facial video stream, voice signal and tactile interaction data and performs format standardization. The feature extraction module extracts facial action features of the facial video stream, prosodic feature vectors of the voice signal and spatial distribution parameters of the tactile interaction data. Step 2: The model building module establishes a cross-modal association model. The cross-modal association model calculates the association matrix between facial motion features and prosodic feature vectors. The environmental noise intensity and light sensor readings are input into the weight dynamic allocator to generate the adjustment coefficient of spatial distribution parameters in real time. Step 3: Input the fused features into the hierarchical classifier. The first level outputs the basic emotion category and intensity value, and the second level outputs the probability distribution of complex psychological states. When the anomaly detection module detects that the KL divergence value of the facial action feature and the prosodic feature vector is >0.3, it sets it to the compensation activation state and activates the emotion compensation channel in the user's historical interaction database. Combined with real-time heart rate variability data, it generates the emotion classification offset used to correct the output of the hierarchical classifier. Step 4: The offset correction module uses the emotion classification offset to correct the output of the hierarchical classifier. Using the output of the hierarchical classifier corrected by the offset correction module, it retrieves intervention plans from the psychological knowledge base. Based on the historical value of the execution success rate of the behavioral suggestions in the intervention plan, it adjusts the weight of the empathy expression template, generates a response strategy, converts the response strategy into a voice signal, and synchronously controls the PID gain parameter of the temperature control device to dynamically adjust with the emotion intensity value. It also drives the vibration device to generate tactile pulses that match the user's skin conductance response curve collected by the physiological sensor. Step 5: The feedback evaluation module records the rate of change of the user's facial micro-expressions and the delay time of the voice response. It calculates the consistency metric between the corrected hierarchical classifier output and the emotional state indicated by the user's response behavior using the Pearson correlation coefficient. When the consistency metric is lower than the preset threshold, the feedback evaluation module updates the user's emotion expression library and feeds the consistency metric back to the cross-modal association model for parameter tuning.

[0019] This invention provides a method for emotional interaction in intelligent devices based on multimodal understanding, achieving accurate identification and response optimization of user emotional states through multi-step collaboration. In step 1, the intelligent device collects facial video streams, voice signals, and tactile interaction data. The feature extraction module adds timestamps to these data and performs format standardization processing to achieve temporal alignment and compatibility of multi-source data. The feature extraction module further analyzes the facial video stream to extract facial action features, processes the voice signal to generate prosodic feature vectors, and parses the tactile interaction data to obtain spatial distribution parameters. Further, this invention includes activating an infrared supplementary lighting mode when ambient light is insufficient, using a face detection algorithm to optimize video stream quality, combining dual-microphone beamforming to suppress noise and segment effective voice segments, and simultaneously converting tactile data into a high-frequency pressure distribution heatmap using a capacitive pressure sensor matrix to improve the robustness of data acquisition. The feature extraction module also calculates the Mel frequency cepstral coefficients of the voice signal to generate Mel feature vectors, inputting them into the deployed semantic understanding model to output emotional keyword confidence vectors, which serve as input features for a hierarchical classifier, forming the foundation of multimodal features.

[0020] In step 2, the model building module establishes a cross-modal association model. This model calculates the association matrix between facial motion features and prosodic feature vectors, revealing the synergistic relationship between visual and auditory modalities. Environmental noise intensity and illumination sensor readings are input into a dynamic weight allocator, which dynamically generates adjustment coefficients for spatial distribution parameters. An adaptive weighting mechanism optimizes the fusion weights of tactile features. This invention relates to the use of tensor decomposition algorithms to calculate the association matrix, achieving effective association of spatiotemporal features. The dynamic weight allocator adjusts coefficients in real time based on environmental parameters, enhancing the model's adaptability to interfering environments and providing a dynamic adjustment basis for subsequent feature fusion.

[0021] Step 3 integrates the input features into the hierarchical classifier. The first level outputs basic emotion categories and intensity values, while the second level outputs probability distributions of complex psychological states, achieving hierarchical emotion recognition. The anomaly detection module checks whether the KL divergence value of facial movement features and prosodic feature vectors exceeds a threshold. If it does, a compensation activation state is activated, triggering the emotion compensation channel in the user's historical interaction database. This is combined with real-time heart rate variability data to generate an emotion classification offset. Further details of this invention include the calculation process of the KL divergence value. When the threshold is exceeded, facial movement amplitude thresholds, speech rate preference values, and stress response sensitivity parameters from the user's historical interaction scenarios are retrieved. These are combined with environmental parameters to perform a weighted average calculation to generate an offset. This offset is input into the offset correction module, which improves the accuracy of emotion recognition through historical preferences and environmental compensation, avoiding misjudgments caused by modal conflicts.

[0022] In step 4, the offset correction module uses the emotion classification offset to correct the output of the hierarchical classifier. The corrected output is used to retrieve intervention plans from the psychological knowledge base. Based on the historical success rate of behavioral suggestions in the intervention plan, the weight of the empathy expression template is adjusted to generate a response strategy. This response strategy is converted into a speech signal output, and simultaneously, the PID gain parameter of the temperature control device is dynamically adjusted according to the emotion intensity value, driving the vibration device to generate tactile pulses matching the user's skin conductance response curve. Further, this invention relates to generating emotion confirmation statements by calling the empathy expression template library stored in the psychological knowledge base, combining it with the user's historical interaction response records to generate progressive guidance statements including graded guidance intensity, matching cognitive reconstruction exercises or interactive game plans as behavioral suggestions through node similarity calculation, and dynamically fusing these elements to generate the final response strategy. The device control section extracts the basic emotion intensity value as a pleasure parameter to set the PID adjustment command, and combines the activation value and skin conductance response curve to control the vibration device to output a sinusoidal pulse wave, achieving synchronous execution of multimodal responses.

[0023] Step 5 involves the feedback evaluation module recording the user's facial micro-expression change rate and voice response delay time. It then calculates a consistency metric between the corrected hierarchical classifier output and the user's response behavior indicating emotional state using the Pearson correlation coefficient. If the consistency metric is below a preset threshold, the module updates the user's emotion expression database and feeds the metric back to the cross-modal association model for parameter tuning. Further, this invention includes separating conflicting feature samples and corresponding environmental data to generate a fine-tuned training dataset. The parameters of the feature extraction module are frozen, and only the weights of the fully connected layers in the cross-modal association model are updated. The updated parameters are then fed back to the model building module to replace the deployed model, forming a closed-loop optimization mechanism to enhance the model's generalization ability.

[0024] The above steps form a closed-loop process: data acquisition and feature processing in steps 1 and 2 provide input for emotion recognition in step 3; offset correction in step 3 optimizes response generation in step 4; the execution result of step 4 is input for feedback evaluation in step 5; the output of step 5 is fed back to model optimization in step 2, achieving adaptive iteration. The overall method improves the accuracy and reliability of emotional interaction through multimodal feature fusion, anomaly compensation, dynamic response, and feedback loop.

[0025] Specifically, the multimodal understanding-driven emotional interaction method for intelligent devices of the present invention includes step 1 as follows: When the ambient light intensity is below the threshold, the intelligent device activates the infrared supplementary light mode, uses a face detection algorithm to process the facial video stream, and outputs the processed facial area video stream to the feature extraction module. The intelligent device uses dual-microphone beamforming to suppress ambient noise, segments the speech signal through an endpoint detection algorithm, and outputs the segmented effective speech segments to the feature extraction module. The intelligent device collects tactile interaction data through a matrix of capacitive pressure sensors, converts the tactile interaction data into a pressure distribution heatmap with a sampling frequency of not less than 10Hz, and outputs it to the feature extraction module.

[0026] The intelligent device data acquisition and processing scheme in step 1 of this invention improves the quality of raw data through a multi-level environmental adaptation mechanism. In the visual data processing stage, when the ambient light intensity is below a preset threshold, the system automatically activates the infrared supplementary lighting mode and uses a face detection algorithm based on HOG features and an SVM classifier to process the facial video stream and extract the effective facial region video stream. This scheme solves the problem of video feature extraction distortion in low-light environments and provides a standardized visual input source for the feature extraction module.

[0027] In the speech signal processing stage, the intelligent device deploys a dual-microphone beamforming array to generate directional beams to suppress environmental noise through phase difference calculation. An endpoint detection algorithm based on short-time energy and zero-crossing rate is used to segment the speech signal, filtering out silent segments and non-human voice interference, and outputting valid speech segments to the feature extraction module. This invention eliminates the influence of reverberation on speech features and improves the accuracy of subsequent prosodic feature vector extraction.

[0028] The tactile interaction data processing uses a capacitive pressure sensor matrix to collect raw tactile data. Discrete pressure values ​​are then converted into a continuous pressure distribution heatmap using a Gaussian interpolation algorithm. The heatmap sampling frequency is maintained above 10Hz to fully record the spatial gradient changes in tactile pressure intensity, and the data is output to the feature extraction module for spatial distribution parameter calculation. This implementation scheme fully preserves the spatiotemporal characteristics of tactile interaction, providing high-fidelity tactile input for multimodal fusion.

[0029] The feature extraction module performs timestamp synchronization on the three types of data, using the Unix timestamp format to achieve millisecond-level precision alignment and eliminate temporal misalignment issues in multi-source data. Format standardization processing follows the MPEG-4 video coding standard, the PCM audio coding standard, and the IEEE haptic data protocol, establishing a unified data processing pipeline. This invention constructs a cross-modal data spatiotemporal benchmark, laying the foundation for subsequent feature association.

[0030] Specifically, in the multimodal understanding-driven emotional interaction method for intelligent devices of the present invention, step 1 further includes: Calculate the Mel frequency cepstral coefficients of the speech signal and generate a Mel feature vector of a specified dimension; The Mel feature vector is input into the semantic understanding model deployed in the feature extraction module, and the output sentiment keyword confidence vector is fed into the hierarchical classifier as the input feature of the speech modality.

[0031] The speech feature processing scheme in step 1 of this invention achieves emotional semantic parsing through multi-level acoustic analysis. The feature extraction module performs pre-emphasis processing on the effective speech segments after endpoint detection to compensate for high-frequency component attenuation and improve the clarity of spectral features. After the signal is framed by a Hamming window, a Fast Fourier Transform is performed to obtain the short-time power spectrum distribution of the speech signal.

[0032] The Mel frequency cepstral coefficient calculation process uses a triangular filter bank to simulate the characteristics of human hearing, and performs a nonlinear transformation of the power spectrum on the Mel scale. The dimension is compressed through discrete cosine transform to generate a Mel feature vector of a specified dimension. This invention simulates the pitch perception characteristics of the human auditory system, preserving the emotional prosodic features of the speech signal.

[0033] The semantic understanding model employs a Transformer architecture based on a multi-head attention mechanism, deployed within the feature extraction module. The model receives a sequence of Mel feature vectors and captures temporal dependencies through positional encoding. In the encoder-decoder structure, a self-attention layer identifies emotion-related acoustic patterns, and the output layer generates emotion keyword confidence vectors.

[0034] The sentiment keyword confidence vector includes the probability distribution of emotion categories such as anger, joy, and sadness, serving as the speech modality input feature for the hierarchical classifier. The confidence vector, along with facial action features and tactile spatial distribution parameters, forms a spatiotemporally aligned multimodal input, providing standardized speech feature representations for subsequent cross-modal association.

[0035] The processing flow forms an acoustic feature transformation chain: the physical signal processing layer (pre-emphasis / framing / FFT) extracts basic spectral features, the perceptual feature transformation layer (Mel filter bank / DCT) generates feature vectors that conform to auditory characteristics, and the semantic parsing layer (Transformer) outputs emotional semantic representations. This three-stage processing enables the hierarchical classifier to obtain speech feature inputs with emotional discriminative power.

[0036] Specifically, step 3 of the multimodal understanding-driven intelligent device emotion interaction method of the present invention includes: Calculate the KL divergence value based on facial movement features and prosodic feature vectors; When the KL divergence value exceeds 0.3, the emotion compensation channel in the user's historical interaction database is activated to retrieve the facial movement amplitude threshold, speech rate preference value and stress response sensitivity parameters recorded by the user in historical interaction scenarios. The parameters are weighted and averaged by combining the ambient noise intensity and the light sensor readings to generate the emotion classification offset, which is then input into the offset correction module.

[0037] The anomaly detection and compensation mechanism in step 3 of this invention improves the robustness of emotion recognition through cross-modal difference analysis. The anomaly detection module receives facial motion features and prosodic feature vectors output by the feature extraction module, and calculates the difference value between the feature distributions of the two modalities using the KL divergence algorithm. This difference value reflects the degree of consistency between the emotional expression of the visual and speech modalities, providing a quantitative basis for anomaly judgment.

[0038] When the KL divergence value exceeds a preset threshold, the system activates the compensation activation state, triggering the emotion compensation channel in the user's historical interaction database. The emotion compensation channel retrieves personalized expression parameters recorded in the user's historical interaction scenarios, including facial movement amplitude thresholds, speech rate preference values, and stress response sensitivity parameters. These parameters characterize the user's unique emotion expression habits, establishing a personalized compensation benchmark.

[0039] The environmental adaptive compensation unit combines real-time acquired environmental noise intensity and illumination sensor readings to dynamically weight historical parameters. Noise intensity affects the reliability weight of the speech modality, while illumination conditions affect the credibility weight of the visual modality, generating an environmental compensation coefficient matrix. This matrix is ​​then element-wise multiplied with the historical parameters to output calibrated, personalized compensation parameters.

[0040] The emotion classification offset generation module linearly combines calibration parameters with KL divergence values, mapping them to a dimension-matched offset vector via a fully connected neural network. This offset vector is input to the offset correction module, where it is element-wise added to the original output of the hierarchical classifier to correct emotion recognition biases caused by environmental interference or individual differences in expression.

[0041] This invention establishes a three-level compensation mechanism: a modal difference detection layer (KL divergence calculation) identifies abnormal states, a personalized parameter extraction layer (historical database retrieval) obtains user baselines, and an environment adaptation layer (weighted calculation) achieves dynamic compensation. The three-level outputs are fused by a neural network to generate offsets, forming a closed-loop correction system.

[0042] Specifically, in the multimodal understanding-driven emotional interaction method for intelligent devices of the present invention, step 4 includes: Based on the probability distribution of complex psychological states in the output of the hierarchical classifier after the offset correction module, the empathy expression template library stored in the psychological knowledge base is called to generate emotion confirmation statements. By combining user emotional interaction response records stored in the user's historical interaction database, progressive guidance statements with three levels of guidance intensity are generated; Based on the node similarity-based intervention matching mechanism in the psychological knowledge base, cognitive restructuring exercises or interactive games are output as behavioral suggestions. The system dynamically integrates emotional affirmation statements, progressive guidance statements, and behavioral suggestions, and generates a final response strategy based on the adjusted empathy expression template weights from step 4.

[0043] The response strategy generation scheme in step 4 of this invention achieves precise emotional intervention through multi-source information fusion. A psychological knowledge base stores a structured empathic expression template library, which stores the mapping relationship between grammatical frameworks and emotional vocabulary according to emotion categories. The invocation process parses the probability distribution of complex psychological states output by the hierarchical classifier, matches the template group corresponding to the highest probability emotional state, and fills in the real-time emotional intensity parameters to generate an emotional confirmation statement. This statement directly responds to the user's current emotional state, establishing a basis for emotional resonance.

[0044] The user history interaction database records user response data to various guidance strategies, and generates progressive guidance statements based on response effectiveness. The system analyzes the positive feedback rate in historical interaction response records and dynamically selects a statement structure with three levels of guidance intensity: Level 1 uses open-ended questions, Level 2 embeds cognitive restructuring prompts, and Level 3 provides explicit behavioral options. The statement content is optimized with vocabulary based on the characteristics of the user's recent emotional interaction response records to improve guidance receptivity.

[0045] The intervention program matching mechanism is implemented based on the node similarity calculation of a graph database. A semantic relationship network is constructed between emotional state type nodes and cognitive behavioral therapy strategy nodes in a psychological knowledge base. The starting node is located based on the probability distribution of complex psychological states output by a hierarchical classifier. The Dijkstra algorithm is used to traverse the path with the highest edge weight, outputting cognitive restructuring exercises or interactive games as behavioral suggestions. The complexity of the program content is adjusted based on user profile feature nodes to form personalized behavioral interventions.

[0046] The dynamic fusion module employs a weighted attention mechanism to integrate three types of elements. The weight of the empathic expression template is dynamically adjusted based on the historical success rate of the behavioral suggestions, with templates achieving high success rates receiving higher attention weights. The module projects the emotion confirmation statements, progressive guidance statements, and behavioral suggestions into a vector space, performs feature-level fusion under weight control, and generates a grammatically coherent final response strategy via a GRU decoder.

[0047] This invention forms a four-stage processing chain: the emotional resonance layer (emotional affirmation statements) establishes a trust relationship; the cognitive guidance layer (progressive statements) stimulates independent thinking; the behavioral intervention layer (cognitive / game-based solutions) provides solutions; and the multimodal fusion layer (weighted attention) optimizes the expression effect. The outputs of each stage form a complete emotional support path through semantic connection.

[0048] There is logical synergy between the functional modules: the template library calls on the emotion recognition results of the hierarchical classifier, the selection of guiding statement strength is based on historical database feedback analysis, the behavior suggestion matching utilizes the topological relationships of the knowledge graph, and the final fusion stage integrates the output of the first three. The system continuously improves the naturalness and effectiveness of emotional interaction by iteratively optimizing the weight of empathic expression templates through real-time feedback.

[0049] Specifically, in the multimodal understanding-driven emotional interaction method for intelligent devices of the present invention, step 4 further includes: Extract the basic emotional intensity value from the output of the hierarchical classifier as the pleasure parameter, set a linear function of the basic temperature value, and generate a PID gain adjustment command for the temperature control device. Based on the activation value in the basic emotional intensity value in the output of the hierarchical classifier and the skin conductance response curve collected by the physiological sensor, the vibration device is controlled to generate a sinusoidal pulse wave of the corresponding frequency as a tactile pulse. The PID gain adjustment command and the tactile pulse signal are synchronously output to the temperature control device and the vibration device for execution.

[0050] The device control scheme in step 4 of this invention enhances the emotional interaction effect through physiological response matching and environmental feedback. The basic emotional intensity value output by the hierarchical classifier is standardized, and the pleasure parameter is extracted as the temperature control benchmark. The system presets the user's comfortable temperature value as the origin of a linear function, and the pleasure parameter is mapped to a temperature offset to generate the target temperature setpoint. The temperature control device receives the target temperature value and dynamically calculates the PID gain parameter through a proportional-integral-derivative control algorithm to form a temperature adjustment command.

[0051] The tactile pulse generation module synchronously processes the skin conductance response curves acquired by the physiological sensors, extracting the rising slope of the skin conductance response as a representation of physiological activation. This representation is weighted and fused with the activation values ​​output by the hierarchical classifier, and mapped to the tactile sensitive frequency range through a lookup table. The vibration device generates sinusoidal pulse waves of the corresponding frequencies based on the fused values, with the waveform envelope changing synchronously with the temporal characteristics of the skin conductance response curve.

[0052] The device command synchronization engine adds a unified timestamp to the PID gain adjustment command and the tactile pulse signal, establishing millisecond-level precision in execution timing alignment. Upon receiving the command, the temperature control device adjusts the power of the semiconductor cooler according to the exponential decay curve, and the vibration device modulates the amplitude of the linear resonator based on the command frequency. The dual-device execution process is synchronized via timestamps, ensuring the spatiotemporal consistency between the rate of temperature change and the intensity of tactile stimulation.

[0053] This invention achieves a three-level control closed loop: the emotion parameter conversion layer maps psychological states to physical control quantities; the physiological response adaptation layer achieves dynamic matching of tactile features and physiological signals; and the device coordination layer achieves precise synchronization of multimodal feedback. The execution effect forms an emotional reinforcement loop: the direction of temperature changes enhances the perception of pleasure, the amplitude of vibration frequency changes corresponds to the level of physiological activation, and the synergistic effect deepens the user's emotional experience.

[0054] The functional logic of each component is closely integrated: the emotional intensity value provides an input source for device control, the skin conductance response curve optimizes the biocompatibility of tactile feedback, and the timestamp mechanism solves the problem of multi-device coordination. The gradual change characteristics of the temperature control device and the transient response of the vibration device complement each other, jointly constructing a multimodal emotional feedback system that conforms to the laws of human perception.

[0055] Specifically, in the multimodal understanding-driven emotional interaction method for intelligent devices of the present invention, step 5 includes: When the consistency metric is lower than the preset threshold, the feature samples that conflict with the user response behavior after the hierarchical classifier output and the environmental noise intensity and light sensor reading data recorded in step 2 during the conflict are separated and stored to generate a fine-tuned training dataset. Based on the fine-tuned training dataset, the parameters of the feature extraction module are frozen, and only the weights of the fully connected layers of the cross-modal association model are updated. The updated cross-modal correlation model parameters are fed back to the model building module, replacing the cross-modal correlation model already deployed in the model building module to complete parameter tuning.

[0056] The feedback optimization mechanism in step 5 of this invention achieves adaptive model evolution through closed-loop training. When the consistency metric between the corrected hierarchical classifier output calculated by the feedback evaluation module and the user's response behavior indicating emotional state is lower than a preset threshold, the system initiates the model optimization process. The conflict feature recognition unit separately stores two types of key data: feature vector samples in the corrected hierarchical classifier output that differ significantly from the user's actual response behavior, and environmental noise intensity and light sensor readings synchronously recorded during the conflict in step 2. These three types of data are timestamped to generate a fine-tuned training dataset, fully preserving the environmental context of the emotion recognition conflict scenario.

[0057] The model update module employs a transfer learning strategy to fine-tune the training dataset. During training, all convolutional layer parameters of the feature extraction module are frozen to maintain the underlying feature extraction capabilities. Only the weights of the fully connected layers in the cross-modal association model are unlocked for backpropagation calculation, and the classification decision boundary is adjusted using a mini-batch gradient descent algorithm. This approach achieves a balance between knowledge transfer and incremental learning, avoiding overfitting scenarios.

[0058] The parameter deployment engine feeds back the updated weight matrix of the fully connected layer of the cross-modal association model to the model building module. The deployment process employs hot-swapping technology, loading the new weight matrix into the memory buffer while maintaining real-time system operation. The updated cross-modal association model parameters are then invoked through the version identifier control module. This invention eliminates service interruptions and achieves seamless replacement of model parameters.

[0059] The optimized process forms a three-stage closed loop: the data layer establishes a snapshot of the conflict scenario, the training layer performs targeted parameter adjustments, and the deployment layer completes dynamic model switching. The updated cross-modal association model automatically applies the new decision rules in the next round of computation, specifically improving the accuracy of emotion recognition in specific environments.

[0060] The invention exhibits a logical progression across its various stages: conflict data storage provides training material for model optimization, selective parameter updates ensure the model's generalization ability, and a hot-switching mechanism enables continuous system operation. The associated storage of environmental context data and feature samples allows the model to learn feature compensation patterns under noise interference, and ultimately, parameter replacement enables the continuous evolution of the system's overall recognition capability.

[0061] Specifically, in the multimodal understanding-driven emotional interaction method for intelligent devices of the present invention, step 4 further includes: Extract continuous values ​​of pleasure and activation from the basic emotion intensity values ​​in the output of the hierarchical classifier; When a smart device is configured with a screen, it generates and displays particle animations with color-coded continuous values ​​of pleasure. When the smart device is equipped with a screenless device, the LED light ring outputs a light signal whose hue is inversely proportional to the continuous value of pleasure and whose brightness is directly proportional to the continuous value of activation.

[0062] The additional device response scheme in step 4 of this invention enhances the visual expression of the user's emotional state through visual feedback. The basic emotional intensity values ​​output by the hierarchical classifier include continuous values ​​of pleasure and activation. The system normalizes these continuous values ​​to a standard range using a linear interpolation algorithm, providing a unified input benchmark for feedback generation from different device types. This process ensures stable transmission of emotional intensity data, avoiding signal fluctuations from affecting subsequent processing.

[0063] In a screen-based device configuration scenario, the system invokes the graphics rendering engine to process the normalized continuous values ​​of pleasure. The engine maps these pleasure values ​​to the hue dimension of the HSV color space, with high pleasure corresponding to warm tones and low pleasure corresponding to cool tones. The particle animation generation module creates a dynamic particle system based on the mapping results. The particle movement speed is positively correlated with the continuous activation value, forming a color-coded particle animation. The display control unit outputs the animation to the screen device, and the visual complexity of the animation dynamically adjusts according to the user's emotional intensity, intuitively presenting changes in emotional state.

[0064] In screenless device configurations, the light signal generation module processes the normalized continuous values ​​of pleasure and activation. The module establishes an inverse relationship model between hue and pleasure values, shifting the hue towards the blue spectrum as pleasure decreases; simultaneously, it sets a direct relationship model between brightness and activation values, linearly increasing brightness as activation increases. The LED ring control unit converts the output of this relationship model into a PWM dimming signal, driving the LED array to generate a continuously varying light signal. The rate of color temperature change in the light signal is synchronized with the frequency of emotional fluctuations, enhancing the immersive experience of the emotional feedback.

[0065] The device response logic forms a branching processing structure: a data preprocessing layer (normalization) standardizes the input signal; a device adaptation layer (rendering / light control) selects the feedback mode according to the hardware type; and a signal output layer (animation / light signal) realizes the visualization. Particle animation on screen devices provides high-resolution emotional visualization, while LED light signals on screenless devices enable emotional perception in a low-power environment. Both are based on the same emotional intensity input, forming a complementary feedback mechanism.

[0066] A causal chain exists between functional modules: emotion intensity value extraction provides a data source for feedback from all devices, device type detection determines the specific execution path, feedback content generates a dependency mapping model, and output execution achieves final visualization. Through unified emotion-physical mapping rules, the system achieves consistency and understandability of emotional feedback across different hardware platforms, enhancing the coherence of the user's emotional interaction experience.

[0067] Specifically, step 3 of the multimodal understanding-driven intelligent device emotion interaction method of the present invention includes: The first level outputs continuous values ​​of pleasure and activation as the basic emotional intensity value; The second level maps the basic emotional intensity value to the annular emotional space through polar coordinate transformation, and outputs a joint probability distribution composed of the probability values ​​of anxiety, loneliness and ambivalence as the probability distribution of complex psychological states to step 4.

[0068] The hierarchical classifier processing scheme in step 3 of this invention achieves refined analysis of psychological states through a circular emotion model. The first level of the hierarchical classifier receives fused feature input, processes it through a fully connected neural network, and outputs basic emotion intensity values, including two dimensions: continuous pleasure value and continuous activation value. The continuous pleasure value represents the positive or negative direction of emotional valence, while the continuous activation value reflects the intensity of physiological arousal level; the two constitute the basic quantitative indicators of emotional state.

[0069] The polar coordinate transformation module uses basic emotion intensity values ​​as input vectors for spatial mapping. Continuous values ​​of pleasure are mapped to polar coordinate angular components, and continuous values ​​of activation are mapped to polar radius components. This data transformation is achieved through a conversion formula from Cartesian to polar coordinates. This transformation projects linear emotional features into a circular emotion space, establishing a two-dimensional expression structure that conforms to the psychological circular emotion model.

[0070] Within the eight emotional quadrants defined by the annular emotional space, the system focuses on monitoring the distribution areas of anxiety, loneliness, and ambivalence. The second-level classifier employs a radial basis function neural network to calculate the similarity between the input vector and the prototype vectors of various psychological states, outputting a joint probability distribution composed of the probability values ​​of anxiety, loneliness, and ambivalence. This probability distribution vector, as the probability distribution of complex psychological states, is output to step 4, providing a refined basis for psychological state diagnosis in subsequent emotional interventions.

[0071] This invention employs a two-tiered processing structure: a basic quantification layer extracts core dimensions of emotion, while a spatial mapping layer classifies psychological states. Continuous values ​​of pleasure and activation are used as basic outputs, transformed into circular spatial coordinates through polar coordinate transformation, and then output as complex psychological state diagnostic results through probability calculation.

[0072] There is a geometric mapping relationship between the functional modules: the basic emotion intensity values ​​constitute the coordinate input of the annular space, polar coordinate transformation establishes a dimensional transformation bridge, and the radial basis function network completes the spatial partition probability calculation. This invention realizes the mathematical derivation from the basic emotion dimension to complex psychological states, and the emotion recognition results not only conform to the psychological theoretical model, but also have machine-processable quantitative characteristics. The output probability distribution establishes the data foundation for the precise intervention in step 4.

[0073] Specifically, in the multimodal understanding-driven emotional interaction method for intelligent devices of the present invention, step 4 includes: Emotional state type, cognitive behavioral therapy strategy, and user profile features are used as nodes, with the user profile feature nodes derived from the user's historical interaction database. The correlation strength between emotion state type nodes and user profile feature nodes is calculated using the Pearson correlation coefficient, and the correlation strength is stored as the relation edge weight attribute value in the graph database. A knowledge graph structure is generated based on the relationship edges between nodes and including weight attribute values, and the structured topological relationship of the knowledge graph is invoked when retrieving intervention plans.

[0074] The psychological knowledge base construction scheme in step 4 of this invention optimizes the retrieval efficiency of emotional intervention strategies through a graph structure. The knowledge base architecture defines three core node types: emotional state type nodes labeled with discrete emotional tags such as "joy" and "sadness"; cognitive behavioral therapy strategy nodes including intervention methods such as "thought recording" and "exposure therapy"; and user profile feature nodes extracting behavioral pattern tags from the user's historical interaction database, such as personalized features like "high avoidance tendency" and "fast-paced response." These node definitions establish the knowledge representation foundation for emotional intervention.

[0075] The correlation strength calculation module uses Pearson correlation coefficient to analyze the co-occurrence frequency of emotion state type nodes and user profile feature nodes. Statistical correlation of node pairs is calculated on the historical interaction dataset, outputting a standardized correlation coefficient in the [-1,1] interval. The correlation coefficient is normalized using the Sigmoid function, transforming it into edge weight values ​​in the [0,1] interval, which are stored as core attributes of the relationship edges in the Neo4j graph database. This invention quantifies the degree of matching between user features and emotion states, providing data support for intervention strategy recommendations.

[0076] The knowledge graph generation engine constructs a directed graph structure based on a set of nodes and weighted relational edges. The graph employs a hierarchical topology design: emotion state type nodes form the root layer, user profile feature nodes form the intermediate layer, and cognitive behavioral therapy strategy nodes form the leaf layer. The weight attributes of relational edges control the connection strength between nodes, with higher-weighted edges forming preferred retrieval paths. This structure achieves a networked organization of knowledge elements, conforming to the associative memory model of human cognition.

[0077] The structured topological relationship retrieval mechanism is activated when querying intervention plans. The system receives the probability distribution of complex psychological states output by the hierarchical classifier and uses the emotional state with the highest probability as the starting point for retrieval. The system traverses the relationship edge paths with the highest weights using Dijkstra's algorithm, outputting the cognitive behavioral therapy strategy with the best matching degree at the leaf layer nodes. The weight parameters in the topological relationship dynamically adjust the query direction, ensuring that the recommended strategy matches both the current emotional state and the user's historical behavioral characteristics.

[0078] This invention forms a three-tiered processing chain: the data layer (nodes / edges) constructs the knowledge framework, the computation layer (relevance coefficients) injects semantic logic, and the application layer (topology retrieval) achieves accurate recommendations. A closed-loop information flow exists between the functional modules: node definitions provide computational input, association strength calculations generate weight parameters, the graph structure integrates knowledge elements, and topology retrieval outputs decision results. The system transforms discrete knowledge into a reasonable semantic network through a weighted graph, significantly improving the personalization level of emotional intervention strategies.

[0079] This invention addresses the issue of inaccurate user emotion recognition in intelligent device systems due to insufficient collaborative feature extraction and comprehensive semantic understanding of data from different modalities. First, during the data acquisition phase, the intelligent device employs an environment-adaptive mechanism to eliminate external interference: it activates an infrared supplementary lighting mode to optimize the facial video stream under low-light conditions, suppresses noise and segments effective speech segments using dual-microphone beamforming, and converts tactile data into a high-frequency pressure distribution heatmap using a capacitive pressure sensor matrix. The feature extraction module adds millisecond-level timestamps to the facial video stream, speech signals, and tactile interaction data and performs standardized format processing, adhering to MPEG-4 video encoding, PCM audio encoding, and the IEEE tactile data protocol, thus establishing a spatiotemporal alignment basis for multi-source data. This step eliminates the heterogeneity problem of multimodal data, providing a unified input framework for collaborative feature extraction.

[0080] At the feature fusion level, the model building module establishes a cross-modal association model and applies tensor decomposition to calculate the spatiotemporal association matrix of facial motion features and prosodic feature vectors, revealing the collaborative relationship between visual and speech modalities. A dynamic weight allocator receives real-time environmental noise intensity and illumination sensor readings, dynamically generating adjustment coefficients for spatial distribution parameters to adaptively adjust the fusion weights of tactile features. This scheme optimizes feature weight allocation through environmental parameter feedback, addressing the insufficient robustness of existing methods in feature fusion under dynamic environments and improving the accuracy of multimodal data collaborative representation.

[0081] When processing fused features, the hierarchical classifier outputs continuous values ​​of pleasure and activation as basic emotional intensity values. The second level maps these basic emotional intensity values ​​to a circular emotional space through polar coordinate transformation, outputting a joint probability distribution composed of probability values ​​for anxiety, loneliness, and ambivalence as the probability distribution for complex psychological states. The anomaly detection module calculates the KL divergence between facial movement features and prosodic feature vectors in real time. When the divergence exceeds a threshold, it activates the emotion compensation channel in the user's historical interaction database, retrieving the user's facial movement amplitude threshold, speech rate preference value, and stress response sensitivity parameters. This data is then combined with environmental noise intensity and light sensor readings for a weighted average calculation to generate the emotion classification offset input to the offset correction module. This mechanism enhances the reliability of comprehensive semantic understanding by correcting modal conflicts through personalized historical data and environmental compensation, avoiding emotion recognition bias caused by misjudgment of a single modality.

[0082] In the response strategy generation phase, the system uses the modified hierarchical classifier output to call the empathy expression template library stored in the psychological knowledge base to generate emotion confirmation statements. It also combines this with user historical interaction database records to generate progressive guidance statements including graded guidance intensities. Furthermore, it uses graph database node similarity calculations to match cognitive reconstruction exercises or interactive game schemes as behavioral suggestions. The dynamic fusion module integrates these elements based on the empathy expression template weights to generate the final response strategy. Simultaneously, it extracts basic emotion intensity values ​​to drive the PID gain adjustment of the temperature control device and the tactile pulse output of the vibration device. The application of the structured topological relationships of the knowledge graph enables precise matching of intervention schemes, ensuring that the response strategy aligns with the user's psychological state characteristics.

[0083] In the closed-loop feedback process, the feedback evaluation module records the rate of change in the user's facial micro-expressions and the voice response delay time, and calculates a consistency metric using the Pearson correlation coefficient. When the metric falls below a preset threshold, the system separates and stores conflict feature samples and corresponding environmental data to generate a fine-tuned training dataset. The parameters of the feature extraction module are frozen, and only the weights of the fully connected layers in the cross-modal association model are updated. The updated parameters are then fed back to the model building module to replace the deployed cross-modal association model. This process achieves incremental model optimization through targeted training of conflict scenario samples, continuously improving the generalization ability of multimodal collaborative processing, forming a closed-loop iteration from data collection to model update, fundamentally solving the problem of insufficient semantic understanding.

[0084] The invention is implemented as follows: In a children's cognitive education application scenario, the intelligent device first collects multimodal user data: when the ambient light intensity is below a threshold, an infrared supplementary lighting mode is activated to process the facial video stream; dual-microphone beamforming suppresses noise and segments effective speech segments; and a capacitive pressure sensor matrix generates a pressure distribution heatmap above 10Hz. The feature extraction module adds Unix timestamps to the three types of data and performs MPEG-4 / PCM / IEEE format standardization, extracting facial movement features, prosodic feature vectors, and spatial distribution parameters, thus solving the problem of difficulty in coordinating multi-source heterogeneous data.

[0085] The model building module establishes a cross-modal association model and applies tensor decomposition to calculate the spatiotemporal association matrix between facial motion features and prosodic feature vectors. A dynamic weight allocator generates spatial distribution parameter adjustment coefficients based on real-time environmental noise intensity and illumination sensor readings, dynamically optimizing the tactile feature fusion weights. This approach overcomes the shortcomings of existing methods in feature fusion robustness under dynamic environments, providing environmental adaptability for subsequent emotion recognition.

[0086] After fusing the feature input hierarchical classifier, the first level outputs continuous values ​​of pleasure and activation as basic emotion intensity values. The second level maps to a circular emotion space through polar coordinate transformation, outputting a joint probability distribution composed of probability values ​​of anxiety, loneliness, and ambivalence. The anomaly detection module calculates the KL divergence value between facial motion features and prosodic feature vectors. When the value exceeds a threshold, the emotion compensation channel in the user's historical interaction database is activated. This channel retrieves facial motion amplitude thresholds, speech rate preference values, and stress response sensitivity parameters recorded in the user's historical interaction scenarios, and generates an emotion classification offset by weighted averaging with environmental parameters. This mechanism addresses the misjudgment problem caused by the contradiction between facial expressions and speech through personalized compensation.

[0087] In the response strategy generation phase, the system calls upon the empathy expression template library stored in the psychological knowledge base to generate emotion confirmation statements. It then combines this with user history interaction database records to generate progressive guidance statements with three levels of guidance intensity. Based on the similarity calculation of nodes in the graph database, it outputs cognitive reconstruction exercises or interactive game solutions. The dynamic fusion module adjusts weights based on the success rate of behavioral suggestions to generate the final response strategy. Simultaneously, it extracts basic emotion intensity values ​​to drive the PID adjustment of the temperature control device and the tactile pulse output of the vibration device, achieving multimodal emotional intervention in educational scenarios.

[0088] The feedback evaluation module calculates a consistency metric using the Pearson correlation coefficient. When the metric falls below a threshold, conflict feature samples and environmental data are separated and stored to generate a fine-tuned training dataset. The parameters of the feature extraction module are frozen, and only the weights of the fully connected layers in the cross-modal association model are updated. A hot-swapping technique replaces the deployed model in the model building module. This closed-loop optimization mechanism continuously improves the system's generalization ability, fundamentally solving the problem of insufficient semantic understanding. In the specific application of this invention in children's cognitive training scenarios, the probability distribution of complex psychological states output by the hierarchical classifier triggers targeted intervention strategies. When an increase in the anxiety probability value is detected, the system calls upon the cognitive reconstruction exercise scheme associated in the knowledge graph. After establishing trust through emotional confirmation statements, progressive guiding statements are output. Simultaneously, a temperature control device raises the ambient temperature, and a vibration device generates low-frequency tactile pulses to alleviate anxiety. All device responses are synchronized at the millisecond level via timestamps, forming a multi-sensory intervention that conforms to the principles of educational psychology.

[0089] The feature extraction module output maintains consistent terminology for facial action features, prosodic feature vectors, and spatial distribution parameters. When the hierarchical classifier performs two-level emotion recognition, the basic emotion intensity values, including continuous values ​​of pleasure and continuous values ​​of activation, are used as the core output. The psychological knowledge base is constructed using a ternary structure of emotion state type nodes, cognitive behavioral therapy strategy nodes, and user profile feature nodes. In the feedback loop, conflict samples are stored in relation to environmental parameters from step 2.

Claims

1. A method for driving emotional interaction in intelligent devices based on multimodal understanding, characterized in that, include: Step 1: The smart device collects the user's facial video stream, voice signal and tactile interaction data. The feature extraction module adds timestamps to the facial video stream, voice signal and tactile interaction data and performs format standardization. The feature extraction module extracts facial action features of the facial video stream, prosodic feature vectors of the voice signal and spatial distribution parameters of the tactile interaction data. Step 2: The model building module establishes a cross-modal association model. The cross-modal association model calculates the association matrix between facial motion features and prosodic feature vectors. The environmental noise intensity and light sensor readings are input into the weight dynamic allocator to generate the adjustment coefficient of spatial distribution parameters in real time. Step 3: Input the fused features into the hierarchical classifier. The first level outputs the basic emotion category and intensity value, and the second level outputs the probability distribution of complex psychological states. When the anomaly detection module detects that the KL divergence value of the facial action feature and the prosodic feature vector is >0.3, it sets it to the compensation activation state and activates the emotion compensation channel in the user's historical interaction database. Combined with real-time heart rate variability data, it generates the emotion classification offset used to correct the output of the hierarchical classifier. Step 4: The offset correction module uses the emotion classification offset to correct the output of the hierarchical classifier. Using the output of the hierarchical classifier corrected by the offset correction module, it retrieves intervention plans from the psychological knowledge base. Based on the historical value of the execution success rate of the behavioral suggestions in the intervention plan, it adjusts the weight of the empathy expression template, generates a response strategy, converts the response strategy into a voice signal, and synchronously controls the PID gain parameter of the temperature control device to dynamically adjust with the emotion intensity value. It also drives the vibration device to generate tactile pulses that match the user's skin conductance response curve collected by the physiological sensor. Step 5: The feedback evaluation module records the rate of change of the user's facial micro-expressions and the delay time of the voice response. It calculates the consistency metric between the corrected hierarchical classifier output and the emotional state indicated by the user's response behavior using the Pearson correlation coefficient. When the consistency metric is lower than the preset threshold, the feedback evaluation module updates the user's emotion expression library and feeds the consistency metric back to the cross-modal association model for parameter tuning.

2. The method for emotional interaction of intelligent devices based on multimodal understanding according to claim 1, characterized in that, Step 1 includes: When the ambient light intensity is below the threshold, the intelligent device activates the infrared supplementary light mode, uses a face detection algorithm to process the facial video stream, and outputs the processed facial area video stream to the feature extraction module. The intelligent device uses dual-microphone beamforming to suppress ambient noise, segments the speech signal through an endpoint detection algorithm, and outputs the segmented effective speech segments to the feature extraction module. The intelligent device collects tactile interaction data through a matrix of capacitive pressure sensors, converts the tactile interaction data into a pressure distribution heatmap with a sampling frequency of not less than 10Hz, and outputs it to the feature extraction module.

3. The method for emotional interaction of intelligent devices based on multimodal understanding according to claim 2, characterized in that, Step 1 also includes: Calculate the Mel frequency cepstral coefficients of the speech signal and generate a Mel feature vector of a specified dimension; The Mel feature vector is input into the semantic understanding model deployed in the feature extraction module, and the output sentiment keyword confidence vector is fed into the hierarchical classifier as the input feature of the speech modality.

4. The method for emotional interaction of intelligent devices based on multimodal understanding according to claim 3, characterized in that, Step 3 includes: Calculate the KL divergence value based on facial movement features and prosodic feature vectors; When the KL divergence value exceeds 0.3, the emotion compensation channel in the user's historical interaction database is activated to retrieve the facial movement amplitude threshold, speech rate preference value and stress response sensitivity parameters recorded by the user in historical interaction scenarios. The parameters are weighted and averaged by combining the ambient noise intensity and the light sensor readings to generate the emotion classification offset, which is then input into the offset correction module.

5. The method for emotional interaction of intelligent devices based on multimodal understanding according to claim 4, characterized in that, Step 4 includes: Based on the probability distribution of complex psychological states in the output of the hierarchical classifier after the offset correction module, the empathy expression template library stored in the psychological knowledge base is called to generate emotion confirmation statements. By combining user emotional interaction response records stored in the user's historical interaction database, progressive guidance statements with three levels of guidance intensity are generated; Based on the node similarity-based intervention matching mechanism in the psychological knowledge base, cognitive restructuring exercises or interactive games are output as behavioral suggestions. The system dynamically integrates emotional affirmation statements, progressive guidance statements, and behavioral suggestions, and generates a final response strategy based on the adjusted empathy expression template weights from step 4.

6. The method for emotional interaction of intelligent devices based on multimodal understanding according to claim 5, characterized in that, Step 4 also includes: Extract the basic emotional intensity value from the output of the hierarchical classifier as the pleasure parameter, set a linear function of the basic temperature value, and generate a PID gain adjustment command for the temperature control device. Based on the activation value in the basic emotional intensity value in the output of the hierarchical classifier and the skin conductance response curve collected by the physiological sensor, the vibration device is controlled to generate a sinusoidal pulse wave of the corresponding frequency as a tactile pulse. The PID gain adjustment command and the tactile pulse signal are synchronously output to the temperature control device and the vibration device for execution.

7. The method for emotional interaction of intelligent devices based on multimodal understanding according to claim 6, characterized in that, Step 5 includes: When the consistency metric is lower than the preset threshold, the feature samples that conflict with the user response behavior after the hierarchical classifier output and the environmental noise intensity and light sensor reading data recorded in step 2 during the conflict are separated and stored to generate a fine-tuned training dataset. Based on the fine-tuned training dataset, the parameters of the feature extraction module are frozen, and only the weights of the fully connected layers of the cross-modal association model are updated. The updated cross-modal correlation model parameters are fed back to the model building module, replacing the cross-modal correlation model already deployed in the model building module to complete parameter tuning.

8. The method for emotional interaction of intelligent devices based on multimodal understanding according to claim 7, characterized in that, Step 4 also includes: Extract continuous values ​​of pleasure and activation from the basic emotion intensity values ​​in the output of the hierarchical classifier; When a smart device is configured with a screen, it generates and displays particle animations with color-coded continuous values ​​of pleasure. When the smart device is equipped with a screenless device, the LED light ring outputs a light signal whose hue is inversely proportional to the continuous value of pleasure and whose brightness is directly proportional to the continuous value of activation.

9. The method for emotional interaction of intelligent devices based on multimodal understanding according to claim 8, characterized in that, Step 3 includes: The first level outputs continuous values ​​of pleasure and activation as the basic emotional intensity value; The second level maps the basic emotional intensity value to the annular emotional space through polar coordinate transformation, and outputs a joint probability distribution composed of the probability values ​​of anxiety, loneliness and ambivalence as the probability distribution of complex psychological states to step 4.

10. The method for emotional interaction of intelligent devices based on multimodal understanding according to claim 9, characterized in that, Step 4 includes: Emotional state type, cognitive behavioral therapy strategy, and user profile features are used as nodes, with the user profile feature nodes derived from the user's historical interaction database. The correlation strength between emotion state type nodes and user profile feature nodes is calculated using the Pearson correlation coefficient, and the correlation strength is stored as the relation edge weight attribute value in the graph database. A knowledge graph structure is generated based on the relationship edges between nodes and including weight attribute values, and the structured topological relationship of the knowledge graph is invoked when retrieving intervention plans.

Images