Method and device based on multi-modal emotion recognition and self-adaptive feedback

Abstract

The invention provides a method and device based on multi-modal emotion recognition and self-adaptive feedback. According to the multi-modal emotion recognition and adaptive feedback-based method and device provided by the embodiment of the invention, the robustness of user emotion recognition in a complex scene can be improved, cross-device adaptive collaboration based on a real-time emotion state is realized, and the naturalness and personalization of interaction are enhanced.

Description

Technical Field

[0001] This invention relates to the field of multimodal emotion recognition and cross-device collaboration technology, and particularly to a method and apparatus based on multimodal emotion recognition and adaptive feedback. Background Technology

[0002] Affective computing, as a core technology in the field of human-computer interaction, is widely used in scenarios such as smart homes and health monitoring. Its development has driven the evolution of interactive systems from task execution to emotional intelligence services. Among related technologies, emotion classification is achieved by extracting features such as tone and energy from speech signals, or facial expression analysis is achieved using convolutional neural networks (CNNs) based on visual images. These constitute the two major existing technology systems for emotion recognition. Specifically, this technology ecosystem covers the entire process from data collection and feature extraction to classification decisions, including key links such as visual perception, speech processing, and text analysis. With the popularization of IoT devices, cross-device collaboration has gradually become a research focus for improving service continuity. Existing solutions mostly rely on cloud state synchronization and static rule triggering, initially achieving basic linkage between devices.

[0003] However, existing emotion recognition methods directly use a single modality (such as speech or vision) for classification without considering the dynamic weight allocation of multimodal information. This can lead to a significant drop in recognition accuracy in complex environments (such as noisy scenes or low-light conditions) or misjudgments due to modal interference. Based on this, while existing technologies have introduced multimodal fusion methods, most remain at the feature-level or decision-level static fusion level, failing to dynamically adjust the contribution of each modality according to environmental context (such as noise intensity and light changes), resulting in insufficient system robustness. Furthermore, cross-device collaboration logic is based solely on preset rules or static preferences, lacking the ability to perceive and reason about the user's real-time emotional state, thus failing to achieve emotion-driven dynamic service composition. In particular, the state synchronization and command forwarding mechanisms between devices fail to integrate emotional context, resulting in delayed service response, fragmented experience, and difficulty in meeting users' deep needs for natural and smooth interaction. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this invention is to propose a method based on multimodal emotion recognition and adaptive feedback.

[0006] The second objective of this invention is to propose a device based on multimodal emotion recognition and adaptive feedback.

[0007] To achieve the above objectives, a first aspect of the present invention proposes a method based on multimodal emotion recognition and adaptive feedback, comprising: S1, by using the visual perception unit, voice perception unit and text perception unit deployed on the terminal device, the user's facial image, audio stream and text information are obtained respectively, and the facial image is subjected to face detection and feature extraction, the audio stream is subjected to noise reduction and acoustic feature extraction, and the text information is subjected to semantic analysis and sentiment classification to generate corresponding visual feature vector, voice feature vector and text sentiment feature vector. S2, based on the cross-modal attention mechanism, the visual feature vector, speech feature vector and text sentiment feature vector are mapped to a high-dimensional common feature space, and the attention weight of each modality is dynamically calculated according to the environmental context. The weighted and concatenated feature vector is input into a multilayer perceptron for fusion calculation to generate a sentiment state vector representing the confidence score of N types of sentiment states. The subsequent collaborative process is triggered only when the maximum confidence value of the sentiment state vector exceeds a preset threshold. S3. Based on the emotional state vector, user ID, and scene tag, match the feedback action sequence from the preset emotional feedback device strategy library, and combine the device priority parameters, energy consumption constraints, and historical strategy matching success rate to dynamically optimize the strategy matching result through a reinforcement learning framework to generate cross-device collaborative instructions. S4. The cross-device collaboration command is sent to the target device combination for execution via the Internet of Things protocol, and the execution status of each device is monitored in real time. If a device execution failure is detected, the device combination is automatically adjusted based on the backup strategy library to ensure the continuity and robustness of the collaborative task.

[0008] In one embodiment of the present invention, S1 includes: S11, the visual perception unit uses the SeetaFace algorithm to detect faces and extracts facial feature points based on the AU action unit, wherein the AU action unit includes at least 12 key facial action features. S12, the speech perception unit acquires the audio stream through a ring microphone array and preprocesses it using a deep learning-based noise reduction model, wherein the deep learning noise reduction model is a WaveNet or Conv-TasNet structure.

[0009] In one embodiment of the present invention, S2 includes: S21, each modality feature vector is mapped to a common feature space of the same dimension through the embedding layer, wherein the visual feature vector V∈R^(d1), the speech feature vector A∈R^(d2), and the text sentiment feature vector T∈R^(d3) are all mapped to the common feature space R^D, where D is a preset uniform dimension parameter; S22, dynamically adjust the attention weights based on noise intensity and light illuminance in the environmental context, wherein the calculation formula for the attention weights α_i is as follows: α_i=exp(w_i^T·v_i+b_i) / Σ_{j=1}^3exp(w_j^T·v_j+b_j), Furthermore, w_i and b_i are updated online based on real-time environmental parameters through the backpropagation algorithm.

[0010] In one embodiment of the present invention, S3 includes: S31 employs the Actor-Critic reinforcement learning framework, using the user's positive / negative response to feedback as the reward signal R. The policy network parameters are optimized online through the gradient update formula θ←θ+η·▽_θlogπ(a|s)·(RV(s)), where η is the dynamically adjusted learning rate. S32, calculate the device priority parameters based on the device status score, the weighting coefficient of the physical distance between the user and the device, and the exponential decay function of the device usage frequency.

[0011] In one embodiment of the present invention, S4 includes: S41 uses the MQTToverTLS protocol for inter-device communication, where the TLS protocol version is TLS1.3, and the communication data packets contain device authentication fields and sentiment state vector summaries; S42, when a device execution failure is detected, the device combination is recalculated based on preset rules in the backup strategy library. The preset rules include device type replacement priority, energy consumption threshold reset and user distance compensation mechanism.

[0012] To achieve the above objectives, a second aspect of the present invention provides an apparatus based on multimodal emotion recognition and adaptive feedback, comprising: The multimodal perception module is used to acquire the user's facial image, audio stream and text information through the visual perception unit, voice perception unit and text perception unit deployed on the terminal device, and to perform face detection and feature extraction on the facial image, noise reduction and acoustic feature extraction on the audio stream, and semantic analysis and sentiment classification on the text information to generate corresponding visual feature vectors, voice feature vectors and text sentiment feature vectors. The cross-modal fusion module is used to map the visual feature vector, speech feature vector and text sentiment feature vector to a high-dimensional common feature space based on the cross-modal attention mechanism, and dynamically calculate the attention weight of each modality according to the environmental context. The weighted concatenated feature vector is then input into a multilayer perceptron for fusion calculation to generate a sentiment state vector representing the confidence score of N types of sentiment states. The subsequent collaborative process is triggered only when the maximum confidence value of the sentiment state vector exceeds a preset threshold. The strategy optimization module is used to match feedback action sequences from a pre-set emotional feedback device strategy library based on the emotional state vector, user ID, and scene tags, and dynamically optimize the strategy matching results through a reinforcement learning framework by combining device priority parameters, energy consumption constraints, and historical strategy matching success rates to generate cross-device collaborative instructions. The collaborative execution monitoring module is used to send the cross-device collaborative instructions to the target device combination for execution via the Internet of Things protocol, and monitor the execution status of each device in real time. If a device execution failure is detected, the device combination is automatically adjusted based on the backup strategy library to ensure the continuity and robustness of the collaborative task.

[0013] The method and apparatus of this invention can improve the robustness of user emotion recognition in complex scenarios and realize cross-device adaptive collaboration based on real-time emotional states, thereby enhancing the naturalness and personalization of interaction.

[0014] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0015] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a method based on multimodal emotion recognition and adaptive feedback according to an embodiment of the present invention; Figure 2 This is a system architecture diagram based on multimodal emotion recognition and adaptive feedback according to an embodiment of the present invention; Figure 3 This is a structural diagram of a device based on multimodal emotion recognition and adaptive feedback according to an embodiment of the present invention. Detailed Implementation

[0016] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0017] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0018] The following description, with reference to the accompanying drawings, describes a method and apparatus based on multimodal emotion recognition and adaptive feedback according to an embodiment of the present invention.

[0019] Example 1 Figure 1 This is a flowchart of a method based on multimodal emotion recognition and adaptive feedback according to an embodiment of the present invention, such as... Figure 1 As shown, it includes: S1, through the visual perception unit, voice perception unit and text perception unit deployed on the terminal device, respectively, acquire the user's facial image, audio stream and text information, and perform face detection and feature extraction on the facial image, noise reduction and acoustic feature extraction on the audio stream, and semantic analysis and sentiment classification on the text information to generate corresponding visual feature vectors, voice feature vectors and text sentiment feature vectors.

[0020] Specifically, this step is the core implementation of the multimodal perception layer. It aims to simultaneously collect multi-dimensional user interaction data through the visual, speech, and text perception units of the terminal device, and perform feature extraction and preliminary sentiment analysis on each, providing high-quality input feature vectors for subsequent multimodal fusion and adaptive decision-making. In some implementations, this step adopts a modular design, with each perception unit operating independently without interference, ensuring the real-time performance and stability of data acquisition.

[0021] Furthermore, the system acquires user facial images in real time using a high-definition camera (resolution ≥1080p, frame rate ≥30fps) deployed on the terminal device. After image preprocessing (grayscale conversion, histogram equalization), a lightweight face detection algorithm (such as SeetaFace) is used for face localization and key point extraction. Then, a pre-trained ResNet-18 variant model is used to extract emotion-related features such as facial action units (AUs), ultimately outputting a visual feature vector. ,in The output dimension of the model is typically 128 dimensions.

[0022] Furthermore, the system employs a circular microphone array (at least 4 microphones) to acquire the audio stream and filters out non-speech segments using speech activity detection (VAD). After noise reduction processing (such as a deep learning-based dereverberation model or spectral subtraction), acoustic features including MFCC (12-dimensional), F0 (fundamental frequency), energy, and speech rate are extracted to form a speech feature vector. ,in It is usually 32-dimensional.

[0023] Furthermore, the system receives text converted from user speech recognition (ASR) or directly input text information, performs semantic analysis and sentiment classification using a fine-tuned model based on BERT or RoBERTa, and outputs a text sentiment feature vector. ,in The emotional label dimension is usually 7 dimensions (such as happy, sad, angry, surprised, fearful, disgusted, and neutral).

[0024] The technical value of this step lies in achieving a comprehensive capture of the user's emotional state through multimodal perception, overcoming the limitations of single-modal recognition in complex environments. By standardizing the feature extraction process and unifying vector representation, it provides a structured and computable data foundation for subsequent cross-modal attention fusion and adaptive decision-making, making it a crucial preliminary step in building an emotion-driven intelligent interaction system.

[0025] Furthermore, S1 includes: S11, the visual perception unit uses the SeetaFace algorithm to detect faces and extracts facial feature points based on the AU action unit, wherein the AU action unit includes at least 12 key facial action features.

[0026] Specifically, in the multimodal perception layer of this invention, the visual perception unit uses the SeetaFace algorithm for face detection and extracts facial feature points based on AU (Action Unit) action units, wherein the AU action unit includes at least 12 key facial action features. This step is a fundamental step in realizing user emotional state recognition, and its technical implementation principle and operation method are as follows: Furthermore, the visual perception unit acquires user facial images in real time using a high-definition camera (resolution ≥1080p, sampling rate ≥30fps) deployed on the terminal device. The SeetaFace algorithm, a lightweight and high-precision face detection model based on a deep convolutional neural network (CNN) structure, can quickly locate face regions in images and output face bounding boxes and key point coordinates. This algorithm achieves high detection accuracy (mAP ≥ 0.92) on public datasets such as Casia and LFW, and maintains good robustness even in complex environments such as low light, side profiles, and occlusion.

[0027] Furthermore, after detecting a face, the system uses a Facial Action Unit (AU) model to extract feature points. AUs are facial muscle movement units defined by the Facial Action Coding System (FACS), used to quantify subtle changes in facial expressions. In this invention, the system extracts at least 12 key AU features, such as AU1 (inner eyebrow lift), AU2 (outer eyebrow lift), AU4 (frowning), AU6 (zygomatic muscle lift), and AU12 (corner of the mouth lift), etc. These AU features are significantly correlated with basic emotions (such as happiness, sadness, and anger). The feature extraction process is typically based on a pre-trained facial expression recognition model (such as FACSNet or OpenFace), using a combination of convolutional feature extraction and regression prediction to output an intensity score for each AU, ranging from 0 to 5, conforming to the FACS standard system.

[0028] Optionally, the system can normalize the extracted AU features and combine them with time series analysis (such as LSTM or Transformer) to capture the dynamic changes in facial expressions, thereby enhancing the temporal sensitivity of emotion recognition. Finally, these facial feature points are encoded into a visual feature vector V∈R^(d1), which serves as one of the inputs to the emotion fusion computing center for cross-modal attention fusion with speech and text modalities, improving the overall accuracy and stability of emotion recognition.

[0029] This step plays a crucial role in the system, as its high-precision face detection and AU feature extraction capabilities provide a reliable data foundation for subsequent multimodal fusion and emotion-based decision-making. Through standardized AU feature modeling, the system can capture user emotional changes more precisely, maintaining high recognition robustness even under complex lighting and occlusion conditions, thereby significantly improving the perception capabilities and naturalness of emotional responses in intelligent interaction systems.

[0030] S12, the speech perception unit acquires the audio stream through a ring microphone array and preprocesses it using a deep learning-based noise reduction model, wherein the deep learning noise reduction model is a WaveNet or Conv-TasNet structure.

[0031] Specifically, the speech perception unit acquires the audio stream through a circular microphone array and preprocesses it using a deep learning-based noise reduction model, which is either WaveNet or Conv-TasNet. This step plays a crucial role in the front-end audio signal processing of the multimodal emotion recognition system. Its technical implementation principle is based on the joint optimization of acoustic signal acquisition and deep neural networks, aiming to improve the signal-to-noise ratio (SNR) of the speech signal, thereby providing high-quality input for subsequent emotion feature extraction.

[0032] Furthermore, a circular microphone array typically consists of 4 to 8 omnidirectional or directional microphones arranged in a ring around the edge of the terminal device (such as a smart speaker or interactive robot) to achieve 360° sound source localization and spatial filtering. During audio acquisition, the system records in real time at a sampling rate of 16kHz or 48kHz, using PCM encoding format to ensure the temporal integrity of the audio signal. In the preprocessing stage, the system first performs speech activity detection (VAD) to identify valid speech segments and avoid invalid noise interference. Subsequently, the audio signal is input into a deep learning noise reduction model for enhancement processing.

[0033] Optionally, the noise reduction model can adopt a WaveNet structure, which is based on stacked causal convolutional layers and residual connections. It can model the temporal dependencies of speech signals and achieve end-to-end speech enhancement. WaveNet models typically use multi-channel mixed speech datasets (such as the DNSChallenge dataset) during the training phase, with noisy speech as input and clean speech as output. The model is optimized in the spectral or temporal domain, and the objective function is the mean squared error (MSE) or a spectral distortion metric (such as STOI or PESQ). Furthermore, the system can also adopt a Conv-TasNet structure. This model is based on a convolutional temporal analysis network and separates speech and noise through mask estimation, exhibiting high real-time processing capabilities and suitability for embedded deployment scenarios.

[0034] Furthermore, the system sets the input length of the denoising model to 256ms and the output step size to 128ms to ensure the continuity and real-time performance of the speech signal. The model inference latency should be controlled within 50ms to meet the low latency requirements of the emotion recognition system for speech input. In addition, the system extracts acoustic features such as MFCC, F0, energy, and speech rate from the denoised speech signal to form a feature vector A∈R^(d2), which serves as one of the inputs to the emotion fusion computing center.

[0035] This step is widely applicable in practical applications for multimodal interaction scenarios such as smart homes, in-vehicle assistants, and remote customer service. For example, in noisy home environments, a circular microphone array can effectively suppress background noise (such as television or air conditioner sounds), while WaveNet or Conv-TasNet models can further improve speech clarity, thereby enhancing the accuracy of emotion recognition. In terms of technical effects, this step significantly improves the signal-to-noise ratio (SNR improvement ≥5dB) of the speech signal and reduces the speech recognition false positive rate (VAD false positive rate ≤5%), providing stable and reliable speech feature input for subsequent multimodal emotion fusion. It is an important foundation for realizing emotion-driven intelligent interaction systems.

[0036] S2, based on the cross-modal attention mechanism, maps the visual feature vector, speech feature vector and text sentiment feature vector to a high-dimensional common feature space, and dynamically calculates the attention weight of each modality according to the environmental context. The weighted concatenated feature vector is input into a multilayer perceptron for fusion calculation to generate a sentiment state vector representing the confidence score of N types of sentiment states. The subsequent collaborative process is triggered only when the maximum confidence value of the sentiment state vector exceeds a preset threshold.

[0037] Specifically, this step, based on a cross-modal attention mechanism, maps visual feature vectors, speech feature vectors, and text sentiment feature vectors to a high-dimensional common feature space. It then dynamically calculates the attention weights for each modality based on the environmental context. Finally, the weighted concatenated feature vectors are input into a multilayer perceptron (MLP) for fusion computation, generating sentiment state vectors representing confidence scores for N types of sentiment states. This step is the core processing flow of the sentiment fusion computation center of this invention, aiming to solve the problems of weight allocation and context adaptation during the fusion process of multimodal heterogeneous data, thereby improving the robustness and accuracy of sentiment recognition.

[0038] Furthermore, firstly, the feature vectors of each modality (V∈R^(d1), A∈R^(d2), T∈R^(d3)) are dimensionally aligned and semantically mapped through independent embedding layers, uniformly encoding the features of different modalities into a high-dimensional common feature space (e.g., dimension 512). The embedding layer can be implemented using a fully connected layer or a learnable projection matrix to ensure semantic comparability among modalities. Subsequently, the system introduces a cross-modal attention mechanism, dynamically adjusting the attention weights of each modality by calculating the correlation between modalities. Specifically, the formula for calculating the attention weight α_i is: α_i=exp(w_i^T·v_i+b_i) / Σ_{j=1}^3exp(w_j^T·v_j+b_j) Here, w_i and b_i are learnable parameters, α_i∈[0,1], and Σα_i=1 to ensure a reasonable weight distribution. This mechanism can dynamically adjust the contribution of each modality according to the current environmental context (such as noise intensity, lighting conditions, user behavior patterns, etc.), for example, reducing the weight of the speech modality in a noisy environment and enhancing the influence of the visual and text modalities.

[0039] The weighted feature vectors are concatenated to form a fused feature vector F∈R^(d1+d2+d3), which is then input into a multilayer perceptron (MLP) for nonlinear fusion and sentiment classification. An MLP typically consists of 2-3 fully connected layers, each followed by a ReLU activation function and Dropout (e.g., Dropout=0.3) to prevent overfitting. The final output is a sentiment state vector E∈R^N, where N is the number of sentiment categories, and each dimension represents the confidence score of the corresponding sentiment.

[0040] Furthermore, the system sets a comprehensive confidence threshold T_confidence = 0.7. Subsequent adaptive decision-making and cross-device collaboration processes are only triggered when the maximum confidence value max(E) in the emotional state vector E exceeds T_confidence. This threshold can be fine-tuned according to the actual application scenario; for example, it can be set to 0.8 in a medical monitoring scenario to improve the reliability of recognition.

[0041] Furthermore, this step is widely applicable to multimodal interaction systems such as smart homes, intelligent customer service, and health monitoring. For example, after recognizing a user's emotion, the system can dynamically adjust services such as lighting, music, and push notifications to achieve emotion-driven personalized responses. By introducing attention mechanisms and confidence threshold control, the system can maintain high recognition accuracy and response consistency even in complex environments, significantly improving the naturalness of the interaction and user satisfaction.

[0042] In summary, this step, through the fusion of cross-modal attention mechanism and MLP model, achieves dynamic weighting and high-precision classification of multimodal emotional features. It is a key technical link in realizing emotion-driven cross-device collaboration in this invention, and has significant innovation and practicality.

[0043] Furthermore, S2 includes: S21, each modality feature vector is mapped to a common feature space of the same dimension through the embedding layer, wherein the visual feature vector V∈R^(d1), the speech feature vector A∈R^(d2), and the text sentiment feature vector T∈R^(d3) are all mapped to the common feature space of R^D, where D is a preset uniform dimension parameter.

[0044] Specifically, this step involves mapping feature vectors from different modalities (visual feature vector V∈R^(d1), speech feature vector A∈R^(d2), and text sentiment feature vector T∈R^(d3)) to a common feature space R^D of the same dimension through an embedding layer, where D is a preset unified dimension parameter. This step is a crucial preliminary step for realizing multimodal sentiment fusion computation. Its core technology lies in solving the alignment problem between heterogeneous feature spaces, providing a unified vector representation basis for subsequent cross-modal attention mechanisms and sentiment state fusion.

[0045] Furthermore, the embedding layer is typically implemented using a fully connected neural network or a trainable linear projection matrix. Specifically, for the feature vector of each modality, the system constructs an independent embedding sub-network with input dimensions d1, d2, and d3, and a uniform output dimension D. Optionally, the embedding layer can introduce non-linear activation functions (such as ReLU or Tanh) to enhance feature representation capabilities. Furthermore, residual connections can be combined to improve model stability.

[0046] Furthermore, the value of D needs to be set according to the system's requirements for feature representation complexity, typically between 128 and 512, with 256 or 512 being preferred, to balance computational efficiency and representation accuracy. The embedding layer can be initialized using Xavier or He initialization to ensure gradient stability in the early stages of training. In addition, the output of the embedding layer must satisfy the feature space alignment constraint, meaning that the semantic distribution of each modality in the common space should be consistent, which can be quantitatively evaluated using inter-modal cosine similarity or KL divergence.

[0047] Furthermore, this step is widely applicable to multimodal emotion recognition systems, such as smart homes, in-vehicle human-machine interaction, and remote customer service. In these scenarios, visual, speech, and text information often have different sampling frequencies and feature dimensions. The unified mapping of the embedding layer ensures the operability and computational consistency of subsequent fusion modules.

[0048] The technical effect of this step is that by mapping heterogeneous feature vectors to a unified high-dimensional common space, it provides a structured and computable input for subsequent cross-modal attention mechanisms and emotion state fusion, thereby significantly improving the robustness of emotion recognition and multimodal collaborative efficiency of the system in complex environments.

[0049] S22, dynamically adjust the attention weights based on noise intensity and light illuminance in the environmental context, wherein the calculation formula for the attention weights α_i is: α_i=exp(w_i^T·v_i+b_i) / Σ_{j=1}^3exp(w_j^T·v_j+b_j), Furthermore, w_i and b_i are updated online based on real-time environmental parameters through the backpropagation algorithm.

[0050] Specifically, this step involves dynamically adjusting attention weights based on noise intensity and light illuminance in the environmental context. Its core lies in real-time optimization of the weight allocation for each modality in the multimodal emotion recognition model by sensing changes in environmental parameters, thereby improving the system's robustness and accuracy in complex environments. In some implementations, this step uses sensors deployed on the terminal device (such as microphone arrays and ambient light sensors) to collect environmental parameters in real time and inputs them as contextual features into the attention weight calculation module. Specifically, noise intensity is analyzed in real-time using audio signals collected by the microphone array to calculate the signal-to-noise ratio (SNR) or decibel (dB) value of the current environment; light illuminance is obtained through an ambient light sensor, typically measured in lux, reflecting whether the current lighting conditions are suitable for visual modality feature extraction.

[0051] In the attention weight calculation, the feature vector of each modality With the corresponding weight parameters and bias Perform a dot product operation, and normalize the result using the Softmax function to obtain the attention weights. .in, and The feature vectors of speech modalities are updated online using a backpropagation algorithm to adapt to environmental changes. For example, in high-noise environments, the feature vectors of speech modalities are updated online. If interference occurs, the system will automatically reduce the weight of the speech modality. Furthermore, the weights of visual or text modalities are increased to maintain the stability of emotion recognition. In this step, the backpropagation update frequency can be set to once every 500ms, and the learning rate... It is usually set between 0.001 and 0.01 to ensure that the model maintains convergence and stability during dynamic adjustment.

[0052] This step plays a crucial role in the entire system and is the core mechanism for achieving adaptive multimodal emotion recognition. By introducing environmental context information, the system can dynamically adjust the contribution of each modality, thus maintaining high recognition accuracy even in complex scenarios such as low lighting and high background noise. For example, in dimly lit environments, the recognition confidence of the visual modality may decrease, and the system will automatically enhance the weight of the speech or text modality to compensate for the lack of visual information. This technical solution effectively solves the problem of existing technologies relying on static weights for multimodal fusion and being unable to adapt to environmental changes, significantly improving the environmental robustness of the emotion recognition model and the overall naturalness of the system's interaction.

[0053] S3. Based on the emotional state vector, user ID, and scene label, match the feedback action sequence from the preset emotional feedback device strategy library, and combine the device priority parameters, energy consumption constraints, and historical strategy matching success rate to dynamically optimize the strategy matching result through a reinforcement learning framework to generate cross-device collaborative instructions.

[0054] Specifically, in some implementations, the core step of the adaptive decision layer of this invention involves matching feedback action sequences from a pre-set emotional feedback device policy library based on the emotional state vector, user ID, and scene label. This is combined with device priority parameters, energy consumption constraints, and historical policy matching success rates. A reinforcement learning framework is then used to dynamically optimize the policy matching results, generating cross-device collaborative instructions. This step aims to transform multimodal emotion recognition results into personalized feedback policies with context awareness, enabling efficient and intelligent collaborative execution in a multi-device environment.

[0055] Furthermore, the system first takes the emotional state vector E (with dimension N, representing the confidence level of N types of emotions) along with the user ID and scene label as input to query a pre-set emotional feedback device strategy library. This strategy library uses a structured storage method, such as a relational database or graph database. Each strategy entry includes emotional state matching conditions, a sequence of feedback actions (such as light adjustment, music playback, message push, etc.), and a set of target devices. The matching process can employ rule-based retrieval or similarity-based vector retrieval. For example, it can calculate the matching degree between the current emotional vector and the pre-set emotional vectors in the strategy library using cosine similarity to filter out Top-K candidate strategies.

[0056] Furthermore, the system introduces a device priority parameter β, calculated as β = f(device status, user distance, usage frequency). Device status includes online status, response latency, and load conditions; user distance can be estimated based on Bluetooth signal strength or Wi-Fi RSSI; and usage frequency is obtained through user behavior log statistics. In addition, the system considers energy consumption constraints, such as setting the total energy consumption limit for a single collaborative task to E_total ≤ 500mWh, to ensure the sustainability of device collaboration.

[0057] Furthermore, the system adopts an Actor-Critic structure, treating the policy matching result as action 'a', and the current emotional state, user ID, and scene label as state 's'. The user's positive or negative response to feedback serves as a reward signal R, used to update the policy network parameters 'θ', optimizing the accuracy of policy matching and user satisfaction. The specific update formula is θ←θ+η·▽_θlogπ(a|s)·(RV(s)), where η is the learning rate and V(s) is the estimated value function.

[0058] Furthermore, for example, in the smart home scenario, when the system recognizes that the user is in a tired state, it can match the strategy to relieve fatigue and dynamically optimize the execution plan in combination with the user's current location (living room), device availability (lights, speakers, mobile phone), and historical matching success rate (γ = 0.85). The finally generated cross-device collaboration instructions are sent to the target devices through the MQTT over TLS protocol to achieve multi-device linkage operations such as dimming the lights, playing music, and pushing care messages.

[0059] The technical effect of this step is that by integrating multi-dimensional context information and the reinforcement learning mechanism, the personalization degree and execution efficiency of policy matching are significantly improved, while ensuring the robustness and user satisfaction of the system in complex environments.

[0060] Furthermore, S3 includes: S31, adopting the Actor-Critic reinforcement learning framework, using the positive / negative reaction of the user to the feedback as the reward signal R, and online optimizing the policy network parameters through the gradient update formula θ←θ+η·▽_θlogπ(a|s)·(R - V(s)), where η is the dynamically adjusted learning rate.

[0061] Specifically, this step adopts the Actor-Critic reinforcement learning framework, uses the positive / negative reaction of the user to the feedback as the reward signal R, and online optimizes the policy network parameters through the gradient update formula θ←θ+η·▽_θlogπ(a|s)·(R - V(s)), where η is the dynamically adjusted learning rate. This step is the core optimization mechanism in the adaptive decision-making layer of the present invention, aiming to achieve the real-time response of the system to the user's emotional state and the continuous evolution of the policy.

[0062] Furthermore, the Actor-Critic framework consists of a policy network (Actor) and a value network (Critic). The Actor network is responsible for outputting the action a to be taken in the current state s, that is, the feedback policy; the Critic network evaluates the value V(s) of this action, that is, the expected return of the system for the current state. In the present invention, the state s is composed of a multi-modal emotional state vector E, user ID, scene label, etc., and the action a is the feedback behavior sequence executed by the system (such as adjusting lights, playing music, pushing messages, etc.). The reward signal R is quantified according to the positive / negative reaction of the user to the feedback, for example, binary or continuous value encoding is performed through voice emotion classification results, user operation behaviors (such as clicking, ignoring, repeating requests), etc., and R ∈ [-1, 1].

[0063] Furthermore, the learning rate η in the gradient update formula employs a dynamic adjustment mechanism, adaptively adjusting based on the stability of the current policy execution and the volatility of user feedback. For example, when user feedback is negative multiple times consecutively (i.e., R < 0), the system will reduce η to avoid drastic policy oscillations; when feedback tends to stabilize and the R value is high, η will be appropriately increased to accelerate policy convergence. In addition, the value function V(s) is output through a Critic network, typically a multilayer fully connected network (MLP), with state s as input and a scalar value as output, used to measure the long-term expected return of the current policy. The policy function π(a|s) is based on an Actor network, outputting the action probability distribution. Its gradient calculation uses the policy gradient theorem, adjusting the policy direction using logπ(a|s).

[0064] Furthermore, this step is embedded in the online learning process of the intelligent interaction system, responding in real time to changes in the user's emotional state. For example, in a smart home scenario, when the system recognizes that a user is in a low mood and performs the feedback action of playing music, if the user subsequently exhibits positive behavior (such as nodding or smiling), R takes a positive value, and the policy network will increase the probability of this action in similar states; conversely, if the user exhibits a negative reaction (such as turning off the music or frowning), R takes a negative value, and the system will adjust the strategy to avoid repeatedly executing invalid feedback.

[0065] The technical advantage of this step lies in the fact that, by introducing a reinforcement learning mechanism, the system can continuously optimize its strategies based on user feedback signals, thereby achieving personalized and adaptive emotional interaction. Compared with traditional static strategy libraries, the online optimization strategy of this invention has stronger environmental adaptability and user preference learning capabilities, significantly improving the intelligence of the interactive system and the natural fluency of the user experience.

[0066] S32, calculate the device priority parameters based on the device status score, the weighting coefficient of the physical distance between the user and the device, and the exponential decay function of the device usage frequency.

[0067] Specifically, in some implementations, the calculation of the device priority parameter β is a key step in the adaptive decision layer of this invention. Its core purpose is to dynamically adjust the device's priority in collaborative tasks based on the device's current state, the physical distance between the user and the device, and the device's usage frequency, thereby improving the efficiency of cross-device collaboration and the consistency of the user experience. Specifically, the calculation of β comprehensively considers three dimensions: device state score, a weighted coefficient for the physical distance between the user and the device, and an exponential decay function based on the usage frequency over time intervals.

[0068] At the technical implementation level, device status scores are typically composed of indicators such as the device's online status, remaining battery power, current load, and response latency. Each indicator can be assigned a different weight; for example, online status has a weight of 0.4, battery power has a weight of 0.3, load has a weight of 0.2, and latency has a weight of 0.1. The total score is a normalized weighted sum, ranging from [0,1]. The weighting coefficient for the physical distance between the user and the device is obtained through positioning technologies such as Bluetooth, Wi-Fi RSSI, or UWB. The closer the distance, the higher the weight. It is usually converted into a value in the range [0,1] using a linear or non-linear mapping function.

[0069] Furthermore, the exponential decay function for device usage frequency is in the form β=α·exp(-λ·t), where α is the product of the device status score and the distance weighting coefficient, t represents the time interval between the last use of the device (in hours), and λ is a preset decay coefficient, typically ranging from [0.01, 0.1], used to control the strength of the impact of usage frequency on priority. For example, when λ=0.05, if the device was last used 24 hours ago, its usage frequency weight decays to approximately 0.30, significantly reducing its priority.

[0070] Specifically, in a smart home scenario, when a user is fatigued, the system needs to quickly select the optimal combination of devices to adjust the environment. Through dynamic calculation of β, the system can prioritize devices that are nearby, in good condition, and frequently used recently, such as smart speakers or smart lights, to ensure timely response and continuity of user habits.

[0071] Furthermore, the design of this exponential decay function effectively avoids the problem of devices being prioritized even after long periods of inactivity, improving the system's adaptability to user behavior. Simultaneously, the weighted calculation combining device status and distance makes device selection more scientific and rational, enhancing the robustness of cross-device collaboration and the consistency of user experience. This method provides crucial support for achieving emotion-driven intelligent collaboration in complex multi-device environments.

[0072] S4. The cross-device collaboration command is sent to the target device combination for execution via the Internet of Things protocol, and the execution status of each device is monitored in real time. If a device execution failure is detected, the device combination is automatically adjusted based on the backup strategy library to ensure the continuity and robustness of the collaborative task.

[0073] Specifically, this step involves sending cross-device collaborative instructions to a combination of target devices via IoT protocols for execution, and monitoring the execution status of each device in real time. If a device execution failure is detected, the device combination is automatically adjusted based on a backup strategy library to ensure the continuity and robustness of the collaborative task. This step is one of the core functions of the device collaboration layer in this invention. Its technical implementation is based on standard IoT communication protocols and a distributed task scheduling mechanism, combined with device status awareness and dynamic invocation of the strategy library to achieve intelligent and adaptive cross-device collaborative control.

[0074] Furthermore, the system employs the MQTT over TLS protocol for command issuance and status feedback, ensuring communication security and real-time performance. The MQTT protocol is widely used in IoT scenarios due to its lightweight nature, low latency, and support for QoS (Quality of Service) levels (QoS 0 / 1 / 2). In this system, the QoS level is dynamically set based on the urgency of the task; for example, QoS 2 is used for critical emotional feedback tasks (such as emergency care) to ensure reliable message delivery. The executor module obtains the real-time operating status (online / offline, executing / failed, etc.) of each device by subscribing to device status topics and maintains a device status table recording key parameters such as device ID, availability, response latency, and energy consumption level.

[0075] When a device fails to execute (e.g., timeout, execution error, or insufficient resources), the system triggers the backup strategy library. This library pre-stores various device combination schemes and their execution conditions; for example, if the smart speaker fails, the system switches to playing music through the smart TV speaker. Each strategy in the library includes parameters such as device combination rules, execution priority, energy consumption budget, and response time threshold. Based on current task requirements, device status, and strategy matching, the system uses a weighted scoring mechanism (e.g., based on the β device priority parameter and the γ strategy success rate history) to select the optimal backup scheme and reissues instructions to the new device combination via the MQTT protocol.

[0076] Furthermore, the system sets a device response timeout threshold of 500ms, a maximum of 3 retries for failed execution, and a backup strategy switching latency of less than 200ms to ensure a continuous user experience. In addition, the system supports dynamically adjusting weight parameters in the strategy library, such as prioritizing low-power device combinations under high load scenarios or increasing the priority of a device when a user approaches it.

[0077] This step is widely applicable in practical scenarios such as smart homes, health monitoring, and remote work, enabling multi-device collaboration. For example, when a user's emotion is recognized and a task of playing soothing music is triggered, if the main speaker malfunctions, the system can automatically switch to a secondary speaker or TV speaker to ensure the task is not interrupted. Through this mechanism, the system can maintain high availability and service continuity even when facing uncertainties such as device malfunctions and network fluctuations, significantly improving the robustness of cross-device collaborative systems and the consistency of user experience.

[0078] Furthermore, S4 includes: S41 uses the MQTToverTLS protocol for inter-device communication, with the TLS protocol version being TLS1.3, and the communication data packets containing device authentication fields and sentiment state vector summaries.

[0079] Specifically, in some implementations, this invention employs the MQTT over TLS protocol for inter-device communication, with the TLS protocol version being TLS 1.3. The communication data packets include device authentication fields and sentiment state vector digests. This step is a key communication mechanism in the device collaboration layer of the system architecture. Its technical implementation principle is based on the combination of the lightweight messaging protocol MQTT and the modern encrypted communication protocol TLS, ensuring the security, real-time performance, and traceability of inter-device communication.

[0080] Furthermore, the MQTT protocol, as a widely used publish / subscribe communication protocol in the Internet of Things (IoT), possesses advantages such as low bandwidth consumption, low latency, and high scalability, making it suitable for heterogeneous terminal communication in multi-device collaborative scenarios. To enhance communication security, this invention introduces the TLS 1.3 encryption mechanism on top of the MQTT protocol. This version supports more efficient key exchange algorithms (such as Elliptic Curve Key Agreement (ECDHE)) and employs forward secrecy to ensure that even if keys are leaked long-term, historical communication content will not be decrypted. Before establishing a connection, both communicating parties must complete authentication based on an X.509 certificate. The device authentication field is the Subject field in the certificate or the device's unique identifier (such as a UUID), used for identity verification and access control on the server side.

[0081] Furthermore, an emotion state vector digest is embedded in the communication data packet. This digest is a hash value (such as SHA-256) of the emotion state vector E∈R^N output by the multimodal emotion recognition module, used to quickly verify the integrity and consistency of the emotion state data at the receiving end. This digest field, together with the device identity field, constitutes the metadata part of the communication data packet, ensuring that the source of the emotion state information is trustworthy and the content has not been tampered with during cross-device collaboration.

[0082] Furthermore, MQTT over TLS communication employs QoS level 1 (ExactlyOnce) to ensure reliable transmission of critical sentiment state information. The communication port is typically set to 8883 (the default TLS port), and the TLS 1.3 protocol stack is implemented in accordance with RFC 8446. It is recommended that the device authentication field be at least 128 bits long, and the sentiment state vector digest be 32 bytes (256 bits) long, to balance transmission efficiency and security.

[0083] Furthermore, this communication mechanism is widely used in multi-device collaboration scenarios such as smart homes and health monitoring. For example, after user emotion recognition, the system sends an emotion state summary to smart lights, speakers, and other devices via MQTToverTLS, triggering corresponding adaptive feedback strategies. This step acts as a bridge in the entire system, securely and efficiently transmitting the emotion recognition results to collaborative devices, providing a reliable data foundation for subsequent personalized service execution.

[0084] Furthermore, this step ensures end-to-end encryption and authentication of device communication through the TLS 1.3 protocol, effectively preventing man-in-the-middle attacks and data tampering. At the same time, the introduction of sentiment state vector summaries enhances the system's ability to reliably verify sentiment state data, ensuring the accuracy and consistency of collaborative decision-making, thereby significantly enhancing the system's security and intelligence level.

[0085] S42, when a device execution failure is detected, the device combination is recalculated based on preset rules in the backup strategy library. The preset rules include device type replacement priority, energy consumption threshold reset and user distance compensation mechanism.

[0086] Specifically, when a device execution failure is detected, the system recalculates the device combination based on preset rules in the backup strategy library. This step is a key fault tolerance and optimization mechanism in the device collaboration layer of this invention, aiming to improve the system's robustness and service continuity in complex environments. In some implementations, this mechanism dynamically adjusts the device execution path through three types of rules: device type replacement priority, energy consumption threshold reset, and user distance compensation, ensuring that the emotion-driven interaction strategy can be continuously and effectively executed.

[0087] Furthermore, when the actuator detects that a device has failed to complete a specified task (such as lighting adjustment failure or speaker unresponsiveness), the system will trigger a call to the backup strategy library. First, based on the task type (such as environmental adjustment, information push, or voice interaction), the system retrieves the corresponding device type replacement priority table from the strategy library. This table ranks alternative devices based on device functional similarity, response latency, and user habits. For example, in the case of lighting adjustment failure, the system can prioritize smart curtains or projector backlight adjustment as alternatives. Second, the system resets the energy consumption threshold based on the current environmental conditions (such as power supply and network bandwidth) to ensure that the new device combination meets emotional feedback needs while complying with energy management standards (such as maximum power consumption not exceeding 200W). Furthermore, the user distance compensation mechanism calculates the physical distance between the user and candidate devices using real-time location data (such as Bluetooth beacons, Wi-Fi RSSI, or UWB positioning), prioritizing devices within a 3-meter distance to improve response speed and interaction naturalness.

[0088] Furthermore, the device type replacement priority table adopts a weighted scoring mechanism, with weights including device function matching degree (0.4), user usage frequency (0.3), response latency (0.2), and energy consumption level (0.1), with a total score range of 0-100. The energy consumption threshold reset module dynamically adjusts the threshold according to the energy management strategy of the device's location (such as home energy-saving mode, peak power rationing in commercial areas), supporting multi-level configurations (such as maximum power consumption ≤150W in low-power mode). In the user distance compensation mechanism, the distance calculation error must be controlled within ±0.5 meters to ensure the accuracy of device selection.

[0089] Furthermore, this step is widely applicable to multi-device collaboration scenarios such as smart homes, health monitoring, and office automation. For example, when a music playback strategy is triggered after user emotion recognition, if the main speaker malfunctions, the system can select a secondary speaker or mobile phone speaker for alternative playback based on the backup strategy library. At the same time, the system adjusts the volume and playback device combination according to the user's location to ensure a consistent experience.

[0090] The technical advantage of this step lies in its ability to effectively address service interruptions caused by equipment failures or environmental changes by dynamically adjusting the equipment combination, ensuring the continuous execution of emotion-driven strategies. Simultaneously, by combining multi-dimensional parameter optimization, the system meets users' emotional needs while also considering energy efficiency and response efficiency, significantly improving the stability and intelligence level of the cross-device collaborative system.

[0091] The cross-device collaboration method based on multimodal dynamic fusion in this invention can improve the robustness and accuracy of user emotion recognition in complex scenarios, realize adaptive interaction strategies based on real-time emotion states and cross-device intelligent collaboration, and enhance the naturalness of interaction and the consistency of services.

[0092] Example 2 This invention constructs a closed-loop intelligent interaction system integrating multimodal perception, emotion computing, intelligent decision-making, and cross-device collaboration. Its core innovations lie in proposing a multimodal dynamic fusion model based on an attention mechanism, a context-aware quantitative decision-making method, and a policy optimization mechanism based on reinforcement learning, ultimately achieving precise emotion-driven cross-device collaboration. The system architecture is as follows: Figure 2 As shown, a four-level processing architecture is adopted: multimodal perception layer, intelligent fusion layer, adaptive decision-making layer and device collaboration layer.

[0093] In one embodiment of the present invention, a multimodal perception layer: this module is responsible for the acquisition and preprocessing of raw data, and constitutes the perception layer of the system.

[0094] Furthermore, the visual perception unit: a high-definition camera (resolution ≥ 1080p, sampling rate ≥ 30fps) deployed on the terminal device captures the user's facial image in real time. A lightweight face detection algorithm based on SeetaFace or a similar structure is employed. After detecting a face, continuous facial feature points (such as AU action units) are extracted and input into a pre-trained deep learning model (such as a ResNet-18 variant) for preliminary sentiment analysis, outputting a visual feature vector V∈R^(d1).

[0095] Furthermore, the speech perception unit: acquires the audio stream through a circular microphone array (at least 4 microphones), and performs noise reduction (using spectral subtraction or a deep learning-based noise reduction model) and speech activity detection (VAD) in the preprocessing stage. Subsequently, it extracts acoustic feature vectors A∈R^(d2), including MFCC (Mel-frequency cepstral coefficients), F0 (fundamental frequency), energy, speech rate, etc.

[0096] Furthermore, the text perception unit receives text converted by the user through speech recognition (ASR) or directly input text information in real time. It employs a fine-tuned model based on BERT or RoBERTa for semantic analysis and sentiment classification (Positive / Negative / Neutral and specific sentiment labels), outputting a text sentiment feature vector T∈R^(d3).

[0097] In one embodiment of the present invention, the emotion fusion computing center is responsible for solving the consistency and collaborative processing problem of multimodal heterogeneous data.

[0098] Furthermore, the model architecture employs a deep fusion network based on cross-modal attention. The feature vectors of each modality are first mapped to a high-dimensional common feature space through independent embedding layers. Subsequently, a module for calculating inter-modal attention weights is designed to dynamically calculate the importance weights of different modalities in the current context.

[0099] Furthermore, attention weight calculation: For each modality feature, the attention weight is calculated as follows: α_i=exp(w_i^T·v_i+b_i) / Σ_{j=1}^3exp(w_j^T·v_j+b_j) Where v_i represents the feature vector of the i-th mode, w_i and b_i are learnable parameters, α_i∈[0,1] represents the weight of the i-th mode, and satisfies Σα_i=1.

[0100] Furthermore, fusion and decision-making: The weighted feature vectors are concatenated and input into a multilayer perceptron (MLP) for fusion computation. F=MLP([α_v·V;α_a·A;α_t·T]) The final output is a unified multimodal sentiment state vector E∈R^N, representing the confidence scores of N sentiment states. The system sets a comprehensive confidence threshold: T_confidence=0.7, The subsequent decision-making process is triggered only when max(E) > T_confidence.

[0101] Furthermore, historical data utilization: The system maintains a time-series-based emotional state profile for the user, recording historical emotional vectors, triggering scenarios, and feedback effects.

[0102] In one embodiment of the present invention, the adaptive decision layer is responsible for transforming emotional states into personalized interaction strategies and has online learning capabilities.

[0103] Furthermore, the strategy library and matching logic: The engine has a pre-built strategy library called Emotion Feedback Device Mapping. This strategy library takes the emotion vector E, user ID, and scene label as input, and the feedback action sequence as output.

[0104] Furthermore, quantitative decision-making: The decision-making process introduces multiple quantitative parameters: feedback intensity parameter α=max(E), which is positively correlated with emotional confidence; device priority parameter β=f(device status, user distance, usage frequency); and historical data of strategy matching success rate γ∈[0,1].

[0105] Furthermore, reinforcement learning optimization: The system employs an Actor-Critic-based reinforcement learning framework for online optimization. The user's positive or negative feedback serves as a reward signal R, used to update the parameters of the policy network. θ←θ+η·▽_θlogπ(a|s)·(RV(s)) Where η is the learning rate, π(a|s) is the policy function, and V(s) is the value function.

[0106] In one embodiment of the present invention, the device collaboration layer is responsible for transforming the abstract strategies output by the decision engine into specific, cross-device collaborative actions.

[0107] Furthermore, the coordination protocol: the actuator communicates with other terminals in a device network via standard IoT protocols such as MQTT over TLS.

[0108] Furthermore, the collaborative logic is as follows: After receiving the instructions from the decision engine, the executor selects the optimal combination of devices from the registered devices to execute the task based on the current device availability, β priority parameter, and energy consumption constraints.

[0109] Furthermore, execution and monitoring: After the executor issues instructions, it monitors the execution status of each device. If a device fails to execute, a backup plan will be automatically activated to ensure the continuity and robustness of the service experience.

[0110] The embodiments of the present invention also have the following technical effects: Specifically, dynamic multimodal emotion fusion: adopts a cross-modal attention mechanism to dynamically calculate the weights of vision, speech, and text based on the environmental context, significantly improving the recognition robustness in complex scenarios; data-driven adaptive decision-making: establishes an emotion feedback device mapping strategy library, introduces a reinforcement learning mechanism, and optimizes decision-making strategies online based on users' historical feedback data to achieve personalized emotional interaction; emotion-driven cross-device collaboration: uses the emotion state vector as a unified input, and dynamically generates and executes cross-device collaboration schemes based on multi-dimensional constraints such as device status and priority, to achieve intelligent linkage between services and emotions.

[0111] Example 3 To achieve the above embodiments, such as Figure 3 As shown, this embodiment also provides a device 10 based on multimodal emotion recognition and adaptive feedback, including: The multimodal perception module 100 is used to acquire the user's facial image, audio stream and text information respectively through the visual perception unit, voice perception unit and text perception unit deployed on the terminal device, and to perform face detection and feature extraction on the facial image, noise reduction and acoustic feature extraction on the audio stream, and semantic analysis and sentiment classification on the text information to generate corresponding visual feature vectors, voice feature vectors and text sentiment feature vectors. The cross-modal fusion module 200 is used to map the visual feature vector, speech feature vector and text sentiment feature vector to a high-dimensional common feature space based on the cross-modal attention mechanism, and dynamically calculate the attention weight of each modality according to the environmental context. The weighted concatenated feature vector is then input into a multilayer perceptron for fusion calculation to generate a sentiment state vector representing the confidence score of N types of sentiment states. The subsequent collaborative process is triggered only when the maximum confidence value of the sentiment state vector exceeds a preset threshold. The strategy optimization module 300 is used to match feedback action sequences from a pre-set emotional feedback device strategy library based on the emotional state vector, user ID and scene tags, and dynamically optimize the strategy matching results through a reinforcement learning framework by combining device priority parameters, energy consumption constraints and historical strategy matching success rates to generate cross-device collaborative instructions. The collaborative execution monitoring module 400 is used to send the cross-device collaborative instructions to the target device combination for execution via the Internet of Things protocol, and monitor the execution status of each device in real time. If a device execution failure is detected, the device combination is automatically adjusted based on the backup strategy library to ensure the continuity and robustness of the collaborative task.

[0112] Furthermore, the multimodal sensing module 100 is also used for: The visual perception unit uses the SeetaFace algorithm for face detection and extracts facial feature points based on the AU action unit, wherein the AU action unit includes at least 12 key facial action features. The speech perception unit acquires the audio stream through a ring microphone array and preprocesses it using a deep learning-based noise reduction model, which is either WaveNet or Conv-TasNet.

[0113] Furthermore, the cross-modal fusion module 200 is also used for: The embedding layer maps each modality feature vector to a common feature space of the same dimension. The visual feature vector V∈R^(d1), speech feature vector A∈R^(d2), and text sentiment feature vector T∈R^(d3) are all mapped to the common feature space R^D, where D is a preset uniform dimension parameter. The attention weight is dynamically adjusted based on the noise intensity and light illuminance in the environmental context, wherein the calculation formula for the attention weight α_i is: α_i=exp(w_i^T·v_i+b_i) / Σ_{j=1}^3exp(w_j^T·v_j+b_j), Furthermore, w_i and b_i are updated online based on real-time environmental parameters through the backpropagation algorithm.

[0114] Furthermore, the strategy optimization module 300 is also used for: The Actor-Critic reinforcement learning framework is adopted, with the user's positive / negative response to the feedback as the reward signal R. The policy network parameters are optimized online through the gradient update formula θ←θ+η·▽_θlogπ(a|s)·(RV(s)), where η is the dynamically adjusted learning rate. The device priority parameters are calculated based on the device status score, the weighting coefficient of the physical distance between the user and the device, and the exponential decay function of the device usage frequency.

[0115] Furthermore, the collaborative execution monitoring module 400 is also used for: The MQTT over TLS protocol is used for inter-device communication, with the TLS protocol version being TLS 1.3, and the communication data packets include device authentication fields and sentiment state vector summaries; When a device failure is detected, the device combination is recalculated based on preset rules in the backup strategy library. The preset rules include device type replacement priority, energy consumption threshold reset, and user distance compensation mechanism.

[0116] Furthermore, it also includes: The Time Series Emotional Profile module is used to construct a time series emotional profile based on the user's historical emotional state vector, and to calculate the deviation between the current emotional state and the historical emotional pattern through a sliding window mechanism. When the deviation exceeds a preset threshold, an active intervention strategy is triggered. The proactive intervention module is used to initiate video call requests to preset contacts and push health advice information through smart terminals. The triggering condition for the video call request is that three consecutive detections of low mood and voice activity detection values ​​below a preset threshold are made.

[0117] The cross-device collaborative device based on multimodal dynamic fusion in this invention can improve the robustness and accuracy of user emotion recognition in complex scenarios, realize adaptive interaction strategies based on real-time emotional states and cross-device intelligent collaboration, and enhance the naturalness of interaction and the consistency of services.

[0118] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0119] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method based on multi-modal emotion recognition and adaptive feedback, characterized in that, Comprise: S1, through the visual perception unit, speech perception unit and text perception unit deployed in the terminal device, respectively, obtain the face image, audio stream and text information of the user, and perform face detection and feature extraction on the face image, noise reduction processing and acoustic feature extraction on the audio stream, and semantic analysis and emotion classification on the text information, to generate corresponding visual feature vector, speech feature vector and text emotion feature vector; S2, based on the cross-modal attention mechanism, map the visual feature vector, speech feature vector and text emotion feature vector to a high-dimensional common feature space, dynamically calculate the attention weight of each modality according to the environmental context, input the weighted spliced feature vector into a multilayer perception machine for fusion calculation to generate an emotion state vector representing the confidence score of N emotion states, and only when the maximum confidence value of the emotion state vector exceeds a preset threshold, trigger the subsequent collaborative process; S3, according to the emotion state vector, user ID and scene label, match the feedback action sequence from the pre-set emotion feedback device strategy library, combine the device priority parameter, energy consumption constraint and historical strategy matching success rate, and dynamically optimize the strategy matching result through the reinforcement learning framework to generate a cross-device collaborative instruction; S4, the cross-device collaborative instruction is issued to the target device combination through the Internet of Things protocol for execution, and the execution state of each device is monitored in real time, and if a device execution failure is detected, the device combination is automatically adjusted based on the backup strategy library to ensure the continuity and robustness of the collaborative task.

2. The method of claim 1, wherein, The S1 further comprises: S11, the visual perception unit uses SeetaFace algorithm for face detection, and extracts facial feature points based on AU action unit, wherein the AU action unit includes at least 12 key facial action features; S12, the speech perception unit collects audio stream through a ring microphone array, and uses a deep learning-based noise reduction model for preprocessing, wherein the deep learning noise reduction model is WaveNet or Conv-TasNet structure.

3. The method of claim 1, wherein, The S2 further comprises: S21, map each modality feature vector to a common feature space with the same dimension through an embedding layer, wherein the visual feature vector V∈R^(d1), the speech feature vector A∈R^(d2), and the text emotion feature vector T∈R^(d3) are all mapped to a common feature space R^D, and D is a preset unified dimension parameter; S22, dynamically adjust the attention weight according to the noise intensity and light intensity in the environmental context, wherein the calculation formula of the attention weight α_i is: α_i=exp(w_i^T·v_i+b_i) / Σ_{j=1}^3exp(w_j^T·v_j+b_j), and w_i and b_i are updated online according to real-time environmental parameters through the back propagation algorithm.

4. The method of claim 1, wherein, The S3 further comprises: S31, using an Actor-Critic reinforcement learning framework, taking the positive / negative reaction of the user to the feedback as a reward signal R, and online optimizing the policy network parameters through the gradient update formula θ←θ+η·▽_θlogπ(a|s)·(R-V(s)), where η is a dynamically adjusted learning rate; S32, calculating the device priority parameter according to the device state score, the physical distance weighting coefficient of the user and the device, and the exponential decay function of the device usage frequency.

5. The method of claim 1, wherein, The S4 further comprises: S41, using the MQTToverTLS protocol for inter-device communication, wherein the TLS protocol version is TLS1.3, and the communication data packet contains a device identity authentication field and an emotional state vector summary; S42, when detecting device execution failure, recalculating the device combination based on preset rules in the backup strategy library, wherein the preset rules include device type replacement priority, energy consumption threshold reset, and user distance compensation mechanism.

6. A device based on multimodal emotion recognition and adaptive feedback, characterized in that, Comprise: A multi-modal perception module for acquiring facial images, audio streams and text information of a user through visual perception units, voice perception units and text perception units deployed on terminal devices, performing face detection and feature extraction on the facial images, performing noise reduction processing and acoustic feature extraction on the audio streams, and performing semantic analysis and emotion classification on the text information to generate corresponding visual feature vectors, voice feature vectors and text emotional feature vectors; A cross-modal fusion module for mapping the visual feature vectors, voice feature vectors and text emotional feature vectors to a high-dimensional common feature space based on a cross-modal attention mechanism, dynamically calculating attention weights of each modality according to environmental context, inputting the weighted spliced feature vectors into a multi-layer perception machine for fusion calculation to generate an emotional state vector representing confidence scores of N emotional states, and triggering a subsequent collaborative process only when the maximum confidence value of the emotional state vector exceeds a preset threshold; A strategy optimization module for matching feedback action sequences from a preconfigured emotional feedback device strategy library according to the emotional state vector, user ID and scene label, and dynamically optimizing the strategy matching results through a reinforcement learning framework to generate cross-device collaboration instructions in combination with device priority parameters, energy consumption constraints and historical strategy matching success rates; A collaborative execution monitoring module for issuing the cross-device collaboration instructions to a target device combination for execution through an Internet of Things protocol, and monitoring the execution status of each device in real time, and automatically adjusting the device combination based on a backup strategy library if a device execution failure is detected to ensure the continuity and robustness of the collaborative task.

7. The apparatus of claim 6, wherein, The multi-modal perception module is further configured to: The visual perception unit uses the SeetaFace algorithm for face detection and extracts facial feature points based on AU action units, wherein the AU action units include at least 12 key facial action features; The voice perception unit acquires audio streams through a ring microphone array and pre-processes them using a deep learning-based noise reduction model, which is a WaveNet or Conv-TasNet structure.

8. The apparatus of claim 6, wherein, The cross-modal fusion module is further configured to: The modality feature vectors are respectively mapped to a common feature space with the same dimension by embedding layers, wherein the visual feature vector V∈R^(d1), the speech feature vector A∈R^(d2), and the text sentiment feature vector T∈R^(d3) are all mapped to a common feature space R^D, and D is a preset unified dimension parameter; The attention weight is dynamically adjusted according to the noise intensity and light illumination in the environmental context, wherein the calculation formula of the attention weight α_i is: α_i=exp(w_i^T·v_i+b_i) / Σ_{j=1}^3exp(w_j^T·v_j+b_j), and w_i and b_i are updated online according to real-time environmental parameters by a back propagation algorithm.

9. The apparatus of claim 6, wherein, The policy optimization module is further configured to: adopt an Actor-Critic reinforcement learning framework, take the positive / negative reaction of the user to the feedback as a reward signal R, and update the policy network parameters online through a gradient update formula θ←θ+η·▽_θlogπ(a|s)·(R-V(s)), wherein η is a dynamically adjusted learning rate; calculate the device priority parameter according to the device state score, the physical distance weighting coefficient between the user and the device, and an exponential decay function of the device usage frequency.

10. The apparatus of claim 6, wherein, The cooperative execution monitoring module is further configured to: adopt an MQTT over TLS protocol for inter-device communication, wherein the TLS protocol version is TLS1.3, and the communication data packet contains a device identity authentication field and a sentiment state vector digest; when detecting a device execution failure, recalculate the device combination based on preset rules in a backup policy library, wherein the preset rules include a device type replacement priority, an energy consumption threshold reset, and a user distance compensation mechanism.

Images