A Personalized Cockpit Interaction System and Method Based on User Behavior Analysis and Emotion Recognition

By using multimodal data acquisition and emotion recognition technologies, dynamic user profiles are constructed, solving the problem of the single perception and interaction strategies of intelligent cockpits. This enables full-dimensional perception and high-precision emotion recognition in autonomous driving scenarios, improving driving safety and comfort.

CN123104545APending Publication Date: 2026-11-17DONGFENG MOTOR GRP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611049091.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-15
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing smart cockpit technologies have limited perception dimensions, making them unsuitable for the autonomous monitoring needs within vehicles in autonomous driving scenarios. They also suffer from insufficient robustness in emotion recognition, rigid interaction strategies lacking dynamic user profile construction, delayed anomaly responses, and inadequate security.

Method used

A multimodal data acquisition module is used, combined with image, audio, physiological and environmental sensors. The user behavior modeling module performs preprocessing and feature fusion to establish an emotion computing framework that integrates discrete emotion and dimensional emotion. The interaction strategy is dynamically adjusted and iteratively optimized through the interaction execution and feedback module.

Benefits of technology

It achieves full-dimensional perception of the in-vehicle status, improves the accuracy and robustness of emotion recognition, enhances the ability to personalize interaction, and improves driving safety and comfort.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123104545A_ABST
    Figure CN123104545A_ABST
Patent Text Reader

Abstract

This invention discloses a personalized cockpit interaction system and method based on user behavior analysis and emotion recognition, relating to the field of automotive intelligent cockpit technology. The system includes: a multimodal data acquisition module, a user behavior modeling module, an emotion recognition module, a personalized interaction strategy module, and an interaction execution and feedback module. The multimodal data acquisition module collects multi-dimensional data within the cockpit; the user behavior modeling module performs data preprocessing, feature extraction, fusion, and behavior pattern recognition; the emotion recognition module constructs a discrete-dimensional fusion emotion model and employs a three-level fusion strategy to achieve high-precision emotion recognition; the personalized interaction strategy module constructs a dynamic user profile and generates environmental adjustment, emotional interaction, and service recommendation strategies; and the interaction execution and feedback module performs strategy execution, feedback collection, and model iterative optimization. This invention improves the comprehensiveness of cockpit environmental perception, the robustness of emotion recognition, and the personalized adaptability of interaction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of automotive intelligent cockpit technology, and in particular to a personalized cockpit interaction system and method based on user behavior analysis and emotion recognition. Background Technology

[0002] With the rapid development of vehicle electrification, intelligence, and connectivity, intelligent cockpits have become a core competitive advantage for vehicle manufacturers, and their interactive experience directly determines driving comfort and safety. Currently, most intelligent cockpit interaction technologies focus on basic driver / passenger status monitoring, such as fatigue and distraction detection, or single-modal emotional interaction, such as facial expression emotion recognition. These technologies struggle to achieve a deep integration and perception of user behavior, emotions, and the cockpit environment.

[0003] The existing technology has four major shortcomings: First, it has a single perception dimension, focusing on human body status monitoring while neglecting comprehensive perception of the cabin environment (cleanliness, maintenance, gas anomalies), and cannot adapt to the autonomous monitoring needs of the vehicle in autonomous driving scenarios; Second, the robustness of emotion recognition is insufficient, with single-modal recognition being susceptible to light and noise interference, and multimodal fusion lacking a dynamic weighting mechanism, resulting in low recognition accuracy and weak anti-interference ability; Third, the interaction strategy is fixed, relying on preset rules, lacking dynamic user profile construction and real-time iterative optimization, and has poor personalized adaptation capabilities; Fourth, the abnormal response is lagging, unable to automatically identify cleaning, maintenance, and emergency situations and trigger corresponding decisions, resulting in insufficient security.

[0004] Therefore, there is an urgent need for a personalized cockpit interaction system and method based on user behavior analysis and emotion recognition to solve the existing technical problems. Summary of the Invention

[0005] This invention aims to solve at least one of the technical problems existing in the prior art, and proposes a personalized cockpit interaction system based on user behavior analysis and emotion recognition.

[0006] In a first aspect, embodiments of the present invention provide a personalized cockpit interaction system based on user behavior analysis and emotion recognition, comprising: a multimodal data acquisition module, a user behavior modeling module, an emotion recognition module, a personalized interaction strategy module, and an interaction execution and feedback module;

[0007] The multimodal data acquisition module is used to collect raw multimodal data of users in the cockpit;

[0008] The user behavior modeling module is connected to the multimodal data acquisition module and is used to preprocess the multimodal raw data to construct a user behavior model.

[0009] The emotion recognition module is connected to the user behavior modeling module and is used to establish an emotion computing framework that integrates discrete emotion models and dimensional emotion models to identify the user's real-time emotional state.

[0010] The personalized interaction strategy module is connected to the emotion recognition module and is used to dynamically adjust the cockpit's interaction strategy and service content based on the user behavior model and the user's real-time emotional state.

[0011] The interaction execution and feedback module is connected to the personalized interaction strategy module. It is used to execute interaction strategies, collect user behavior feedback, and iteratively optimize user profiles, interaction strategy rules, and model parameters based on feedback data to achieve continuous optimization of the interaction experience.

[0012] Furthermore, the multimodal data acquisition module includes an image sensor, an audio sensor, a physiological sensor, and an environmental sensor, used to acquire multimodal raw data of user facial images, voice signals, physiological parameters, and cabin environmental parameters within the cockpit.

[0013] Furthermore, the user behavior modeling module preprocesses the multimodal raw data. The specific methods include: denoising, standardizing, and reducing the dimensionality of the multimodal raw data; extracting temporal, spatial, interactive, and semantic features; using an attention mechanism that combines early and late fusion to complete feature fusion; identifying user behavior patterns through unsupervised clustering and deep learning models; and constructing a user behavior model.

[0014] Furthermore, the multimodal raw data is preprocessed using algorithms including wavelet transform filtering for denoising, Z-score or Min-Max standardization, and PCA or t-SNE dimensionality reduction. Feature extraction employs CNN to extract visual features, RNN to capture temporal features, and TF-IDF and word embedding to extract semantic features. Feature fusion uses an attention mechanism to dynamically allocate modality weights. Behavioral pattern recognition uses K-means clustering, hybrid neural networks, and isolated forest anomaly detection algorithms.

[0015] Furthermore, the emotion recognition module is used to establish an emotion computing framework that integrates discrete emotion models and dimensional emotion models. The specific methods include: extracting facial expression, speech acoustic and physiological signal features, adopting a three-level fusion strategy of data level, feature level and decision level, and outputting discrete emotion categories and valence-arousal dimension scores through a hybrid neural network to identify the user's real-time emotional state.

[0016] Furthermore, the discrete emotion model of the emotion recognition module sets feature thresholds and weights based on Ekman's six basic emotions, and the dimensional emotion model maps emotions based on Russell's valence-arousal two-dimensional model; in the three-level fusion strategy, data-level fusion is feature vector concatenation, feature-level fusion dynamically allocates modality weights according to emotion type, and decision-level fusion uses a voting mechanism or Bayesian inference to integrate classification results; the hybrid neural network includes CNN, LSTM and fully connected networks, with a network depth of 8 layers and the number of hidden layer neurons is 128-256-512-256-128.

[0017] Furthermore, the personalized interaction strategy module is used to dynamically adjust the cockpit's interaction strategy and service content based on the user behavior model and the user's real-time emotional state. The specific method includes: constructing a dynamic hierarchical user profile based on the user behavior model, the user's real-time emotional state, and spatiotemporal scene information, and generating interaction strategies for cockpit environment adjustment, emotional interaction support, and personalized service recommendation by matching the strategy template library.

[0018] Furthermore, the user profile of the personalized interaction strategy module is divided into long-term stable features and short-term changing features, and is dynamically updated using a time decay model; the interaction strategy generation includes strategy matching, personalized adjustment, multimodal fusion, priority ranking, and conflict elimination; the cabin environment adjustment includes gradual adjustment of temperature, light, audio, seat, and air parameters; emotional interaction support includes voice tone and speech rate adjustment, visual feedback optimization, and proactive intervention for negative emotions; personalized service recommendations include navigation, entertainment, dining, rest reminders, and social connection services, adapted to driving safety status.

[0019] Furthermore, the feedback collection of the interaction execution and feedback module includes voice feedback, touch score, behavioral feedback, and emotional feedback; feedback analysis adopts SVM, random forest, or deep learning models; model optimization adopts incremental learning and reinforcement learning, preset feedback triggers parameter updates, and an emergency intervention mechanism is established to respond to sudden feedback problems.

[0020] Secondly, this invention also discloses a personalized cockpit interaction method based on user behavior analysis and emotion recognition, comprising:

[0021] S100. Collect raw multimodal data of users in the cockpit;

[0022] S200. Preprocess multimodal data to identify user behavior patterns and build behavior models;

[0023] S300 extracts facial, voice, and physiological emotional features, and identifies the user's real-time emotional state through a three-level fusion strategy;

[0024] S400 integrates behavioral models, emotional states, and contextual information to construct dynamic user profiles and generate personalized interaction strategies.

[0025] S500. Execute the interaction strategy, collect user feedback, iteratively optimize the profile, strategy and model parameters, and repeat steps S200-S500.

[0026] This invention discloses a personalized cockpit interaction system and method based on user behavior analysis and emotion recognition. The system includes: a multimodal data acquisition module, a user behavior modeling module, an emotion recognition module, a personalized interaction strategy module, and an interaction execution and feedback module. The multimodal data acquisition module collects multi-dimensional data within the cockpit through image, audio, physiological, and environmental sensors. The user behavior modeling module completes data preprocessing, feature extraction, fusion, and behavior pattern recognition. The emotion recognition module constructs a discrete-dimensional fusion emotion model and employs a three-level fusion strategy to achieve high-precision emotion recognition. The personalized interaction strategy module constructs a dynamic user profile and generates environmental adjustment, emotional interaction, and service recommendation strategies. The interaction execution and feedback module completes strategy execution, feedback collection, and model iterative optimization. This invention improves the comprehensiveness of cockpit environmental perception, the robustness of emotion recognition, and the personalized adaptability of interaction, adapting to autonomous driving scenarios, enhancing driving safety and comfort, and is applicable to the field of intelligent cockpit interaction technology.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. Comprehensive perception: Multi-sensor fusion covers human behavior, emotions, and cabin environment, enabling full-dimensional perception of the in-vehicle status and adapting to the autonomous monitoring needs of unmanned driving scenarios;

[0029] 2. High accuracy in recognition: Dual emotion model fusion, three-level multimodal feature fusion, dynamic weight allocation, resistance to illumination and noise interference, high emotion recognition accuracy and strong robustness;

[0030] 3. Personalized Interaction: Dynamic layered user profiles, closed-loop feedback and iterative optimization, interaction strategies adapted to user habits, real-time emotions and scenario needs, and improved satisfaction;

[0031] 4. Intelligent response: Automatically identifies cleaning, maintenance, and emergency situations, triggering environmental adjustments, alarms, and maintenance requests to improve driving and passenger safety. Attached Figure Description

[0032] Figure 1 A structural block diagram of a personalized cockpit interaction system based on user behavior analysis and emotion recognition provided in an embodiment of the present invention;

[0033] Figure 2This is a flowchart illustrating a personalized cockpit interaction method based on user behavior analysis and emotion recognition, provided as an embodiment of the present invention. Detailed Implementation

[0034] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0035] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0036] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0037] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0038] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0039] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0040] In related technologies, intelligent cockpit interaction systems mainly focus on monitoring and serving drivers and passengers. For example, existing technologies disclose an intelligent cockpit emotion interaction method. This method collects multimodal feature data of users and environmental feature data in real time, processes the data based on a feature fusion algorithm, obtains emotion recognition results, and generates interaction strategies accordingly. Existing technologies also propose an intelligent cockpit interaction method that uses a deep learning model to classify the emotions of users' facial expressions and voice information, and adjusts the cabin environment atmosphere based on the classification results. Existing technologies also consider error propagation and multimodal data in automotive intelligent cockpit emotion analysis methods, improving the accuracy of emotion analysis by fusing data from different modalities.

[0041] In the field of multimodal data acquisition and processing, existing technologies disclose an intelligent voice care system based on physiological and emotional characteristics. This system uses sensor modules to collect biometric and voice data, extracts physiological and emotional features through deep learning models, and generates a comprehensive feature vector using multimodal data fusion technology. Existing technologies also propose a multi-screen voice interaction system for use in car cabins. This system collects behavioral information and physiological data of drivers and passengers through multi-sensor fusion, enabling the extraction and processing of multimodal information.

[0042] However, existing technologies have the following significant shortcomings: First, current intelligent cockpit systems primarily focus on monitoring the status of the driver and passengers, with limited comprehensive perception capabilities of the vehicle's interior environment, making it difficult to meet the autonomous monitoring needs of the vehicle's interior in future autonomous driving scenarios. Second, existing technologies lack effective mechanisms for identifying abnormal conditions inside the vehicle (such as the need for cleaning, maintenance, or emergencies), making it impossible to automatically make corresponding decisions in an autonomous driving environment. Third, existing systems are mostly designed for manned driving scenarios and do not fully consider the special needs of in-vehicle environment monitoring under autonomous driving conditions, resulting in insufficient system adaptability. Fourth, existing technologies still have room for improvement in multimodal data fusion and analysis, particularly in terms of robustness and accuracy in complex environments.

[0043] To address at least one of the technical problems existing in the aforementioned related technologies, the present invention provides a personalized cockpit interaction system and method based on user behavior analysis and emotion recognition.

[0044] This implementation discloses a personalized cockpit interaction system based on user behavior analysis and emotion recognition, such as... Figure 1 It includes: a multimodal data acquisition module, a user behavior modeling module, an emotion recognition module, a personalized interaction strategy module, and an interaction execution and feedback module;

[0045] The multimodal data acquisition module is used to collect multimodal raw data of the user in the cockpit. In this embodiment, the multimodal data acquisition module includes an image sensor, an audio sensor, a physiological sensor, and an environmental sensor, used to collect multimodal raw data of the user's facial images, voice signals, physiological parameters, and cabin environmental parameters. The image sensor is a high-definition infrared camera with a frame rate of 30fps and a resolution of 1280×720, locating 68 facial key points to capture micro-expression changes. The audio sensor is a 4-microphone array with a 44.1kHz sampling rate, achieving 360-degree sound source localization. The physiological sensors include a steering wheel heart rate monitor, a seat pressure sensor, and a skin conductivity sensor, collecting heart rate, respiratory rate, and skin conductivity signals. The environmental sensors include temperature, humidity, gas, pressure, and motion sensors, collecting cabin environmental parameters.

[0046] The user behavior modeling module is connected to the multimodal data acquisition module and is used to preprocess the multimodal raw data to construct a user behavior model. In this embodiment, the user behavior modeling module preprocesses the multimodal raw data, specifically by: performing denoising, standardization, and dimensionality reduction preprocessing on the multimodal raw data; extracting temporal, spatial, interactive, and semantic features; using an attention mechanism combining early and late fusion to complete feature fusion; and identifying user behavior patterns through unsupervised clustering and deep learning models to construct a user behavior model.

[0047] The preprocessing of the multimodal raw data includes wavelet transform filtering for denoising, Z-score or Min-Max standardization, and PCA or t-SNE dimensionality reduction. Feature extraction uses CNN to extract visual features, RNN to capture temporal features, and TF-IDF and word embedding to extract semantic features. Feature fusion uses an attention mechanism to dynamically allocate modality weights. Behavioral pattern recognition uses K-means clustering, hybrid neural networks, and isolated forest anomaly detection algorithms.

[0048] Specifically, the user behavior modeling module models user behavior, which includes two main steps: behavior data preprocessing and behavior pattern recognition.

[0049] S1: Behavioral data preprocessing:

[0050] This step processes the collected multimodal data to make it suitable for subsequent behavioral pattern analysis. Specifically, it includes the following sub-steps:

[0051] First, the collected multimodal data undergoes denoising processing. Multimodal data typically includes user clickstream data, browsing trajectory, dwell time, eye movement data, facial expression data, etc. Denoising processing employs wavelet transform filtering algorithms to filter out random noise and outliers in the data, reducing the signal-to-noise ratio. For clickstream data, the system identifies and removes robot behaviors and misoperations; for eye movement data, it filters out interfering signals such as blinking and head movements.

[0052] Secondly, feature extraction is performed. The system extracts valuable feature vectors from the preprocessed data, including temporal features (such as visit duration and frequency), spatial features (such as click hotspots and browsing paths), and interactive features (such as scrolling speed and dwell point distribution). For textual data, TF-IDF and word embedding techniques are used to extract semantic features; for image data, convolutional neural networks are used to extract visual features; and for time-series data, recurrent neural networks are used to capture temporal dependencies.

[0053] Finally, data fusion is performed. Features from different modalities and sources are integrated to construct a unified representation of user behavior. Data fusion employs a combination of early and late fusion: early fusion directly concatenates data from different modalities at the feature level; late fusion integrates the analysis results from each modality at the decision level. The system also uses an attention mechanism to assign dynamic weights to data from different modalities, enhancing the fusion effect. Through this multi-layered fusion strategy, the accuracy and robustness of subsequent behavior analysis are effectively improved.

[0054] In a preferred embodiment, data preprocessing further includes a data standardization step, which unifies features at different scales to the same dimensional range by using Z-score or Min-Max standardization methods to ensure that each feature has the same importance in model training.

[0055] In another preferred embodiment, data preprocessing further includes a dimensionality reduction step, which maps high-dimensional features to a low-dimensional space using principal component analysis (PCA) or t-SNE algorithms, reducing computational complexity while retaining key information of the data.

[0056] S2: Behavioral Pattern Recognition

[0057] This step uses machine learning algorithms to model user behavior based on the preprocessed data. The specific implementation is as follows:

[0058] First, the system employs an unsupervised learning method for preliminary behavioral clustering. The K-means clustering algorithm is used to divide user behavior data into several categories, initially identifying user behavior patterns. The system automatically determines the optimal number of clusters using silhouette coefficients and the elbow rule, avoiding biases introduced by manual settings. After visualization of the clustering results, the distribution characteristics and main categories of user behavior can be intuitively displayed.

[0059] Secondly, the system employs a deep learning model to further refine behavioral features. A hybrid neural network containing multilayer perceptrons, convolutional layers, and recurrent layers is constructed to capture complex nonlinear relationships in user behavior data. The network uses batch normalization and dropout techniques to prevent overfitting, and the Adam optimizer is used for parameter updates with a learning rate of 0.001. During training, the system evaluates the validation set performance every 50 epochs and dynamically adjusts the learning rate based on the validation loss.

[0060] Finally, the system constructs user behavior profiles based on the trained model. These profiles include multi-dimensional features such as user interests, usage habits, activity levels, and spending tendencies. The system also calculates the temporal evolution trend of user behavior to predict future behavioral tendencies. The behavior model employs an incremental learning mechanism, enabling continuous updates and optimization as new data arrives, maintaining the model's timeliness and accuracy.

[0061] In a preferred embodiment, behavior pattern recognition also includes an abnormal behavior detection function, which uses algorithms such as isolated forest or autoencoder to identify user behaviors that deviate from the normal pattern, and promptly discover potential risks or special needs.

[0062] In another preferred embodiment, the behavior pattern recognition employs an ensemble learning framework, combining the prediction results of multiple algorithms such as random forest, gradient boosting tree, and deep neural network, and deriving the final behavior model through a weighted voting mechanism, which significantly improves the model's generalization ability and prediction accuracy.

[0063] Through the above-mentioned behavioral data preprocessing and behavioral pattern recognition steps, this method can accurately capture users' behavioral characteristics and patterns, providing data support for personalized service recommendations, user experience optimization, and precision marketing, while ensuring the interpretability and real-time performance of the model.

[0064] The emotion recognition module, connected to the user behavior modeling module, is used to establish an emotion computing framework that integrates discrete emotion models and dimensional emotion models to identify the user's real-time emotional state. In this embodiment, the emotion recognition module is used to establish an emotion computing framework that integrates discrete emotion models and dimensional emotion models. The specific method includes: extracting facial expression, speech acoustic and physiological signal features, adopting a three-level fusion strategy of data level, feature level and decision level, and outputting discrete emotion categories and valence-arousal dimension scores through a hybrid neural network to identify the user's real-time emotional state.

[0065] In this embodiment, the discrete emotion model of the emotion recognition module sets feature thresholds and weights based on Ekman's six basic emotions, and the dimensional emotion model maps emotions based on Russell's valence-arousal two-dimensional model. In the three-level fusion strategy, data-level fusion is feature vector concatenation, feature-level fusion dynamically allocates modality weights according to emotion type, and decision-level fusion uses a voting mechanism or Bayesian inference to integrate classification results. The hybrid neural network includes CNN, LSTM, and fully connected networks, with a network depth of 8 layers and the number of hidden layer neurons being 128-256-512-256-128.

[0066] Specifically, in this embodiment, the emotion recognition module identifies the user's real-time emotional state, including the following steps:

[0067] S1: Establish an emotion computing model

[0068] An emotion computing framework incorporating discrete and dimensional emotion models is established. The discrete emotion model is based on Ekman's six basic emotions (happiness, sadness, anger, fear, disgust, and surprise), with specific recognition feature thresholds and weight coefficients assigned to each emotion. The dimensional emotion model employs Russell's valence-arousal two-dimensional model, mapping emotional states to a two-dimensional plane. The valence dimension ranges from -10 to 10, representing the positive or negative nature of the emotion; the arousal dimension ranges from 0 to 10, representing the degree of emotional activation. The two models work collaboratively through a weighted fusion approach, with a weight of 0.6 for the discrete model and 0.4 for the dimensional model, to improve the accuracy and stability of emotion recognition.

[0069] S2: Multimodal Feature Extraction

[0070] Data is collected and features are extracted simultaneously from three modalities: sound, facial expressions, and physiological signals.

[0071] Audio feature extraction: Audio signals with a sampling rate of 44.1kHz were acquired, and fundamental frequency (pitch) features were extracted, with a frequency range of 75-500Hz; speech rate features were extracted, calculating the number of syllables per second, with a normal range of 4-6 syllables / second; sound quality features were extracted, including harmonic noise ratio (HNR) and jitter. Spectral features of the sound were extracted using short-time Fourier transform (STFT) and Mel-frequency cepstral coefficients (MFCC), with 13-dimensional MFCC features extracted for each frame.

[0072] Facial expression feature extraction: Facial video was acquired at a frame rate of 30fps and a resolution of 1280×720 pixels. Sixty-eight facial key points were located, with a focus on tracking eye muscle movements (including eyelid opening and closing, eyebrow position changes), mouth shape changes (angle of the corners of the mouth up / down, lip opening and closing), and the movement features of facial muscle groups (such as the orbicularis oculi, frontalis, and zygomaticus major muscles). Euclidean distances and angular changes between key points were calculated to construct a facial expression feature vector.

[0073] Physiological signal feature extraction: Electrocardiogram (ECG) signals were acquired at a sampling rate of 250 Hz, and heart rate variability (HRV) indices were calculated, including SDNN (standard deviation of adjacent RR intervals) and RMSSD (root mean square of the difference between adjacent RR intervals); skin conductance (EDA) signals were acquired at a sampling rate of 128 Hz, and skin conductance level (SCL) and skin conductance response (SCR) features were extracted; body temperature changes were measured with an accuracy of ±0.1℃; respiratory rate was recorded, with a normal range of 12-20 breaths / minute.

[0074] S3: Feature Preprocessing and Normalization

[0075] The extracted raw features were preprocessed, including denoising, outlier detection, and missing value handling. Audio features were filtered using a bandpass filter (80-400Hz) to remove background noise; facial expression features were smoothed using a Kalman filter to remove keypoint trajectories; and physiological signals were processed using wavelet transform to remove baseline drift and high-frequency interference. All features were normalized to unify the numerical range to the [0,1] interval to eliminate the influence of different units on subsequent fusion.

[0076] S4: Multimodal Feature Fusion

[0077] A three-level fusion strategy is adopted to achieve effective integration of multimodal features:

[0078] Data-level fusion: The original feature vectors of the three modalities are directly concatenated to form a high-dimensional joint feature vector.

[0079] Feature-level fusion: An adaptive weighting mechanism is used to dynamically adjust the weights of each modality according to different emotion types. For happy emotions, the weight of facial expression is 0.5, the weight of voice features is 0.3, and the weight of physiological signals is 0.2; for sad emotions, the weight of voice features is 0.45, the weight of facial expression is 0.35, and the weight of physiological signals is 0.2; for angry emotions, the weight of physiological signals is increased to 0.4, the weight of facial expression is 0.35, and the weight of voice features is 0.25.

[0080] Decision-level fusion: Each modality is trained with a separate classifier to produce emotion judgment results. The decision results of multiple classifiers are then integrated through a voting mechanism or Bayesian inference. The voting threshold is set to 0.67, meaning that the emotion category is confirmed only when at least two-thirds of the classifiers reach a consensus.

[0081] S5: Emotional State Recognition and Output

[0082] Based on fused multimodal features, a deep learning model is used for emotion state recognition. The model employs a hybrid neural network architecture, including a convolutional neural network (CNN) for processing facial expression sequence features, a long short-term memory network (LSTM) for processing temporal sound features, and a fully connected neural network for processing physiological signal features. The network has a depth of 8 layers, including 3 convolutional layers, 2 LSTM layers, and 3 fully connected layers, with a hidden layer neuron count of 128-256-512-256-128.

[0083] The output includes both discrete emotion categories (six basic emotions and their combinations) and dimensional emotion scores (valence and arousal values). The system also calculates a confidence score (0-100%) for emotion recognition. When the confidence score is below 60%, the system marks it as "uncertain" and triggers a re-collection of multimodal data.

[0084] The emotion recognition results can be visualized in real time, including emotion radar charts, emotion change trend charts, and two-dimensional emotion spatial distribution maps, supporting dynamic tracking and analysis of emotion changes. The system response time is less than 200 milliseconds, meeting the needs of real-time emotion interaction.

[0085] This emotion recognition method overcomes the limitations of single-modal recognition by fusing multimodal information, significantly improving the accuracy and robustness of emotion recognition, and maintaining high recognition performance under different environmental conditions and individual differences.

[0086] The personalized interaction strategy module is connected to the emotion recognition module and is used to dynamically adjust the cockpit's interaction strategy and service content based on the user behavior model and the user's real-time emotional state.

[0087] In this embodiment, the personalized interaction strategy module is used to dynamically adjust the interaction strategy and service content of the cabin based on the user behavior model and the user's real-time emotional state. The specific method includes: constructing a dynamic hierarchical user profile based on the user behavior model, the user's real-time emotional state, and spatiotemporal scene information, and generating interaction strategies for cabin environment adjustment, emotional interaction support, and personalized service recommendation by matching the strategy template library.

[0088] The user profile of the personalized interaction strategy module is divided into long-term stable features and short-term changing features, and is dynamically updated using a time decay model; the interaction strategy generation includes strategy matching, personalization adjustment, multimodal fusion, priority ranking, and conflict elimination; the cabin environment adjustment includes gradual adjustment of temperature, light, audio, seat, and air parameters; the emotional interaction support includes voice tone and speech rate adjustment, visual feedback optimization, and proactive intervention for negative emotions; the personalized service recommendation includes navigation, entertainment, dining, rest reminders, and social connection services, adapted to driving safety status.

[0089] Specifically, the methods used by the personalized interaction strategy module to build user profiles and achieve personalized interactions with users include:

[0090] S1: Data Acquisition

[0091] First, the system continuously collects data generated by the user during interaction, including but not limited to user operation records, click behavior, dwell time, input content, and selection preferences. Simultaneously, the system collects facial expression data via the front-facing camera and voice data via the microphone; this data will be used for subsequent emotion recognition analysis. During data collection, the system performs real-time encryption to ensure user privacy and security.

[0092] S2: User Behavior Model Construction:

[0093] Based on the collected user operation records, the system uses machine learning algorithms to build a user behavior model. This model mainly includes the following aspects:

[0094] (1) Operation habit analysis: The system records and analyzes the user's operation path, function usage frequency, operation speed, etc., to identify the user's operation patterns and habits. For example, the system can identify whether the user tends to use shortcut keys, whether they are used to accessing functions through the menu bar, and whether they frequently use specific functions.

[0095] (2) Usage scenario identification: The system identifies the user's usage scenario based on the user's usage time, location and environmental factors, such as work scenario, leisure scenario, commuting scenario, etc.

[0096] (3) Behavioral sequence analysis: The system uses time-series analysis to identify typical sequences and patterns of user behavior and predict the user's possible next action.

[0097] S3: Sentiment Recognition and Analysis

[0098] The system uses multimodal emotion recognition technology to analyze users' emotional states in real time.

[0099] (1) Facial expression analysis: Using computer vision technology, the system analyzes the subtle changes in the user's facial expressions and identifies basic emotion types, such as happiness, sadness, anger, surprise, fear, disgust, etc.

[0100] (2) Voice emotion analysis: The system identifies the emotional information contained in the voice by analyzing the acoustic features of the user's voice, such as pitch, volume, speech rate, and pauses.

[0101] (3) Text sentiment analysis: Natural language processing is performed on the text content input by the user to extract the sentiment tendency and intensity in the text.

[0102] (4) Emotion Fusion Analysis: The system weighted and fused the emotion recognition results from facial, voice, and text modalities to obtain a more accurate judgment of emotional state. The system uses a deep learning model for multimodal emotion feature extraction and fusion, achieving an emotion recognition accuracy of over 85%.

[0103] S4: User Profile Building

[0104] Based on user behavior models and sentiment recognition results, the system constructs a comprehensive user profile:

[0105] (1) Basic profile construction: Record the user's basic information, such as age group, gender, occupation type, etc. This information can be inferred from the user's registration information or usage behavior.

[0106] (2) Behavioral feature extraction: Extract user operation habit features from user behavior model, including frequently used function list, operation path preference, interaction method preference, etc.

[0107] (3) Emotional feature analysis: Record the emotional change patterns of users in different scenarios and during the use of different functions, and construct an emotional response model.

[0108] (4) Preference model construction: Based on the user's selection history, establish a user preference model for content, functions, interface style, etc.

[0109] (5) Dynamic Update Mechanism: User profiles are not static. The system is designed with a dynamic update mechanism to continuously optimize and update user profiles based on the latest user behavior and emotional feedback. The system adopts a time decay model, assigning different weights to historical user behavior data, with recent behavior having a higher weight.

[0110] In a preferred embodiment, the system manages user profiles in a hierarchical manner, dividing user characteristics into long-term stable characteristics and short-term changing characteristics. Long-term stable characteristics include users' basic operating habits, functional preferences, etc., which change slowly; short-term changing characteristics include users' current emotional state, immediate needs, etc., which may change at any time. The system employs different update and application strategies for characteristics at different levels.

[0111] In another preferred embodiment, the system implements a multi-dimensional visualization function for user profiles. Administrators or analysts can intuitively understand various dimensions of the user profile through a visual interface, including behavioral characteristic radar charts, emotional state heatmaps, and function usage frequency bar charts. This visualization helps to better understand user characteristics and behavioral patterns.

[0112] By constructing user profiles through the above steps, the system can accurately grasp users' operating habits, preferences, and emotional states, providing data support for subsequent personalized interaction strategies and achieving a more intelligent and human-computer interaction experience. Experimental results show that user profiles constructed based on this method can improve user satisfaction with the interactive system by more than 20%, and significantly reduce user operation error rates and learning costs.

[0113] The interaction execution and feedback module is connected to the personalized interaction strategy module. It is used to execute interaction strategies, collect user behavior feedback, and iteratively optimize user profiles, interaction strategy rules, and model parameters based on feedback data to achieve continuous optimization of the interaction experience.

[0114] In this embodiment, the feedback collection of the interaction execution and feedback module includes voice feedback, touch score, behavioral feedback, and emotional feedback; feedback analysis adopts SVM, random forest, or deep learning models; model optimization adopts incremental learning and reinforcement learning, preset number of feedback triggers parameter updates, and an emergency intervention mechanism is established to respond to sudden feedback problems.

[0115] Specifically, the interaction execution and feedback module can be applied to intelligent cockpit systems, generating suitable interaction strategies based on the user's personal characteristics and current scenario requirements to improve the user experience.

[0116] The method for generating this interaction strategy includes the following steps:

[0117] Step 1: User Profile Building

[0118] The system acquires user information and builds user profiles through various data collection methods. Specifically, this includes:

[0119] 1) Collect basic user information, including demographic characteristics such as age, gender, and occupation;

[0120] 2) Record user history behavior data, including cabin environment preference settings, frequently used functions, usage frequency, etc.;

[0121] 3) Real-time collection of user physiological data, such as heart rate, respiratory rate, and facial expression, through in-vehicle sensors;

[0122] 4) Analyze user interaction patterns, including voice command habits, touch operation characteristics, and feedback to system responses;

[0123] 5) Establish user interest models, including music preferences, news categories, and frequently used applications.

[0124] The system integrates and processes the above data to construct a multi-dimensional user profile. This profile is stored in vector format and includes information such as user characteristics, preferences, and habits. The user profile data is updated after each interaction to ensure its timeliness and accuracy.

[0125] Step 2: Interactive Scene Recognition

[0126] The system identifies the current interaction scenario in real time, mainly through the following methods:

[0127] 1) Spatiotemporal information acquisition: Acquire information such as current time, vehicle location, and driving status;

[0128] 2) Environmental perception: Acquire environmental parameters such as temperature, humidity, light, and noise through sensors inside and outside the vehicle;

[0129] 3) User status monitoring: Monitoring user's emotional state, fatigue level, etc. through in-vehicle cameras and sensors;

[0130] 4) Interaction Intent Analysis: Analyze the user's current operation or voice command to infer the user's intent;

[0131] 5) Task scenario judgment: Identify the user's current task, such as driving, resting, working, or entertaining.

[0132] The system integrates and processes this information to form a feature description of the current interaction scenario, providing a basis for subsequent strategy generation.

[0133] Step 3: Interaction Strategy Generation

[0134] Based on user profiles and the current interaction scenario, the system generates personalized interaction strategies:

[0135] 1) Strategy matching: Match the current user profile and scenario characteristics with the preset strategy template library to select suitable basic strategies;

[0136] 2) Personalized adjustments: Adjusting the basic strategy based on the user's specific preferences and current needs;

[0137] 3) Multimodal strategy integration: Integrating strategies from different dimensions such as environmental adjustment, emotional support, and service recommendation;

[0138] 4) Strategy Prioritization: Prioritize multiple possible interaction strategies based on the importance of the current scenario and user preferences;

[0139] 5) Conflict detection and resolution: Detect and resolve potential strategy conflicts.

[0140] The generated interaction strategy includes three main aspects: cabin environment adjustment strategy, emotional interaction strategy, and personalized service recommendation strategy.

[0141] Step 4: Automatic adjustment of the cabin environment

[0142] The system automatically adjusts cabin environment parameters based on the generated environmental adjustment strategy:

[0143] 1) Temperature control: Automatically adjusts the air conditioner temperature and airflow based on user preferences, external temperature, and the user's current physiological state;

[0144] 2) Light control: Adjust the brightness and color temperature of the interior lighting according to external lighting conditions and user visual comfort requirements;

[0145] 3) Audio environment optimization: Adjust music type, volume, and sound quality parameters to create a sound environment that meets the user's current needs;

[0146] 4) Seat adjustment: Automatically adjusts the seat angle, height, and support level based on the user's physical condition and current task requirements;

[0147] 5) Air quality management: Adjust the ventilation and air purification system according to the external air quality and user health needs.

[0148] The environmental adjustment process is gradual to avoid discomfort caused by abrupt changes. At the same time, the system monitors the user's reaction to the adjustment and makes timely fine-tuning.

[0149] Step 5: Providing Emotional Support

[0150] The system provides appropriate emotional support based on the user's emotional state and the needs of the scenario:

[0151] 1) Emotion recognition: Identifying the user's current emotional state through multimodal data such as facial expressions, voice features, and physiological indicators;

[0152] 2) Support strategy selection: Select appropriate emotional support methods based on the identified emotional state and personality traits in the user profile;

[0153] 3) Voice interaction adjustment: Adjust the system's voice parameters such as tone, speed, and word choice to match the user's emotional needs;

[0154] 4) Visual feedback optimization: Adjust interface colors, animation effects, etc., to create a visual experience that meets the user's emotional needs;

[0155] 5) Proactive emotional intervention: When negative emotions are detected in users, provide interactive content that offers comfort, encouragement, or distraction.

[0156] Emotional support strategies take into account users' acceptance levels and privacy preferences to avoid excessive intervention that could cause resentment.

[0157] Step Six: Personalized Service Recommendations

[0158] The system intelligently recommends personalized services based on user profiles and current scenario requirements:

[0159] 1) Navigation service: Recommends the most suitable route and driving mode based on the user's preferred route, current time, and destination type;

[0160] 2) Entertainment content: Recommend music, news, podcasts, or video content based on user interests and current context;

[0161] 3) Food and beverage services: Recommend nearby restaurants or coffee shops based on time, location, and user dietary preferences;

[0162] 4) Rest reminders: Monitor driving time and fatigue level, and recommend rest stops and rest methods as needed;

[0163] 5) Social Connections: Provide convenient communication options or social media updates based on users' social habits and current status.

[0164] The recommendation service takes into account current driving safety factors and adjusts the complexity and interaction of recommended content under different driving conditions.

[0165] Step 7: Strategy Implementation and Feedback Collection

[0166] The system executes the generated interaction strategy and collects user feedback:

[0167] 1) Step-by-step execution: Execute each strategy in order of priority to avoid user discomfort caused by executing multiple adjustments at the same time;

[0168] 2) User reaction monitoring: Monitor users' real-time reactions to policy execution through sensors, such as changes in facial expressions and physiological indicators;

[0169] 3) Explicit feedback collection: Obtain user evaluation of the strategy through simple voice questions or interface interactions;

[0170] 4) Implicit Feedback Analysis: Analyze user behaviors such as whether they accept recommendations and whether they manually adjust system settings;

[0171] 5) Feedback data storage: Store user feedback data in a structured manner for subsequent strategy optimization.

[0172] The system will adjust the current strategy in real time based on user feedback and use the feedback data for long-term strategy model optimization.

[0173] Step 8: Strategy Model Update and Optimization

[0174] The system continuously optimizes the interaction strategy model based on execution results and user feedback.

[0175] 1) Strategy effectiveness evaluation: Analyze changes in user status before and after strategy implementation to evaluate the effectiveness of the strategy;

[0176] 2) User profile update: Update user profiles based on newly collected user feedback and behavioral data;

[0177] 3) Strategy model adjustment: Optimize the strategy generation algorithm and parameters based on accumulated execution data;

[0178] 4) New strategy template generation: Extract patterns from successful cases to form new strategy templates;

[0179] 5) Cold start problem handling: For new users or new scenarios, optimize the strategy and recommend initialization methods.

[0180] The system employs an incremental learning approach to maintain the continuous optimization and adaptability of the model while preserving personalized features.

[0181] In a preferred embodiment, the user profile building step further includes emotional feature analysis, which analyzes users' emotional tendencies, stress coping methods, and emotional fluctuation patterns through long-term interaction data, providing a more accurate basis for emotional support strategies.

[0182] In another preferred embodiment, the interaction strategy generation step employs a reinforcement learning algorithm. By continuously trying different strategy combinations and analyzing user feedback, the strategy selection model is gradually optimized to improve the accuracy of personalized recommendations.

[0183] In another preferred embodiment, the system automatically adjusts the focus of the interaction strategy according to different stages of the user's driving state (such as before departure, during high-speed driving, in congested areas, and before reaching the destination). For example, it focuses more on emotional relief in congested areas and more on fatigue monitoring and reminders during long-distance driving.

[0184] Through the above methods, the system can provide a highly personalized interactive experience based on the user's personal characteristics and current needs, thereby improving user satisfaction and security. At the same time, as the usage time increases, the system's understanding of the user and its service capabilities will continue to improve.

[0185] This embodiment discloses a personalized cockpit interaction system based on user behavior analysis and emotion recognition, including: a multimodal data acquisition module, a user behavior modeling module, an emotion recognition module, a personalized interaction strategy module, and an interaction execution and feedback module. The multimodal data acquisition module collects multi-dimensional data within the cockpit through image, audio, physiological, and environmental sensors; the user behavior modeling module completes data preprocessing, feature extraction, fusion, and behavior pattern recognition; the emotion recognition module constructs a discrete-dimensional fusion emotion model and employs a three-level fusion strategy to achieve high-precision emotion recognition; the personalized interaction strategy module constructs a dynamic user profile and generates environmental adjustment, emotional interaction, and service recommendation strategies; the interaction execution and feedback module completes strategy execution, feedback collection, and model iterative optimization. This invention improves the comprehensiveness of cockpit environmental perception, the robustness of emotion recognition, and the personalized adaptability of interaction, adapting to autonomous driving scenarios, enhancing driving safety and comfort, and is applicable to the field of intelligent cockpit interaction technology.

[0186] Based on the same inventive concept, embodiments of the present invention also provide a personalized cockpit interaction method based on user behavior analysis and emotion recognition, such as... Figure 2 ,include:

[0187] S100. The multimodal data acquisition module collects raw multimodal data from users inside the cockpit;

[0188] S200. The user behavior modeling module preprocesses multimodal data, identifies user behavior patterns, and builds behavior models.

[0189] The S300 emotion recognition module extracts facial, voice, and physiological emotion features, and identifies the user's real-time emotional state through a three-level fusion strategy.

[0190] S400. The personalized interaction strategy module integrates behavioral models, emotional states, and scene information to construct dynamic user profiles and generate personalized interaction strategies.

[0191] S500. The interaction execution and feedback module executes the interaction strategy, collects user feedback, iteratively optimizes the profile, strategy and model parameters, and cyclically executes steps S200-S500.

[0192] The specific working methods of the multimodal data acquisition module, user behavior modeling module, emotion recognition module, personalized interaction strategy module, and interaction execution and feedback module have been described in detail in the above system, and will not be repeated here in this embodiment.

[0193] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A personalized cockpit interaction system based on user behavior analysis and emotion recognition, characterized in that, include: Multimodal data acquisition module, user behavior modeling module, emotion recognition module, personalized interaction strategy module, and interaction execution and feedback module; The multimodal data acquisition module is used to collect raw multimodal data of users in the cockpit; The user behavior modeling module is connected to the multimodal data acquisition module and is used to preprocess the multimodal raw data to construct a user behavior model. The emotion recognition module is connected to the user behavior modeling module and is used to establish an emotion computing framework that integrates discrete emotion models and dimensional emotion models to identify the user's real-time emotional state. The personalized interaction strategy module is connected to the emotion recognition module and is used to dynamically adjust the cockpit's interaction strategy and service content based on the user behavior model and the user's real-time emotional state. The interaction execution and feedback module is connected to the personalized interaction strategy module. It is used to execute interaction strategies, collect user behavior feedback, and iteratively optimize user profiles, interaction strategy rules, and model parameters based on feedback data to achieve continuous optimization of the interaction experience.

2. The system according to claim 1, characterized in that, The multimodal data acquisition module includes an image sensor, an audio sensor, a physiological sensor, and an environmental sensor, used to acquire multimodal raw data of user facial images, voice signals, physiological parameters, and cabin environmental parameters in the cockpit.

3. The system according to claim 1, characterized in that, The user behavior modeling module preprocesses the multimodal raw data. The specific methods include: denoising, standardizing, and reducing the dimensionality of the multimodal raw data; extracting temporal, spatial, interactive, and semantic features; using an attention mechanism that combines early and late fusion to complete feature fusion; identifying user behavior patterns through unsupervised clustering and deep learning models; and constructing a user behavior model.

4. The system according to claim 3, characterized in that, The multimodal raw data is preprocessed using algorithms including wavelet transform filtering for denoising, Z-score or Min-Max normalization, and PCA or t-SNE dimensionality reduction. Feature extraction employs CNN to extract visual features, RNN to capture temporal features, and TF-IDF and word embedding to extract semantic features. Feature fusion employs an attention mechanism to dynamically allocate modality weights; behavioral pattern recognition utilizes K-means clustering, hybrid neural networks, and isolated forest anomaly detection algorithms.

5. The system according to claim 1, characterized in that, The emotion recognition module is used to establish an emotion computing framework that integrates discrete emotion models and dimensional emotion models. The specific methods include: extracting facial expression, speech acoustic and physiological signal features, adopting a three-level fusion strategy of data level, feature level and decision level, and outputting discrete emotion categories and valence-arousal dimension scores through a hybrid neural network to identify the user's real-time emotional state.

6. The system according to claim 1, characterized in that, The discrete emotion model of the emotion recognition module is based on Ekman's six basic emotions, setting feature thresholds and weights. The dimensional emotion model is based on Russell's valence-arousal two-dimensional model to map emotions. In the three-level fusion strategy, data-level fusion is feature vector concatenation, feature-level fusion dynamically allocates modality weights according to emotion type, and decision-level fusion uses a voting mechanism or Bayesian inference to integrate classification results. The hybrid neural network includes CNN, LSTM, and fully connected networks, with a network depth of 8 layers and the number of hidden layer neurons being 128-256-512-256-128.

7. The system according to claim 1, characterized in that, The personalized interaction strategy module is used to dynamically adjust the cockpit's interaction strategy and service content based on the user behavior model and the user's real-time emotional state. The specific method includes: constructing a dynamic hierarchical user profile based on the user behavior model, the user's real-time emotional state, and spatiotemporal scene information, and matching the strategy template library to generate interaction strategies for cockpit environment adjustment, emotional interaction support, and personalized service recommendation.

8. The system according to claim 1, characterized in that, The user profile of the personalized interaction strategy module is divided into long-term stable features and short-term changing features, and is dynamically updated using a time decay model; the interaction strategy generation includes strategy matching, personalization adjustment, multimodal fusion, priority ranking, and conflict elimination; the cabin environment adjustment includes gradual adjustment of temperature, light, audio, seat, and air parameters; the emotional interaction support includes voice tone and speech rate adjustment, visual feedback optimization, and proactive intervention for negative emotions; the personalized service recommendation includes navigation, entertainment, dining, rest reminders, and social connection services, adapted to driving safety status.

9. The system according to claim 1, characterized in that, The feedback collection of the interaction execution and feedback module includes voice feedback, touch score, behavioral feedback, and emotion feedback; feedback analysis adopts SVM, random forest, or deep learning models; model optimization adopts incremental learning and reinforcement learning, preset feedback triggers parameter updates, and an emergency intervention mechanism is established to respond to sudden feedback problems.

10. A personalized cockpit interaction method based on user behavior analysis and emotion recognition, characterized in that, include: S100. Collect raw multimodal data of users in the cockpit; S200. Preprocess multimodal data to identify user behavior patterns and build behavior models; S300 extracts facial, voice, and physiological emotional features, and identifies the user's real-time emotional state through a three-level fusion strategy; S400 integrates behavioral models, emotional states, and contextual information to construct dynamic user profiles and generate personalized interaction strategies. S500. Execute the interaction strategy, collect user feedback, iteratively optimize the profile, strategy and model parameters, and repeat steps S200-S500.