A multi-modal data-driven intelligent cockpit parameter adaptation method and system

Abstract

The embodiment of the application discloses a multi-modal data-driven intelligent cabin parameter adaptation method and system, and relates to the technical field of intelligent cabin human-computer interaction. The method synchronously collects physiological data, multi-modal interaction data, driving behavior and scene data, extracts core features after preprocessing, adopts an evidence theory attention mechanism combination algorithm to construct a dynamic fusion adaptation model, realizes accurate and real-time adaptation of multiple cabin parameters such as sitting posture, vision, control and environment, and covers special groups such as the elderly, children and disabled users and all driving scenes. The application solves the defects of the prior art, such as insufficient data fusion dimension, static adaptation model, lack of psychological support and special group adaptation, parameter prediction error and whole-process delay, significantly improves driving safety and driving comfort, and is suitable for parameter optimization of various intelligent cabin products.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology in intelligent cockpits, specifically to a multimodal data-driven intelligent cockpit parameter adaptation method and system. Background Technology

[0002] With the rapid development of automotive intelligent technology, the intelligent cockpit has become a core carrier for improving driving experience and safety. Its parameter adaptation (such as seat adjustment, display settings, interactive control, and environmental control) directly affects the user's driving experience and driving safety. Currently, intelligent cockpit parameter adaptation technology is developing in multiple directions, but it generally suffers from the following core problems:

[0003] Insufficient data fusion dimensions: Existing technologies mostly rely on single-type data (such as only based on driving behavior or single interaction modality data), failing to achieve full-dimensional data fusion of physiological state, multimodal interaction, driving behavior, and scene dynamics, resulting in one-sided adaptation criteria and limited accuracy;

[0004] Static model adaptation: Most solutions use preset parameter templates or rule-based algorithms, which cannot adapt to real-time physiological changes in users (such as sudden fatigue or distracted driving) and scene switching. , Dynamic adjustments are not possible, resulting in insufficient flexibility and real-time adaptability.

[0005] Lack of scientific theoretical support: It does not fully integrate psychological experimental paradigms and human factors engineering principles, ignores the influence of psychological factors such as cognitive load and emotional state on parameter fitting, and the scientific nature of the fitting logic needs to be improved.

[0006] Lack of adaptation for special groups: No special adaptation plan was designed for the physiological characteristics and interaction habits of the elderly, children, and disabled users (such as slow response, height difference, and limited operation ability), resulting in insufficient universality;

[0007] Limited fitting ability of algorithms: Most existing modeling algorithms are single models (such as logistic regression and basic neural networks), which have weak fitting ability for nonlinear and time-series multimodal data associations and have high parameter prediction errors (generally above 8%).

[0008] While some existing technologies involve data acquisition, model building, or scenario adaptation, these solutions focus on individual technical aspects and fail to form a systematic solution. For example, some solutions only focus on the synchronous acquisition of multimodal data but lack effective data fusion and dynamic modeling methods; some solutions propose adaptation logic for single-type parameters but fail to cover all dimensions of cockpit parameters; some solutions use static models for basic adaptation but cannot meet real-time requirements and the needs of special user groups. Even combining these technical solutions fails to address the core pain points of "all-dimensional data fusion + dynamic and accurate modeling + adaptation to all user groups + real-time response," making it difficult to meet the high demands of intelligent cockpits for personalization, safety, and comfort. Therefore, a comprehensive, accurate, and flexible intelligent cockpit parameter adaptation solution is urgently needed. Summary of the Invention

[0009] To address the aforementioned technical problems, embodiments of this application provide a multimodal data-driven intelligent cockpit parameter adaptation method and system.

[0010] According to one aspect of the embodiments of this application, a multimodal data-driven intelligent cockpit parameter adaptation method is provided, comprising the following steps: S1: Multimodal data acquisition: acquiring human physiological data, cockpit interaction behavior data, vehicle operating status data, external environment perception data, and subjective feedback data, wherein the multimodal data is synchronously acquired through standardized acquisition equipment; S2: Data preprocessing: cleaning, synchronizing, and extracting features from the acquired multi-source data to obtain physiological features, interaction features, behavioral features, and scene features; S3: Multimodal fusion adaptation model construction and training: using... Evidence theory integrates multimodal features, through An attention mechanism algorithm is used to construct a dynamic fusion adaptation model. The model is trained in multiple rounds using multi-dimensional measured data and synthetic supplementary data as samples, so that the parameter prediction error of the dynamic fusion adaptation model is less than a preset threshold, and the trained dynamic fusion adaptation model is obtained. S4: Real-time dynamic parameter adaptation: The preprocessed multimodal data is input into the trained dynamic fusion adaptation model, the dynamic fusion adaptation model outputs the optimal parameter configuration, and it is sent to each control module through a high-speed bus, and is dynamically updated once every preset time period.

[0011] In one embodiment of this application, the collected human physiological data includes eye movement data, electroencephalogram (EEG) signals, electrocardiogram (ECG) data, skin conductance levels, and the human body's position on the cushion. Backrest pressure distribution and contact area; cabin interaction behavior data including voice interaction data, gesture recognition data, eye tracking data, touch interaction data, and facial recognition data; vehicle operating status data including vehicle speed, acceleration, steering angle, braking frequency, cabin environmental control parameters, and system operating resource utilization rate; external environment perception data including light intensity, ambient temperature, noise level, rainfall / visibility, road type, and traffic density; subjective feedback data including satisfaction, cognitive load, and emotional pleasure score.

[0012] In one embodiment of this application, the collected multi-source data is cleaned, including: IQR outlier removal and Kalman filtering noise smoothing. IQR outlier removal includes: calculating a set threshold for the interquartile range (IQR), the calculation formula being... , , The outlier threshold is Data values ​​exceeding the outlier threshold range are removed. Here, Q1 is the first quartile, reflecting the lower limit of the data distribution; Q3 is the third quartile, reflecting the upper limit of the data distribution; n is the number of data samples, taking a positive integer value; IQR is the interquartile range, which is the difference between the third quartile and the first quartile, used to quantify the stability of the data; 1.5IQR is the outlier determination coefficient, a fixed coefficient.

[0013] The Kalman filter equation is:

[0014]

[0015] in, The Kalman gain at time k, For the observed values, For the observation matrix, The estimated value is the value predicted at time k. for The optimal estimate at time. The value is the observation at time k.

[0016] In one embodiment of this application, synchronizing collected multi-source data includes:

[0017] Data synchronization employs timestamp alignment and linear interpolation to compensate for missing data. The linear interpolation formula is as follows:

[0018]

[0019] in, t1 represents the target timestamp data value, i.e., the multimodal data corresponding to the target timestamp t; t is the target timestamp: the time point when the data is missing, with a precision of 1ms, and a non-negative integer value; t1 is the preceding valid timestamp: the timestamp corresponding to the most recent valid data before t, ensuring that the time interval with t is minimized, and a non-negative integer value. t2 is the subsequent valid timestamp, the timestamp corresponding to the most recent valid data after t, and its value is a non-negative integer (and ; For previous valid data values: valid collected data corresponding to timestamp t1, with values ​​consistent with data types; For subsequent valid data values: valid collected data corresponding to timestamp t2, with values ​​consistent with data type.

[0020] In one embodiment of this application, feature extraction is performed on the collected multi-source data, including: feature extraction includes calculating eye-tracking focus, EEG fatigue index, heart rate stress coefficient, multimodal interaction success rate, driving stability coefficient, and scene complexity. Normalization adopts Standardization, the formula is:

[0021]

[0022] in, These are the standardized data values; The original data value; The minimum value of the data. This represents the maximum value of the data.

[0023] In one embodiment of this application, the control module includes vehicle control components, including but not limited to seats, backrests, steering wheels, HUDs, screens, rearview mirrors, vehicle infotainment systems, vision recognition modules, touch response modules, cabin temperature control modules, and fragrance control modules.

[0024] In one embodiment of this application, dynamic parameter adaptation includes, but is not limited to, adapting parameters such as seat height, backrest angle, steering wheel distance, HUD position, screen brightness, rearview mirror angle, screen color temperature, steering wheel pressure, voice sensitivity, gesture control sensitivity, touch response speed, cabin temperature, humidity, lighting brightness, and fragrance concentration.

[0025] In one embodiment of this application, the preset threshold is set to .

[0026] According to one aspect of the embodiments of this application, a multimodal data-driven intelligent cockpit parameter adaptation system is provided, including: a multimodal data acquisition module configured to collect human physiological data, cockpit interaction behavior data, vehicle operating status data, external environment perception data and subjective feedback data, wherein the multimodal data is synchronously acquired through standardized acquisition equipment;

[0027] Data preprocessing module: configured to clean, synchronize, and extract features from collected multi-source data to obtain physiological features, interaction features, behavioral features, and scene features; Multimodal fusion adaptation model construction and training module: configured to use... Evidence theory integrates multimodal features, through The attention mechanism algorithm constructs a dynamic fusion adaptation model. The model is trained iteratively in multiple rounds using multi-dimensional measured data and synthetic supplementary data as samples, so that the parameter prediction error of the dynamic fusion adaptation model is less than a preset threshold, and the trained dynamic fusion adaptation model is obtained. The parameter execution module is configured to input the preprocessed multimodal data into the trained dynamic fusion adaptation model. The dynamic fusion adaptation model outputs the optimal parameter configuration and sends it to each control module through a high-speed bus, and updates it dynamically once every preset time period.

[0028] In the technical solution provided in the embodiments of this application, human physiological data, cockpit interaction behavior data, vehicle operating status data, external environment perception data, and subjective feedback data are collected, cleaned, synchronized, and feature extracted to obtain physiological features, interaction features, behavioral features, and scene features. Multimodal features are then fused using evidence theory. The dynamic fusion adaptation model, which combines attention mechanism algorithm construction with key feature enhancement, is trained iteratively in multiple rounds using multi-dimensional measured data, associated data, and synthetic supplementary data as samples, until the parameter prediction error of the dynamic fusion adaptation model is less than a preset threshold. At this point, the dynamic fusion adaptation model training is complete. When applied in the vehicle, multiple optimal configurations of cockpit parameters are dynamically distributed periodically via a high-speed bus. This achieves a leap from "static template adaptation" to "multimodal data-driven real-time accurate adaptation for all users and all scenarios" for intelligent cockpit parameters. This not only significantly improves parameter adaptation accuracy, response speed, and universality, filling the adaptation gap for special groups, but also reduces user cognitive load and error rate, comprehensively improving driving safety and ride comfort.

[0029] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0031] Figure 1 This is a schematic diagram illustrating a multimodal data-driven intelligent cockpit parameter adaptation method in an exemplary embodiment of this application;

[0032] Figure 2 This is a schematic diagram of a multimodal data-driven intelligent cockpit parameter adaptation system, as illustrated in an exemplary embodiment of this application. Detailed Implementation

[0033] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0034] The block diagrams shown in the attached figures are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, within one or more hardware modules or integrated circuits, or across different networks and... or processor device and These functional entities may be implemented in a microcontroller device.

[0035] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all content and operations. The steps do not necessarily have to be performed in the order described. For example, some operations... The steps can be broken down further, and some operations... The steps can be combined or partially combined, so the actual execution order may change depending on the actual situation.

[0036] In this application, "multiple" refers to two or more. "or" describes the relationship between related objects, indicating that three types of relationships can exist, for example, A and "or B" can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. The character " "Generally, this indicates that the objects before and after are in an "or" relationship.

[0037] The embodiments of this application provide a contactor control circuit and a contactor control method, which will be described in detail below.

[0038] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating a multimodal data-driven smart cockpit parameter adaptation method, as shown in an exemplary embodiment of this application.

[0039] In one exemplary embodiment, a multimodal data-driven intelligent cockpit parameter adaptation method includes:

[0040] S1: Synchronous acquisition of multimodal data, including the acquisition of human physiological data, cockpit interaction behavior data, vehicle operating status data, external environment perception data, and subjective feedback data.

[0041] In one exemplary embodiment, the multimodal data can be synchronously acquired using a standardized acquisition device, with data acquired every fixed time interval, the fixed time interval being less than or equal to 5ms, to ensure synchronization accuracy. The multimodal data may also include a portion of pre-collected and memorized data to facilitate matching based on real-time data collection.

[0042] Understandably, the aforementioned human physiological data includes eye movement data, electroencephalogram (EEG) signals, electrocardiogram (ECG) data, skin conductance levels, and the human body's position on the cushion. Backrest pressure distribution and contact area. The eye movement data can include fixation points, fixation duration, and saccade amplitude collected by an eye tracker; the electroencephalogram (EEG) signals can include data collected by an EEG device. Wave power, ECG data can include HRV and PR interval acquired by ECG sensors, and skin conductance level can be obtained from the human body's position on the cushion by skin conductance sensors. The pressure distribution and contact area of ​​the backrest can be obtained using body pressure distribution acquisition equipment.

[0043] Understandably, the cockpit interaction behavior data includes voice interaction data, gesture recognition data, eye tracking data, touch interaction data, and face recognition data. Specifically, it includes voice recognition accuracy and response time, gesture recognition accuracy and response latency, eye tracking interaction effectiveness, touch response speed, positioning accuracy, false touch rate, face recognition accuracy, and liveness detection anti-attack capability.

[0044] Understandably, the vehicle operating status data includes vehicle speed, acceleration, steering angle, braking frequency, cabin environment control parameters, and system operating resource utilization.

[0045] Understandably, external environment perception data includes light intensity, ambient temperature, noise level, rainfall / visibility, road type, and traffic density.

[0046] Understandably, subjective feedback data is obtained based on the SUS and PADS scales, including satisfaction, cognitive load, and emotional well-being scores. It is also understood that the SUS and PADS scales are pre-collected data tables, which can be directly used in this application's embodiments.

[0047] In the embodiments of this application, regarding the acquisition of human physiological data, a high-precision eye tracker can be used to collect physiological data, mainly collecting fixation point coordinates, fixation duration, and saccade amplitude; brainwave devices can also be used for data collection. Wave , Wave , Wave , Wave Power values; high-precision ECG sensors acquire heart rate variability (HRV) and PR interval; skin conductance sensors acquire skin conductance levels (EDA); body pressure distribution acquisition devices acquire data from the cushion. Backrest pressure distribution, contact area, etc.

[0048] In specific embodiments, the multimodal data acquisition device can be selected according to the actual application requirements, and the core performance parameters of each device can be flexibly configured within the following ranges, with exemplary values ​​as follows: high-precision eye tracker, sampling frequency range (Example values: 120Hz, 180Hz), gaze point accuracy range (Example values) , ); Channel EEG acquisition equipment, sampling rate range (Example values: 500Hz, 800Hz), resolution range (Example values) , High-precision ECG sensor, sampling rate range (Example values: 1000Hz, 1500Hz), Measurement range (Example values ​​cover this range and are adapted to different heart rate scenarios); Electrodermal sensor, measurement range (Example values ​​cover this range and are adapted to different skin conductance states); Body pressure distribution acquisition device, sensor density range point (Example values) , High-definition depth camera, frame rate range (Example values: 30fps, 50fps), resolution range (Example values) , ); Voice interaction module, sampling rate range (Example values: 16kHz, 32kHz), accuracy range for noise-free environment recognition (Example values: 95%, 97%); High-precision touch display module, sampling rate range (Example values: 240Hz, 360Hz), positioning accuracy range (Example values: 0.1mm, 0.2mm); Vehicle environmental sensor, temperature accuracy range (Example values) ), light measurement range (Example values ​​cover this range and are adaptable to different lighting scenarios). The above-mentioned acquisition device communicates with the data preprocessing module via CAN / Ethernet to ensure the accuracy of synchronous acquisition of multimodal data. This provides high-quality, multi-dimensional raw data support for subsequent feature extraction and model training.

[0049] S2: Data Preprocessing: The collected multi-source data is cleaned, synchronized, and feature extracted to obtain physiological features, interaction features, behavioral features, and scene features. In this embodiment, feature extraction can be based on… "Relational model implementation."

[0050] The cleaning of the collected multi-source data includes: IQR outlier removal and Kalman filtering noise smoothing. Specifically, the IQR outlier removal includes: calculating a set threshold for the interquartile range, calculated using the following formula: First quartile: Third and quartiles: Interquartile range: The outlier threshold is , Data values ​​that exceed the outlier threshold range will be removed.

[0051] Wherein, Q1 (first quartile): the quartile in which the collected multimodal data (such as EEG signals, ECG data, interactive behavior data, etc.) are sorted in ascending order. The value of the position is Q1, which reflects the lower limit of the data distribution and is used to define the lower bound of the normal range of the data. Q3 (Third Quartile): The position of the data after sorting in ascending order. The value of the position is Q3, which reflects the upper limit of the data distribution and is used to define the upper bound of the normal range of the data. n (number of data samples): the total number of data points in a single group to be processed, such as the number of EEG alpha wave power data points or ECG HRV data points collected within a certain period. It is a positive integer and varies with the collection duration and frequency in practical applications, typically ranging from 1 to several thousand. IQR (interquartile range): the difference between the third quartile and the first quartile, reflecting the dispersion of the middle 50% of the data. It is not affected by extreme values ​​and is used to quantify the stability of the data. It is a non-negative real number. (Outlier determination coefficient): A fixed coefficient used to expand the coverage of the interquartile range, ensuring that extreme outliers are reasonably removed while retaining discrete data within the normal range. In this embodiment, the industry-standard coefficient of 1.5 is used, which is a fixed constant.

[0052] The Kalman filter equation is as follows:

[0053]

[0054] in, Predicted value at time k: based on The optimal estimate of time k is used to predict the data estimate at time k, and is used to initially estimate the real data at the current time (such as physiological signals, environmental parameters, etc.). The value is consistent with the data type of the measured data (such as EEG signal power value, temperature value, etc.). for Optimal estimate at time: The optimal data value obtained after the Kalman filter update at time k is the basis for the prediction at time k, and its value is consistent with the data type of the measured data. The observation value at time k: the raw data (including noise) directly collected by the sensor at time k, such as the fixation point accuracy data collected by the eye tracker, the temperature data collected by the environmental sensor, etc., is taken as the actual measured value of the sensor. H is the observation matrix, which describes the mapping relationship between the system state and the observation values. In the embodiment of this application, the observation values ​​directly reflect the system state, so let H be the observation matrix. The value is a real number matrix (here it is...) (Matrix, with elements equal to 1). Let K be the Kalman gain at time k, a coefficient that balances the reliability of the predicted estimate and the observed value, determining the contribution of the observed value to the updated estimate. Its value range is... (like . For the observed values, This is the observation matrix.

[0055] Furthermore, data synchronization employs timestamp alignment and linear interpolation to compensate for... With missing data, the linear interpolation formula is:

[0056]

[0057] in, (Target timestamp data value): The missing data value to be supplemented, i.e., the multimodal data (such as physiological data, interaction data, etc.) corresponding to the target timestamp t, with the value consistent with the missing data type. t (target timestamp): The time point when the data is missing, with a precision of 1ms (data synchronization timestamp precision), and a non-negative integer value (unit: ms). t1 (preceding valid timestamp): The timestamp corresponding to the most recent valid data before t, ensuring the minimum time interval with t, and a non-negative integer value (unit: ms). t2 (subsequent valid timestamp): The timestamp corresponding to the most recent valid data after t, with a non-negative integer value (unit: ms). . (Preceding valid data value): Valid data collected corresponding to timestamp t1, with values ​​consistent with data type (such as EEG signal value, touch response speed, etc.). (Subsequent valid data values): Valid collected data corresponding to timestamp t2, with values ​​consistent with data type.

[0058] The feature extraction includes calculating eye-tracking focus, EEG fatigue index, heart rate stress coefficient, multimodal interaction success rate, driving stability coefficient, and scene complexity. Normalization adopts Standardization, the calculation formula is:

[0059]

[0060] in, (Standardized data values): Mapping the original data to... The normalized result after the interval is used to eliminate the dimensional differences between data of different dimensions (such as different units of light intensity and noise level), which facilitates multimodal feature fusion calculation. The value range is [value range missing]. (Light intensity standardized in the embodiments of this application) ). (Raw data values): Multimodal data to be standardized, such as light intensity, noise level, traffic density, etc., are taken as the original sensor acquisition values. (Minimum value of data): The theoretical or measured minimum value of the data in this dimension, such as the minimum value of light intensity. (No light) Minimum noise level (Lower limit for environmental sensor noise measurement), the value is taken as the reasonable minimum value of the data in this dimension. (Maximum value of data): The theoretical or measured maximum value of the data in this dimension, such as the maximum light intensity. (Upper limit for ambient light measurement by environmental sensor), maximum noise level) (Upper limit for environmental sensor measurements), which is the reasonable maximum value for the data in this dimension.

[0061] S3: Adopt Evidence theory integrates multimodal features, through An attention mechanism algorithm is used to construct a dynamic fusion adaptation model. The model is trained in multiple rounds of iterations using multi-dimensional measured data and synthetic supplementary data as samples, so that the parameter prediction error of the dynamic fusion adaptation model is less than a preset threshold, thus obtaining a trained dynamic fusion adaptation model.

[0062] For example, the preset threshold in this application embodiment can be set according to the actual application scenario. For instance, the preset threshold can be set to... That is, the parameter prediction error of the trained dynamic fusion adaptation model. In one exemplary embodiment, first based on " "Associative logic, through" Evidence theory assigns probabilities and resolves conflicts based on preprocessed physiological, interaction, behavioral, and scene characteristics, thus assigning modal features. The fusion confidence level is used to address the uncertainty problem of multi-source data; then, a dynamic fusion adaptation model is constructed. In this embodiment, the dynamic fusion adaptation model is " The "attention mechanism" combined model, in which the LSTM layer has 256 neurons and a dropout rate of 0.2, captures temporal data correlations through the synergistic effect of the forgetting gate, input gate, and output gate. The attention mechanism layer calculates the feature weights at each time step using the softmax function, enhancing the feature contributions corresponding to key physiological states (such as fatigue and focus) and scene changes (such as congestion and bright light). The model is trained using real-world data from 200 different groups (ordinary users, the elderly, children, and disabled users). “ Using correlated data and long-tail scenario synthetic data as samples, the Adam optimizer was iterated for 1000 rounds, and the parameters were continuously optimized through the MSE loss function, ultimately reducing the prediction error of 23 cockpit parameters in four categories. It also supports independent training and switching adaptation of model branches specific to certain population groups.

[0063] Furthermore, in this embodiment of the application, the construction and training of the multimodal fusion adaptation model includes the following steps:

[0064] 1. Multimodal feature fusion preprocessing: based on " "The association model filters out 32 core features obtained after preprocessing (including eye movement focus, EEG fatigue index, heart rate stress coefficient, scene complexity, etc.) to construct a recognition framework." Assign a base probability to each type of feature (e.g., 0.8 for physiological features and 0.75 for interaction features).

[0065] 2. Evidence theory fusion is achieved by quantifying the confidence of each modality feature through basic probability assignment and calculating the fusion confidence of the feature intersection events using Dempster's composition rule. The formula is as follows:

[0066]

[0067] in, The confidence score is the reliability score after multimodal feature fusion, used to determine whether the features can be used for parameter prediction. For example, After fusion The model determines that the feature combination is reliable and can be directly used to calculate parameters such as seats and screens; if If so, the model will reduce the weight of that feature to avoid misleading parameter adaptation. To integrate target events: reliability determination events based on multimodal features, the identification framework in this application embodiment... , It can be a single event (such as "physiological characteristics are reliable") or an overlapping event (such as " ), covering the core dimensions of cockpit data. For single-modal confidence, it is a reliability score for single-modal data. For example, , , , The values ​​are based on the consistency and validity of the single-modal data. For single-modal events, reliability determination is based on four categories of features: physiological, interactive, behavioral, and contextual. For example, "Reliable physiological characteristics" "Reliable interaction features" corresponds to the credibility assessment of four types of data, providing a foundation for fusion computing. The conflict coefficient quantifies the degree of conflict in multimodal data. For example, physiological data shows user fatigue (confidence 0.8), but interaction data shows frequent user actions (confidence 0.7), indicating a conflict, and the K value increases (e.g., to 0.2); if all modal data show the user's state is stable, A value close to 0 indicates a better fusion effect, ensuring the confidence level of the core features after fusion. This addresses the issue of low reliability in existing multimodal data fusion technologies.

[0068] 3. Composite model structure design: Constructing " The "attention mechanism" deep model uses an LSTM layer with 256 neurons and a dropout rate of 0.2. The forgetting gate, input gate, and output gate employ the sigmoid activation function, while the cell state uses the tanh activation function. Formulas capture the temporal correlation between physiological states and scene changes; the attention mechanism layer... The feature attention score is calculated and normalized to enhance the weights of key features such as fatigue and strong light, thus addressing the limitation of single models in accurately capturing nonlinear temporal correlations. Among these, Attention scoring: Quantifying the importance of features for parameter fitting, for example, eye-tracking attention features in high-speed driving scenarios. (High score, key features) Fragrance concentration characteristics (Low score, minor feature); Touch response speed characteristics in congested scenarios. (High scores, key features) The model prioritizes adapting the parameters corresponding to the key features. Transpose the attention score weight vector: A dimensional vector, optimized through training. For example, for scenarios involving the elderly. It will increase the score weight of features related to font size and voice sensitivity, making these features easier to strengthen. Here is the attention weight matrix for the hidden state: A 3D matrix, optimized through training. For example, for scenarios involving users with disabilities, It will strengthen the weight of hidden states related to voice interaction and gesture control, and improve the attention scores of these features. The hidden state at time t is a 128-dimensional output vector that fuses historical and current features. For example, It includes the user's historical seating preferences, current physiological state, and scene complexity, which are directly input into the attention mechanism layer for subsequent parameter prediction. Input feature attention weight matrix: A 3D matrix, optimized through training. For example, in a rainstorm scenario, This will increase the weight of input features related to light intensity and wiper frequency, allowing the model to prioritize the adaptation to visual parameters. The input features at time t are 32-dimensional multimodal fusion features, including current physiological, interaction, and core scene information. Adjust the scoring baseline for the attention score bias term: a 128-dimensional vector. For example, for children's scenarios, The optimization will prioritize safety-related features (such as seatbelt pressure and temperature) to ensure higher scores and better fit. The output of the forget gate controls whether historical time-series information is retained. For example, if a user remains focused while driving at high speed... (Retains most historical focus information); If a user switches from congested traffic to smooth traffic, (Forget some fatigue information during congestion) to ensure the model adapts to changes in state. The input features at time t can be multimodal fusion features, including current physiological, interaction, and core scene information. The input gate output controls the degree of influence of the current feature on the model. For example, a user suddenly experiences fatigue signals (EEG fatigue index 0.9). (To enhance the impact of this fatigue characteristic), the model quickly adjusts cockpit parameters (such as reducing screen brightness and playing soothing music). The cell state at time t: The core memory unit of the LSTM stores key temporal information. For example, the user fatigue state over three consecutive update cycles (900ms). All are stored in Based on this, the model judges the user's fatigue trend and dynamically adjusts the seat angle and air conditioning temperature. Candidate cell states are 128-dimensional vectors that store the core information to be updated at the current moment. For example, the user's satisfaction rating of 9.1 and cognitive load of 4.1 at the current moment constitute candidate cell states, which will be incorporated into the model's memory.

[0069] 4. Model Training Optimization: The training sample includes measured data from 200 subjects (112 ordinary users, 24 elderly people, 16 children, and 8 disabled users). “ "The model incorporates data from related scenarios and synthesized data from long-tail scenarios such as heavy rain and extreme traffic congestion; it employs the Adam optimizer (learning rate 0.001), batch size 32, and 1000 iterations, continuously optimizing model parameters through the MSE loss function. Separate model branches are designed for elderly, children, and disabled users, with targeted optimization of feature weights. Ultimately, this achieves accurate prediction of 23 cabin parameters across four categories." It is significantly better than the existing technology with an error level of more than 8%.

[0070] The core innovation of step S3 lies in... Evidence theory and " "Combining association models to achieve high-confidence fusion of multimodal features (confidence level)" ), and at the same time build " This model combines an "attention mechanism" approach with an additional branch for specific population groups. It captures the temporal correlation between physiological states and scene changes, strengthens the weights of key features, and employs a mixed sample of measured data, correlated data, and long-tail scene-synthesized data for training across the entire population. This overcomes the limitations of existing technologies, such as low reliability of multi-source data fusion, simple model architecture, and incomplete sample coverage. Its beneficial effects include reducing parameter prediction error to [a lower value]. It is far superior to the existing technology with an error level of more than 8%, effectively resolves multimodal data conflicts, improves the scientific nature of the adaptation logic, and meets the personalized needs of elderly, children and disabled users, filling the gap in adaptation for special groups.

[0071] S4: Real-time dynamic parameter adaptation: The preprocessed multimodal data is input into the trained dynamic fusion adaptation model. The dynamic fusion adaptation model outputs the optimal parameter configuration and distributes it to each control module via a high-speed bus, dynamically updating it once every preset time period. For example, the preset time period can be set to 300ms.

[0072] It is understood that the control module in the embodiments of this application can be a vehicle control component, such as a seat, seat back, steering wheel, HUD, screen, rearview mirror, vehicle system, vision recognition module, touch response module, cabin temperature adjustment module, fragrance control module, etc., which can be set and adapted according to needs, and will not be described in detail here.

[0073] In this embodiment, dynamic parameter adaptation includes adapting 23 parameters in four categories: seat height, backrest angle, steering wheel distance, HUD position, screen brightness, rearview mirror angle, screen color temperature, steering wheel pressure, voice sensitivity, gesture control sensitivity, touch response speed, cabin temperature, humidity, lighting brightness, and fragrance concentration.

[0074] In one exemplary embodiment, the parameter range for dynamic parameter adaptation may include the following: seat height. Backrest angle Steering wheel distance HUD position Screen brightness Rearview mirror angle Screen color temperature Steering wheel force Voice sensitivity Gesture control sensitivity Touch response speed cabin temperature ,humidity Lighting brightness Fragrance concentration It includes 23 parameters in four categories, including levels.

[0075] In this embodiment of the application, the training of the dynamic fusion adaptation model used experimental data from 200 subjects (including 112 ordinary users, 24 elderly people, 16 children, and 8 disabled users) and over 100,000 data points. "The associated data, combined with synthetic data from long-tail scenarios, was iterated through the Adam optimizer for 1000 rounds, and the confidence level was then integrated." The core purpose of this step is to enable the intelligent cockpit parameters to be accurately and dynamically adapted to different groups of people and scenarios. This is achieved by covering 23 key parameters across four categories: posture, vision, control, and environment, combined with real-world test data from ordinary users, the elderly, children, and disabled users. “ "The associated data and long-tail scene synthetic data were trained by the Adam optimizer for 1000 rounds, with..." By integrating confidence scores to construct a highly reliable model, the problems of limited applicability, static nature, poor universality, and high error in existing technologies are addressed, ultimately achieving improved parameter prediction accuracy. This transforms the cabin from "passive adjustment" to "active adaptation," allowing it to quickly adjust according to physiological states and scene changes without manual user operation. It covers all people and various typical and extreme scenarios, significantly improving driving safety, ease of operation, and ride comfort.

[0076] It is understood that the real-time dynamic parameter adaptation described in this application embodiment supports the triggering of adaptation logic specific to special populations. The adaptation logic specific to special populations includes: adaptation for the elderly: font enlargement by 20%, voice sensitivity reduced to level 2, seat height increased by 2cm, and backrest angle increased. Touch response speed increased by 10%; Child-friendly: Rear seat temperature remains constant. Entertainment and interactive content safety filtering, automatic seat belt tension adjustment, simplified voice commands; Adaptation for disabled users: voice control. Gesture-based operation is prioritized, touch area is enlarged by 30%, and operation process is simplified. The system includes adaptive adjustment of environmental parameters (temperature 24℃, humidity 50%, light intensity 500 lux). The core purpose of this step is to provide customized solutions for the physiological characteristics and interaction habits of elderly, children, and disabled users. Through targeted settings such as enlarged fonts, constant temperature, and voice / gesture-based operation, it fills the gaps in adaptation for special groups and addresses the lack of universality in existing technologies. Its significance lies in upgrading the smart cockpit from "single-user adaptation" to "full-user coverage," achieving personalized service for each individual. The practical effect is a significant reduction in cognitive load and error rate for special groups, improving their driving safety, ease of operation, and satisfaction, while simultaneously expanding the application scenarios and user base of the smart cockpit.

[0077] The core innovation of the S4 step lies in constructing a closed-loop adaptation process with dynamic updates within 300ms. It leverages the high-speed transmission of the CANFD bus to achieve low-latency parameter command delivery. Simultaneously, it designs a dual-trigger mechanism combining face recognition and physiological feature detection to automatically activate dedicated adaptation logic for elderly, children, and disabled users. This overcomes the limitations of existing technologies such as static parameter templates, manual switching of adaptation modes, and lag in response. Its beneficial effect is manifested in improved overall adaptation latency. It can respond in real time to changes in the user's physiological state and scene switching, and can provide accurate and personalized cabin parameter configurations for different groups of people without human intervention, significantly reducing the cognitive load and error rate of special groups of people, and improving driving safety and comfort.

[0078] Through steps S3 and S4 above, a model is first trained using multi-dimensional data to accurately integrate physiological, interactive, and scene information. It has also been specifically optimized for the elderly, children, and people with disabilities. A fast-response adaptation system is then built, updating cockpit parameters every 300 milliseconds and automatically identifying user types and switching to customized settings without manual adjustment. Compared to previous technologies, it solves the problems of inaccurate data, single-model approach, fixed parameters, and slow response. In practical use, parameters are tuned more accurately (error reduced to within 3.4%) and the response is faster (total latency less than 0.6 seconds). Whether for ordinary people or those with special needs, everyone can directly enjoy a cockpit experience tailored to their requirements, reducing misoperations and making driving safer and more comfortable.

[0079] In one embodiment of this application, the multimodal data-driven intelligent cockpit parameter adaptation method of the present invention, in addition to the steps described above, It may also include step S5 for continuous iteration and optimization of the model. Step S5: Adaptation effect verification and feedback: verify the adaptation effect through objective indicators and subjective scores, accumulate data to iteratively optimize the model, and support online model updates and scenario expansion.

[0080] According to one aspect of the embodiments of this application, a multimodal data-driven intelligent cockpit parameter adaptation system is provided, which is applied to the above-described multimodal data-driven intelligent cockpit parameter adaptation method. Figure 2 As shown, it includes the following modules:

[0081] Multimodal data acquisition module: configured to collect human physiological data, cockpit interaction behavior data, vehicle operating status data, external environment perception data, and subjective feedback data. Multimodal data is acquired synchronously through standardized acquisition equipment.

[0082] Data preprocessing module: configured to clean, synchronize and extract features from collected multi-source data to obtain physiological features, interaction features, behavioral features and scene features.

[0083] Multimodal fusion adaptation model construction and training module: configured to use Evidence theory integrates multimodal features, through The attention mechanism algorithm constructs a dynamic fusion adaptation model. The model is trained in multiple rounds using multi-dimensional measured data and synthetic supplementary data as samples, so that the parameter prediction error of the dynamic fusion adaptation model is less than a preset threshold, and the trained dynamic fusion adaptation model is obtained.

[0084] Parameter execution module: It is configured to input preprocessed multimodal data into the trained dynamic fusion and adaptation model, output the optimal parameter configuration, and send it to each control module through the high-speed bus. It is dynamically updated once every preset time period.

[0085] In the technical solution provided in the embodiments of this application, human physiological data, cockpit interaction behavior data, vehicle operating status data, external environment perception data, and subjective feedback data are collected, cleaned, synchronized, and feature extracted to obtain physiological features, interaction features, behavioral features, and scene features. Multimodal features are then fused using evidence theory. The dynamic fusion adaptation model, which combines attention mechanism algorithm construction with key feature enhancement, is trained iteratively in multiple rounds using multi-dimensional measured data, associated data, and synthetic supplementary data as samples, until the parameter prediction error of the dynamic fusion adaptation model is less than a preset threshold. At this point, the dynamic fusion adaptation model training is complete. When applied in the vehicle, multiple optimal configurations of cockpit parameters are dynamically distributed periodically via a high-speed bus. This achieves a leap from "static template adaptation" to "multimodal data-driven real-time accurate adaptation for all users and all scenarios" for intelligent cockpit parameters. This not only significantly improves parameter adaptation accuracy, response speed, and universality, filling the adaptation gap for special groups, but also reduces user cognitive load and error rate, comprehensively improving driving safety and ride comfort.

[0086] Please see Figures 1-2 This invention provides a multimodal data-driven intelligent cockpit parameter adaptation method; the invention performs calculations and tests on the above method using example data, as detailed below:

[0087] Test 1: Full-Scenario Adaptation Test for Ordinary Users

[0088] Test subjects: 112 ordinary users ( age, Driving experience Year).

[0089] Test scenario: High-speed driving ( ), urban congestion Night driving ), high noise environment ( ).

[0090] Core verification:

[0091] Multimodal data acquisition: synchronization accuracy .

[0092] Preprocessing results: Outlier removal rate of 98%, signal-to-noise ratio improved by 35% after noise smoothing.

[0093] Model performance: Parameter prediction error fusion confidence .

[0094] Dynamic adaptation: End-to-end latency Update cycle: 300ms.

[0095] The following are the full-scenario adaptation effects for ordinary users:

[0096] Table 1. Adaptation effect for ordinary users across all scenarios

[0097]

[0098] Test 2: Adaptability Test Specifically for Seniors

[0099] Test subjects: 24 elderly people ).

[0100] Test scenario: City roads ( ), community roads ( ).

[0101] Core verification:

[0102] Enlarging the font by 20%: Information recognition accuracy increased from...

[0103] Voice sensitivity level 2: False wake-up rate from .

[0104] Raising the seat by 2cm reduces lumbar pressure by 30%.

[0105] The following are the specific adaptation effects for seniors:

[0106] Table 2. Adaptation Effects Specifically for the Elderly

[0107]

[0108] Test 3: Child-Specific Adaptability Test

[0109] Test subjects: 16 children ( ) One accompanying parent.

[0110] Test scenario: Heavy rain ( Foggy weather ).

[0111] Core verification:

[0112] Child-friendly: Seat belt pressure distribution uniformity improved by 35%, and entertainment content filtering accuracy reached 100%.

[0113] Extreme weather: Glare is reduced by 45% in rainstorm scenes and information recognition speed is improved by 20% in fog scenes.

[0114] The adaptation effect is as follows:

[0115] Table 3. Adaptation Effects Specifically for Children

[0116]

[0117] Test 4: Adaptation Test for Users with Disabilities

[0118] Test subjects: 8 people with disabilities (6 with limited physical activity and 4 with mild visual impairment).

[0119] Test scenario: City roads ( ), parking lot scene ( ).

[0120] Core verification:

[0121] Voice-based operation replacement: Core function completion rate increased from .

[0122] Increased touch area: Accidental touch rate from .

[0123] Environmental Adaptation: The accuracy of visually impaired user information recognition has increased from [previous level]. .

[0124] The customized features for users with disabilities are as follows:

[0125] Table 4. Adaptation Effects Specifically for Users with Disabilities

[0126]

[0127] The comparison of the core indicators in tests 1-4 above is as follows:

[0128]

[0129] Conclusion: All embodiments verify the core technical features of the embodiments of this application, and the parameter prediction error is... End-to-end delay The device is highly effective in adapting to special populations, reducing the error rate by more than 70% and increasing satisfaction by more than 38%.

[0130] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

Claims

1. A multimodal data-driven intelligent cockpit parameter adaptation method, characterized in that, Includes the following steps: S1: Multimodal data acquisition: Collect human physiological data, cockpit interaction behavior data, vehicle operating status data, external environment perception data, and subjective feedback data. The multimodal data is acquired synchronously through standardized acquisition equipment. S2: Data preprocessing: Cleaning, synchronizing and feature extraction of the collected multi-source data to obtain physiological features, interaction features, behavioral features and scene features; S3: Construction and training of multimodal fusion adaptation model: Multimodal features are fused using evidence theory, and a dynamic fusion adaptation model is constructed using LSTM + attention mechanism algorithm. The model is trained in multiple rounds of iterations using multi-dimensional measured data and synthetic supplementary data as samples, so that the parameter prediction error of the dynamic fusion adaptation model is less than a preset threshold, and the trained dynamic fusion adaptation model is obtained. S4: Real-time dynamic parameter adaptation: The preprocessed multimodal data is input into the trained dynamic fusion adaptation model. The dynamic fusion adaptation model outputs the optimal parameter configuration and sends it to each control module through the high-speed bus. It is dynamically updated every preset time period.

2. The multimodal data-driven intelligent cockpit parameter adaptation method according to claim 1, characterized in that: The human physiological data includes eye movement data, electroencephalogram (EEG) signals, electrocardiogram (ECG) data, skin conductance levels, and the human body's position on the cushion. Backrest pressure distribution and contact area; The cockpit interaction data includes voice interaction data, gesture recognition data, eye tracking data, touch interaction data, and facial recognition data. The vehicle operating status data includes vehicle speed, acceleration, steering angle, braking frequency, cabin environment control parameters, and system operating resource utilization rate. The external environment perception data includes light intensity, ambient temperature, noise level, rainfall / visibility, road type, and traffic density; The subjective feedback data includes satisfaction, cognitive load, and emotional well-being scores.

3. The multimodal data-driven intelligent cockpit parameter adaptation method according to claim 1, characterized in that, The cleaning of the collected multi-source data includes: IQR outlier removal and Kalman filtering noise smoothing are employed. The IQR outlier removal includes: calculating a set threshold for the interquartile range (IQR), using the following formula: , , The outlier threshold is Data values ​​exceeding the outlier threshold range are removed. Here, Q1 is the first quartile, reflecting the lower limit of the data distribution; Q3 is the third quartile, reflecting the upper limit of the data distribution; n is the number of data samples, which takes a positive integer value; IQR is the interquartile range, which is the difference between the third quartile and the first quartile. IQR is used to quantify the stability of the data. The outlier determination coefficient is a fixed coefficient. The Kalman filter equation is as follows: in, The Kalman gain at time k, For the observed values, For the observation matrix, The estimated value is the value predicted at time k. for The optimal estimate at time. The value is the observation at time k.

4. The multimodal data-driven intelligent cockpit parameter adaptation method according to claim 3, characterized in that, Synchronizing multi-source data includes: Data synchronization employs timestamp alignment and linear interpolation to compensate for missing data. The linear interpolation formula is as follows: in, t1 represents the target timestamp data value, i.e., the multimodal data corresponding to the target timestamp t; t is the target timestamp: the time point when the data is missing, with a precision of 1ms, and a non-negative integer value; t1 is the preceding valid timestamp: the timestamp corresponding to the most recent valid data before t, ensuring that the time interval with t is minimized, and a non-negative integer value. t2 is the subsequent valid timestamp, the timestamp corresponding to the most recent valid data after t, and its value is a non-negative integer (and ; For previous valid data values: valid collected data corresponding to timestamp t1, with values ​​consistent with data types; For subsequent valid data values: valid collected data corresponding to timestamp t2, with values ​​consistent with data type.

5. The multimodal data-driven intelligent cockpit parameter adaptation method according to claim 4, characterized in that, Feature extraction is performed on the collected multi-source data, including: the feature extraction includes calculating eye-tracking focus, EEG fatigue index, heart rate stress coefficient, multimodal interaction success rate, driving stability coefficient, and scene complexity, wherein the scene complexity... Normalization adopts Standardization, the formula is: in, These are the standardized data values; The original data value; The minimum value of the data. This represents the maximum value of the data.

6. The multimodal data-driven intelligent cockpit parameter adaptation method according to claim 1, characterized in that, The control module includes vehicle control components, including but not limited to seats, backrests, steering wheels, HUDs, screens, rearview mirrors, vehicle infotainment systems, vision recognition modules, touch response modules, cabin temperature control modules, and fragrance control modules.

7. The multimodal data-driven intelligent cockpit parameter adaptation method according to claim 1, characterized in that, The dynamic parameter adaptation includes, but is not limited to, adapting parameters such as seat height, backrest angle, steering wheel distance, HUD position, screen brightness, rearview mirror angle, screen color temperature, steering wheel pressure, voice sensitivity, gesture control sensitivity, touch response speed, cabin temperature, humidity, lighting brightness, and fragrance concentration.

8. The multimodal data-driven intelligent cockpit parameter adaptation method according to claim 1, characterized in that, The preset threshold is set to .

9. A multimodal data-driven intelligent cockpit parameter adaptation system, characterized in that, include: Multimodal data acquisition module: configured to collect human physiological data, cockpit interaction behavior data, vehicle operating status data, external environment perception data, and subjective feedback data. The multimodal data is acquired synchronously through standardized acquisition equipment. Data preprocessing module: configured to clean, synchronize, and extract features from collected multi-source data to obtain physiological features, interaction features, behavioral features, and scene features; Multimodal fusion adaptation model construction and training module: configured to use Evidence theory integrates multimodal features, through An attention mechanism algorithm is used to construct a dynamic fusion adaptation model. The model is trained in multiple rounds of iterations using multi-dimensional measured data and synthetic supplementary data as samples, so that the parameter prediction error of the dynamic fusion adaptation model is less than a preset threshold, and the trained dynamic fusion adaptation model is obtained. Parameter execution module: It is configured to input preprocessed multimodal data into the trained dynamic fusion and adaptation model, output the optimal parameter configuration, and send it to each control module through the high-speed bus. It is dynamically updated once every preset time period.

Images