Distributed edge computing architecture of emotion cockpit and intelligent cockpit

Abstract

The invention provides a distributed edge computing architecture of an emotion cockpit and an intelligent cockpit, and the distributed edge computing architecture comprises a local edge node layer, a regional coordination node layer and a central control unit layer which are connected through low-delay communication interfaces, and cooperatively achieve the collection, fusion, judgment and strategy response of multi-modal emotion data. Wherein the local edge node layer comprises a plurality of intelligent terminals deployed near passengers in a vehicle, and each terminal comprises a modal sensing module and a feature extraction module; the area coordination node layer comprises a plurality of area processing units, and each unit comprises an area fusion module; the central control unit layer comprises an emotion map module, an emotion decision module and a strategy response module. By adopting an edge-region-center three-level collaborative architecture, the method has low delay, high robustness and privacy protection capability, can realize emotion-driven in-vehicle environment intelligent linkage control, and improves driving safety and user experience.

Description

Technical Field

[0001] The present invention relates to the field of distributed edge computing technology, and in particular to a distributed edge computing architecture and a smart cockpit for an emotional cockpit. Background Art

[0002] Smart cockpits and in-vehicle emotion recognition systems are currently receiving widespread attention for their user experience and driving safety. With advances in in-vehicle sensors and multimodal data processing, emotion recognition technology is increasingly being applied to intelligent in-vehicle environmental control. However, existing emotion recognition systems still suffer from the following drawbacks:

[0003] 1. Insufficient privacy protection. Currently, many in-vehicle emotion recognition systems upload users' facial expressions, voice, and physiological signals to the cloud for processing. This poses a potential risk to users' privacy protection, especially when processing personal sensitive information, which may lead to data leakage or abuse.

[0004] 2. Latency: Existing emotion recognition systems often rely on cloud computing or centralized data processing architectures, resulting in high latency during data transmission. This makes it difficult to meet the demand for real-time feedback, especially in emergency driving scenarios, where emotion-driven environmental adjustments cannot take effect immediately.

[0005] 3. High computing resource requirements: Most existing emotion recognition systems concentrate computing tasks on central servers or in the cloud, which requires high bandwidth and a large amount of computing resources. This not only increases hardware costs but may also lead to excessive energy consumption, affecting the vehicle's energy efficiency and endurance.

[0006] The patent document (application number: CN201910089030.7) discloses a multimodal emotion recognition method and system that integrates cloud and edge computing. The system includes a data acquisition module, an edge computing module, a network transmission module, a cloud computing module, and an emotional state determination module. The data acquisition module is responsible for collecting facial images and voice-related data; the edge computing module calculates local emotion recognition results based on facial expression recognition technology; the network transmission module transmits voice data to the cloud computing module; the cloud computing module is responsible for calculating online emotion recognition results based on voice emotion recognition technology, and pushes the results to the emotional state determination module through the network transmission module; the emotional state determination module implements a comprehensive judgment of the emotional state based on the local emotion recognition results and the online emotion recognition results. However, this patent relies on the network to transmit voice data to the cloud for processing, which may affect the real-time performance of emotion recognition when the network is delayed or unstable; the weights of the multimodal data fusion algorithm are fixed and cannot be dynamically adjusted according to the real-time scenario, and may not be able to adapt to the emotion recognition needs of different users or environments.

[0007] Patent document (application number: CN202210451569.4) discloses a distributed multimodal emotion detection method that combines three physiological signals—electroencephalogram (EEG), electrocardiogram (ECG), and electrodermal conductance—to detect emotions. This method utilizes a multi-terminal architecture to enhance its performance. The stimulus generation terminal is responsible for eliciting the subject's emotions. The subject's physiological signals are collected by the signal acquisition terminal, and the data transceiver integrates the signals from the signal acquisition terminal. The main control terminal receives data from the data transceiver and sends it to the algorithm processing terminal. The algorithm processing terminal splits, filters, and downsamples the received data, extracts features from each modality's physiological signals, performs feature fusion and classification, and determines the subject's emotional state. The algorithm processing terminal sends the physiological signal features, downsampled physiological signals, and emotion recognition results to the main control terminal. The main control terminal receives data from the algorithm processing terminal and sends it to the result display terminal. The result display terminal is responsible for displaying the subject's current physiological signals, physiological signal feature maps, and emotion recognition results in real time. However, the data collection of this patent is complex and relies on professional equipment, multimodal fusion needs to be optimized, the distributed architecture is susceptible to communication delays, the emotion induction effect is unstable, the model generalization ability is limited, and its applicability to complex scenarios is insufficient. Summary of the Invention

[0008] In view of the defects in the prior art, the purpose of the present invention is to provide a distributed edge computing architecture and a smart cockpit for an emotional cockpit.

[0009] The distributed edge computing architecture of the emotional cockpit provided by the present invention includes:

[0010] The three-layer structure of local edge node layer, regional coordination node layer and central control unit layer is connected by low-latency communication interface, collaboratively realizing the collection, fusion, judgment and strategic response of multimodal emotion data;

[0011] in:

[0012] The local edge node layer includes multiple smart terminals deployed near the passengers in the vehicle, each terminal including:

[0013] Modal perception module M1, used to collect multimodal emotional data of passengers in real time, including images, voice and physiological signals;

[0014] Feature extraction module M2 is used to perform local preprocessing, feature extraction and preliminary emotion recognition on the raw data, and generate a structured data packet containing emotion labels, confidence levels and timestamps;

[0015] The regional coordination node layer includes multiple regional processing units, each of which includes:

[0016] The regional fusion module M3 is used to receive data uploaded by local edge nodes in the region, synchronize time through a sliding time window mechanism, and use a multimodal fusion model to perform feature fusion and output the regional dominant emotion label and confidence level.

[0017] The central control unit layer includes:

[0018] Emotional map module M4, used to receive emotional data from each area and construct a structured emotional map of all passengers in the vehicle;

[0019] Emotional decision module M5, used to combine the emotional map, the internal and external environmental conditions and vehicle driving information to generate a multimodal response strategy;

[0020] The strategy response module M6 is used to map the strategy instructions into specific device control commands to drive the coordinated adjustment of the cockpit environment.

[0021] Preferably, the modality perception module M1 includes:

[0022] Image acquisition submodule M1.1 collects occupant facial expression data through in-car cameras, supports infrared / visible light dual-channel imaging, and performs local expression classification based on a lightweight convolutional neural network model;

[0023] The speech acquisition submodule M1.2 collects speech data through a distributed microphone array, extracts Mel-frequency cepstral coefficients (MFCCs) and pitch features, and performs speech emotion recognition using a CNN-BiLSTM model.

[0024] The physiological signal acquisition submodule M1.3 collects heart rate, heart rate variability (HRV) and skin conductance (EDA) signals through steering wheel sensors or wearable devices, and generates physiological emotion indicators based on rule models or small neural networks.

[0025] Preferably, the regional fusion module M3 adopts a sliding time window mechanism to align the timestamps of the multimodal data and performs feature fusion through a lightweight version of the FusionNet-Lite model;

[0026] The multimodal fusion model includes the following steps: extracting feature vectors from images, voices and physiological signals respectively; performing weighted summation or concatenation on the feature vectors to form a comprehensive feature vector; inputting the feature vectors into a classification network to output an emotion label and a confidence level;

[0027] Taking the emotional state and the environmental state as model input, the fusion model is used to learn the relationship between emotion and environment, which can be expressed as:

[0028] E=f(S,C)

[0029] Among them, E represents the emotional state, S represents the environmental state, C represents the emotional characteristics, and the function f() represents the reasoning process of joint modeling;

[0030] The model generates decisions based on the comprehensive information of emotional state and environmental state, using a joint neural network model in the form of:

[0031] E=NN(S,C)

[0032] Among them, the neural network NN() represents the joint learning and reasoning of these two inputs.

[0033] Preferably, the emotion map module M4 dynamically maintains the whole vehicle emotion state map through JSON objects or graph database, including occupant positions, dominant emotion labels, confidence levels and historical trajectory information;

[0034] The graph module provides a standardized interface, supports data query by time interval, passenger ID or emotion type, and supports historical data export for offline model training.

[0035] Preferably, the emotion decision module M5 generates a response strategy in the following manner:

[0036] Rule engine: preset emotional state-environmental conditions-response strategy mapping rules, triggering safety intervention strategies when driver anxiety is detected and vehicle speed exceeds thresholds;

[0037] Machine learning model: Based on decision trees or reinforcement learning networks, it inputs emotion feature vectors and environmental parameters and outputs recommendation strategies;

[0038] The response strategy includes at least one of air conditioning adjustment, lighting control, music playback, and voice interaction.

[0039] Preferably, the training process of the decision tree model includes:

[0040] Collect historical emotional data and environmental parameters and mark the optimal response strategy;

[0041] Construct a decision tree based on the CART algorithm, using Gini impurity or information gain rate as the splitting criterion;

[0042] Improve model generalization capabilities through pruning and hyperparameter tuning;

[0043] Regular incremental training is performed to update the model.

[0044] Preferably, the policy response module M6 converts abstract policy instructions into device control commands via the control bus CAN / LIN, supporting concurrent scheduling and conflict management;

[0045] The execution order is dynamically adjusted according to the device status and task priority, and the execution results are fed back to the emotional decision module M5 to optimize the strategy closed loop.

[0046] Preferably, a privacy protection mechanism is adopted, whereby the original image, voice, and physiological data are processed only at the local edge node, and only the structured emotion feature vector is uploaded to the parent node;

[0047] Regional coordination nodes and central control units do not store or transmit original sensitive data.

[0048] Preferably, it supports multiple rounds of emotional response linkage, dynamically adjusting the strategy intensity or switching the strategy type according to the emotional trend, forming a continuous interactive process;

[0049] The response strategy includes a combination of at least two of seat vibration, temperature gradient adjustment, and adaptive music recommendation.

[0050] The smart cockpit provided by the present invention is equipped with an on-board emotion recognition system, and the on-board emotion recognition system adopts the distributed edge computing architecture of the emotional cockpit.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] (1) The present invention solves the problems of high processing delay, large bandwidth consumption and data privacy leakage risks in centralized processing architecture by constructing a distributed edge computing architecture. This architecture sinks the modal perception and feature extraction functions to local edge nodes, allowing sensitive data such as raw images and voice to be pre-processed and emotion recognized locally, and only structured results are uploaded, thereby reducing data transmission volume, lowering system latency, and effectively protecting passenger privacy.

[0053] (2) The present invention solves the problems of asynchronous uploading of multimodal data, large fusion errors, and recognition inconsistency by introducing a modal fusion mechanism and a time synchronization window at the regional coordination node. A sliding time window mechanism is set at the regional layer to unify the alignment standards of each modal data, and combined with a multimodal attention mechanism and a feature splicing fusion model (such as FusionNet-Lite), the fusion efficiency and consistency are significantly improved, ensuring the reliability of regional emotion judgment.

[0054] (3) The present invention solves the problem of single response and poor scenario adaptability of traditional strategies by establishing a strategy judgment mechanism based on a hybrid drive of rules and models. The present invention supports the intelligent generation of multimodal response strategy solutions based on rule conditions (such as "anxiety + high-speed driving") or strategy models trained based on historical data, including air conditioning adjustment, music playback, lighting control, voice interaction, etc., to achieve personalized and humanized emotional comfort and guidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0056] Figure 1 This is a diagram of the distributed edge computing architecture used in the emotional cockpit of the present invention. DETAILED DESCRIPTION

[0057] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0058] Example

[0059] like Figure 1 The present invention provides a distributed edge computing architecture for an emotional cockpit, which adopts a three-layer distributed edge computing architecture, including: a local edge node layer, a regional coordination node layer, and a central control unit layer. The three-layer structure is connected by a low-latency communication interface to collaboratively complete the collection, fusion, judgment, and strategic response control of the occupant's emotional state, with the characteristics of strong real-time performance, fast response speed, and high system scalability.

[0060] At the local edge node layer, multiple intelligent terminals deployed near the vehicle occupants (e.g., steering wheel cameras, headrest microphones, and physiological signal collectors) form the core components of this layer. Each local node integrates a modal perception module and a feature extraction module, which are used to respectively realize the collection and preliminary analysis and processing of multimodal emotional signals. The modal perception module can collect raw signals such as images, voice, and physiological parameters. The feature extraction module extracts features such as facial expressions, voice emotion indicators, or heart rate fluctuations based on artificial intelligence algorithms, and generates standardized emotional feature vectors for uploading to the upper-level node.

[0061] At the regional coordination node layer, the system deploys regional processing units according to the functional areas in the vehicle (such as the front row, rear row, driver's seat, etc.), which are used to aggregate the emotional data of all local nodes under its jurisdiction. This layer is equipped with a regional fusion module to fuse multiple modal inputs with data from multiple node sources. It uses confidence weighting, time series alignment, multimodal complementarity and other methods to comprehensively output the current dominant emotional label and its credibility score for the region. The regional coordination node can effectively suppress misidentification noise and improve the accuracy and stability of overall judgment. At the same time, it acts as an intermediate hub to report the emotional status of the region to the central control unit.

[0062] At the central control unit level, the core processor deployed in the cockpit's central computing platform assumes global control of the system. This layer houses multiple modules, including an emotion map module, an emotion decision module, and a strategy response module. The emotion map module receives emotion information reported by each area and constructs a structured vehicle-wide emotion map based on information such as occupant location and vehicle ID. The emotion decision module, based on this map and the current internal and external vehicle operating conditions, determines whether there are any emotional events requiring intervention and generates corresponding adjustment strategies. The strategy response module is responsible for mapping emotion strategies into executable cockpit control commands, driving coordinated responses from systems such as air conditioning, lighting, audio, voice announcements, and seat adjustment, thereby achieving intelligent adaptive control of the occupant's emotional state.

[0063] The modules used are as follows:

[0064]

[0065] Module M1: Modal Perception Module

[0066] This module is used to collect multimodal emotion-related signals from passengers in real time, including images, voice, and physiological parameters. Its main function is to convert multi-source sensory data from the vehicle environment into effective raw information required for emotion recognition and analysis. It specifically includes the following submodules:

[0067] Module M1.1: Acquisition of image modality data;

[0068] The microphone array collects the passenger's voice data and identifies their voice characteristics such as intonation, speaking speed, and emotional vocabulary. The specific process includes:

[0069] The camera can be deployed in areas such as the lower A-pillar, the upper center console, and the rearview mirror housing to ensure an unobstructed frontal image view for passengers.

[0070] The collected facial images are used to identify key facial features (such as raised eyebrows, drooping mouth corners, closed eyes, etc.) to assist in judging emotions such as anger, surprise, and fatigue;

[0071] Supports infrared / visible light dual-channel imaging to ensure image quality at night or in backlight conditions;

[0072] Perform pre-processing operations on the image within the edge node (such as grayscale conversion, face alignment, and expression region cropping), and call a lightweight convolutional neural network model to complete local expression classification or emotion embedding generation;

[0073] The output format includes emotion labels (such as "Happy") and confidence levels (such as 0.86).

[0074] Module M1.2: Collect voice modality data;

[0075] The microphone array collects the passenger's voice data and identifies their voice characteristics such as intonation, speaking speed, and emotional vocabulary. The specific process includes:

[0076] Microphones can be embedded in the roof light control panel, both sides of the central control screen, above the seat backs, etc., to form a spatially distributed sound pickup structure;

[0077] The system collects speech waveform data in real time and performs pre-processing operations such as noise reduction, endpoint detection, and speech framing;

[0078] Extract Mel frequency cepstral coefficients MFCC, Chroma features, energy envelope, pitch, formant and other characteristic parameters;

[0079] The data is fed into a lightweight speech emotion recognition model (such as CNN-BiLSTM) deployed on the edge to classify the speech emotion (such as anger, calmness, anxiety) or perform emotion curve regression.

[0080] The output includes speech emotion labels, confidence scores, and related time-domain and frequency-domain feature embeddings for fusion processing.

[0081] Module M1.3: Acquisition of physiological modality data;

[0082] Through the steering wheel, capacitive hand grip or wearable device, the physiological state signals of the occupants are obtained in real time to determine their inner arousal level. The details are as follows:

[0083] Sensor collection parameters include but are not limited to: heart rate (HR), heart rate variability (HRV), skin conductance (EDA), pulse wave transit time (PTT), etc.

[0084] The data acquisition frequency is generally not less than 50Hz to ensure the identification of sudden changes (such as emotional peaks and instantaneous excitement);

[0085] Use the rule model or small deep neural network inside the edge node to extract features, such as calculating SDNN, RMSSD, LF / HF ratio, etc.

[0086] Output structured physiological and emotional characteristics: such as "HRV decreases → tension increases", and convert them into emotional state indicators (tension / relaxation, excitement / calmness) and confidence.

[0087] Module M2: feature extraction module;

[0088] The feature extraction module, deployed on local edge nodes, preprocesses the raw data collected by the modal perception module, extracts emotional features, and performs preliminary emotion recognition. This module possesses local independent computing capabilities, supports lightweight model execution, and enables real-time emotional state determination without relying on higher-level nodes. Its processing includes the following aspects:

[0089] Module M2.1: Local preprocessing of modal data;

[0090] After receiving the raw data uploaded by the modal perception module, each edge node first performs local preprocessing operations to ensure that the input data meets the requirements of subsequent model calculations and removes redundant or invalid information.

[0091] Image modality preprocessing includes image denoising, grayscale conversion, face detection, key point alignment, ROI extraction and size normalization;

[0092] Speech modality preprocessing includes voice endpoint detection (VAD), background noise suppression, frame segmentation, and feature extraction (such as MFCC, Chroma, and pitch).

[0093] Physiological modality preprocessing includes bandpass filtering, trend removal, and normalization of signals such as heart rate, HRV, and EDA, and can calculate derivative indicators such as RMSSD and LF / HF;

[0094] All preprocessing processes are completed independently within the edge node without relying on central control communication, ensuring data privacy and low-latency processing capabilities.

[0095] Module M2.2: Lightweight emotion recognition model reasoning;

[0096] After completing the preprocessing, the feature extraction module calls the locally deployed lightweight emotion recognition model to infer and judge the current data. The model used supports the operation of edge devices and has the characteristics of small computational complexity, fast loading speed, and high real-time response.

[0097] The form of lightweight emotion recognition model:

[0098] Structure: three-way sub-network + feature fusion + fully connected classification;

[0099] Image subnet: MobileNetV2 → facial expression features;

[0100] Speech subnet: 1D-CNN+BiLSTM→MFCC / Chroma / pitch features;

[0101] Physiological subnetwork: FC layer → HR / HRV / EDA derived indicators;

[0102] Model input: face ROI image, speech feature matrix (such as MFCC), physiological feature vector (such as RMSSD, LF / HF);

[0103] Model output: emotion category probability vector and maximum confidence;

[0104] Reasoning and judgment process: 1. Forward calculation of each sub-network; 2. Feature splicing; 3. Classification output; 4. Take the maximum value.

[0105] Module M2.3: Generate and upload structured emotion data packets;

[0106] After completing the inference, the feature extraction module encapsulates the emotion recognition results at the current moment into a standardized structured data packet for uploading to the regional coordination node for subsequent fusion processing.

[0107]

[0108] Module M3: Regional fusion module;

[0109] The regional fusion module, deployed at the regional coordination node, is primarily responsible for receiving, synchronizing, and integrating emotion feature data uploaded by multiple edge nodes within the region, and determining the current dominant emotional state in the region. By fusing data from different modalities and nodes, this module effectively improves the accuracy and robustness of emotion recognition. Its processing includes the following aspects:

[0110] Module M3.1: Receive and synchronize multimodal emotion data;

[0111] Regional coordination nodes receive data packets from multiple edge nodes. These data have inconsistent sampling frequencies due to different modalities. Therefore, a sliding time window mechanism (e.g., 500ms) is used to align timestamps to ensure unified fusion judgment within the same time period. Time alignment rules can use a "maximum time difference threshold" filtering principle to eliminate severely out-of-sync data.

[0112] Module M3.2: Modal fusion and feature enhancement processing;

[0113] The regional fusion module uses a locally deployed multimodal fusion model to perform a fusion analysis of synchronized image, speech, and physiological and other emotional features. A lightweight fusion architecture (such as FusionNet-Lite) can be used, supporting execution within the vehicle's central control processor, with an average fusion latency of less than 50ms.

[0114] Multimodal fusion models improve the accuracy and robustness of emotion recognition by fusing information from different modalities (such as images, speech, and physiological data). The model typically consists of the following steps:

[0115] 1. Feature extraction: Extract features from each modality. For example, facial expression features are extracted from image data using a convolutional neural network (CNN), speech features are extracted from speech data using an RNN (such as BiLSTM) or CNN, and features such as heart rate and skin conductance are extracted from physiological data using a DNN.

[0116] 2. Feature fusion: The features extracted from each modality are weighted summed or concatenated to form a comprehensive feature vector.

[0117] 3. Emotion classification: The fused feature vector is input into the classification network, and the emotion label and confidence level are output.

[0118] Model input:

[0119] Image modality: facial expression feature vector extracted by CNN.

[0120] Speech modality: Speech feature vectors processed by RNN or CNN, such as MFCC and pitch.

[0121] Physiological modality: Physiological signal feature vectors such as heart rate and HRV extracted by DNN.

[0122] Model output:

[0123] Emotion labels: such as "Happy", "Sad", "Angry", etc.

[0124] Confidence: The confidence level of each sentiment label.

[0125] The specific process of fusion analysis:

[0126] Data reception and synchronization: First, we receive multimodal sentiment data from multiple edge nodes. Due to different sampling frequencies and timestamps, this data needs to be synchronized using a sliding time window mechanism.

[0127] Feature extraction: The data of each modality (such as images, speech, and physiological signals) will undergo preprocessing and feature extraction to form an independent feature vector.

[0128] Modal fusion: The features of each modality are weighted summed or concatenated to obtain a fused feature vector.

[0129] Emotion classification and output: Use the classification model to classify the fused features and output the emotion label and confidence level.

[0130] Based on the fused feature vectors, the regional coordination node outputs the dominant emotion label for that region. This regional-level emotion judgment is sent as input to the central control unit to contribute to the construction of the vehicle-wide emotional state map.

[0131] Module M4: Emotional mapping module;

[0132] The emotion graph module, deployed within the central control unit, aggregates emotion assessments from regional coordination nodes, constructs a structured emotional state graph for all vehicle occupants, and continuously maintains emotion history and trend information. This module is the core of the system's global emotion modeling, trend prediction, and strategy generation. Its processing includes the following aspects:

[0133] Module M4.1: Receive and standardize regional sentiment data;

[0134] The emotion graph module receives structured emotion judgment results from regional coordination nodes. Each record typically includes fields such as the region number, dominant emotion label, confidence score, and generation timestamp. The module performs field validation, data standardization, and chronological ordering on the received data to ensure a consistent data structure and complete format. For data records with missing fields or abnormal timestamps, fault tolerance mechanisms can be implemented or they can be marked as "abnormal input" to ensure graph data consistency.

[0135] Module M4.2: Constructing a structured emotional map in the car;

[0136] Normalized regional emotion data is mapped to corresponding locations in the graph structure based on occupant location, node number, or region number. The emotion graph is maintained in a structured format, which can be represented as a JSON object, a sparse matrix, or a graph database structure. Each node record includes the current dominant emotion label, confidence level, region, and timestamp, and supports the addition of historical trajectory information. The graph is dynamically updated to reflect the emotional state of all occupants in real time.

[0137] Module M4.3: Update node status and historical trajectory;

[0138] For existing nodes in the graph, the historical sentiment records of the node are updated in chronological order, and its sentiment trend indicators are calculated, such as the confidence change range, the frequency of sentiment label switching, and the average sentiment intensity. If a node has not been updated for a set period of time, its status can be marked as "unknown" or "inactive" to prevent old data from misleading current judgments.

[0139] Module M4.4: Provides graph interface services to the outside world;

[0140] The emotion graph module provides a standardized interface that allows other modules in the central control system (such as the emotion decision module) to access graph snapshot data in real time. It also supports querying and analyzing historical graph trajectories. The interface supports filtering by time interval, occupant ID, emotion type, and other conditions. It also supports the export of graph data sequences for offline model training and over-the-air (OTA) remote optimization analysis.

[0141] Module M5: Emotional decision-making module;

[0142] The emotion decision module deployed in the central control unit is used to comprehensively analyze the emotion map, the in-vehicle environment status, and vehicle driving information to determine whether there are emotional events that require intervention and generate corresponding multimodal cockpit control strategies. As the strategic center of the system, this module's processing process includes the following aspects:

[0143] Module M5.1: Analyze global sentiment map data;

[0144] The Emotional Decision Module periodically calls upon snapshot data provided by the Emotional Graph Module to extract the dominant emotion label and confidence level for each occupant at the current moment. This module can set emotion trigger thresholds or trend warning rules to identify occupant nodes with abnormal or fluctuating emotions, or those exhibiting negative emotions such as anxiety and anger.

[0145] Module M5.2: Fusion of context-aware information;

[0146] In addition to the emotion map, the module also simultaneously reads contextual data from both inside and outside the vehicle, including vehicle speed, acceleration, navigation path, external temperature and humidity, current road conditions, and other contextual information. By jointly modeling emotional and environmental states, contextual correlation can be determined. For example, "high-speed driving combined with driver anxiety" is identified as a high-risk state, while "an anxious child in the back seat combined with parking" triggers a soothing interaction mode.

[0147] The specific process of "joint modeling of emotional state and environmental state" is as follows:

[0148] 1. Data Collection:

[0149] Emotional data: The modal perception module collects occupants' emotional data (such as facial expressions, voice, heart rate, etc.) and performs real-time analysis to derive each occupant's emotional state (such as anxiety, anger, calmness, etc.).

[0150] Environmental data: This data also collects environmental information such as vehicle speed, acceleration, navigation path, external temperature and humidity, and current road conditions. This information can be obtained through various vehicle sensors (such as GPS, onboard sensors, and climate monitoring sensors).

[0151] 2. Data preprocessing and synchronization:

[0152] To ensure data accuracy and consistency, timestamps are used to align various data types, especially between sentiment data and environmental data. A sliding time window mechanism (e.g., 500ms) is used to synchronize these data.

[0153] 3. Joint Modeling:

[0154] Taking emotional state and environmental state as model input, a fusion model (such as multi-layer perceptron, support vector machine, LSTM, etc.) is usually used to learn the relationship between emotion and environment.

[0155] Joint modeling can be expressed by the following formula:

[0156] E=f(S,C)

[0157] Among them, E represents the emotional state, S represents the environmental state (such as vehicle speed, acceleration, etc.), C represents the emotional characteristics (such as anger, anxiety), and the function f() represents the reasoning process of joint modeling.

[0158] 4. Decision making:

[0159] The model generates decisions based on a combination of emotional and environmental information. For example, the system can determine that a driver is anxious and, if the vehicle is traveling at high speed and experiencing drastic acceleration changes, trigger safety interventions.

[0160] Model expression:

[0161] A joint neural network model can be used, in the form of:

[0162] E=NN(S,C)

[0163] Among them, S is the environment state vector, C is the emotional state vector, and the neural network NN() represents the joint learning and reasoning of these two inputs, and outputs the predicted emotional state E.

[0164] The specific process of “situational relevance judgment”:

[0165] 1. Context Identification:

[0166] Emotional state recognition: Identify the occupant's current emotional state, such as anxiety, anger, fatigue, etc., through the emotion perception module.

[0167] Environmental status analysis: Analyze environmental information such as vehicle speed, acceleration, and navigation path to determine whether there are situations that may affect emotions.

[0168] 2. Contextual Matching:

[0169] The system uses preset rules or trained machine learning models to match emotional states with environmental conditions. For example, it can combine "high-speed driving" with "anxiety" to determine whether a high-risk driving state exists.

[0170] 3. Generate response strategy:

[0171] If the situation meets the predetermined intervention conditions, the system will trigger the corresponding emotion regulation strategy or safety measures.

[0172] For example:

[0173] Example 1: Scenario: "Driver Anxiety" + "High-Speed Driving"

[0174] Judgment: High-speed driving will increase anxiety, so the system can trigger a safe driving mode or soothing in-car environment adjustment (such as adjusting music, seat temperature, etc.).

[0175] Example 2: Scenario: "Children in the back seat are anxious" + "Parking status"

[0176] Judgment: A child's anxious state may affect the driver's attention and safety, so the system can trigger a soothing interaction mode (such as playing soothing music, adjusting the temperature in the car, etc.).

[0177] Module M5.3: Implement emotional decision-making logic;

[0178] After acquiring the emotional map status and in-vehicle and out-of-car context information, the emotional decision module executes the strategy judgment logic to determine whether emotional response intervention is necessary and generates the corresponding response strategy content. This judgment process can be implemented based on the following two methods:

[0179] 1. Decisions are made through a pre-set rule engine. A mapping rule is established between "emotional state - environmental conditions - response strategy." When a combination of scenarios that meet specific trigger conditions is detected (e.g., "driver is anxious" and "current vehicle speed is greater than 60 km / h"), the corresponding strategy response logic is triggered.

[0180] 2. The strategy generation method based on the training model can use a trained machine learning model (such as decision tree, logistic regression, reinforcement learning network, etc.) to input the current emotional state and context parameters, and automatically output the recommended response strategy combination to improve the intelligence and adaptability of strategy judgment.

[0181] Taking the decision tree as an example, its implementation process is as follows:

[0182] Step S1, feature vector construction;

[0183] 1.1. Synchronously obtain the occupant's emotional state label from the emotional map module;

[0184] 1.2. Synchronously obtain environmental parameters such as vehicle speed, acceleration, external temperature and humidity, and road condition code from the environmental perception module;

[0185] 1.3. Assemble the above emotional states and environmental parameters into feature vectors in a fixed order:

[0186] X=[x1=emotion_state,x2=speed,x3=acceleration,x4=out_temperature,x5

[0187] =road_condition,…]

[0188] Step S2, decision tree model reasoning;

[0189] 2.1. Input the feature vector X into the pre-trained decision tree classifier T;

[0190] 2.2. The decision tree T iterates from the root node to a leaf node according to the rule "if xj≤t, enter the left subtree, otherwise enter the right subtree";

[0191] 2.3. Each leaf node is directly mapped to the response strategy label set Y, and the leaf node outputs the optimal strategy y = T(X);

[0192] Step S3, strategic intent generation;

[0193] 3.1. Determine the target occupant, emotion type, and environment summary based on y;

[0194] 3.2. Encapsulate the response policy content (such as "play light music", "safety reminder" or "adjust temperature") into a policy intent data packet according to the preset priority;

[0195] 3.3. Send the policy intent data packet to the central control unit for execution.

[0196] Step S4, decision tree model training and updating;

[0197] 4.1. Collect historical samples containing emotional states and corresponding environmental parameters, and annotate the optimal response strategy;

[0198] 4.2. Divide the samples into training set and validation set in proportion, and perform feature encoding and normalization;

[0199] 4.3. Based on the CART algorithm, a decision tree is recursively constructed using Gini impurity or information gain as the splitting criterion.

[0200] 4.4. Prune the tree structure or tune hyperparameters based on the performance of the validation set to derive the final decision rule;

[0201] 4.5. Regularly add new samples, retrain offline, and update model files online to improve decision accuracy.

[0202] The judgment results are organized in the form of strategic intent, including fields such as target occupant, emotional state, processing priority, and recommended response method.

[0203] Module M5.4: Generate multimodal response strategy solutions;

[0204] The decision-making module generates structured strategy instructions based on the judgment results, which may include response plans in multiple dimensions such as air conditioning adjustment, lighting control, voice broadcast, music switching, and seat vibration.

[0205] Module M5.5: Submit the strategy to the execution module;

[0206] The final strategy is delivered to the strategy response module in the form of a structured instruction package, along with response priorities and timeout handling rules. The system can set strategy numbers and execution status callback mechanisms to achieve closed-loop response tracking. If the current emotional state changes, the module can trigger strategy interruption, replacement, or dynamic adjustment to ensure that the response plan aligns with the evolving emotional state.

[0207] Using a multimodal feature fusion lightweight network, the emotion vector output formula is:

[0208]

[0209] Among them, f, s, p are facial, voice, and physiological, x f 、x s 、x p Respectively represent facial image input, voice signal input, and physiological signal input; W m represents the weight of mode m; represents the feature extraction function; x m is the input data; Softmax is the normalized exponential function; MobileNetV3, LSTM, and FCN are feature extraction functions for facial images, speech signals, and physiological signals.

[0210] Module M6: Strategy response module;

[0211] The strategic response module is deployed in the central control unit and connected to various controllable devices in the cockpit via a control bus. It receives strategic instructions generated by the emotional decision module and implements multimodal device linkage control. This module is the key execution unit for implementing occupant emotional state adjustment feedback. Its processing includes the following aspects:

[0212] Module M6.1: Receive and parse policy instructions;

[0213] The policy response module receives structured policy data packets from the emotion decision module. These packets typically include fields such as the response target (e.g., "driver"), policy type (e.g., "soothing"), control sub-items (e.g., "lower temperature," "play soothing music"), adjustment parameters (e.g., wind speed, color temperature, volume), and response duration. The module first parses the policy content and determines its feasibility based on the current device state to avoid invalid commands or device conflicts.

[0214] Module M6.2: Control instruction conversion and device mapping;

[0215] The parsed policy instructions are mapped to specific device control commands. The policy response module has a built-in control instruction conversion engine that converts abstract policy intent into physical execution commands based on the preset device control protocol (such as CAN, LIN, Zonal domain bus, etc.).

[0216] Module M6.3: Concurrency Scheduling and Conflict Management;

[0217] When a policy includes multiple subcommands or involves multiple devices, the policy response module performs concurrent scheduling and priority management. The module determines the order and timing of each subcommand based on the policy's urgency, device occupancy, and task conflict levels. If a device is in manual control mode or a higher-priority task is available, the system can skip, delay, or replace the policy response.

[0218] Module M6.4: Strategy execution and status feedback;

[0219] After receiving a control command, each device executes the corresponding action and returns the execution status (success / failure / partial execution) to the policy response module via a communication interface. The module summarizes all execution results and generates a feedback log, which the emotional decision module uses for closed-loop policy optimization. Furthermore, if a critical device failure or an abnormal policy response is detected, the system can trigger a downgraded response mechanism or issue a user prompt.

[0220] Module M6.5: Supports multi-round emotional response linkage;

[0221] The Strategy Response Module supports chained scheduling of multiple response strategies, such as maintaining strategy consistency or dynamically adjusting response intensity across consecutive emotion recognition cycles. Based on emotional trends, the module determines whether a strategy should be continued, strengthened, or canceled, forming a continuous interactive process that dynamically adapts to changes in occupant emotions.

[0222] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0223] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A distributed edge computing architecture for an emotional cockpit, characterized by: include: The three-layer structure of local edge node layer, regional coordination node layer and central control unit layer is connected by low-latency communication interface, collaboratively realizing the collection, fusion, judgment and strategic response of multimodal emotion data; in: The local edge node layer includes multiple smart terminals deployed near the passengers in the vehicle, each terminal including: Modal perception module M1, used to collect multimodal emotional data of passengers in real time, including images, voice and physiological signals; Feature extraction module M2 is used to perform local preprocessing, feature extraction and preliminary emotion recognition on the raw data, and generate a structured data packet containing emotion labels, confidence levels and timestamps; The regional coordination node layer includes multiple regional processing units, each of which includes: The regional fusion module M3 is used to receive data uploaded by local edge nodes in the region, synchronize time through a sliding time window mechanism, and use a multimodal fusion model to perform feature fusion and output the regional dominant emotion label and confidence level. The central control unit layer includes: Emotional map module M4, used to receive emotional data from each area and construct a structured emotional map of all passengers in the vehicle; Emotional decision module M5, used to combine the emotional map, the internal and external environmental conditions and vehicle driving information to generate a multimodal response strategy; The strategy response module M6 is used to map the strategy instructions into specific device control commands to drive the coordinated adjustment of the cockpit environment.

2. The distributed edge computing architecture of the emotional cockpit according to claim 1 is characterized in that: The modality perception module M1 includes: Image acquisition submodule M1.1 collects occupant facial expression data through in-car cameras, supports infrared / visible light dual-channel imaging, and performs local expression classification based on a lightweight convolutional neural network model; The speech acquisition submodule M1.2 collects speech data through a distributed microphone array, extracts Mel-frequency cepstral coefficients (MFCCs) and pitch features, and performs speech emotion recognition using a CNN-BiLSTM model. The physiological signal acquisition submodule M1.3 collects heart rate, heart rate variability (HRV) and skin conductance (EDA) signals through steering wheel sensors or wearable devices, and generates physiological emotion indicators based on rule models or small neural networks.

3. The distributed edge computing architecture of the emotional cockpit according to claim 1 is characterized in that: The regional fusion module M3 uses a sliding time window mechanism to align the timestamps of multimodal data and performs feature fusion through a lightweight version of the FusionNet-Lite model; The multimodal fusion model includes the following steps: extracting feature vectors from images, voices and physiological signals respectively; performing weighted summation or concatenation on the feature vectors to form a comprehensive feature vector; inputting the feature vectors into a classification network to output an emotion label and a confidence level; Taking the emotional state and the environmental state as model input, the fusion model is used to learn the relationship between emotion and environment, which can be expressed as: E=f(S,C) Among them, E represents the emotional state, S represents the environmental state, C represents the emotional characteristics, and the function f() represents the reasoning process of joint modeling; The model generates decisions based on the comprehensive information of emotional state and environmental state, using a joint neural network model in the form of: E=NN(S,C) Among them, the neural network NN() represents the joint learning and reasoning of these two inputs.

4. The distributed edge computing architecture of the emotional cockpit according to claim 1 is characterized in that: The emotion map module M4 dynamically maintains the entire vehicle’s emotion state map through JSON objects or a graph database, including occupant positions, dominant emotion labels, confidence levels, and historical trajectory information; The graph module provides a standardized interface, supports data query by time interval, passenger ID or emotion type, and supports historical data export for offline model training.

5. The distributed edge computing architecture of the emotional cockpit according to claim 1 is characterized in that: The emotional decision module M5 generates a response strategy in the following way: Rule engine: preset emotional state-environmental conditions-response strategy mapping rules, triggering safety intervention strategies when driver anxiety is detected and vehicle speed exceeds thresholds; Machine learning model: Based on decision trees or reinforcement learning networks, it inputs emotion feature vectors and environmental parameters and outputs recommendation strategies; The response strategy includes at least one of air conditioning adjustment, lighting control, music playback, and voice interaction.

6. The distributed edge computing architecture of the emotional cockpit according to claim 5 is characterized in that: The training process of the decision tree model includes: Collect historical emotional data and environmental parameters and mark the optimal response strategy; Construct a decision tree based on the CART algorithm, using Gini impurity or information gain rate as the splitting criterion; Improve model generalization capabilities through pruning and hyperparameter tuning; Regular incremental training is performed to update the model.

7. The distributed edge computing architecture of the emotional cockpit according to claim 1 is characterized in that: The policy response module M6 converts abstract policy instructions into device control commands via the control bus CAN / LIN, supporting concurrent scheduling and conflict management; The execution order is dynamically adjusted according to the device status and task priority, and the execution results are fed back to the emotional decision module M5 to optimize the strategy closed loop.

8. The distributed edge computing architecture of the emotional cockpit according to claim 1 is characterized in that: Adopting a privacy protection mechanism, raw images, voice, and physiological data are only processed at local edge nodes, and only structured emotion feature vectors are uploaded to the upper node; Regional coordination nodes and central control units do not store or transmit original sensitive data.

9. The distributed edge computing architecture of the emotional cockpit according to claim 1 is characterized in that: Supports multiple rounds of emotional response linkage, dynamically adjusts strategy intensity or switches strategy type according to emotional trends, and forms a continuous interactive process; The response strategy includes a combination of at least two of seat vibration, temperature gradient adjustment, and adaptive music recommendation.

10. A smart cockpit, characterized in that: It is equipped with an on-board emotion recognition system, which adopts the distributed edge computing architecture of the emotion cockpit according to any one of claims 1 to 9.

Images