A dynamic expression display method and system based on vehicle state information

Abstract

The application discloses a dynamic expression display method and system based on vehicle state information, and relates to the technical field of vehicle-mounted man-machine interaction and intelligent cabin. The method comprises the following steps: collecting multi-modal data and preprocessing, and constructing a state feature vector; inputting the vector into a time series prediction model to output a future emotion prediction sequence; based on the sequence, calling a base element from a preset expression base element library, pre-generating an expression animation key frame sequence through an element stack and an animation mixing tree, and storing the sequence into a ring buffer pool; obtaining real-time emotional needs, comparing the real-time emotional needs with the emotion prediction sequence to generate a unified emotional description; based on the description, calling or generating an animation in real time from the ring buffer pool, and combining scene-related dimensional data in the preprocessed multi-modal data to render and display on a vehicle-mounted display interface. The application realizes forward-looking prediction, real-time generation, intelligent fusion and scene adaptation of vehicle-mounted expression animation, and improves the real-time performance, accuracy and emotional richness of vehicle-mounted man-machine interaction under the premise of ensuring driving safety.

Description

Technical Field

[0001] This invention relates to the field of in-vehicle human-machine interaction and smart cockpit technology, and in particular to a dynamic facial expression display method and system based on vehicle status information. Background Technology

[0002] With the continuous development of intelligent cockpit and human-computer interaction technologies, the functional attributes of vehicles have gradually upgraded from a single means of transportation to a mobile intelligent space integrating travel, entertainment, and emotional communication. In-vehicle facial expression displays, as a core carrier for realizing emotional connection between people and vehicles, have become crucial for enhancing the driving experience through their level of intelligence and contextualization, leading to the rapid development of related interactive technologies.

[0003] Currently, existing in-vehicle facial expression display and human-computer interaction technologies still have the following shortcomings. First, existing solutions mostly rely on user input or vehicle transient events to trigger preset facial expressions and voice feedback, which is essentially an event-response passive interaction. This lack of predictability regarding future changes in the driver's emotional state results in facial animation lagging behind the driver's actual emotional changes, making it difficult to achieve proactive and natural emotional interaction. Second, although existing solutions propose static priority coverage rules based on scene type, their mechanisms for handling conflicts between multi-source interaction needs and emotional expression are still imperfect. In non-safety event scenarios, simple high-priority coverage can abruptly interrupt the original interaction experience, disrupting interaction continuity. In safety event scenarios, there may be issues with untimely response or insufficient warning expression, making it difficult to simultaneously ensure driving safety and maintain interaction continuity. Third, existing facial animation solutions are mostly based on fixed templates or script-driven approaches, and real-time calculation and generation lead to significant response delays. Furthermore, because they do not fully consider the dynamic changes in scene factors such as vehicle driving status, ambient light, and autonomous driving level, inconsistencies between animation display and driving scene can easily occur, affecting the driving experience and potentially distracting the driver. Summary of the Invention

[0004] To address the problems of passive and delayed interaction, inadequate conflict handling, delayed animation generation, and insufficient scene adaptability in existing solutions, this invention proposes a dynamic facial expression display method and system based on vehicle status information. This method utilizes an integrated technical architecture encompassing multimodal data fusion, vehicle-cloud collaborative temporal emotion prediction, hierarchical dynamic fusion of facial expression primitives, priority adjudication of emotional needs, and full-scene adaptive rendering. This achieves forward-looking prediction, real-time generation, intelligent fusion, and scene-specific adaptation of in-vehicle facial expression animations, improving the real-time performance, accuracy, and emotional richness of in-vehicle human-computer interaction while ensuring driving safety.

[0005] The present invention achieves the above objectives through the following technical solutions: A method for displaying dynamic facial expressions based on vehicle status information includes: Multimodal data is collected and preprocessed to construct state feature vectors; the multimodal data includes at least vehicle dynamic data, driver behavior data, driver physiological characteristic data, and environmental data. The state feature vector is input into a pre-trained time series prediction model, which outputs the probability distribution of each time node corresponding to each sentiment category within a preset time period in the future, forming a sentiment prediction sequence. Based on the emotion prediction sequence, the corresponding primitive is retrieved from the preset expression primitive library, and the corresponding expression animation key frame sequence is pre-generated through the superposition and combination logic of primitive stack and animation hybrid tree, and the expression animation key frame sequence is stored in the circular cache pool. The system acquires real-time sentiment needs from interactive events, compares and adjudicates these real-time sentiment needs with a sentiment prediction sequence, and generates a unified sentiment description based on the adjudication results. Based on a unified emotion description, the system retrieves the corresponding keyframe sequence of facial expression animation from the circular cache pool by timestamp matching or generates animation parameters in real time. Combined with vehicle dynamic data, driver behavior data, and environmental data from the preprocessed multimodal data, the system performs scene-adaptive rendering and display of facial expression animation on the in-vehicle display interface.

[0006] As a preferred embodiment of the present invention, the driver's physiological characteristic data includes at least heart rate, skin conductance response, and voice emotion characteristics; the vehicle dynamic data includes at least vehicle speed, acceleration, steering angle, and brake pedal depth; the driver's behavioral data includes at least facial expression, head posture, gaze direction, and fatigue state; and the environmental data includes at least weather conditions, light intensity, and road type. The preprocessing includes outlier detection, missing value imputation, and format standardization.

[0007] As a preferred embodiment of the present invention, the time series prediction model adopts an architecture that combines a lightweight prediction model on the vehicle side with a large model in the cloud. The lightweight prediction model is deployed on the vehicle domain controller and performs real-time sentiment inference based on state feature vectors, outputting a sentiment prediction sequence; the sentiment prediction sequence includes at least sentiment category and probability value. The cloud-based large model is deployed on the vehicle manufacturer's cloud server. It is periodically trained offline based on non-real-time user feedback data uploaded by multiple vehicles and historical multimodal data to generate optimized model parameters. These parameters are then sent to the vehicle via vehicle-cloud communication when the vehicle is parked or charging. This allows for dynamic iteration and updates of the lightweight prediction model on the vehicle side, while the vehicle side retains historical model versions as rollback backups. The non-real-time user feedback data includes at least one of the following: user satisfaction rating of the facial animation, interaction delay data, and user facial expression reaction data.

[0008] As a preferred embodiment of the present invention, the step of retrieving corresponding primitives from a preset expression primitive library based on the emotion prediction sequence, and pre-generating the corresponding expression animation keyframe sequence through the superposition and combination logic of the primitive stack and the animation hybrid tree, includes: The emotion categories and probability values ​​in the emotion prediction sequence are mapped to basic emotion primitives, action primitives and modification primitives in the preset expression primitive library, and the initial fusion weight of each primitive is determined based on the proportion of each probability value in the total probability value. Input the selected basic emotion primitives, action primitives, and modification primitives into the primitive stack, and the primitive stack will perform hierarchical management and maintenance operations on each primitive. The primitives in the primitive stack are input into the animation blending tree. Based on the initial blending weights of each primitive and the preset blending rules, the blended animation parameters are generated. The animation blending tree is a tree structure, with leaf nodes being primitives and intermediate nodes being blending operations of weighted superposition or linear blending. Based on the mixed animation parameters, a sequence of key frames for facial animation is generated frame by frame according to a preset frame rate; the sequence of key frames for facial animation includes timestamps, position and rotation information of each control point on the face, transparency and color parameters of the modification primitives, and transition weights with the preceding and following key frame sequences. The generated keyframe sequence of facial animation, along with timestamps and transition weights, is stored in a circular cache pool; the circular cache pool uses timestamps as indexes and employs a first-in-first-out (FIFO) strategy to evict data.

[0009] As a preferred embodiment of the present invention, the step of obtaining real-time emotional needs from interactive events and comparing and deciding on real-time emotional needs with emotional prediction sequences includes: The system acquires the emotional category and interaction event type corresponding to the real-time emotional needs, compares them with the emotional category at the current time node in the emotional prediction sequence, and determines the emotional conflict type based on the comparison results. The emotional conflict type includes at least strong conflict, weak conflict, and no conflict. The interaction event type includes at least safety warning events, driving task events, and infotainment events, with safety warning events having a higher priority than driving task events, and driving task events having a higher priority than infotainment events. Differentiated arbitration procedures are implemented based on the type of emotional conflict.

[0010] As a preferred embodiment of the present invention, determining the type of emotional conflict based on the comparison results includes: If the interaction event type for real-time emotional needs is a security alert event, it is determined to be a strong conflict; If the interaction event type of the real-time emotional need is a driving task event or an infotainment event, and the emotional category of the need does not belong to the same basic emotional category, nor to a similar derived category or a neutral category, as the emotional category of the current time node in the emotional prediction sequence, then it is determined to be a weak conflict. If the emotion category of the real-time emotion demand and the emotion category of the current time node in the emotion prediction sequence are the same basic emotion category, a similar derived category of the same basic emotion, or both are neutral categories without obvious emotion tendency, then it is determined that there is no conflict. The basic emotion categories include at least pleasure, calmness, irritability, drowsiness, tension, and surprise; the neutral category refers to a basic state without a clear emotional tendency; and the similar derived categories refer to similar emotions extended from the same basic emotion, with consistent core emotional characteristics.

[0011] As a preferred embodiment of the present invention, the step of performing differentiated arbitration operations based on the type of emotional conflict includes: When the emotional conflict type is strong conflict, based on the principle of safety as the highest priority, the currently playing or pre-generated facial animation keyframe sequence is forcibly interrupted, and the facial animation that matches the real-time emotional needs corresponding to the safety warning event is immediately played. The interrupted facial animation status data is stored in the interruption task temporary storage queue. After the triggering conditions of the safety warning event are removed and the vehicle detects that the driving status has returned to normal, the interrupted facial animation is resumed from the temporary storage queue and played again according to the preset smooth transition rules. When the emotional conflict type is weak, if the interaction event type is a driving task event, the expression primitives in the preset expression primitive library corresponding to the driving task event will be used as the main layer, and the expression primitives in the preset expression primitive library corresponding to the emotional prediction sequence will be used as the auxiliary layer. The weighted superposition display will be performed according to the preset animation layer superposition rules. If the interaction event type is an infotainment event, while retaining the expression animation corresponding to the emotional prediction sequence as the main display content, the display weight of the expression primitives in the preset expression primitive library corresponding to the infotainment event will be reduced to below the preset weight threshold. When the emotional conflict type is no conflict, the emotional state at the current time point in the emotional prediction sequence is used directly.

[0012] As a preferred embodiment of the present invention, the unified sentiment description includes sentiment category, sentiment intensity, confidence level, and mixing ratio; The emotion category is determined by the arbitration result, which specifies either a single emotion category or a composite emotion category; the arbitration result is the execution result of the differentiated arbitration operation. The emotional intensity includes the intensity of a single emotion and the intensity of a compound emotion; the intensity of a single emotion is taken as the corresponding probability value of the emotional prediction sequence or the calibration value of the real-time emotional demand; the intensity of the compound emotion is taken as the average value of the weighted calculation of the intensities of each single emotion; wherein, the weighting coefficient is determined according to the priority of the interactive event. The confidence level is a weighted fusion value of the real-time emotional demand confidence level of the corresponding interactive event type and the emotional prediction confidence level of the emotional prediction sequence; the emotional prediction confidence level is taken as the maximum emotional probability at the corresponding time node; The mixing ratio is determined based on the arbitration results and the priority of interactive events.

[0013] As a preferred embodiment of the present invention, the vehicle display interface includes at least a vehicle central control screen, a digital instrument panel, a head-up display (HUD), and a rear entertainment screen, and each vehicle display interface adopts a synchronous playback mechanism for facial animation. The rendering and display of facial animations on the in-vehicle display interface in a scene-adaptive manner includes at least one of the following methods: Based on environmental data and vehicle dynamic data, the brightness, contrast, and playback speed of the facial expression animation are dynamically adjusted. Based on driver behavior data and vehicle dynamic data, the current driving state is identified. When a dangerous driving state is detected, the facial animation, text, and audio-visual warning information are rendered and displayed in an integrated manner with location linkage and synchronized motion effects. The warning facial animation has the highest visual hierarchy on all in-vehicle display interfaces. The dangerous driving states include at least fatigue, distraction, making or receiving phone calls, and speeding. The display duration of the warning facial animation is bound to the state of danger being lifted. The warning facial animation plays in a loop while the dangerous driving state is in effect, and smoothly transitions to normal facial animation after the state is lifted. Based on the autonomous driving level classification in vehicle dynamic data, the facial expression animation display mode is as follows: For L2 and below assisted driving modes, the facial expression animation mainly focuses on safety reminders and driving status feedback. The head-up display (HUD) only displays monochrome safety facial expression animations without gradients, and the digital instrument panel simplifies the facial expression animation effects. The monochrome safety facial expression animations without gradients are generated by alternating between on and off states at a preset frequency. For L3 and above autonomous driving modes, the facial expression animation mainly focuses on entertainment interaction and scene services. The in-vehicle central control screen and rear entertainment screen display full-motion color interactive facial expression animations, and the head-up display (HUD) selectively turns off the facial expression animation display.

[0014] A dynamic facial expression display system based on vehicle status information includes: The data acquisition and preprocessing module is used to acquire multimodal data, perform preprocessing, and construct state feature vectors; the multimodal data includes at least vehicle dynamic data, driver behavior data, driver physiological characteristic data, and environmental data; The sentiment prediction module is used to input the state feature vector into a pre-trained time series prediction model and output the probability distribution of each sentiment category at each time point within a preset time period in the future, forming a sentiment prediction sequence. The expression pre-generation module is used to retrieve the corresponding primitives from the preset expression primitive library based on the emotion prediction sequence, pre-generate the corresponding expression animation key frame sequence through the superposition and combination logic of primitive stack and animation hybrid tree, and store the expression animation key frame sequence into a circular cache pool. The unified sentiment description generation module is used to obtain real-time sentiment needs from interactive events, compare and adjudicate the real-time sentiment needs with the sentiment prediction sequence, and generate a unified sentiment description based on the adjudication result. The facial expression animation generation and rendering module is used to match and call the corresponding facial expression animation keyframe sequence from the circular cache pool according to the timestamp based on the unified emotion description, or to generate animation parameters in real time. Combined with vehicle dynamic data, driver behavior data and environmental data in the preprocessed multimodal data, the module performs scene-adaptive rendering and display of facial expression animation on the in-vehicle display interface.

[0015] The beneficial effects of this invention are as follows: By collecting four core multimodal data types—vehicle dynamics, driver behavior, driver physiological characteristics, and environment—and constructing a unified state feature vector through preprocessing, a comprehensive representation of the influencing factors of people, vehicles, and the environment in driving scenarios is achieved. This effectively solves the analysis error problems caused by the single data dimension, sensor anomalies, or inconsistent formats in traditional solutions, providing high-quality input for subsequent emotion prediction. Furthermore, a temporal prediction model is introduced to proactively anticipate driver emotions. Compared to the passive mode of existing technologies that only trigger fixed expressions based on immediate commands or transient events, this allows the in-vehicle system to predict the trend of driver emotional changes in advance, making the generation and display of facial expression animations predictive and forward-looking, significantly improving the real-time performance and natural fit of human-computer interaction. Furthermore, breaking away from the traditional design approach of fixed emoji templates, this system pre-generates keyframe sequences of emoji animations and stores them in a circular cache pool through the superposition and combination logic of primitive stacks and animation hybrid trees. This mechanism supports personalized dynamic synthesis of both single and complex emotions, ensuring both the richness and accuracy of emotional expression. It also decouples animation generation from playback, enabling rapid retrieval based on timestamps. This effectively overcomes the latency issues of real-time calculation and generation under limited vehicle-side computing power, ensuring smooth in-vehicle interaction. Simultaneously, a comparison and adjudication mechanism between real-time emotional needs and predicted sequences is established to generate a unified emotional description. This solves the problems of chaotic feedback and contradictory expressions in traditional in-vehicle emojis when triggered by multiple sources of needs. It can distinguish between different conflict scenarios, ensuring rapid response to safety incidents while maintaining interactive continuity in non-safety incidents, achieving dual protection for driving safety and interactive experience. Finally, based on a unified emotional description, and combined with vehicle dynamic data, driver behavior data, and environmental data from the preprocessed multimodal data, the scene-based adaptive rendering and display of facial expression animations are achieved on the in-vehicle display interface. This allows the display effect of the facial expression animations to be deeply adapted to the actual driving scenario, enriching the human-vehicle emotional interaction experience while avoiding interference with driving operations. It also supports multi-screen synchronous display, ensuring the consistency and coordination of the in-vehicle display, making the in-vehicle facial expression animations an intelligent interactive carrier that combines emotional interaction value with driving safety assistance functions. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 A flowchart of a dynamic facial expression display method based on vehicle status information provided by the present invention; Figure 2This is a diagram of the vehicle-cloud collaborative time-series prediction model architecture in an embodiment of the present invention; Figure 3 This is a flowchart of the expression animation keyframe sequence generation process in an embodiment of the present invention; Figure 4 This is a flowchart of the emotional conflict adjudication process in an embodiment of the present invention; Figure 5 This is a schematic diagram of a modular structure for a dynamic facial expression display system based on vehicle status information provided by the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0018] like Figure 1 As shown, this is an embodiment of the present invention, which provides a dynamic facial expression display method based on vehicle status information, including: S1: Collect multimodal data, perform preprocessing, and construct state feature vectors.

[0019] With user authorization, multimodal data includes at least vehicle dynamic data, driver behavior data, driver physiological characteristic data, and environmental data.

[0020] Vehicle dynamic data includes at least vehicle speed, acceleration, steering angle, and brake pedal depth.

[0021] Specifically, vehicle dynamic data is collected by the on-board vehicle controller, vehicle speed sensor, acceleration sensor, steering angle sensor and brake pedal displacement sensor. After being transmitted to the on-board domain controller via CAN bus, the on-board domain controller directly extracts the data into dimensionless quantized values. The quantization range is consistent with the sensor's factory calibrated range.

[0022] Driver behavior data includes at least facial expressions, head posture, gaze direction, and fatigue level.

[0023] Specifically, driver behavior data is collected by the vehicle-mounted DMS driver monitoring camera. After the image is analyzed frame by frame by a lightweight visual recognition algorithm on the vehicle side (such as a combination of MTCNN and MobileNet), quantified behavioral feature parameters are output. These behavioral feature parameters include facial emotion labels and their matching degree, head posture Euler angles, gaze direction 3D vector, and fatigue state confidence. The specific algorithm logic is as follows: MTCNN (Multi-Task Convolutional Neural Network) quickly detects and locates 18 core facial key points, such as the brow ridge, corner of the eye, and corner of the mouth. Then, the MobileNet lightweight convolutional neural network classifies and infers the positional changes and morphological features of the key points, quantifying them into emotion labels such as pleasure, calmness, irritability, and drowsiness, and simultaneously outputs the matching degree value of each emotion label, with a value range of 0 to 1. The head posture Euler angles are calculated from the 3D coordinates of the key points, the gaze direction 3D vector is generated based on the eyeball key point tracking model, and the fatigue state confidence is calculated by combining continuous frame facial feature changes (such as blinking frequency and eyelid closure) with the PERCLOS algorithm.

[0024] Driver physiological characteristics data should include at least heart rate, skin conductance response, and vocal emotion characteristics.

[0025] Specifically, the driver's physiological characteristic data is collected through the smart steering wheel, the in-vehicle voice acquisition microphone, and the voice emotion recognition module. After the physiological signal is converted into an electrical signal by the sensor, it is successively subjected to second-order low-pass filtering (filter cutoff frequency 10Hz) and amplification processing (amplification factor 1000 times) to finally output a quantized value. The voice signal is collected by the microphone and first undergoes adaptive noise reduction processing. Then, the voice emotion algorithm (such as the combination of MFCC and SVM algorithm) is used to extract core features such as Mel-frequency cepstral coefficients and fundamental frequency. The features are classified by support vector machine to output voice emotion labels and their matching degree values ​​(the value range is 0 to 1).

[0026] Environmental data should include at least weather conditions, light intensity, and road type.

[0027] Specifically, environmental data is collected collaboratively by onboard environmental perception sensors, light sensors, vehicle-to-everything (V2X) modules, and high-precision map modules; external environmental data is collected in real-time by onboard sensors; weather conditions are obtained through real-time meteorological data of the location via the V2X module; road type is derived by combining road attribute data from the high-precision map with real-time vehicle positioning information, and is analyzed using preset multi-level classification rules. These multi-level classification rules are constructed based on road width, speed limit information, and the number of lanes. In one specific embodiment, the construction steps of the multi-level classification rules are as follows: extract the speed limit information of the currently located road segment, and determine whether the speed limit information is greater than or equal to a first preset speed threshold; if the speed limit information is greater than or equal to the first preset speed threshold, then extract the number of lanes and determine the number of lanes. If the number of lanes is greater than or equal to the first preset lane threshold, the current road type is determined to be a highway. If the speed limit information is less than the second preset speed threshold, the road width is extracted, and it is determined whether the road width is less than the preset width threshold. If the road width is less than the preset width threshold, the current road type is determined to be an urban branch road; otherwise, the current road type is determined to be a regular road. The first preset speed threshold is set to 80 km / h, the first preset lane threshold is set to a two-way four-lane road, the second preset speed threshold is set to 40 km / h, and the preset width threshold is set to 10 meters. The first preset speed threshold, the second preset speed threshold, the first preset lane threshold, and the preset width threshold are based on fixed parameter values ​​pre-calibrated according to industry standards.

[0028] Preprocessing includes outlier detection, missing value imputation, and format normalization.

[0029] Specifically, outlier detection employs a joint detection mechanism combining a threshold-based hard-judgment method with the 3σ principle. Data marked as outliers will not participate in subsequent imputation calculations, preventing invalid data generated by sensor malfunctions or signal interference from interfering with the input of the time-series prediction model. The thresholds for each data dimension are pre-calibrated based on the factory calibration parameters of the vehicle sensors and the distribution range of baseline values ​​collected under typical driving scenarios.

[0030] Missing value imputation selects an appropriate method based on the data type and cause of the missing value to ensure data integrity. For continuous numerical data such as vehicle speed, acceleration, heart rate, skin conductance, light intensity, and line-of-sight angle, a sliding window averaging method based on the vehicle data acquisition frequency calibration is used for imputation. This involves taking the average of the most recent preset number of valid acquisition frames as the imputation value. In one specific embodiment, for a typical acquisition frequency of 30fps, the preset number is set to 20, corresponding to a historical data window of approximately 0.67 seconds, which effectively smooths out instantaneous noise without excessively delaying the response. If consecutive missing frames exceed a preset missing threshold, it is determined to be a sensor fault, and a fault warning is output to the vehicle diagnostic system. Simultaneously, historical normal values ​​from the same driving scenario in that dimension are used for imputation. In one specific embodiment, considering that continuous data interruptions exceeding 1 second may indicate serious problems, the preset missing threshold for data with an acquisition frequency of 30fps is set to 50 frames, meaning that a sensor fault determination is triggered when consecutive missing frames exceed approximately 1.67 seconds. The aforementioned preset quantity and preset missing threshold need to be comprehensively determined based on the frequency of vehicle data collection and the sensitivity of data dimension changes, balancing the requirements of data continuity and real-time performance. For discrete categorical data such as facial expressions, gaze direction category labels, weather conditions, and road types, the nearest neighbor method is used for imputation, taking the nearest valid label of the data as the imputation value; if there is no voice interaction input, the voice emotion feature-related data is imputed with a neutral label and the matching degree is marked as 0; if it cannot be directly recognized due to insufficient lighting, driver obstruction, etc., the imputation is indirectly derived by combining the multi-dimensional features of the driver's facial expressions, head posture, and gaze direction through a Naive Bayes inference algorithm; if it still cannot be derived, the default label is used and the confidence level is marked as 0.

[0031] Format standardization transforms all data into numerical data in a unified format through normalization and encoding. For continuous numerical data such as vehicle speed, acceleration, heart rate, skin conductance, and light intensity, min-max normalization is used to map the data to the range of 0 to 1. For discrete labeled data such as facial expressions, weather conditions, and road types, a one-hot encoding algorithm is used to convert them into multi-dimensional numerical vectors, with each dimension taking the value of 0 or 1, without the need for additional normalization.

[0032] After format standardization, the standardized features within each of the four dimensions—vehicle dynamics data, driver behavior data, driver physiological characteristics data, and environmental data—are reduced or filtered based on preset feature selection rules to obtain core feature values ​​for each dimension. Specifically, the preset feature selection rules are as follows: for driver behavior data already quantified into emotion tags, the emotion tag with the highest matching degree and its intensity value are directly selected as the core feature of that dimension; for vehicle dynamics data, four original dimensions—vehicle speed, acceleration, steering angle, and brake pedal depth—are selected as core features; for physiological characteristics data, the matching degree of heart rate, skin conductance response value, and voice emotion tags is selected as core features; for environmental data, weather condition coding, normalized light intensity value, and road type coding are selected as core features. Subsequently, a concatenated fusion method is used to sequentially concatenate the core features extracted from the above four dimensions in the order of vehicle dynamics data, driver behavior data, driver physiological characteristics data, and environmental data to form a one-dimensional standardized state feature vector.

[0033] After the state feature vector is constructed, it needs to undergo final verification to ensure that all feature values ​​within the vector are within the range of 0 to 1 and that there are no invalid values. If the verification passes, the vector is transmitted to the input layer of the time-series prediction model for subsequent sentiment prediction. If the verification fails, the number of failures is recorded, and the process returns to the preprocessing stage to re-execute outlier detection and missing value imputation. When the number of consecutive verification failures exceeds a preset verification threshold, the most recently successfully constructed and verified state feature vector is used as a replacement, and a data anomaly alarm is output to the vehicle diagnostic system. In one specific embodiment, the preset verification threshold is set to 5 times, which is determined based on the vehicle-side data processing efficiency and fault tolerance requirements.

[0034] S2 inputs the state feature vector into a pre-trained time-series prediction model and outputs the probability distribution of different emotion categories for each time node within a preset time period in the future, forming an emotion prediction sequence.

[0035] like Figure 2 As shown, the time series prediction model adopts an architecture that combines a lightweight prediction model on the vehicle side with a large model in the cloud.

[0036] Specifically, the design of the time-series prediction model not only meets the low-latency requirements of real-time inference on the vehicle side, but also has the ability to continuously optimize the model accuracy, which is one of the core innovations.

[0037] The preset duration is dynamically configured based on the dynamic changes in the driving scenario and the vehicle-side inference efficiency. In one specific embodiment, the preset duration is set to 5 seconds for conventional driving scenarios such as urban roads or highways, and 3 seconds for low-speed scenarios such as parking or traffic jams. This configuration is based on the fact that the driver's emotional change cycle is approximately 5 seconds in conventional driving scenarios, while environmental changes are more frequent in low-speed scenarios. Shortening the preset duration to 3 seconds can effectively improve the timeliness of prediction.

[0038] The lightweight prediction model is deployed on the vehicle domain controller and performs real-time sentiment inference based on state feature vectors, outputting a sentiment prediction sequence. This sentiment prediction sequence includes at least the sentiment category and its probability value.

[0039] Specifically, the emotion categories include at least six basic emotions: pleasure, calmness, irritability, drowsiness, tension, and surprise, as well as composite emotions generated by fusing two or more basic emotions with different weights. The sum of the probability values ​​of all emotion categories is 1.

[0040] The lightweight prediction model for the vehicle-side primarily employs temporal convolutional networks or gated recurrent units. The number of parameters in this model must be controlled within a preset parameter threshold to accommodate the limited computing resources of the vehicle's domain controller. The preset parameter threshold is set considering the storage capacity and inference latency requirements of typical vehicle domain controllers, with a recommended value of 5MB to ensure millisecond-level inference under limited computing power conditions on the vehicle side. The model takes a state feature vector as input. The hidden layer captures the temporal dependencies of the input features through temporal convolutional kernels or gated units. The fully connected layer maps the high-dimensional features output from the hidden layer to the sentiment category dimension, obtaining a sentiment dimension feature vector. The Softmax layer normalizes the sentiment dimension feature vector and outputs a list of sentiment categories and corresponding probability values ​​for each time point within a preset future time period, forming a sentiment prediction sequence. This sequence is sorted by timestamp and stored in a circular cache pool on the vehicle side in the form of a sliding window. For example, an emotion prediction sequence can be represented as [t1: {pleasant: 0.70, calm: 0.25, irritable: 0.05}, t2: {……}, ……], where t1 and t2 represent the prediction results in the 1st and 2nd seconds of the future, respectively, and the sum of the probability values ​​of all emotion categories is 1. The time interval is dynamically determined based on the preset duration and the model output frequency, and is conventionally set to 1 second.

[0041] The large cloud model is deployed on the vehicle manufacturer's cloud server. It is periodically trained offline based on non-real-time user feedback data uploaded by multiple vehicles and historical multimodal data to generate optimized model parameters. These parameters are then sent to the vehicle via vehicle-cloud communication when the vehicle is parked or charging. This allows for dynamic iteration and updates of the lightweight prediction model on the vehicle, while the vehicle retains historical model versions as rollback backups.

[0042] Specifically, the cloud-based large-scale model is built on the Transformer architecture, with a parameter count far exceeding that of the lightweight prediction model on the vehicle side, reaching hundreds of megabytes, and possessing stronger feature extraction and pattern learning capabilities. This model employs offline training, using de-identified historical multimodal data uploaded from multiple vehicles and non-real-time user feedback data for periodic iteration to generate optimized model parameters such as network weights, bias terms, hyperparameters, and feature mapping configurations. The de-identified historical multimodal data includes de-identified vehicle dynamic data, driver behavior data, driver physiological characteristic data, and environmental data collected from past driving scenarios for each vehicle. Network weights refer to the weight values ​​of connections between neurons in each layer of the model. For example, the self-attention weights and fully connected layer weights of large cloud models, and the convolutional kernel weights and gating unit weights of lightweight prediction models for vehicles. These are core parameters that determine the model's feature extraction and inference accuracy. Bias terms refer to the offset parameters that accompany the network weights, used to adjust the output baseline of each layer of neurons and optimize the model's fitting ability. Hyperparameters include fixed configurations related to model training and adaptation parameters related to inference. Fixed configurations include learning rate, number of iterations, and batch size, while adaptation parameters include inference stride and feature pruning threshold of the vehicle model. Feature mapping configuration refers to the mapping relationship rules between multimodal data and the model's input layer, and the correspondence table between sentiment categories and output vectors, ensuring that the lightweight prediction model for vehicles accurately parses input features and outputs a matching sentiment probability distribution.

[0043] The timing of model parameter delivery is limited to when the vehicle is parked or charging to avoid consuming vehicle communication bandwidth or affecting the stability of the vehicle's infotainment system during driving, thus ensuring driving safety. After delivery, the new model parameters are validated locally. Validation includes verifying whether the model loading is complete (no missing parameters or format errors), whether the inference latency meets real-time requirements (single inference time less than 100 milliseconds), and whether the sentiment prediction accuracy on the vehicle-side local validation dataset (including 5000 frames of labeled multimodal data) is not lower than the preset accuracy percentage of historical versions. In one specific embodiment, the preset accuracy percentage is set to 95%, which is calibrated based on the balance between real-time performance and accuracy requirements of vehicle-side inference. After successful validation, the iterative update of the vehicle-side lightweight prediction model can be completed. Simultaneously, the vehicle retains the three most recent versions of the historical model as a rollback backup. If the new vehicle-side lightweight prediction model experiences inference anomalies or a sudden drop in accuracy during local validation, a rollback mechanism will be immediately triggered, automatically rolling back to the previous stable version.

[0044] Non-real-time user feedback data includes at least one of the following: user satisfaction ratings for facial animations, interaction latency data, and user facial expression reaction data.

[0045] Specifically, user satisfaction with the facial animation is collected through explicit or implicit feedback. Explicit feedback includes actions like clicking "like" or "dislike" buttons on a touchscreen, while implicit feedback includes behavioral characteristics such as whether the user continues to linger after the animation plays or repeatedly triggers similar interactions. Interaction latency data records the complete time stamp chain from triggering the emotional need to the completion and rendering of the facial animation, including the emotional need generation time stamp, the time stamp of the vehicle-side lightweight prediction model's inference completion, the animation pre-generation hit time stamp, and the rendering completion time stamp. By statistically analyzing the average and outlier percentages of the time consumed at each stage and the total latency (outliers are defined as values ​​exceeding the average latency ±3σ), the cache hit rate and pre-generation strategy are optimized. User facial expression reaction data is collected in real-time from the vehicle's DMS driver monitoring camera during the facial animation playback. The changes in facial key points are analyzed frame-by-frame by an emotion recognition algorithm (including facial key point detection and emotion classification sub-algorithms), quantifying the match between the user's real emotion and the emotion expressed in the facial animation. This serves as a supervisory signal for training the time-series prediction model and as a basis for optimizing the loss function.

[0046] Non-real-time user feedback data is recorded by the vehicle without the user's awareness. When the vehicle is parked and connected to WiFi, sensitive data such as user identity information and vehicle location information are anonymized and then uploaded to the cloud server for periodic offline training of large cloud models.

[0047] S3, based on the emotion prediction sequence, retrieves the corresponding primitive from the preset expression primitive library, pre-generates the corresponding expression animation keyframe sequence through the superposition and combination logic of primitive stack and animation hybrid tree, and stores the expression animation keyframe sequence into the circular cache pool.

[0048] like Figure 3 As shown, the specific process for generating the keyframe sequence of facial expression animation is as follows: S31, map the emotion category and probability value in the emotion prediction sequence to the basic emotion primitive, action primitive and modification primitive in the preset expression primitive library, and determine the initial fusion weight of each primitive based on the proportion of each probability value in the total probability value.

[0049] Specifically, the preset facial expression primitive library is a collection of basic facial animation units pre-designed and stored in the vehicle's onboard memory. It is divided into a basic emotion primitive sub-library, an action primitive sub-library, and a modification primitive sub-library, with each primitive corresponding to an independent animation segment. This primitive library is created using offline design tools such as 3D modeling and animation software, exported to standard animation formats compatible with vehicles, such as FBX and GLTF, and then compressed and optimized using a dedicated onboard compression algorithm to suit the limited storage and computing resources of the vehicle before being deployed on the vehicle's infotainment system. The compression algorithm includes at least one of the following: quantization compression of geometric vertex data, ASTC or ETC2 hardware compression of textures, inter-frame differential encoding of keyframe animations, and redundant node removal and LOD simplification of the 3D model.

[0050] The mapping is implemented using a table lookup method, as follows: Establish a one-to-one correspondence between emotion categories and basic emotion primitive IDs. For example, pleasure corresponds to basic emotion primitive ID_00001, and calmness corresponds to basic emotion primitive ID_00002. If the highest probability emotion category at a certain time point in the emotion prediction sequence is pleasure, and the probability value is 0.70, then select basic emotion primitive ID_00001, and use the probability value of 0.70 as the initial fusion weight for this primitive.

[0051] Each basic emotion primitive corresponds to a continuous animated segment of a single emotion category. This can be a looping animation or a single-shot animation with a fade-in / fade-out effect. For example, a looping animation expressing joy includes facial changes such as a raised corner of the mouth and eyes curving into crescents. Each basic emotion primitive contains multiple animation frames. To ensure smooth display of in-vehicle animations, the frame rate is set to 30fps, and the duration is typically 1 to 3 seconds. It supports looping and is the core component of facial expression animations.

[0052] Action primitives are divided into two categories: mandatory actions and optional actions. Mandatory actions include default periodic actions such as blinking and breathing, which are triggered directly according to preset rules (e.g., inserting a blinking primitive every 3 seconds) without depending on the emotion category. Optional actions are added dynamically based on interaction events, such as adding a head-turning primitive when navigation prompts are displayed. Action primitives are mainly short-duration actions, typically lasting between 0.5 and 1 second. They can be superimposed on basic emotion primitives on the timeline without changing the core expression of the basic emotion primitives. Their weight is preset to 0.3 to 0.7 based on visual saliency and does not participate in the probability value calculation.

[0053] Each basic emotion category is pre-associated with a set of modifier primitives. For example, happiness is associated with blush and glitter effects, while irritability is associated with flame textures and red halos. The display intensity of the modifier primitive (such as saturation, transparency, and scaling) is linearly related to the emotion probability value; the higher the probability value, the more significant the modifier effect. If the emotion probability value is lower than a preset probability threshold (e.g., 0.3), the modifier primitive is not loaded. This probability threshold is based on subjective perception experiments: when the probability value is lower than 0.3, more than 80% of testers cannot clearly perceive the existence of the modifier effect; therefore, the preset probability threshold is set to this value. Modifier primitives are usually implemented as semi-transparent textures or particle effects, which can be superimposed on the basic emotion primitives and have independent animation durations, typically 0.2 to 0.8 seconds.

[0054] If multiple emotion categories exist simultaneously at a certain time point in the emotion prediction sequence (such as pleasure 0.80, calm 0.15, and irritability 0.05), then the initial fusion weights of each basic emotion primitive are 0.80, 0.15, and 0.05, respectively, and after normalization, they are used for weighted superposition in the animation hybrid tree.

[0055] S32: Input the selected basic emotion primitives, action primitives, and modification primitives into the primitive stack, and the primitive stack performs hierarchical management and maintenance operations on each primitive.

[0056] Specifically, the primitive stack is divided into a bottom layer, a middle layer, and a top layer, strictly following the last-in-first-out stack operation principle. Newly added primitives directly overwrite old primitives in the same layer, and primitives in different layers can coexist on the timeline without interfering with each other. Among them, the bottom layer is specifically used to manage basic emotion primitives, the middle layer is used to manage action primitives, and the top layer is specifically used to manage modification primitives.

[0057] S33: Input the primitives in the primitive stack into the animation blending tree, and generate the blended animation parameters based on the initial blending weights of each primitive and the preset blending rules. The animation blending tree is a tree structure, with leaf nodes being primitives and intermediate nodes representing weighted superposition or linear blending operations.

[0058] Specifically, the mixed animation parameters include at least the position and rotation angle of each facial control point in the current frame, as well as the real-time fusion weights of each primitive.

[0059] For the same time point in the emotion prediction sequence, if only one basic emotion primitive is selected, its original animation parameters are directly used as the parameters for the blended sequence. If multiple basic emotion primitives are superimposed, the blended animation parameters are generated through an animation blending tree according to preset blending rules. The preset blending rules are set differently based on the primitive type. Each basic emotion primitive is weighted and superimposed using its normalized probability value as the fusion weight, forming the main body of the facial expression animation. Action primitives are additively superimposed onto the timeline of the basic emotion primitives. The fade-in and fade-out of the actions are controlled by a cubic Bézier easing curve on the timeline, ensuring a smooth and seamless transition between the actions and the basic emotions. Their weights are preset based on the visual salience of the actions, typically between 0.3 and 0.7. This range is determined by the visual salience of the action primitive; the stronger the salience, the higher the weight. Modification primitives are multiplicatively superimposed on the fused basic emotion and action primitives, and dynamically adjusted according to changes in probability values. The higher the probability value, the greater the weight of the modification primitive, and the more obvious the display effect. For example, modification primitives such as changing facial color or adding halo effects have their basic weight preset by designers in combination with in-vehicle display effects, usually set to 0.2 to 0.5.

[0060] During the animation playback and rendering phase, the duration of the facial expression display is determined by the time interval of the emotion prediction sequence, with a default time interval of 1 second. This interval is consistent with the inference output frame rate of the lightweight prediction model on the vehicle side, ensuring the synchronization of prediction and display. The duration of the facial expression display is consistent with the interval. For a single basic emotion primitive, if the original facial expression display duration is less than 1 second, it will be looped until the time slice is filled; if the original facial expression display duration is more than 1 second, the animation frames of the previous second are extracted, and the last frame is retained as the transition starting point with the next time node animation. The default setting for the animation frame rate is 30fps to ensure the smoothness of the animation on the in-vehicle display interface. When the load on the vehicle system is detected to exceed the preset load threshold (such as CPU utilization exceeding 70%), a dynamic frame rate adjustment mechanism will be triggered to dynamically reduce the animation frame rate to a preset low frame rate (such as 15fps), thereby ensuring the overall smooth operation of the vehicle system; when the load on the vehicle system falls back below the preset load threshold and continues to fall back for a preset duration (such as 5 seconds), the animation frame rate will automatically return to 30fps. Color parameters are adjusted within the HSV color space based on probability values. Hue is mapped to a preset hue range according to emotional category, such as yellow for happiness, red for irritability, and blue for calmness; saturation changes linearly with probability values, with higher probability values ​​resulting in higher saturation.

[0061] S34, based on the mixed animation parameters, calculates and generates a sequence of facial expression animation keyframes frame by frame at a preset frame rate. The sequence of facial expression animation keyframes includes timestamps, position and rotation information of each control point on the face, transparency and color parameters of the modification primitives, and transition weights with the preceding and following keyframe sequences.

[0062] Specifically, the facial expression animation keyframe sequence consists of several consecutive facial expression animation keyframes arranged in chronological order, corresponding to a complete animation segment at a given time point. The transition weights include the fusion weights with the previous facial expression animation keyframe sequence and the fusion weights with the next facial expression animation keyframe sequence: the former is used for linear interpolation transitions between adjacent time points to achieve smooth facial expression switching; the latter is used by the vehicle to prefetch and preload animation data for the next time point to ensure seamless transitions.

[0063] Employing in-vehicle 3D engines such as Unreal Engine, the keyframe generation logic runs in a separate background thread. Based on the definition of the animation blending tree, this thread calculates the blending results of basic emotion primitives, action primitives, and modification primitives within the primitive stack frame by frame, generating facial expression animation keyframes and combining them in real-time into a single facial expression animation keyframe sequence. The overall generation time of each facial expression animation keyframe sequence is controlled within a preset latency threshold (e.g., 50 milliseconds) to ensure that the generation operation does not consume main thread resources and does not affect the real-time performance of other in-vehicle functions such as navigation and voice interaction.

[0064] S35, the generated facial animation keyframe sequence, along with timestamps and transition weights, is stored in a circular buffer pool. The circular buffer pool uses timestamps as indexes and employs a first-in, first-out (FIFO) strategy for data eviction.

[0065] Specifically, the timestamp is the vehicle-side relative time corresponding to the sentiment prediction sequence time node, used to accurately locate the required animation segment when calling up expressions. When the storage capacity of the circular cache pool reaches the preset storage capacity threshold, the oldest animation data is automatically evicted using a first-in, first-out (FIFO) strategy to free up storage space for newly generated expression animation keyframe sequences. If the expression animation at a certain time node is not found in the circular cache pool, primitives are retrieved and calculated in real time based on the preset expression primitive library, and the circular cache pool is updated instantly, storing the real-time generated expression animation keyframe sequence into the circular cache pool so that subsequent calls with the same timestamp can directly hit it.

[0066] In a specific embodiment, assuming that in the emotion prediction sequence, when t1 is 1 second, the core emotion is pleasure with a probability value of 0.70, and no other action is required, only the default blinking action primitive and blush modification primitive need to be added, then the pre-generation process of the facial expression animation at this time point is as follows: Load the basic emotion primitive of "pleasantness" from the preset expression primitive library and input it into the bottom layer of the primitive stack as the core animation; insert the blinking action primitive at the 0.2-second mark on the "pleasantness" primitive timeline and input it into the middle layer of the primitive stack. Set the blending weight to 0.5 in the animation blending tree, use an additive overlay method, and configure an easing curve to achieve a smooth presentation and fading of the blinking action; starting from the 0-second mark on the "pleasantness" primitive timeline, overlay the blush enhancement primitive and input it into the top layer of the primitive stack. Its color saturation is adjusted to 70% based on the probability value of "pleasantness" (0.70), and it is multiplicatively overlaid on the blending animation of the bottom and middle layers. The blush transparency changes linearly with the emotion probability; according to... A preset frame rate of 30fps is used to calculate and generate a sequence of 30 keyframes for a joyful facial expression animation. Each frame records the position of the eyebrows, the opening and closing of the eyes, the curvature of the mouth, the RGB color value of the blush, and the transparency parameter. This sequence of keyframes for the facial expression animation, along with the t1 timestamp, a fusion weight of 0.8 with the previous sequence of keyframes for the facial expression animation, and a fusion weight of 0.5 with the next sequence of keyframes for the facial expression animation, is stored in a circular buffer pool. When the vehicle needs to play the facial expression at the t1 time node, the corresponding sequence of keyframes for the facial expression animation is retrieved directly from the circular buffer pool according to the t1 timestamp. After being rendered by the in-vehicle 3D engine, it can be output on the in-vehicle display interface.

[0067] S4: Obtain real-time emotional needs from interactive events, compare and adjudicate the real-time emotional needs with the emotional prediction sequence, and generate a unified emotional description based on the adjudication result.

[0068] like Figure 4 As shown, the specific adjudication process for emotional conflicts is as follows: S41, obtain the emotion category and interaction event type corresponding to the real-time emotion demand, compare it with the emotion category of the current time node in the emotion prediction sequence, and determine the emotion conflict type based on the comparison result.

[0069] Among them, the emotional conflict types include at least strong conflict, weak conflict, and no conflict; the interactive event types include at least safety warning events, driving task events, and infotainment events, with safety warning events having a higher priority than driving task events, and driving task events having a higher priority than infotainment events.

[0070] If the interaction event type for real-time emotional needs is a security alert event, it is determined to be a strong conflict.

[0071] Specifically, each real-time emotional demand includes a demand ID, source, emotional category, emotional intensity, timestamp, and additional information; among which, the additional information includes at least the event trigger threshold, demand confidence level, and associated vehicle module identifier.

[0072] If the interaction event type of the real-time emotional need is a driving task event or an infotainment event, and the emotional category of the need does not belong to the same basic emotional category, nor to a similar derived category or a neutral category, as the emotional category of the current time node in the emotional prediction sequence, then it is determined to be a weak conflict.

[0073] If the emotion category of the real-time emotion demand and the emotion category of the current time node in the emotion prediction sequence are the same basic emotion category, a similar derived category of the same basic emotion, or both are neutral categories without obvious emotion tendency, then it is determined that there is no conflict.

[0074] Among them, the basic emotion categories include at least pleasure, calmness, irritability, drowsiness, tension, and surprise; the neutral category refers to the basic state without a clear emotional tendency; the similar derived categories refer to the same kind of emotion extended from the same basic emotion, and the core emotional characteristics are consistent. For example, the similar derived categories of calmness are relaxation and peace, the similar derived categories of pleasure are happiness and joy, and the similar derived categories of irritability are anxiety and annoyance.

[0075] S42, based on the type of emotional conflict, implement differentiated arbitration procedures, as follows: S421, when the emotional conflict type is strong conflict, based on the principle of safety as the highest priority, the currently playing or pre-generated facial expression animation keyframe sequence is forcibly interrupted, and the facial expression animation that matches the real-time emotional needs corresponding to the safety warning event is immediately played. The interrupted facial expression animation status data is stored in the interruption task temporary storage queue. After the triggering conditions of the safety warning event are removed and the vehicle detects that the driving status has returned to normal, the interrupted facial expression animation is resumed from the temporary storage queue and played again according to the preset smooth transition rules.

[0076] Specifically, the interrupted facial animation state data includes at least the current playback frame index, the position or rotation parameters of each control point on the face, the transparency or color parameters of the modification primitives, the animation playback speed, the rendering adaptation parameters of each display interface, and the transition weight with the previous and next frames.

[0077] The preset smooth transition rule uses linear interpolation for the transition, with a fixed transition duration between 0.5 and 1 second: 0.5 seconds for high-speed or complex road scenes and 1 second for low-speed or urban road scenes. During the transition, interpolated frames are generated through proportional linear interpolation to gradually change the interrupted frame state to the normal playback frame state. The transition is performed synchronously across multiple vehicle display interfaces, with the time synchronization error controlled within 50 milliseconds.

[0078] S422, when the emotional conflict type is weak conflict, if the interaction event type is driving task event, the expression primitives in the preset expression primitive library corresponding to the driving task event are used as the main layer, and the expression primitives in the preset expression primitive library corresponding to the emotional prediction sequence are used as the auxiliary layer, and weighted superposition display is performed according to the preset animation level superposition rules; if the interaction event type is infotainment event, while retaining the expression animation corresponding to the emotional prediction sequence as the main display content, the display weight of the expression primitives in the preset expression primitive library corresponding to the infotainment event is reduced to below the preset weight threshold.

[0079] Specifically, the preset animation layer overlay rules are as follows: the main layer is rendered at the bottom layer, and the auxiliary layer is overlaid on the top layer; the transparency and motion effect parameters of the two layers are merged according to weights, and primitives without motion effect conflicts are displayed independently, while primitives with motion effect conflicts are based on the motion effect of the main layer. The weight of the main layer is fixed at 1.0, and the weight of the auxiliary layer is dynamically adjusted according to the safety of the driving scenario, with an adjustment range of 0.3 to 0.7. The preset weight threshold is calibrated based on the safety of the in-vehicle driving scenario, with a calibration range of 0.1 to 0.4, where 0.1 is used for highways or complex roads, and 0.4 is used for low-speed or urban roads; in addition, when infotainment primitives are overlaid, they are only displayed on the in-vehicle central control screen or rear entertainment screen, and the digital instrument panel and head-up display (HUD) do not load this type of primitive.

[0080] S423, when the emotional conflict type is no conflict, directly use the emotional state of the current time node in the emotional prediction sequence.

[0081] A unified sentiment description includes sentiment category, sentiment intensity, confidence level, and mixing ratio.

[0082] The emotion category is determined by the arbitration result, which specifies either a single emotion category or a composite emotion category. The arbitration result is the execution result of the differentiated arbitration operation. Emotion intensity includes the intensity of a single emotion and the intensity of composite emotions. The intensity of a single emotion is taken as the corresponding probability value of the emotion prediction sequence or the calibrated value of real-time emotion demand. The intensity of composite emotions is taken as the weighted average of the intensities of each single emotion. The weighting coefficients are determined according to the priority of the interaction events. Confidence is the weighted fusion value of the real-time emotion demand confidence of the corresponding interaction event type and the emotion prediction confidence of the emotion prediction sequence. The emotion prediction confidence is taken as the maximum emotion probability at the corresponding time node. The mixing ratio is determined based on the arbitration result and the priority of the interaction events.

[0083] Specifically, emotional intensity refers to the degree of emotional expression, where 0 represents no emotion and 1 represents the highest emotional intensity. The generation rules for each dimension of the unified emotional description will be differentiated based on the adjudication results, as follows: For strong conflict adjudication, the unified sentiment description directly adopts the sentiment category and sentiment intensity of the high-priority sentiment needs corresponding to the security alert event, the confidence level is taken as the confidence level of the real-time sentiment need, and the mixing ratio is a single sentiment, i.e., 100%.

[0084] For weak conflict adjudication, the unified sentiment description adopts a multi-sentiment fusion approach. The sentiment category is composite sentiment, the sentiment intensity is the weighted average of the sentiment intensities calculated according to the weights assigned to each sentiment intensity based on the priority of the interactive event, the confidence level is the weighted average of the confidence levels of each sentiment calculated according to the same priority weight, and the mixing ratio records the proportion of each sentiment, and the sum of the proportions of each sentiment is 100%.

[0085] For conflict-free adjudication, the unified sentiment description directly adopts the sentiment probability distribution of the current time node in the sentiment prediction sequence. The sentiment category is the sentiment category with the highest probability, the sentiment intensity is the corresponding probability value, the confidence level is the probability value, and the mixing ratio is 100%.

[0086] In one specific embodiment, taking the weak-conflict fusion of concurrent navigation and weather reporting as an example, assuming the vehicle is driving at low speed on urban roads, the emotion prediction sequence shows that the current driver's emotional state is calm, with a probability of 0.9 and a confidence level of 0.9. At this time, the navigation system issues a prompt to turn left at the intersection ahead, with a corresponding real-time emotional need of "keep your attention," a probability of 0.6, a confidence level of 0.95, and an interaction event priority of driving task event; while the weather reporting system simultaneously issues a prompt that it will rain today, with a corresponding real-time emotional need of "no special emotion," a probability of 0, a confidence level of 0.9, and an interaction event priority of infotainment event.

[0087] The generation process of the comparative adjudication and the unified sentiment description is as follows: The current vehicle speed is 35 km / h, indicating a low-speed driving state. The corresponding autonomous driving level is L2, the road type is urban, and the weather is cloudy. It was determined that neither of the two real-time emotional requests constitutes a safety event and is inconsistent with the calm state emotional category in the emotional prediction sequence, thus forming a weak conflict. Weight coefficients were assigned based on the priority of the interaction events: the weight coefficient for navigation driving tasks was set to 0.6, the weight coefficient for calm states in the emotional prediction sequence was set to 0.3, and the weight coefficient for weather-related infotainment events was set to 0.1. During animation fusion, the head micro-turning primitive from the preset expression primitive library corresponding to the focused emotion was used as the main layer to convey clear navigation instructions; the basic emotion primitive corresponding to calm emotions was overlaid as an auxiliary layer to maintain the driver's core emotional tone; neutral primitives without special emotions were combined with raindrop effect modifiers and displayed with lower weights, only shown on the in-vehicle central control screen. The transparency of the raindrop effect modifiers was set to 0.5 and they remained displayed until the weather broadcast event ended. The head micro-rotation effect in the main layer cycles once every 3 seconds, while the calm element in the auxiliary layer is continuously displayed, and the raindrop falling effect is presented synchronously and continuously. The emotion category this time is a composite emotion of attention and calmness. The emotion intensity is calculated by weight as 0.6×0.6+0.3×0.9+0.1×0=0.57, the mixing ratio of the two is attention 0.6 and calmness 0.4, and the confidence level is 0.6×0.95+0.3×0.9+0.1×0.9=0.93.

[0088] S5, based on a unified emotion description, retrieves the corresponding facial animation keyframe sequence from the circular cache pool by timestamp matching or generates animation parameters in real time. Combined with vehicle dynamic data, driver behavior data, and environmental data from the preprocessed multimodal data, it performs scene-adaptive rendering and display of facial animation on the in-vehicle display interface.

[0089] The in-vehicle display interface includes at least a central control screen, a digital instrument panel, a head-up display (HUD), and a rear entertainment screen, and each in-vehicle display interface adopts a synchronized animation playback mechanism.

[0090] Specifically, based on the current moment, the system searches the circular cache pool for the expression animation keyframe sequence whose timestamp is closest to the current moment and whose emotion category matches the unified emotion description. The allowed range for timestamp matching error is ±100 milliseconds, which is greater than the multi-screen synchronization error (50 milliseconds). This ensures sufficient processing margin between cache retrieval and rendering startup, avoiding animation loss due to retrieval timeouts. If a matching expression animation keyframe sequence exists in the circular cache pool, it is directly retrieved and sent to the 3D rendering engine for playback. If no matching expression animation keyframe sequence exists in the circular cache pool, the expression animation keyframe sequence is synthesized in real time using the expression primitive library and animation hybrid tree in S3, based on the unified emotion description. The newly generated sequence is then stored in the circular cache pool for subsequent use. The circular cache pool's calling and eviction mechanism is executed locally on the vehicle, ensuring the real-time performance of the animation and avoiding stuttering caused by real-time calculations.

[0091] In the in-vehicle display interface, the rendering and display of facial expression animations should be adapted to the specific context, using at least one of the following methods: Based on environmental and vehicle dynamic data, the brightness, contrast, and playback speed of the facial expression animation are dynamically adjusted.

[0092] Specifically, the system acquires light intensity data from the environment and automatically adjusts the brightness and contrast of the facial expression animation based on a preset linear mapping curve between light and brightness. In one specific embodiment, in a strong light environment with light intensity exceeding 5000 lux, the brightness of the facial expression animation is increased to 80% to 100% of its maximum value, and the contrast is increased by 20% to ensure clear visibility even under direct sunlight. At this time, the color parameters of the modifier primitives in the facial expression animation keyframe sequence (such as the RGB value of blush) are optimized synchronously with the brightness adjustment to avoid color distortion under strong light. In a weak light environment with light intensity below 50 lux, the brightness of the facial expression animation is reduced to 40% to 50% of its maximum value, and the contrast is reduced by 15% to avoid the bright image being glaring and affecting driving. At the same time, the transparency parameters of the modifier primitives in the facial expression animation keyframe sequence are reduced (such as the transparency of the flash effect is reduced from 0.8 to 0.5) to prevent the bright modifiers from interfering with vision in low light. For environments with alternating light and dark, such as entering and exiting tunnels, a smooth transition strategy is adopted, which linearly adjusts the brightness and contrast within 1 to 2 seconds to prevent visual discomfort caused by sudden changes. The brightness adjustment step size is set to adjust 10% of the total difference every 100 milliseconds. During the transition, the animation smooth connection when switching between light and dark is achieved by using the forward and backward transition weights in the keyframe sequence of facial expression animation (such as the forward blending weight linearly increasing from 0.2 to 0.8).

[0093] The playback speed of the facial expression animation is dynamically adjusted based on vehicle dynamic data. In one specific embodiment, when the vehicle speed is greater than 80 km / h (high-speed driving), the animation playback speed is reduced to 50% to 70% of the normal speed, and the number of animation keyframes is simplified, retaining 50% of the keyframes. At this time, only the core position and rotation information of each control point on the face in the facial expression animation keyframe sequence are retained, discarding redundant details to avoid distracting the driver's attention with rapid changes in the image. When the vehicle speed is less than 30 km / h (low-speed driving or parking), the normal playback speed of the animation is maintained to fully display the details. When the acceleration is greater than 0.3g (rapid acceleration or deceleration), the playback speed is temporarily reduced. Non-safety-related facial animation effects are paused and resumed after the vehicle's condition stabilizes (acceleration returns to within ±0.1g) to avoid interfering with the driver when the vehicle is moving violently. When the vehicle is traveling on bumpy roads such as gravel roads and speed bumps (determined by the vibration frequency detected by the vehicle vibration sensor to be greater than 5Hz), the animation playback speed is reduced to 60% of the normal speed. At the same time, the smoothness of the animation is increased by increasing the interpolation weight between keyframes (such as 0.8). Fine-tuning is done by combining the rotation information of facial control points in the facial animation keyframe sequence to reduce the sense of screen jitter caused by vehicle vibration.

[0094] Based on driver behavior data and vehicle dynamic data, the system identifies the current driving state. When a dangerous driving state is detected, it integrates facial animations with text and audio-visual warnings, rendering them in a unified manner with synchronized location and motion effects. Warning-type facial animations have the highest visual hierarchy across all in-vehicle display interfaces. Dangerous driving states include at least fatigue, distraction, making or receiving phone calls, and speeding. The display duration of the warning-type facial animations is linked to the resolution of the dangerous driving state; while the dangerous driving state persists, the warning-type facial animations loop, and after the state is resolved, they smoothly transition to normal facial animations.

[0095] Specifically, based on the type of dangerous driving condition, the corresponding warning-type emoticon animation is retrieved from the preset emoticon library. A text warning is then overlaid in a centered position below the emoticon animation. The text size adaptively adjusts according to the display interface size and moves with the focal area of ​​the emoticon animation to ensure intuitive information delivery. Simultaneously, a warning voice message or sound effect is played through the vehicle's audio system. The warning voice message volume is 20% higher than the current media volume but does not exceed 80 decibels, and its playback timing is synchronized with the keyframes of the emoticon animation. The vehicle's ambient lighting flashes or changes color at a frequency of 1Hz, changing between red and yellow (red for severe danger, yellow for minor danger), maintaining consistency with the visual rhythm of the emoticon animation.

[0096] The in-vehicle central control screen displays warning expressions in the form of a semi-transparent mask or full-screen floating layer with 50% transparency, covering other entertainment content; the digital instrument panel displays the warning expressions at the top layer, side by side with core driving information such as vehicle speed and engine speed, and the display area accounts for 30% of the instrument panel; the head-up display (HUD) projects a high-brightness monochromatic icon onto the central area directly in front of the driver's line of sight on the windshield, with a projection size of 10 cm × 10 cm, ensuring that the driver can obtain warning information without taking their eyes off the road.

[0097] When the dangerous driving state persists, the warning expression animation plays in a loop, and the text and audible and visual warning information continue simultaneously; when the dangerous driving state is解除, the warning expression animation automatically stops and smoothly transitions to the normal expression animation.

[0098] Based on the classification of the autonomous driving level in the vehicle dynamic data, the display mode of the expression animation is divided. If it is an assisted driving mode of level L2 or below, the expression animation focuses on safety reminders and driving state feedback. The HUD only displays a monochromatic and non-gradual safety expression animation, and the digital instrument panel simplifies the animation effect of the expression animation; the monochromatic and non-gradual safety expression animation alternates between the lit state and the extinguished state at a preset frequency; if it is an autonomous driving mode of level L3 or above, the expression animation focuses on entertainment interaction and scenario services. The in-vehicle central control screen and the rear entertainment screen display a full-motion and color interactive expression animation, and the HUD selectively turns off the display of the expression animation.

[0099] Specifically, the core principle of this classification strategy is that when the driving responsibility is borne by the driver (level L2 and below), the information presentation on the HUD and the digital instrument panel should prioritize safety and avoid interference from animations; when the vehicle system bears the driving responsibility (level L3 and above), more screen resources can be released for emotional and entertainment interactions.

[0100] To ensure the consistency and coordination of the expression animations on each display interface, a master-slave synchronization mechanism with the in-vehicle domain controller as the master node and each display interface as the slave node is adopted. Among them, the in-vehicle domain controller uniformly receives the emotional description instructions and generates a global expression animation timestamp; each display interface adapts and renders the global animation according to its own display capabilities; for the multi-screen linkage scenario, each display interface synchronizes through the timestamp to ensure that the key frames of the expression animation start playing simultaneously on all interfaces, and the time synchronization error is controlled within 50 milliseconds.

[0101] In one specific embodiment, taking a dangerous driving warning scenario as an example, when it is detected that the driver has closed his eyes for more than 5 seconds continuously, based on the preprocessed driver behavior data (fatigue state confidence level 0.95) and vehicle dynamic data (vehicle speed 60km / h, urban road), it is determined that the driver is in a dangerous driving state of severe fatigue, and a unified emotion description is generated. The emotion category is severe fatigue warning, the emotion intensity is 1.0, the confidence level is 0.98, and the mixing ratio is 100%. Based on the unified emotion description, the corresponding warning-type facial expression keyframe sequence is matched in the circular cache pool (because this type of warning scene is a high-frequency scene, the cache hit is successful). This sequence includes: timestamp (with an error of no more than 50 milliseconds from the current time), the position of each control point on the face (such as the key point coordinates when the eyelids close, the three-dimensional position of the head drooping 15 degrees) and rotation information, the transparency (0.9) and color parameters (RGB: 255,0,0) of the modification primitive (red warning halo), and the transition weights with the preceding and following facial expression animation keyframe sequences (forward fusion weight 0.1, backward fusion weight 0.9). Then, the integrated rendering warning mechanism is triggered, which is specifically manifested as: the facial expression animation presents a drowsy expression (driven by the position and rotation information of the facial control points in the facial expression animation keyframe sequence), the eyes slowly close and then open in a 2-second cycle, and the head droops slightly (matching the rotation information in the facial expression animation keyframe sequence). The text warning "Severe fatigue, please stop and rest immediately" is displayed below the facial expression animation, and the display state of the text follows the opening and closing of the eyes in the facial expression animation. Linked Changes: When the eyes are open, the text transparency drops to a preset brightness threshold; when the eyes are closed, the text fades in with 100% transparency to enhance the warning effect. When the warning is triggered, a "ding-dong" notification sound is played first, followed by a voice message at a preset high-priority volume level: "Severe fatigue detected, please stop and rest immediately." At the same time, the steering wheel LED light strip flashes red at a frequency of 1Hz. The warning expression animation will cover the entire area of ​​the vehicle's central control screen and digital instrument panel (the high transparency and red parameters of the modifier primitives in the expression animation keyframe sequence enhance the warning effect), and the head-up display (HUD) will display a red rest icon measuring 12cm × 12cm. This warning will continue to be displayed until the DMS camera detects that the driver has maintained a normal open-eye state and a stable head posture for 10 consecutive seconds. After determining that the fatigue state has been relieved, the driver switches to a unified emotional description of normal driving (calm, emotional intensity 0.6), retrieves the corresponding normal expression animation keyframe sequence from the cache, and smoothly transitions to a normal expression based on the transition weight (forward fusion weight 0.8) to avoid abrupt changes in the image. The recovery transition time is 0.5 seconds. In one specific embodiment, the preset brightness threshold is set to 50%, which conforms to the common sense of visual interaction for safety warnings; the preset high priority volume level is set to 80% of the system's maximum volume.

[0102] Taking the switching of autonomous driving modes as an example, assuming the vehicle is driving in Level 3 autonomous driving mode on the highway, the system will generate a unified emotional description based on the autonomous driving level in the vehicle's dynamic data: the emotional category is a composite emotion of pleasure and relaxation, the emotional intensity is 0.7, the confidence level is 0.92, and the mixing ratio is 60% pleasure and 40% relaxation. Subsequently, the system will match the corresponding full-motion color facial expression animation keyframe sequence in the circular cache pool. This sequence includes: a timestamp with an error of no more than 30 milliseconds from the current time, flexible position and rotation information of each control point on the face that supports virtual pet interaction, modification primitives with an opacity of 0.7 (blush RGB values ​​255, 150, 180; glitter RGB values ​​255, 255, 0) and vivid color parameters, and a transition weight of 0.5. During rendering, the system follows a Level 3 display mode: the head-up display (HUD) disables facial animations, retaining only core driving information such as vehicle speed and speed limits; the digital instrument panel displays a minimalist facial expression activated by the autonomous driving mode (retaining only core control point location information); the in-vehicle central control screen displays a virtual pet, which can perform active interactive actions such as slight head turns and mouth opening and closing based on the control point rotation information in the facial animation keyframe sequence, while simultaneously displaying a full-motion color expression to enhance the sense of pleasure (enhanced by modifying primitive color parameters). When the vehicle is about to exit the highway and requires driver intervention at a distance of 2 kilometers from the highway exit, the autonomous driving level in the vehicle dynamic data switches to Level 2, and the unified emotion description is updated synchronously: the emotion category is a composite emotion of safety reminder and attention, with an emotion intensity of 0.5, a confidence level of 0.95, and a mixing ratio of 70% safety reminder and 30% attention. At this point, the circular cache pool did not find the corresponding expression animation keyframe sequence. The system then called the S3 expression primitive library and the animation hybrid tree to synthesize the expression animation keyframe sequence in real time, and automatically switched to L2 level display mode: the head-up display (HUD) immediately displayed a red "Please take over the steering wheel" icon (matching the modifier primitive color parameters), and the icon blinked at a frequency of 1.5Hz; the digital instrument panel displayed the takeover reminder expression animation (based on the control point position information to achieve a simple head turning action); the in-vehicle central control screen paused the entertainment expression and displayed the takeover prompt in the form of a semi-transparent overlay.

[0103] like Figure 5 As shown, another embodiment of the present invention provides a dynamic facial expression display system based on vehicle status information. The system includes a data acquisition and preprocessing module, an emotion prediction module, an expression pre-generation module, a unified emotion description generation module, and an expression animation generation and rendering module.

[0104] The data acquisition and preprocessing module is used to collect multimodal data, perform preprocessing, and construct state feature vectors. The multimodal data includes at least vehicle dynamics data, driver behavior data, driver physiological characteristic data, and environmental data.

[0105] The sentiment prediction module is used to input the state feature vector into a pre-trained time-series prediction model and output the probability distribution of each sentiment category corresponding to a preset future duration, forming a sentiment prediction sequence.

[0106] The expression pre-generation module is used to retrieve the corresponding primitives from the preset expression primitive library based on the emotion prediction sequence, pre-generate the corresponding expression animation keyframe sequence through the superposition and combination logic of primitive stack and animation hybrid tree, and store the expression animation keyframe sequence into a circular cache pool.

[0107] The unified sentiment description generation module is used to obtain real-time sentiment needs from interactive events, compare and adjudicate the real-time sentiment needs with the sentiment prediction sequence, and generate a unified sentiment description based on the adjudication result.

[0108] The facial expression animation generation and rendering module is used to match and call the corresponding facial expression animation keyframe sequence from the circular cache pool according to the timestamp based on the unified emotion description, or to generate animation parameters in real time. Combined with vehicle dynamic data, driver behavior data and environmental data in the preprocessed multimodal data, the module performs scene-adaptive rendering and display of facial expression animation on the in-vehicle display interface.

[0109] The specific functions of each module correspond one-to-one with the steps S1 to S5 above, and will not be repeated here.

[0110] In summary, this invention provides a dynamic facial expression display method and system based on vehicle state information. Through an integrated technical architecture encompassing multimodal data fusion, vehicle-cloud collaborative temporal emotion prediction, hierarchical dynamic fusion of facial expression primitives, priority conflict resolution of emotional needs, and full-scene adaptive rendering, it achieves forward-looking prediction, real-time generation, intelligent fusion, and scene-specific adaptation of in-vehicle facial expression animation. This improves the real-time performance, accuracy, and emotional richness of in-vehicle human-machine interaction while ensuring driving safety. Specifically, firstly, it collects four types of multimodal data: vehicle dynamics, driver behavior, driver physiological characteristics, and environment. Through preprocessing strategies such as threshold and 3σ joint detection, categorical missing value imputation, normalization, and one-hot encoding, it constructs a high-quality unified state feature vector, laying a precise data foundation for forward-looking emotion prediction. Secondly, a sentiment prediction architecture combining lightweight vehicle-side processing with a large cloud-based model is adopted. The vehicle-side model is built using temporal convolutional networks or gated recurrent units, keeping the number of parameters within the vehicle's computing power range to achieve millisecond-level real-time sentiment inference. The cloud-based model is built using a Transformer architecture, periodically trained offline using anonymized multi-vehicle data and user feedback. Optimized parameters are deployed when the vehicle is parked or charging, and version rollback is supported, enabling continuous iterative optimization of model accuracy and accurately outputting driver sentiment prediction sequences within a preset timeframe. Building upon this, a 3D animation synthesis system is designed, incorporating hierarchical management of facial expression primitives, a primitive stack, and an animation hybrid tree. Facial expression primitives are divided into three sub-libraries: basic emotions, actions, and embellishments. The primitive stack enables hierarchical management of different types of primitives, and the animation hybrid tree performs weighted superposition, linear fusion, and multiplicative embellishment of multiple primitives, supporting vivid and accurate expression of both single and complex emotions. Simultaneously, a circular cache pool is used to achieve keyframe pre-generation and rapid retrieval, completely resolving the issues of fixed facial expression templates and high animation generation latency in traditional solutions. To address the issue of multi-source emotional conflict in human-computer interaction, a dynamic priority conflict adjudication mechanism based on driving scenarios is proposed. Interaction events are prioritized from high to low according to safety warning events, driving task events, and infotainment events, corresponding to three categories of emotional conflict: strong, weak, and none, with differentiated arbitration strategies applied. For strong conflicts caused by safety warnings, a high-priority forced overriding and low-priority state temporary storage and smooth recovery mechanism is adopted to ensure driving safety as the core. For weak conflicts caused by non-safety events, a layered fusion of the main and auxiliary layers, weakened display of infotainment elements, and split-screen loading mechanism are used to enrich the interactive performance without interfering with driving. For conflict-free scenarios, the emotional expression result of the current time node in the emotional prediction sequence is directly used to generate a unified emotional description, which serves as the sole basis for facial expression animation rendering.Simultaneously, a multi-dimensional, all-scenario adaptive rendering system for facial expression animations is created. This system integrates information from multiple dimensions, including ambient light intensity, vehicle driving status, autonomous driving level, and driver's driving status, to achieve intelligent dynamic adjustment of facial expression animation brightness, contrast, playback speed, and display mode. A master-slave synchronization mechanism is employed, with the vehicle domain controller as the master and each display interface as a slave node, controlling the time synchronization error of multi-screen displays within 50 milliseconds. This ensures display consistency across the vehicle's central control screen, digital instrument panel, head-up display (HUD), and rear entertainment screen. Specifically, for dangerous driving scenarios, an innovative integrated warning rendering system is implemented, integrating facial expression animations with text, sound, light, and ambient lighting. Warning-type facial expression animations are set at the highest visual level and linked to dangerous driving states for looping playback and smooth transitions. This makes in-vehicle facial expression animations not only an emotional carrier for human-computer interaction but also an intelligent auxiliary reminder for driving safety. Through collaborative innovation across various technical aspects, this invention effectively solves the core technical pain points of traditional in-vehicle facial expression display interaction, such as passive lag, improper conflict handling, delayed animation generation, and insufficient scene adaptability, providing a completely new solution for in-vehicle human-computer emotional interaction.

[0111] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A dynamic expression display method based on vehicle state information, characterized by, include: Multimodal data is collected and preprocessed to construct state feature vectors; the multimodal data includes at least vehicle dynamic data, driver behavior data, driver physiological characteristic data, and environmental data. The state feature vector is input into a pre-trained time series prediction model, which outputs the probability distribution of each time node corresponding to each sentiment category within a preset time period in the future, forming a sentiment prediction sequence. Based on the emotion prediction sequence, the corresponding primitive is retrieved from the preset expression primitive library, and the corresponding expression animation key frame sequence is pre-generated through the superposition and combination logic of primitive stack and animation hybrid tree, and the expression animation key frame sequence is stored in the circular cache pool. The system acquires real-time sentiment needs from interactive events, compares and adjudicates these real-time sentiment needs with a sentiment prediction sequence, and generates a unified sentiment description based on the adjudication results. Based on a unified emotion description, the system retrieves the corresponding keyframe sequence of facial expression animation from the circular cache pool by timestamp matching or generates animation parameters in real time. Combined with vehicle dynamic data, driver behavior data, and environmental data from the preprocessed multimodal data, the system performs scene-adaptive rendering and display of facial expression animation on the in-vehicle display interface.

2. The dynamic facial expression display method based on vehicle status information according to claim 1, characterized in that, The driver's physiological characteristics data include at least heart rate, skin conductance response, and voice emotion characteristics; The vehicle dynamic data includes at least vehicle speed, acceleration, steering angle, and brake pedal depth; the driver behavior data includes at least facial expression, head posture, gaze direction, and fatigue level; the environmental data includes at least weather conditions, light intensity, and road type. The preprocessing includes outlier detection, missing value imputation, and format standardization.

3. The dynamic facial expression display method based on vehicle status information according to claim 1, characterized in that, The time-series prediction model adopts an architecture that combines a lightweight prediction model on the vehicle side with a large model in the cloud. The lightweight prediction model is deployed on the vehicle domain controller and performs real-time sentiment inference based on state feature vectors, outputting a sentiment prediction sequence; the sentiment prediction sequence includes at least sentiment category and probability value. The cloud-based large model is deployed on the vehicle manufacturer's cloud server. It is periodically trained offline based on non-real-time user feedback data uploaded by multiple vehicles and historical multimodal data to generate optimized model parameters. These parameters are then sent to the vehicle via vehicle-cloud communication when the vehicle is parked or charging. This allows for dynamic iteration and updates of the lightweight prediction model on the vehicle side, while the vehicle side retains historical model versions as rollback backups. The non-real-time user feedback data includes at least one of the following: user satisfaction rating of the facial animation, interaction delay data, and user facial expression reaction data.

4. The dynamic facial expression display method based on vehicle status information according to claim 1, characterized in that, The process based on emotion prediction sequences involves retrieving corresponding primitives from a pre-defined expression primitive library and pre-generating corresponding expression animation keyframe sequences through the superposition and combination logic of primitive stacks and animation hybrid trees. This includes: The emotion categories and probability values ​​in the emotion prediction sequence are mapped to basic emotion primitives, action primitives and modification primitives in the preset expression primitive library, and the initial fusion weight of each primitive is determined based on the proportion of each probability value in the total probability value. Input the selected basic emotion primitives, action primitives, and modification primitives into the primitive stack, and the primitive stack will perform hierarchical management and maintenance operations on each primitive. The primitives in the primitive stack are input into the animation blending tree. Based on the initial blending weights of each primitive and the preset blending rules, the blended animation parameters are generated. The animation blending tree is a tree structure, with leaf nodes being primitives and intermediate nodes being blending operations of weighted superposition or linear blending. Based on the mixed animation parameters, a sequence of key frames for facial animation is generated frame by frame according to a preset frame rate; the sequence of key frames for facial animation includes timestamps, position and rotation information of each control point on the face, transparency and color parameters of the modification primitives, and transition weights with the preceding and following key frame sequences. The generated keyframe sequence of facial animation, along with timestamps and transition weights, is stored in a circular cache pool; the circular cache pool uses timestamps as indexes and employs a first-in-first-out (FIFO) strategy to evict data.

5. The dynamic facial expression display method based on vehicle status information according to claim 1, characterized in that, The process of acquiring real-time emotional needs from interaction events and comparing and adjudicating these real-time emotional needs with an emotional prediction sequence includes: The system acquires the emotional category and interaction event type corresponding to the real-time emotional needs, compares them with the emotional category at the current time node in the emotional prediction sequence, and determines the emotional conflict type based on the comparison results. The emotional conflict type includes at least strong conflict, weak conflict, and no conflict. The interaction event type includes at least safety warning events, driving task events, and infotainment events, with safety warning events having a higher priority than driving task events, and driving task events having a higher priority than infotainment events. Differentiated arbitration procedures are implemented based on the type of emotional conflict.

6. The dynamic facial expression display method based on vehicle status information according to claim 5, characterized in that, The determination of the type of emotional conflict based on the comparison results includes: If the interaction event type for real-time emotional needs is a security alert event, it is determined to be a strong conflict; If the interaction event type of the real-time emotional need is a driving task event or an infotainment event, and the emotional category of the need does not belong to the same basic emotional category, nor to a similar derived category or a neutral category, as the emotional category of the current time node in the emotional prediction sequence, then it is determined to be a weak conflict. If the emotion category of the real-time emotion demand and the emotion category of the current time node in the emotion prediction sequence are the same basic emotion category, a similar derived category of the same basic emotion, or both are neutral categories without obvious emotion tendency, then it is determined that there is no conflict. The basic emotion categories include at least pleasure, calmness, irritability, drowsiness, tension, and surprise; the neutral category refers to a basic state without a clear emotional tendency; and the similar derived categories refer to similar emotions extended from the same basic emotion, with consistent core emotional characteristics.

7. The dynamic facial expression display method based on vehicle status information according to claim 6, characterized in that, The differentiated arbitration operation based on the type of emotional conflict includes: When the emotional conflict type is strong conflict, based on the principle of safety as the highest priority, the currently playing or pre-generated facial animation keyframe sequence is forcibly interrupted, and the facial animation that matches the real-time emotional needs corresponding to the safety warning event is immediately played. The interrupted facial animation status data is stored in the interruption task temporary storage queue. After the triggering conditions of the safety warning event are removed and the vehicle detects that the driving status has returned to normal, the interrupted facial animation is resumed from the temporary storage queue and played again according to the preset smooth transition rules. When the emotional conflict type is weak, if the interaction event type is a driving task event, the expression primitives in the preset expression primitive library corresponding to the driving task event will be used as the main layer, and the expression primitives in the preset expression primitive library corresponding to the emotional prediction sequence will be used as the auxiliary layer. The weighted superposition display will be performed according to the preset animation layer superposition rules. If the interaction event type is an infotainment event, while retaining the expression animation corresponding to the emotional prediction sequence as the main display content, the display weight of the expression primitives in the preset expression primitive library corresponding to the infotainment event will be reduced to below the preset weight threshold. When the emotional conflict type is no conflict, the emotional state at the current time point in the emotional prediction sequence is used directly.

8. The dynamic facial expression display method based on vehicle status information according to claim 7, characterized in that, The unified sentiment description includes sentiment category, sentiment intensity, confidence level, and mixing ratio; The emotion category is determined by the arbitration result, which will ultimately adopt either a single emotion category or a composite emotion category. The arbitration result is the result of the execution of differentiated arbitration operations; The emotional intensity includes the intensity of a single emotion and the intensity of a compound emotion; the intensity of a single emotion is taken as the corresponding probability value of the emotional prediction sequence or the calibration value of the real-time emotional demand; the intensity of the compound emotion is taken as the average value of the weighted calculation of the intensities of each single emotion; wherein, the weighting coefficient is determined according to the priority of the interactive event. The confidence level is a weighted fusion value of the real-time emotional demand confidence level of the corresponding interactive event type and the emotional prediction confidence level of the emotional prediction sequence; the emotional prediction confidence level is taken as the maximum emotional probability at the corresponding time node; The mixing ratio is determined based on the arbitration results and the priority of interactive events.

9. The dynamic facial expression display method based on vehicle status information according to claim 1, characterized in that, The in-vehicle display interface includes at least an in-vehicle central control screen, a digital instrument panel, a head-up display (HUD), and a rear entertainment screen, and each in-vehicle display interface adopts a synchronized playback mechanism for facial animations. The rendering and display of facial animations on the in-vehicle display interface in a scene-adaptive manner includes at least one of the following methods: Based on environmental data and vehicle dynamic data, the brightness, contrast, and playback speed of the facial expression animation are dynamically adjusted. Based on driver behavior data and vehicle dynamic data, the current driving state is identified. When a dangerous driving state is identified, the facial animation, text, and sound and light warning information are rendered and displayed in an integrated manner with location linkage and dynamic effect synchronization. The warning facial animation has the highest visual level on all in-vehicle display interfaces. The dangerous driving states include at least fatigue, distraction, making or receiving phone calls, and speeding. The display duration of the warning-type emoticon animations is linked to the state of de-danger. When the dangerous driving state is ongoing, the warning-type emoticon animations play in a loop, and when the state is de-dangered, they smoothly transition to normal emoticon animations. Based on the autonomous driving level classification in vehicle dynamic data, the facial animation display mode is divided into L2 level and below assisted driving mode. The facial animation mainly focuses on safety reminders and driving status feedback. The head-up display (HUD) only displays monochrome safety facial animation without gradient, and the digital instrument panel simplifies the facial animation effects. The monochrome, non-gradient safety expression animation is generated by alternating between the on and off states at a preset frequency. In L3 or higher autonomous driving mode, the expression animation is mainly for entertainment interaction and scene service. The in-vehicle central control screen and rear entertainment screen display the full-motion color interactive expression animation, while the head-up display (HUD) selectively turns off the expression animation display.

10. A dynamic facial expression display system based on vehicle status information, used to execute the dynamic facial expression display method based on vehicle status information as described in any one of claims 1 to 9, characterized in that, include: The data acquisition and preprocessing module is used to acquire multimodal data, perform preprocessing, and construct state feature vectors; the multimodal data includes at least vehicle dynamic data, driver behavior data, driver physiological characteristic data, and environmental data; The sentiment prediction module is used to input the state feature vector into a pre-trained time series prediction model and output the probability distribution of each sentiment category at each time point within a preset time period in the future, forming a sentiment prediction sequence. The expression pre-generation module is used to retrieve the corresponding primitives from the preset expression primitive library based on the emotion prediction sequence, pre-generate the corresponding expression animation key frame sequence through the superposition and combination logic of primitive stack and animation hybrid tree, and store the expression animation key frame sequence into a circular cache pool. The unified sentiment description generation module is used to obtain real-time sentiment needs from interactive events, compare and adjudicate the real-time sentiment needs with the sentiment prediction sequence, and generate a unified sentiment description based on the adjudication result. The facial expression animation generation and rendering module is used to match and call the corresponding facial expression animation keyframe sequence from the circular cache pool according to the timestamp based on the unified emotion description, or to generate animation parameters in real time. Combined with vehicle dynamic data, driver behavior data and environmental data in the preprocessed multimodal data, the module performs scene-adaptive rendering and display of facial expression animation on the in-vehicle display interface.

Images