Digital human emotion recognition and feedback system based on large model

Abstract

The invention discloses a digital human emotion recognition and feedback system based on a large model, and relates to the technical field of large models.During operation of the system, multi-modal data of a driver are collected in real time through an embedded sensor network in various driving environments, and the emotion, driving behaviors and driving environment parameters of the driver are obtained; the method comprises the steps of constructing fatigue, anger and other multi-dimensional emotion features of a driver to form an emotion feature vector, analyzing an emotion change trend through threshold comparison, obtaining an evaluation state, dynamically generating a feedback strategy for optimizing a driving state based on an emotion state and a driving scene of the driver, and embedding a generative large model into a module. Diversified feedback responses including audio reminding, driving mode adjustment and entertainment content recommendation are automatically generated, feedback strength and form are adaptively adjusted through multi-dimensional parameters, a feedback generation effect is dynamically adjusted, and an emotion state analysis result and vehicle behavior monitoring data are integrated, so that a user experience is improved. And evaluating the potential dangerous state of the driver in real time.

Description

Technical Field

[0001] The present invention relates to the technical field of large models, and in particular to a digital human emotion recognition and feedback system based on large models. Background Art

[0002] As a frontier field in the development of artificial intelligence, digital human emotion recognition technology uses multimodal data collection and analysis to accurately judge the user's emotional state. In recent years, emotion recognition technology has been widely used in scenarios such as intelligent customer service, education and training, and medical psychology, and has gradually expanded to the field of intelligent driving for automobiles with high risks and high interactive needs. In the field of intelligent driving, the driver's emotional state is closely related to his behavior. Especially in emotional states such as fatigue and anger, the instability of driving behavior will significantly increase the risk of traffic accidents. Traditional emotion recognition systems are limited to a single data modality or simple algorithm, and it is difficult to evaluate the driver's complex emotional state in real time and accurately.

[0003] However, there are still some shortcomings at this stage. For example, the traditional system has weak processing capabilities for multimodal data, and the recognition results are often unstable due to data noise or missing data; the existing feedback mechanisms are mostly static or preset rules, and it is difficult to generate accurate feedback strategies in real time according to driving situations; in addition, the correlation analysis between emotional state and driving behavior and external environment is insufficient, and potential driving risks cannot be fully assessed. Therefore, the existing system still has significant room for improvement in accuracy, real-time and feedback personalization.

[0004] The shortcomings of the existing system are mainly due to the limitations of multimodal data processing and fusion technology, as well as the complexity of the emotion recognition algorithm model. When faced with the heterogeneity of multimodal data, traditional algorithms often fail to achieve effective feature alignment and fusion, resulting in biased recognition results. In addition, the static design of the emotion recognition and feedback system makes it lack the ability to adapt to dynamic driving situations, especially in high-pressure or emergency situations, the system may not be able to provide effective intervention. These shortcomings directly lead to a variety of abnormal effects, such as the driver cannot get timely reminders when he is tired or angry, which leads to further deterioration of driving behavior; the static feedback generated by the system may interfere with driving concentration and increase potential safety hazards; the emotional state is not accurately classified or the feedback is too delayed, which may not effectively prevent accidents at critical moments. To solve the above problems, the emotion recognition and feedback system based on large models provides a new solution for data fusion, intelligent decision-making and dynamic feedback. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a digital human emotion recognition and feedback system based on a large model, which solves the problems mentioned in the background technology.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a digital human emotion recognition and feedback system based on a large model, including a data acquisition module, a feature extraction module, an emotion recognition and analysis module, a feedback generation module, a system optimization module and a security monitoring and early warning module; The data acquisition module is used to collect multimodal data of the driver in real time under various driving environments through an embedded sensor network, including facial expression information, voice intonation characteristics, driving behavior characteristics and physiological signal data, to obtain the driver's emotions, driving behavior and driving environment parameters; The feature extraction module is used to perform time alignment, noise filtering and feature encoding on the collected multimodal data, form an emotional feature vector by constructing the driver's fatigue, anger and other multidimensional emotional features, and perform modal data decomposition and dimensionless processing as input for subsequent analysis; The emotion recognition and analysis module is used to generate emotion state classification results by analyzing the driver's emotion feature vector, using a multi-layer Transformer network to perform deep learning on the features, combining historical driving data and real-time features, and analyzing emotion change trends through threshold comparison to obtain normal, tired or angry emotion evaluation states; The feedback generation module is used to dynamically generate feedback strategies for optimizing driving status based on the driver's emotional state and driving scenarios. The module has a built-in generative large model that can automatically generate diversified feedback responses including audio reminders, driving mode adjustments, and entertainment content recommendations based on the results of different emotional evaluation states, and adaptively adjust the feedback intensity and form through multi-dimensional parameters, including visual, auditory, or tactile feedback; The system optimization module is used to dynamically adjust the feedback generation effect, collect the driver's behavioral reactions and emotional changes, including whether the feedback is accepted or whether the emotional state is improved, optimize the parameters of the emotion recognition model and the feedback generation model in real time, collect driving data for a long time and update the model weights; The safety monitoring and warning module is used to conduct real-time assessment of the driver's potential dangerous state by integrating the emotional state analysis results with the vehicle behavior monitoring data. When a high-risk emotional state is detected, including extreme fatigue or intense anger, the emergency warning program is automatically initiated, including automatic deceleration, switching to automatic driving mode or issuing high-priority reminders. It supports linkage with the vehicle control system and enhances overall safety through real-time monitoring of safety parameters, including braking reaction time and lane deviation.

[0007] Preferably, the data acquisition module includes an emotion monitoring unit, a driving behavior monitoring unit and an environmental status monitoring unit; The emotion monitoring unit is used to collect the driver's facial expressions, voice intonation and physiological signal characteristics in real time, and extract the differences between micro-expressions and overt expressions based on facial expression sequence analysis by infrared cameras, extract speech speed and pitch fluctuation indicators through voice spectrum features, and extract low-frequency or high-frequency ratios through heart rate variability HRV to obtain: expression inconsistency F di , Voice stress index V sp and fatigue response value P fr ; The driving behavior monitoring unit is used to collect the characteristics of direction control, acceleration and deceleration, and response delay in real time, measure the deviation between the steering wheel angle input and the vehicle's driving trajectory in real time, install pedal pressure sensors in the accelerator and brake pedals, collect the driver's pedal input strength and change rate, and install a high-precision vehicle-mounted radar on the front of the vehicle to generate a brake trigger condition signal to obtain: direction deviation rate S dev 、Acceleration and deceleration fluctuation range A var and brake response delay B rt ; The environmental status monitoring unit is used to perceive the impact of the external environment of roads, weather and traffic flow on driving in real time, measure the slipperiness of the road through tire slip rate, calculate driving visibility through ambient light intensity and road clarity ahead, obtain real-time traffic flow and surrounding vehicle status information through vehicle networking technology, and obtain: road surface friction coefficient R fr , visibility index V ind and traffic dynamic density T dd .

[0008] Preferably, the feature extraction module includes a preprocessing unit, an emotional feature encoding unit and a modal decomposition unit; The preprocessing unit is used to perform time synchronization and noise filtering on the collected facial expression, voice intonation, driving behavior and physiological signal data, align the multimodal data using a dynamic time warping algorithm, remove environmental noise in the voice signal using an adaptive filter, reduce random fluctuations in the physiological signal using a smoothing algorithm, and fill in the missing parts of the multimodal data using an interpolation method based on time series prediction; The emotional feature encoding unit is used to extract key features of the driver's fatigue, anger and other emotional states from the multimodal data to form a high-dimensional emotional feature vector, analyze the fatigue state based on the low-frequency and high-frequency ratio of heart rate variability HRV, and quantify the anger emotion through the sharp frequency features in the speech spectrum energy distribution and the dynamic changes of facial expressions, including the frowning frequency and the eye opening amplitude, combined with the eyelid opening and closing frequency and facial muscle relaxation characteristics; The modal decomposition unit is used to perform modal decomposition, normalization and dimension-independent processing on the multimodal emotional feature vector to eliminate the dimensional differences between different modal data, decompose the multimodal feature vector into sub-vector groups according to physiological signals, behavioral characteristics and expression data, normalize the data distribution of each modality through the Z-score standardization method, eliminate the influence of the dimensional unit of the feature on the calculation result, and calculate the emotional fluctuation coefficient C em , driving behavior stability coefficient C db 、Environmental impact factor coefficient C env and driving prediction model index Jsmx.

[0009] Preferably, the emotion fluctuation coefficient C em Calculated by the following formula: ; In the formula, F di Indicates the inconsistency of facial expression, V sp Represents the speech pressure index, P fr represents the fatigue response value, It represents the interaction between expression inconsistency and fatigue response, and is used to measure the degree of abnormal expression under fatigue emotion. Represents the logarithmic transformation of the speech pressure index, which can handle speech features with large fluctuations. Indicates the nonlinear relationship between fatigue response and expression inconsistency.

[0010] Preferably, the driving behavior stability coefficient C db Calculated by the following formula: ; In the formula, S dev Indicates the direction deviation rate, A var Indicates the fluctuation range of acceleration and deceleration, B rt Indicates brake response delay. It means reducing the negative impact of behaviors with large directional deviations on the coefficients. This means that high-frequency acceleration and deceleration fluctuations significantly reduce stability. The longer the response delay is, the worse the stability is. Represents the interaction term between directional control and acceleration / deceleration fluctuations.

[0011] Preferably, the environmental impact factor coefficient C env Calculated by the following formula: ; In the formula, R fr Represents the friction coefficient of the road surface, V ind Represents the visibility index, T dd represents the traffic dynamic density, Represents the exponential relationship between the coefficient of friction and visibility, used to quantify the positive impact of good road conditions on driving. Represents the negative effect of increased traffic density on driving safety. Represents a significant increase in environmental risk under low friction and low visibility conditions; The driving prediction model index Jsmx is obtained by calculating through the following formula: ; The C em 、C db and C env respectively represent the emotional fluctuation coefficient C em 、the driving behavior stability coefficient C db 、the environmental impact factor coefficient C env , and α, β, and γ are optimization parameters for adjusting the weights of different coefficients respectively.

[0012] Preferably, the emotion recognition and analysis module includes an emotion feature decoding unit and an emotion state evaluation unit; The emotion feature decoding unit is used to perform multi-layer deep learning analysis on the input emotion feature vector, encode and decode the emotion feature vector using a multi-layer Transformer network, extract the key features of the emotion state through the multi-head attention mechanism, construct a personalized baseline emotion feature based on historical driving data for real-time feature comparison analysis, and generate a high-precision emotion classification result by fusing real-time features and baseline features; The emotion state evaluation unit is used to predict the emotion change trend according to the time change pattern of emotion features, monitor the change rate and amplitude of emotion features using time series analysis methods, judge whether the current emotion trend deviates from the normal range through threshold comparison methods, and output the emotion state classification result, including normal, fatigued or angry; Set a preset first driving emotion threshold M and a preset second driving emotion threshold N, and obtain the state evaluation by comparing the driving prediction model index Jsmx with the preset first driving emotion threshold M and the preset second driving emotion threshold N; When Jsmx ≤ M, it indicates that the driver's emotion state is normal. At this time, the driver's emotion and behavior are in a stable state, and the system does not need to take intervention; When M < Jsmx ≤ N, it indicates that the driver is in a fatigued state. At this time, the driver's behavior has a slight deviation, but it does not pose a danger yet, and the system considers providing a reminder or mild intervention; When Jsmx > N, it indicates that the driver is in an angry state, and the system needs to intervene, including enabling the automatic driving mode and taking emergency reminder measures to ensure driving safety.

[0013] Preferably, the feedback generation module includes an emotional response generation unit, a driving mode optimization unit and a multimodal feedback control unit; The emotional response generation unit is used to embed a generative large model, generate personalized feedback content according to the emotional state, adjust the form and intensity of the feedback content in combination with urban driving or highway driving, and support diversified feedback forms, including audio reminders, text suggestions or visual guidance; The driving mode optimization unit is used to optimize the driving mode of the vehicle by analyzing the emotional state and driving scene. For fatigue state, it is recommended to switch the automatic driving mode or reduce the driving load. For anger state, it is used to reduce the vehicle response sensitivity to reduce dangerous operations. For normal state, it is used to maintain the existing driving mode or provide entertainment content recommendations. The multimodal feedback control unit is used to dynamically adjust the intensity and type of feedback forms according to the driver's current state, adaptively adjust the proportion and frequency of visual, auditory or tactile feedback, enhance the priority and intensity of multimodal feedback in high-risk emotional states, and reduce feedback intensity in normal states to reduce driver interference.

[0014] Preferably, the system optimization module includes a dynamic parameter adjustment unit; The dynamic parameter adjustment unit is used to dynamically adjust the weight and bias of the emotion recognition model according to the feedback effect evaluation results, fine-tune the output strategy of the generative model, ensure that the feedback content is more in line with actual needs, use the gradient descent optimization algorithm to improve the overall accuracy of the model, collect long-term driving data, form a personalized driver emotion and behavior database, use new data to regularly update the model weight, maintain the adaptability and robustness of the system, support model federated learning under the data privacy protection mechanism, and optimize the overall system performance.

[0015] Preferably, the safety monitoring and warning module includes a dangerous state detection unit and an emergency warning generation unit; The dangerous state detection unit is used to evaluate driving risks in real time by integrating emotional state and behavioral data, detect high-risk emotional states, including extreme fatigue or strong anger, and combine driving behavior data, including lane deviation and braking reaction time, to further evaluate the danger level and output potential dangerous state signals to provide a basis for subsequent early warning procedures; The emergency warning generation unit is used to initiate an automated emergency response when a high-risk state is detected, issue a high-priority alarm, including audio reminders, seat vibrations or visual guidance, automatically adjust the vehicle status, including slowing down or switching to autonomous driving mode, record emergency event data, and support post-event analysis.

[0016] The present invention provides a digital human emotion recognition and feedback system based on a large model, which has the following beneficial effects: (1) When the system is running, the embedded sensor network collects the driver's multimodal data in real time under various driving environments to obtain the driver's emotions, driving behaviors and driving environment parameters. By constructing the driver's fatigue, anger and other multi-dimensional emotional characteristics, an emotional feature vector is formed. The emotional change trend is analyzed by threshold comparison to obtain the evaluation status. Based on the driver's emotional state and driving scenario, a feedback strategy for optimizing the driving state is dynamically generated. The module has an embedded generative large model that automatically generates a variety of feedback responses including audio reminders, driving mode adjustments and entertainment content recommendations. The feedback intensity and form are adaptively adjusted through multi-dimensional parameters, and the feedback generation effect is dynamically adjusted. By integrating the emotional state analysis results with the vehicle behavior monitoring data, the driver's potential dangerous state is evaluated in real time.

[0017] (2) The digital human emotion recognition and feedback system based on the large model has fully completed the key tasks of real-time monitoring of the driver's emotional state, feature extraction, analysis and classification, feedback generation, system optimization, and safety warning through the collaborative work of six modules. The data acquisition module realizes the comprehensive capture of multimodal data, providing the system with a variety of information inputs such as facial expressions, voice intonation, and physiological signals; the feature extraction module ensures data quality and generates high-dimensional emotional feature vectors through time alignment, noise filtering, and dimensionless processing; the emotion recognition and analysis module combines historical driving data and real-time features, and uses the large model deep learning framework to accurately classify the driver's emotional state; the feedback generation module dynamically generates targeted driving suggestions to optimize the driving experience; the system optimization module adjusts the model parameters according to the feedback effect to improve long-term adaptability; the safety monitoring and warning module timely ensures driving safety through dangerous state detection and emergency warning. This system not only completes the closed-loop control of driver emotional state monitoring, but also realizes data-driven feedback optimization, which comprehensively improves driving safety and experience comfort.

[0018] (3) Compared with current technical means, this system has achieved significant improvements in multimodal data processing, emotional state recognition and feedback strategy generation. Traditional technologies often rely on a single modality (such as facial expressions or voice) for sentiment analysis, which has problems such as inaccurate recognition and long delays. However, this system collects multimodal data in real time through an embedded sensor network, and performs data alignment and normalization in the feature extraction module to ensure the integrity and high quality of the input data. The emotion recognition and analysis module uses the multi-head attention mechanism of the large model to decode the driver's emotional features into a high-dimensional feature space, greatly improving the accuracy and real-time performance of emotion classification. In addition, the feedback generation module uses a generative large model to dynamically adjust the feedback content according to the driving situation and emotional state, significantly improving the singleness and lag of traditional feedback strategies. The system optimization module continuously updates the model parameters based on the feedback effect, achieving personalization and dynamic adaptation, far exceeding the scope of application of static rule systems.

[0019] (4) Through the above technical improvements, the system has achieved significant improvement in many aspects. On the one hand, the accuracy and real-time performance of emotion recognition have been significantly improved, enabling the system to quickly take appropriate intervention measures when the driver's emotions are abnormal; on the other hand, the feedback strategy based on the generative large model has a high degree of personalization and scene adaptability, which not only improves the driving experience, but also effectively reduces the risk of intervention failure or interference with driving. At the same time, the system optimization module and the safety monitoring and early warning module work together to enable the system to have self-learning capabilities and real-time hazard perception capabilities to ensure driving safety. Overall, from data processing to emotion analysis, from feedback generation to system optimization, the system comprehensively solves the shortcomings of traditional technologies in real-time, accuracy and adaptability, and injects a new path to improve safety and user experience into intelligent driving technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a block diagram of the digital human emotion recognition and feedback system based on a large model of the present invention; DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Example 1 The present invention provides a digital human emotion recognition and feedback system based on a large model, please refer to Figure 1 , including data acquisition module, feature extraction module, emotion recognition and analysis module, feedback generation module, system optimization module and security monitoring and early warning module; The data acquisition module is used to collect multimodal data of the driver in real time under various driving environments through an embedded sensor network, including facial expression information, voice intonation characteristics, driving behavior characteristics and physiological signal data, to obtain the driver's emotions, driving behavior and driving environment parameters; The feature extraction module is used to perform time alignment, noise filtering and feature encoding on the collected multimodal data, form an emotional feature vector by constructing the driver's fatigue, anger and other multidimensional emotional features, and perform modal data decomposition and dimensionless processing as input for subsequent analysis; The emotion recognition and analysis module is used to generate emotion state classification results by analyzing the driver's emotion feature vector, using a multi-layer Transformer network to perform deep learning on the features, combining historical driving data and real-time features, and analyzing emotion change trends through threshold comparison to obtain normal, tired or angry emotion evaluation states; The feedback generation module is used to dynamically generate feedback strategies for optimizing driving status based on the driver's emotional state and driving scenarios. The module has a built-in generative large model that can automatically generate diversified feedback responses including audio reminders, driving mode adjustments, and entertainment content recommendations based on the results of different emotional evaluation states, and adaptively adjust the feedback intensity and form through multi-dimensional parameters, including visual, auditory, or tactile feedback; The system optimization module is used to dynamically adjust the feedback generation effect, collect the driver's behavioral reactions and emotional changes, including whether the feedback is accepted or whether the emotional state is improved, optimize the parameters of the emotion recognition model and the feedback generation model in real time, collect driving data for a long time and update the model weights; The safety monitoring and warning module is used to conduct real-time assessment of the driver's potential dangerous state by integrating the emotional state analysis results with the vehicle behavior monitoring data. When a high-risk emotional state is detected, including extreme fatigue or intense anger, the emergency warning program is automatically initiated, including automatic deceleration, switching to automatic driving mode or issuing high-priority reminders. It supports linkage with the vehicle control system and enhances overall safety through real-time monitoring of safety parameters, including braking reaction time and lane deviation.

[0023] In this embodiment, in a variety of driving environments, the driver's multimodal data is collected in real time through an embedded sensor network to obtain the driver's emotions, driving behaviors and driving environment parameters. By constructing the driver's fatigue, anger and other multi-dimensional emotional features, an emotional feature vector is formed. The emotional change trend is analyzed by threshold comparison to obtain the emotional evaluation state of normal, fatigue or anger. Based on the driver's emotional state and driving scene, a feedback strategy for optimizing the driving state is dynamically generated. The module embeds a generative large model to automatically generate a variety of feedback responses including audio reminders, driving mode adjustments and entertainment content recommendations. The feedback intensity and form are adaptively adjusted through multi-dimensional parameters to dynamically adjust the feedback generation effect. By integrating the emotional state analysis results with the vehicle behavior monitoring data, the driver's potential dangerous state is evaluated in real time. When a high-risk emotional state is detected, including extreme fatigue or strong anger, an emergency warning program is automatically started, including automatic deceleration, switching to automatic driving mode or issuing a high-priority reminder. It supports linkage with the vehicle control system and enhances overall safety through real-time monitoring of safety parameters, including braking reaction time and lane deviation.

[0024] Example 2 This embodiment is explained in Example 1, please refer to Figure 1 ,Specifically: the data acquisition module includes an emotion monitoring unit, a driving behavior monitoring unit and an environmental status monitoring unit; The emotion monitoring unit is used to collect the driver's facial expressions, voice intonation and physiological signal characteristics in real time, and extract the differences between micro-expressions and overt expressions based on facial expression sequence analysis by infrared cameras, extract speech speed and pitch fluctuation indicators through voice spectrum features, and extract low-frequency or high-frequency ratios through heart rate variability HRV to obtain: expression inconsistency F di , Voice stress index V sp and fatigue response value P fr ; The driving behavior monitoring unit is used to collect the characteristics of direction control, acceleration and deceleration, and response delay in real time, measure the deviation between the steering wheel angle input and the vehicle's driving trajectory in real time, install pedal pressure sensors in the accelerator and brake pedals, collect the driver's pedal input strength and change rate, and install a high-precision vehicle-mounted radar on the front of the vehicle to generate a brake trigger condition signal to obtain: direction deviation rate S dev 、Acceleration and deceleration fluctuation range A var and brake response delay B rt ; The environmental status monitoring unit is used to perceive the impact of the external environment of roads, weather and traffic flow on driving in real time, measure the slipperiness of the road through tire slip rate, calculate driving visibility through ambient light intensity and road clarity ahead, obtain real-time traffic flow and surrounding vehicle status information through vehicle networking technology, and obtain: road surface friction coefficient R fr , visibility index V ind and traffic dynamic density T dd .

[0025] The feature extraction module includes a preprocessing unit, an emotional feature encoding unit and a modal decomposition unit; The preprocessing unit is used to perform time synchronization and noise filtering on the collected facial expression, voice intonation, driving behavior and physiological signal data, align the multimodal data using a dynamic time warping algorithm, remove environmental noise in the voice signal using an adaptive filter, reduce random fluctuations in the physiological signal using a smoothing algorithm, and fill in the missing parts of the multimodal data using an interpolation method based on time series prediction; The emotional feature encoding unit is used to extract key features of the driver's fatigue, anger and other emotional states from the multimodal data to form a high-dimensional emotional feature vector, analyze the fatigue state based on the low-frequency and high-frequency ratio of heart rate variability HRV, and quantify the anger emotion through the sharp frequency features in the speech spectrum energy distribution and the dynamic changes of facial expressions, including the frowning frequency and the eye opening amplitude, combined with the eyelid opening and closing frequency and facial muscle relaxation characteristics; The modal decomposition unit is used to perform modal decomposition, normalization and dimension-independent processing on the multimodal emotional feature vector to eliminate the dimensional differences between different modal data, decompose the multimodal feature vector into sub-vector groups according to physiological signals, behavioral characteristics and expression data, normalize the data distribution of each modality through the Z-score standardization method, eliminate the influence of the dimensional unit of the feature on the calculation result, and calculate the emotional fluctuation coefficient C em , driving behavior stability coefficient C db 、Environmental impact factor coefficient C env and driving prediction model index Jsmx.

[0026] In this embodiment, through the collaborative work of the data acquisition module and the feature extraction module, the system can realize the comprehensive collection and efficient processing of the driver's multimodal data, significantly improving the accuracy and real-time performance of emotion recognition. The emotion monitoring unit combines infrared cameras, voice spectrum analysis and heart rate variability HRV to accurately capture the driver's facial expression changes, voice stress characteristics and physiological state, and obtain key parameters such as expression inconsistency F di , Voice stress index V sp and fatigue response value P fr , providing a reliable basis for the quantitative analysis of emotional state. The driving behavior monitoring unit collects the direction deviation rate S in real time through the steering wheel torque sensor, pedal pressure sensor and high-precision vehicle radar. dev 、Acceleration and deceleration fluctuation range A var and brake response delay B rt , accurately reflects the stability of the driver's behavior. The environmental status monitoring unit integrates tire slip rate, ambient light intensity and vehicle network data to dynamically obtain the road surface friction coefficient R fr , visibility index V ind and traffic dynamic density T dd , providing key basis for risk assessment of driving environment.

[0027] The feature extraction module completes the time alignment and noise filtering of multimodal data through the preprocessing unit to ensure data quality; the emotional feature encoding unit extracts high-dimensional emotional feature vectors to accurately quantify emotional states such as fatigue and anger; the modal decomposition unit eliminates the dimensional differences between data to ensure the fusion consistency of different modal features. The final generated emotional fluctuation coefficient C em, driving behavior stability coefficient C db 、Environmental impact factor coefficient C env The driving prediction model index Jsmx provides a scientific and accurate decision-making basis for emotion recognition and safety warning, and comprehensively optimizes driving safety and system reliability.

[0028] Example 3 This embodiment is explained in Example 1, please refer to Figure 1 , specifically: the emotional fluctuation coefficient C em Calculated by the following formula: ; In the formula, F di Indicates the inconsistency of facial expression, V sp Represents the speech pressure index, P fr represents the fatigue response value, It represents the interaction between expression inconsistency and fatigue response, and is used to measure the degree of abnormal expression under fatigue emotion. Represents the logarithmic transformation of the speech pressure index, which can handle speech features with large fluctuations. Indicates the nonlinear relationship between fatigue response and expression inconsistency.

[0029] The driving behavior stability coefficient C db Calculated by the following formula: ; In the formula, S dev Indicates the direction deviation rate, A var Indicates the fluctuation range of acceleration and deceleration, B rt Indicates brake response delay. It means reducing the negative impact of behaviors with large directional deviations on the coefficients. This means that high-frequency acceleration and deceleration fluctuations significantly reduce stability. The longer the response delay is, the worse the stability is. Represents the interaction term between directional control and acceleration / deceleration fluctuations.

[0030] The environmental impact factor coefficient C env Calculated by the following formula: ; In the formula, R fr Represents the friction coefficient of the road surface, V ind Represents the visibility index, T dd represents the traffic dynamic density, Represents the exponential relationship between friction coefficient and visibility, and is used to quantify the positive impact of good road conditions on driving. Indicates the negative impact of increased traffic density on driving safety, Indicates that environmental risks increase significantly under low friction and low visibility conditions; The driving prediction model index Jsmx is calculated by the following formula: ; The C em , C db and C env They represent the emotional fluctuation coefficient C em , driving behavior stability coefficient C db 、Environmental impact factor coefficient C env , α, β and γ are optimization parameters for adjusting the weights of different coefficients.

[0031] In this embodiment, the system constructs the emotion fluctuation coefficient C em , driving behavior stability coefficient C db 、Environmental impact factor coefficient C env The driving prediction model index Jsmx realizes the quantitative evaluation of the driver's emotions, behaviors and environmental factors. em Accurately capture the expression inconsistency Fdi and voice stress index V sp and fatigue response value P fr The dynamic interaction relationship between the driver and the driver can fully reflect the trend of the driver's emotional changes; the driving behavior stability coefficient C db The direction deviation rate S dev 、Acceleration and deceleration fluctuation range A var and brake response delay B rt , achieving high-precision quantification of driving operation stability; environmental impact factor coefficient C env The road friction coefficient R fr , visibility index V ind and traffic dynamic density T dd The final driving prediction model index Jsmx combines the three key elements of emotion, behavior and environment, providing a comprehensive and accurate basis for driving risk assessment.

[0032] Compared with the traditional single-index analysis method, the multi-coefficient calculation formula of this system fully considers the interaction between emotions, behaviors and the environment, and achieves significant improvement in situational adaptability and data fusion through the optimization adjustment of dynamic weight parameters α, β and γ. This system not only improves the prediction accuracy of driving safety, but also provides scientific and systematic quantitative support for emotion recognition, driving behavior optimization and hazard warning, significantly enhancing the overall effect of driving safety management and user experience.

[0033] Example 4 This embodiment is an explanatory description carried out in Embodiment 1. Please refer to Figure 1 , specifically: The emotion recognition and analysis module includes an emotion feature decoding unit and an emotion state evaluation unit; The emotion feature decoding unit is used to perform multi-layer deep learning analysis on the input emotion feature vector, encode and decode the emotion feature vector using a multi-layer Transformer network, extract the key features of the emotion state through a multi-head attention mechanism, construct a personalized baseline emotion feature based on historical driving data for real-time feature comparison analysis, and fuse the real-time feature and the baseline feature to generate a high-precision emotion classification result; The emotion state evaluation unit is used to predict the emotion change trend according to the time change pattern of the emotion feature, monitor the change rate and amplitude of the emotion feature using time series analysis methods, judge whether the current emotion trend deviates from the normal range through a threshold comparison method, and output the emotion state classification result, including normal, fatigued or angry; Set a preset first driving emotion threshold M and a preset second driving emotion threshold N, and compare the driving prediction model index Jsmx with the preset first driving emotion threshold M and the preset second driving emotion threshold N to obtain a state evaluation; When Jsmx ≤ M, it indicates that the driver's emotion state is normal. At this time, the driver's emotion and behavior are in a stable state, and the system does not need to take any intervention; When M < Jsmx ≤ N, it indicates that the driver is in a fatigued state. At this time, the driver's behavior has a slight deviation, but it does not pose a danger yet. The system considers providing a reminder or mild intervention; When Jsmx > N, it indicates that the driver is in an angry state, and the system needs to intervene, including enabling the autonomous driving mode and taking emergency reminder measures to ensure driving safety.

[0034] The feedback generation module includes an emotion response generation unit, a driving mode optimization unit, and a multi-modal feedback control unit; The emotion response generation unit is used to embed a generative large model, generate personalized feedback content according to the emotion state, combine urban driving or highway driving, adjust the form and intensity of the feedback content, and support diverse feedback forms, including audio reminders, text suggestions, or visual guidance; The driving mode optimization unit is used to optimize the vehicle's driving mode by analyzing the emotion state and driving scenario. For the fatigued state, it is recommended to switch to the autonomous driving mode or reduce the driving load. For the angry state, it reduces the vehicle response sensitivity to reduce dangerous operations. For the normal state, it maintains the existing driving mode or provides entertainment content recommendations; The multimodal feedback control unit is used to dynamically adjust the intensity and type of feedback forms according to the driver's current state, adaptively adjust the proportion and frequency of visual, auditory or tactile feedback, enhance the priority and intensity of multimodal feedback in high-risk emotional states, and reduce feedback intensity in normal states to reduce driver interference.

[0035] In this embodiment, the collaborative work of the emotion recognition analysis module and the feedback generation module realizes the accurate classification and real-time response of the driver's emotional state, effectively improving driving safety and user experience. The emotion feature decoding unit extracts the key features of the emotional state through deep learning analysis of the multi-layer Transformer network, combined with the multi-head attention mechanism, and uses historical driving data to build a personalized baseline model to ensure the accuracy and dynamic adaptability of the emotion recognition results. The emotion state evaluation unit uses the time series analysis method and threshold comparison mechanism to monitor the trend of emotion changes in real time, and accurately judges whether the driver is in a normal, fatigued or angry state based on the comparison of the driving prediction model index Jsmx with the preset thresholds M and N, providing a reliable emotion classification basis for feedback generation. The feedback generation module embeds a generative large model, and generates personalized feedback strategies such as audio reminders, text suggestions or visual guidance according to different driving scenarios and emotional states through the emotion response generation unit, realizing efficient and diversified feedback forms. The driving mode optimization unit dynamically adjusts the vehicle's driving mode to effectively reduce the driving load in fatigue state and the risk of dangerous operation in anger state; the multimodal feedback control unit adaptively adjusts the form and intensity of feedback according to the driver's state and situation, strengthens the feedback priority in high-risk emotional state, reduces interference in normal state, and ensures the effectiveness of feedback response and user acceptance. Compared with the traditional single rule or static feedback system, this module significantly enhances the accuracy of emotional state recognition and the dynamic adaptability of feedback strategy. By combining emotional analysis with driving mode optimization, the system not only improves the response efficiency to high-risk emotional state, but also optimizes the overall driver experience, reduces unnecessary intervention, and improves the application value of the system in actual driving scenarios.

[0036] Example 5 This embodiment is explained in Example 1, please refer to Figure 1 ,Specifically: the system optimization module includes a dynamic parameter adjustment unit; The dynamic parameter adjustment unit is used to dynamically adjust the weight and bias of the emotion recognition model according to the feedback effect evaluation results, fine-tune the output strategy of the generative model, ensure that the feedback content is more in line with actual needs, use the gradient descent optimization algorithm to improve the overall accuracy of the model, collect long-term driving data, form a personalized driver emotion and behavior database, use new data to regularly update the model weight, maintain the adaptability and robustness of the system, support model federated learning under the data privacy protection mechanism, and optimize the overall system performance.

[0037] The safety monitoring and warning module includes a dangerous state detection unit and an emergency warning generation unit; The dangerous state detection unit is used to evaluate driving risks in real time by integrating emotional state and behavioral data, detect high-risk emotional states, including extreme fatigue or strong anger, and combine driving behavior data, including lane deviation and braking reaction time, to further evaluate the danger level and output potential dangerous state signals to provide a basis for subsequent early warning procedures; The emergency warning generation unit is used to initiate an automated emergency response when a high-risk state is detected, issue a high-priority alarm, including audio reminders, seat vibrations or visual guidance, automatically adjust the vehicle status, including slowing down or switching to autonomous driving mode, record emergency event data, and support post-event analysis.

[0038] In this embodiment, the collaborative design of the system optimization module and the safety monitoring and early warning module significantly improves the accuracy, adaptability and safety performance of the system. The dynamic parameter adjustment unit adjusts the weights and biases of the emotion recognition model in real time, and fine-tunes the output strategy of the generative model to make the feedback content more in line with the actual needs of the driver. Using the gradient descent optimization algorithm and personalized modeling of long-term driving data, the unit not only improves the overall accuracy of the model, but also maintains the high adaptability of the system to different drivers and driving environments by periodically updating the weights. The federated learning capability supported under the data privacy protection mechanism further enhances the robustness and wide applicability of the system. The safety monitoring and early warning module integrates emotional state and driving behavior data through the dangerous state detection unit to evaluate driving risks in real time. Combined with key behavioral data such as lane deviation and braking reaction time, high-risk emotional states such as extreme fatigue or strong anger are accurately identified, and potential danger signals are output to provide a scientific basis for subsequent warnings and interventions. The emergency warning generation unit automatically initiates emergency response in high-risk emotional states, including issuing high-priority alarms, seat vibration reminders, and visual guidance. It also dynamically adjusts the vehicle status (such as slowing down or switching to autonomous driving mode) and records emergency event data to support post-event analysis and improvement. Compared with traditional systems, the dynamic optimization and real-time warning capabilities of this module effectively fill the gaps in delayed emotion recognition and delayed high-risk intervention. By achieving accurate and dynamic emotion recognition and timely response, the system has achieved remarkable results in improving driving safety, optimizing user experience, and preventing potential risks, providing more reliable technical support for intelligent driving.

[0039] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A digital human emotion recognition and feedback system based on a large model, characterized by: It includes data acquisition module, feature extraction module, emotion recognition and analysis module, feedback generation module, system optimization module and security monitoring and early warning module; The data acquisition module is used to collect multimodal data of the driver in real time under various driving environments through an embedded sensor network, including facial expression information, voice intonation characteristics, driving behavior characteristics and physiological signal data, to obtain the driver's emotions, driving behavior and driving environment parameters; The feature extraction module is used to perform time alignment, noise filtering and feature encoding on the collected multimodal data, form an emotional feature vector by constructing the driver's fatigue, anger and other multidimensional emotional features, and perform modal data decomposition and dimensionless processing as input for subsequent analysis; The emotion recognition and analysis module is used to generate emotion state classification results by analyzing the driver's emotion feature vector, using a multi-layer Transformer network to perform deep learning on the features, combining historical driving data and real-time features, and analyzing emotion change trends through threshold comparison to obtain normal, tired or angry emotion evaluation states; The feedback generation module is used to dynamically generate feedback strategies for optimizing driving status based on the driver's emotional state and driving scenarios. The module has a built-in generative large model that can automatically generate diversified feedback responses including audio reminders, driving mode adjustments, and entertainment content recommendations based on the results of different emotional evaluation states, and adaptively adjust the feedback intensity and form through multi-dimensional parameters, including visual, auditory, or tactile feedback; The system optimization module is used to dynamically adjust the feedback generation effect, collect the driver's behavioral reactions and emotional changes, including whether the feedback is accepted or whether the emotional state is improved, optimize the parameters of the emotion recognition model and the feedback generation model in real time, collect driving data for a long time and update the model weights; The safety monitoring and warning module is used to conduct real-time assessment of the driver's potential dangerous state by integrating the emotional state analysis results with the vehicle behavior monitoring data. When a high-risk emotional state is detected, including extreme fatigue or intense anger, the emergency warning program is automatically initiated, including automatic deceleration, switching to automatic driving mode or issuing high-priority reminders. It supports linkage with the vehicle control system and enhances overall safety through real-time monitoring of safety parameters, including braking reaction time and lane deviation.

2. The large-model-based digital human emotion recognition and feedback system according to claim 1, characterized in that: The data acquisition module includes an emotion monitoring unit, a driving behavior monitoring unit and an environmental status monitoring unit; The emotion monitoring unit is used to collect the driver's facial expressions, voice intonation and physiological signal characteristics in real time, and extract the differences between micro-expressions and overt expressions based on facial expression sequence analysis by infrared cameras, extract speech speed and pitch fluctuation indicators through voice spectrum features, and extract low-frequency or high-frequency ratios through heart rate variability HRV to obtain: expression inconsistency F di , Voice stress index V sp and fatigue response value P fr ; The driving behavior monitoring unit is used to collect the characteristics of direction control, acceleration and deceleration, and response delay in real time, measure the deviation between the steering wheel angle input and the vehicle's driving trajectory in real time, install pedal pressure sensors in the accelerator and brake pedals, collect the driver's pedal input strength and change rate, and install a high-precision vehicle-mounted radar on the front of the vehicle to generate a brake trigger condition signal to obtain: direction deviation rate S dev 、Acceleration and deceleration fluctuation range A var and brake response delay B rt ; The environmental status monitoring unit is used to perceive the impact of the external environment of roads, weather and traffic flow on driving in real time, measure the slipperiness of the road through tire slip rate, calculate driving visibility through ambient light intensity and road clarity ahead, obtain real-time traffic flow and surrounding vehicle status information through vehicle networking technology, and obtain: road surface friction coefficient R fr , visibility index V ind and traffic dynamic density T dd .

3. The large-model-based digital human emotion recognition and feedback system according to claim 1, characterized in that: The feature extraction module includes a preprocessing unit, an emotional feature encoding unit and a modal decomposition unit; The preprocessing unit is used to perform time synchronization and noise filtering on the collected facial expression, voice intonation, driving behavior and physiological signal data, align the multimodal data using a dynamic time warping algorithm, remove environmental noise in the voice signal using an adaptive filter, reduce random fluctuations in the physiological signal using a smoothing algorithm, and fill in the missing parts of the multimodal data using an interpolation method based on time series prediction; The emotional feature encoding unit is used to extract the key features of the driver's fatigue, anger and other emotional states from multi-modal data, form a high-dimensional emotional feature vector, analyze the fatigue state based on the low-frequency and high-frequency ratio of heart rate variability (HRV), combine the eyelid opening and closing frequency and facial muscle relaxation features, and quantify the anger emotion through the sharp frequency features in the voice spectrum energy distribution and the dynamic changes of facial expressions, including the frowning frequency and the eye opening amplitude; The modal decomposition unit is used to perform modal decomposition, normalization and dimension-independent processing on the multimodal emotional feature vector to eliminate the dimensional differences between different modal data, decompose the multimodal feature vector into sub-vector groups according to physiological signals, behavioral characteristics and expression data, normalize the data distribution of each modality through the Z-score standardization method, eliminate the influence of the dimensional unit of the feature on the calculation result, and calculate the emotional fluctuation coefficient C em , driving behavior stability coefficient C db 、Environmental impact factor coefficient C env and driving prediction model index Jsmx.

4. The large-model-based digital human emotion recognition and feedback system according to claim 3, characterized in that: The emotional fluctuation coefficient C em Calculated by the following formula: ； In the formula, F di Indicates the inconsistency of facial expression, V sp Represents the speech pressure index, P fr represents the fatigue response value, It represents the interaction between expression inconsistency and fatigue response, and is used to measure the degree of abnormal expression under fatigue emotion. Represents the logarithmic transformation of the speech pressure index, which can handle speech features with large fluctuations. Indicates the nonlinear relationship between fatigue response and expression inconsistency.

5. The large-model-based digital human emotion recognition and feedback system according to claim 3, characterized in that: The driving behavior stability coefficient C db Calculated by the following formula: ； In the formula, S dev Indicates the direction deviation rate, A var Indicates the fluctuation range of acceleration and deceleration, B rt Indicates brake response delay. It means reducing the negative impact of behaviors with large directional deviations on the coefficients. This means that high-frequency acceleration and deceleration fluctuations significantly reduce stability. The longer the response delay is, the worse the stability is. Represents the interaction term between directional control and acceleration / deceleration fluctuations.

6. The large-model-based digital human emotion recognition and feedback system according to claim 3, characterized in that: The environmental impact factor coefficient C env Calculated by the following formula: ； In the formula, R fr Represents the friction coefficient of the road surface, V ind Represents the visibility index, T dd represents the traffic dynamic density, Represents the exponential relationship between friction coefficient and visibility, and is used to quantify the positive impact of good road conditions on driving. Indicates the negative impact of increased traffic density on driving safety, Indicates that environmental risks increase significantly under low friction and low visibility conditions; The driving prediction model index Jsmx is calculated and obtained through the following formula: ； The C em , C db and C env They represent the emotional fluctuation coefficient C em , driving behavior stability coefficient C db 、Environmental impact factor coefficient C env , α, β and γ are optimization parameters for adjusting the weights of different coefficients.

7. The large-model-based digital human emotion recognition and feedback system according to claim 1, characterized in that: The emotion recognition and analysis module includes an emotion feature decoding unit and an emotion state evaluation unit; The emotion feature decoding unit is used to perform multi-layer deep learning analysis on the input emotion feature vector, use a multi-layer Transformer network to encode and decode the emotion feature vector, extract the key features of the emotion state through the multi-head attention mechanism, construct a personalized baseline emotion feature based on historical driving data for real-time feature comparison analysis, and fuse the real-time feature and the baseline feature to generate a high-precision emotion classification result; The emotion state evaluation unit is used to predict the emotion change trend according to the time change pattern of the emotion features, monitor the change rate and amplitude of the emotion features by using the time series analysis method, judge whether the current emotion trend deviates from the normal range through the threshold comparison method, and output the emotion state classification result, including normal, fatigue or anger; Set a preset first driving emotion threshold M and a preset second driving emotion threshold N, and compare the driving prediction model index Jsmx with the preset first driving emotion threshold M and the preset second driving emotion threshold N to obtain the state evaluation; When Jsmx ≤ M, it means that the driver's emotional state is normal. At this time, the driver's emotion and behavior are in a stable state, and no intervention is required by the system; When M < Jsmx ≤ N, it means that the driver is in a fatigue state. At this time, the driver's behavior has a slight deviation, but it does not pose a danger yet. The system considers providing a reminder or mild intervention; When Jsmx > N, it means that the driver is in an angry state, and the system needs to intervene, including enabling the autopilot mode and taking emergency reminder measures to ensure driving safety.

8. The large-model-based digital human emotion recognition and feedback system according to claim 1, characterized in that: The feedback generation module includes an emotion response generation unit, a driving mode optimization unit and a multi-modal feedback control unit; The emotion response generation unit is used to embed a generative large model, generate personalized feedback content according to the emotion state, combine urban driving or highway driving, adjust the form and intensity of the feedback content, and support diverse feedback forms, including audio reminders, text suggestions or visual guidance; The driving mode optimization unit is used to optimize the driving mode of the vehicle by analyzing the emotion state and driving scenario. For the fatigue state, it is recommended to switch to the autopilot mode or reduce the driving load. For the angry state, it reduces the vehicle response sensitivity to reduce dangerous operations. For the normal state, it maintains the existing driving mode or provides entertainment content recommendations; The multimodal feedback control unit is used to dynamically adjust the intensity and type of feedback forms according to the driver's current state, adaptively adjust the proportion and frequency of visual, auditory or tactile feedback, enhance the priority and intensity of multimodal feedback in high-risk emotional states, and reduce feedback intensity in normal states to reduce driver interference.

9. The large-model-based digital human emotion recognition and feedback system according to claim 1, characterized in that: The system optimization module includes a dynamic parameter adjustment unit; The dynamic parameter adjustment unit is used to dynamically adjust the weight and bias of the emotion recognition model according to the feedback effect evaluation results, fine-tune the output strategy of the generative model, ensure that the feedback content is more in line with actual needs, use the gradient descent optimization algorithm to improve the overall accuracy of the model, collect long-term driving data, form a personalized driver emotion and behavior database, use new data to regularly update the model weight, maintain the adaptability and robustness of the system, support model federated learning under the data privacy protection mechanism, and optimize the overall system performance.

10. The digital human emotion recognition and feedback system based on a large model according to claim 1, characterized in that: The safety monitoring and warning module includes a dangerous state detection unit and an emergency warning generation unit; The dangerous state detection unit is used to evaluate driving risks in real time by integrating emotional state and behavioral data, detect high-risk emotional states, including extreme fatigue or strong anger, and combine driving behavior data, including lane deviation and braking reaction time, to further evaluate the danger level and output potential dangerous state signals to provide a basis for subsequent early warning procedures; The emergency warning generation unit is used to initiate an automated emergency response when a high-risk state is detected, issue a high-priority alarm, including audio reminders, seat vibrations or visual guidance, automatically adjust the vehicle status, including slowing down or switching to autonomous driving mode, record emergency event data, and support post-event analysis.

Images