Head-up display adjustment method, system, device and medium based on emotion recognition

Abstract

The present application relates to a kind of based on mood recognition's head-up display adjustment method, system, equipment and medium. Method includes: obtaining the facial video frame sequence of driver and corresponding physiological signal sequence;The facial video frame sequence is input to expression recognition model, and the expression feature sequence of the facial video frame sequence is extracted;According to physiological feature extraction model, physiological feature sequence is extracted from the physiological signal sequence;The expression feature sequence and the physiological feature sequence are input to multimodal emotion recognition model, and generate emotion category;Based on the emotion category, according to the preset adjustment strategy, the content of head-up display on vehicle is dynamically adjusted.The present application dynamically adjusts head-up display content according to the emotion of driver, and improves driving safety.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, and particularly relates to a head-up display method and system based on emotion recognition, a vehicle terminal and a medium. BACKGROUND

[0002] With the development of science and technology, head-up display (HUD) technology has been widely used in vehicles. The head-up display technology projects important driving information such as speed and navigation onto the windshield in front of the driver, allowing the driver to view the key information without turning or looking down, thereby improving driving safety. However, in the actual driving process, the driver needs to adjust the driving strategy according to the changing road conditions and environmental conditions. However, the traditional head-up display system can only provide preset fixed information display, which makes it difficult for the driver to obtain the most critical information at critical moments, thereby affecting driving safety. Therefore, it is necessary to provide a head-up display adjustment method, system, device and medium based on emotion recognition. SUMMARY

[0003] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a head-up display adjustment method, system, device and medium based on emotion recognition, which improves the problem that the prior art cannot adjust the head-up display content in real time based on the driver's emotion.

[0004] To achieve the above-mentioned purpose and other related purposes, the present application provides a head-up display adjustment method based on emotion recognition, comprising: acquiring a sequence of facial video frames of a driver and a corresponding sequence of physiological signals; inputting the sequence of facial video frames into an expression recognition model to extract a sequence of expression features of the sequence of facial video frames; extracting a sequence of physiological features from the sequence of physiological signals according to a physiological feature extraction model; inputting the sequence of expression features and the sequence of physiological features into a multi-modal emotion recognition model to generate an emotion category; and dynamically adjusting the content of the head-up display on the vehicle based on the emotion category and according to a preset adjustment strategy.

[0005] In an embodiment of the present application, the inputting the sequence of facial video frames into the expression recognition model and extracting a sequence of expression features of the sequence of facial video frames comprises: inputting the sequence of facial video frames into a face detection module of the expression recognition model, detecting and cropping each corresponding facial image from the sequence of facial video frames based on a face detection algorithm to generate a sequence of facial images; inputting the sequence of facial images into a key point detection module of the expression recognition model, locating facial key points corresponding to each facial image based on a facial key point detection method to obtain a sequence of facial key points corresponding to the sequence of facial images; and inputting the sequence of facial key points and the sequence of facial images into a detection module of the expression recognition model, aligning each facial image based on each facial key point and extracting corresponding expression features from each aligned facial image to generate a sequence of expression features.

[0006] In an embodiment of the present application, the detection module is a convolutional neural network, and the inputting the sequence of facial key points and the sequence of facial images into the detection module of the expression recognition model, aligning each facial image based on each facial key point and extracting corresponding expression features from each aligned facial image to generate a sequence of expression features comprises: for each facial image, aligning the facial image based on the corresponding facial key point; extracting local features and corresponding global features of the facial image in layers, and fusing the local features and the global features extracted in the last layer to obtain expression features of the facial image; and arranging the expression features of each facial image in sequence to generate a sequence of expression features.

[0007] In an embodiment of the present application, the extracting a sequence of physiological features from the sequence of physiological signals according to the physiological feature extraction model comprises: performing fast Fourier transform on each cardiac cycle signal of the sequence of physiological signals to generate frequency domain features of each cardiac cycle signal; calculating a mean value of each skin conductance of the sequence of physiological signals to obtain a skin conductance feature; combining each frequency domain feature and the mean conductance feature into a comprehensive feature sequence; inputting the comprehensive feature sequence into the physiological feature extraction model, classifying the comprehensive feature sequence based on each weight and corresponding bias learned in advance to determine a sequence of physiological features corresponding to the comprehensive feature sequence; and the physiological feature extraction model is a support vector machine.

[0008] In an embodiment of the present application, the inputting the expression feature sequence and the physiological feature sequence into the multi-modal emotion recognition model to generate an emotion category comprises: inputting the expression feature sequence into an expression feature module of the multi-modal emotion recognition model to generate deep expression features according to the spatiotemporal correlation of each expression feature; wherein the expression feature module is a three-dimensional convolutional neural network; inputting the physiological feature sequence into a physiological feature module of the multi-modal emotion recognition model to generate deep physiological features according to the time dependence of each physiological feature; wherein the physiological feature module is an LSTM; inputting the deep expression features and the deep physiological features into a feature fusion module of the multi-modal emotion recognition model for fusion to generate fusion features; and inputting the fusion features into an output module of the multi-modal emotion recognition model to map the fusion features to preset emotion categories and determine a final emotion category based on the confidence of each emotion category after mapping.

[0009] In an embodiment of the present application, the adjusting the content of the head-up display on the vehicle according to the preset adjustment strategy based on the emotion category comprises: if the emotion category is a positive emotion, keeping the content of the head-up display unchanged; and if the emotion category is a negative emotion, hiding preset non-key information in the head-up display and adjusting the brightness and contrast of the head-up display.

[0010] In an embodiment of the present application, after the dynamic adjustment of the content of the head-up display on the vehicle according to the preset adjustment strategy based on the emotion category, the method further comprises: monitoring whether the emotion category changes in real time, and gradually presenting each non-key information to the interface of the head-up display according to a preset priority order after the emotion category changes from a negative emotion to a positive emotion.

[0011] In an embodiment of the present application, a head-up display adjustment system based on emotion recognition is also provided, which comprises: a data acquisition module configured to acquire a sequence of facial video frames of a driver and a corresponding sequence of physiological signals; an expression recognition module configured to input the sequence of facial video frames into an expression recognition model to extract an expression feature sequence of the sequence of facial video frames; a physiological feature recognition module configured to extract a physiological feature sequence from the sequence of physiological signals according to a physiological feature extraction model; a multi-modal fusion module configured to input the expression feature sequence and the physiological feature sequence into a multi-modal emotion recognition model to generate an emotion category; and an adjustment presentation module configured to dynamically adjust the content of a head-up display on a vehicle according to a preset adjustment strategy based on the emotion category.

[0012] In an embodiment of the present application, an electronic device is also provided, comprising: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement any of the above-mentioned emotion recognition-based head-up display adjustment methods.

[0013] In an embodiment of the present application, a computer-readable storage medium is also provided, which stores a computer program, which, when executed by a processor of a computer, causes the computer to execute any of the above-mentioned emotion recognition-based head-up display adjustment methods.

[0014] As described above, the emotion recognition-based head-up display adjustment method, system, device and medium of the present application have the following beneficial effects: the immune video frame sequence and the physiological signal sequence of the driver are obtained at the same time, and the expression features and the physiological features are extracted respectively, the two kinds of features are input into the multi-modal emotion recognition model, and the multi-dimensional information is combined, so that the emotional state of the driver can be more accurately recognized, and the content of the head-up display is dynamically adjusted according to the emotional state. By recognizing different emotional states, the content of the head-up display can be quickly adjusted when the emotional category of the driver changes, the matching degree of the environment in the cockpit and the emotional state of the driver is improved, so that a safer, more comfortable and more intelligent driving environment is provided for the driver, and the driving safety is improved. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 A flowchart of an emotion recognition-based head-up display adjustment method provided by an embodiment of the present application is shown.

[0016] Figure 2 A structure block diagram of an emotion recognition-based head-up display adjustment system provided by an embodiment of the present application is shown.

[0017] Figure 3 A structure diagram of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0018] The embodiments of the present application will be described in detail below with reference to specific examples. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the specification. The present application can also be implemented or applied by different specific embodiments, and the details in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0019] It is to be understood that the figures provided in the following embodiments are only schematically illustrating the basic concept of the present application, and only the components related to the present application are shown in the figures, not the number, shape and size of the components when actually implemented. The actual implementation of each component may be a random change, and the component layout pattern may be more complex.

[0020] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the application, however, it is obvious to those skilled in the art that the embodiments of the application can be implemented without these specific details, and in other embodiments, the known structures and devices are shown in the form of block diagrams rather than in the form of details, to avoid making the embodiments of the application difficult to understand.

[0021] The inventors found that during driving, the driver needs to adjust the driving strategy according to different road conditions and environmental conditions. The traditional head-up display system can usually only provide fixed information display, and cannot be optimized according to the changes of the actual driving scene. This may cause the driver to fail to obtain the most critical information at critical moments, thereby affecting driving safety.

[0022] In view of the above, the present application provides a head-up display adjustment method based on emotion recognition, which simultaneously obtains a driver's immune video frame sequence and a physiological signal sequence, and extracts facial features and physiological features, respectively, and inputs the two features into a multi-modal emotion recognition model, combining this multi-dimensional information, so as to more accurately identify the emotional state of the driver, and dynamically adjust the content presented by the head-up display according to the emotional state. By recognizing different emotional states, the content of the head-up display can be quickly adjusted when the driver's emotional category changes, improving the matching degree of the environment in the cockpit and the driver's emotions, thereby providing a safer, more comfortable and intelligent driving environment for the driver, and improving driving safety.

[0023] Please refer to Figure 1 The head-up display adjustment method based on emotion recognition includes the following steps:

[0024] S1, obtaining a driver's facial video frame sequence and a corresponding physiological signal sequence.

[0025] During driving, the physiological signals of the driver can be monitored in real time through sensors installed on the steering wheel, wherein the physiological signals refer to physiological activity indicators of the driver, through which the emotional state of the driver can be reflected, and the physiological signals include but are not limited to heart rate, skin conductivity, etc., and correspondingly, the sensors include but are not limited to heart rate sensors, skin conductivity sensors, etc., as long as the physiological signals of the driver can be obtained. Each physiological signal obtained within a preset first sampling duration is sequentially arranged to form a physiological signal sequence, and each physiological signal in the sequence represents a physiological state at a specific time point. It should be noted that each physiological signal is provided with a time stamp, so as to realize time alignment with the facial video frames, so as to realize subsequent multi-modal emotional observation and recognition of the driver at the same time point. Through the high-resolution camera integrated with an infrared sensor installed in the vehicle, video stream data capable of representing the facial state change of the driver can be captured. The video stream collected by the camera is sampled at a preset frame rate to form a facial video frame sequence within a preset first sampling duration, and each facial video frame represents the facial state of the driver at a specific time point.

[0026] S2, inputting the facial video frame sequence into an expression recognition model to extract an expression feature sequence of the facial video frame sequence.

[0027] The facial video frame sequence is input into the trained expression recognition model, and the model analyzes the facial features in each frame of image to identify the features related to the expression in the frame of image. After all the video frames are processed, the expression features of all the frames are summarized to generate an expression feature sequence, which represents the expression change of the driver within the first sampling duration.

[0028] In an embodiment of the present application, the inputting the facial video frame sequence into the expression recognition model to extract the expression feature sequence of the facial video frame sequence comprises:

[0029] The facial video frame sequence is input into the facial detection module of the expression recognition model, and based on the facial detection algorithm, each corresponding facial image is detected and cropped from the facial video frame sequence to generate a facial image sequence;

[0030] The facial image sequence is input into the key point detection module of the expression recognition model, and based on the facial key point detection method, the facial key points corresponding to each facial image are located to obtain a facial key point sequence corresponding to the facial image sequence;

[0031] The facial key point sequence and the facial image sequence are input into the detection module of the expression recognition model, based on each facial key point, the corresponding facial image is aligned and processed, and the corresponding expression features are extracted from each aligned facial image to generate an expression feature sequence.

[0032] The expression recognition model includes three cascaded networks of a face detection module, a key point detection module and a detection module, the output of the previous module is taken as the input of the next module, and the final detection module generates an expression feature sequence, which is used for subsequent combination with a physiological feature sequence to realize multi-modal emotion analysis and recognition of the driver. Specifically, the face video frame is input into the face detection module of the trained expression recognition model, the face region in the video frame is located based on the face detection algorithm, a face detection frame is generated to frame the face of the driver, and the face is cropped from the entire video frame to remove the background and noise effects, so as to ensure that the subsequent processing is only focused on the face features. The cropped face region of each frame is arranged in the order of the video frame to form a face image sequence. The type of the face detection module includes but is not limited to a Haar cascade classifier, an SSD, a YOLO series network and the like, as long as the face region of the driver can be effectively cropped, and the person skilled in the art can select adaptively based on the actual needs, which is not limited herein. The generated face image sequence is input into the key point detection module of the expression recognition model, and for each input face image: a series of face key points in the image are located by analyzing the region features of the face image, and the key points are sequentially arranged to form a face key point combination of the image. The face key point combinations of the face regions are arranged in the order of the face image to form a face key point sequence. The type of the key point detection module includes but is not limited to OpenPose and Dlib face key point detection. The face key point sequence and the face image sequence are input into the detection module of the expression recognition model, the face image is aligned according to the key points to improve the angle and position difference of the face posture, and the expression features are extracted from each aligned face image. The expression features of all face images are arranged in the order of the face image to form an expression feature sequence. The type of the detection module includes but is not limited to a convolutional neural network, ResNet and MTCNN. In the present application, the face image is aligned according to the face key points, which can improve the image error caused by the change of the head position or angle, and ensure that the face features remain consistent in each frame. The extracted expression features are more accurate, the accuracy of expression recognition is effectively improved, and the face expression transformation of the driver in different postures can be recognized, and the robustness of the model is improved.

[0033] In an embodiment of the present application, the detection module is a convolutional neural network, the face key point sequence and the face image sequence are input into the detection module of the expression recognition model, the corresponding face image is aligned based on each face key point, the corresponding expression feature is extracted from each aligned face image, and an expression feature sequence is generated, including:

[0034] For each face image:

[0035] aligning the face image based on the corresponding face key points;

[0036] extracting local features and corresponding global features of the face image in a hierarchical manner, and fusing the local features and global features extracted in the last layer to obtain expression features of the face image;

[0037] sequentially arranging the expression features of each face image to generate an expression feature sequence.

[0038] For each face image, the face feature in the face image is rotated, scaled, translated, etc. to align with the corresponding face key points, thereby obtaining a standardized face image, and realizing the preprocessing of the image. Each standardized face image is input into a convolutional neural network, and the local features (such as edge, texture features, etc.) in the image are extracted through the front several layers of the convolutional neural network, which are used to represent the micro-expression differences (such as mouth up, eye opening, etc.) in the image. Through the last several layers of the convolutional neural network, the local features extracted in the front are gradually fused, and the receptive field of convolution is increased, thereby extracting the global features in the image, which are used to represent the overall expression features (such as smiling, angry, etc.) in the image. The global features and local features are input into the fully connected layer of the convolutional neural network, so that the two kinds of features are fused to obtain the expression features of the standardized face image. The expression features of all standardized face images are arranged in the order of the corresponding standardized face images to form an expression feature sequence.

[0039] S3, extracting a physiological feature sequence from the physiological signal sequence according to a physiological feature extraction model.

[0040] Through the physiological feature extraction model, the key physiological features such as heart rate variability or skin conductance change are extracted from the input physiological signal sequence (such as heart rate, skin conductance, etc.), and a time-sequentially arranged physiological feature sequence is generated. The sequence is used to represent the physiological or microphysiological state changes of the driver, so as to analyze the emotional state of the driver subsequently. The types of the physiological feature extraction model include but are not limited to support vector machine, random forest, deep learning model, etc.

[0041] In an embodiment of the present application, the physiological feature sequence is extracted from the physiological signal sequence according to the physiological feature extraction model, comprising:

[0042] performing fast Fourier transform on each cardiac cycle signal of the physiological signal sequence to generate frequency domain features of each cardiac cycle signal;

[0043] calculating the mean value of each skin conductance of the physiological signal sequence to obtain a skin conductance feature;

[0044] combining each of the frequency domain features and the average conductance feature into a comprehensive feature sequence;

[0045] inputting the comprehensive feature sequence into the physiological feature extraction model, classifying the comprehensive feature sequence based on each weight and corresponding bias learned in advance, and determining a physiological feature sequence corresponding to the comprehensive feature sequence; wherein the physiological feature extraction model is a support vector machine.

[0046] The physiological feature sequence in this embodiment is sequentially composed of a plurality of physiological feature groups, and each physiological feature group includes a cardiac cycle signal and a skin conductance at the same sampling time. For each cardiac cycle signal in the physiological signal sequence: the cardiac cycle signal is subjected to fast Fourier transform, thereby converting it into a frequency domain signal, and generating a frequency domain feature of the cardiac cycle signal. Similarly, since the skin conductance is related to the emotion, for all skin conductances in the physiological signal sequence: the mean value of all skin conductances is calculated to obtain a skin conductance feature, which is used to represent the emotional state of the driver in this first sampling duration. It can be understood that the physiological signal can include both the cardiac cycle signal and the skin conductance, or only one of them, and can also include other signals that can represent the physiological state of the driver. Those skilled in the art can adaptively select based on actual detection needs, which are not limited herein. The skin conductance feature and all frequency domain features are combined to form a comprehensive feature sequence, which integrates dynamic heart rate features and static skin conductance features. Through multi-dimensional physiological feature information, the physiological state of the driver can be more accurately identified. The comprehensive feature sequence is input into the trained support vector machine, and each comprehensive feature is calculated by using the weights and bias parameters learned during training, and each comprehensive feature is mapped to a pre-set classification result. All classification results are summarized to form a physiological feature sequence in the order of each comprehensive feature. Each physiological feature represents a classification result corresponding to the corresponding comprehensive feature.

[0047] S4, inputting the expression feature sequence and the physiological feature sequence into a multi-modal emotion recognition model to generate an emotion category.

[0048] The expression feature sequence and the physiological feature sequence are combined and input into the multi-modal emotion recognition model. The model analyzes the emotional state of the driver by fusing facial expression features and physiological signal features (such as heart rate, skin conductance, etc.), and generates a corresponding emotion category (such as positive emotion, negative emotion, etc.) to reflect the current emotional state of the driver. The type of multi-modal emotion recognition model includes but is not limited to a three-dimensional convolutional neural network and an LSTM hybrid model, a multi-modal support vector machine, a double-flow convolutional neural network, etc.

[0049] In an embodiment of the present application, the inputting the expression feature sequence and the physiological feature sequence into a multi-modal emotion recognition model to generate an emotion category comprises:

[0050] The expression feature sequence is inputted into an expression feature module of the multi-modal emotion recognition model to generate deep expression features according to the spatio-temporal correlation of each expression feature; wherein the expression feature module is a three-dimensional convolutional neural network;

[0051] The physiological feature sequence is inputted into a physiological feature module of the multi-modal emotion recognition model to generate deep physiological features according to the time dependence of each physiological feature; wherein the physiological feature module is an LSTM;

[0052] The deep expression features and the deep physiological features are inputted into a feature fusion module of the multi-modal emotion recognition model for fusion to generate fusion features;

[0053] The fusion features are inputted into an output module of the multi-modal emotion recognition model, the fusion features are mapped to preset emotion categories, and a final emotion category is determined based on the confidence of the mapped emotion categories.

[0054] In this embodiment, the multi-modal emotion recognition model is a mixed model of a three-dimensional convolutional neural network and an LSTM. The three-dimensional convolutional neural network is used to capture spatial information and dynamic changes over time in the facial video frames, and to extract deep expression features. The LSTM is used to extract deep physiological features according to the change trend of physiological features at different time steps. Specifically, the expression feature sequence is input into the three-dimensional convolutional neural network of the multi-modal emotion recognition model. Each expression feature in the expression feature sequence is convoluted in the spatial dimension to extract the spatial features of the facial geometric features corresponding to each expression feature. These spatial features can represent the subtle changes in the driver's facial expressions (such as raised eyebrows and drooping corners of the mouth). In addition, each expression feature in the expression feature sequence is also convoluted in the time dimension to capture the dynamic changes of the expression in the entire expression feature sequence based on the changes between the previous and subsequent expression features, and to obtain the time features of the entire expression feature sequence. The spatial features and the time features are fused to obtain deep expression features. These features not only contain information about the static geometric structure of the face, but also reflect the dynamic changes of the expression over time, making the emotion recognition more accurate. For physiological features, the physiological feature sequence is input into the trained LSTM. The input gate, output gate, and forget gate mechanisms are used to gradually capture the long-term and short-term dependencies of the physiological features at each time step. Through these gating mechanisms, the LSTM can integrate the physiological features at the current time step with the historical features at the historical time steps, and finally generate deep physiological features. The deep expression features and the deep physiological features are input into a feature fusion module to perform feature fusion, and a fusion feature containing multi-dimensional information is generated. The fusion feature is input into an output module to map the fusion feature to the preset emotion categories (such as positive emotions and negative emotions), and the emotion category with the highest confidence is selected as the final emotion category to represent the current emotional state of the driver.

[0055] S5, based on the emotion category, dynamically adjusting the content of the head-up display on the vehicle according to a preset adjustment strategy.

[0056] According to different emotion categories, the corresponding display adjustment in the vehicle head-up display is triggered according to the preset adjustment strategy. For example, according to the emotion adjustment strategy, the information most relevant to the current emotional state of the driver is selected for display. For example, if the driver is in a state of tension, the HUD will reduce the display of non-driving critical information, such as incoming call prompts and surrounding landmark recommendations, and focus on displaying safety-related information, such as vehicle speed, driving risk warnings, and road target markers. When the driver returns to a calm state, the full-featured information will be displayed according to user settings and trigger conditions.

[0057] In an embodiment of the present application, the content of the head-up display on the vehicle is adjusted based on the emotion classification according to a preset adjustment strategy, which includes:

[0058] If the emotion category is positive emotion, the content of the head-up display is kept unchanged;

[0059] If the emotion category is negative emotion, preset non-critical information in the head-up display is hidden, and the brightness and contrast of the head-up display are adjusted.

[0060] After obtaining the current emotion category of the driver through the multi-modal emotion recognition model, the corresponding preset adjustment strategy is triggered according to the emotion classification result, and the corresponding head-up display adjustment strategy is selected for different emotion categories. Specifically, if the emotion category is positive emotion, it indicates that the driver's current emotion is stable and can drive normally, so the original display content on the head-up display can be kept unchanged, and the original brightness and contrast will not be adjusted. If the emotion category is negative emotion, in order to reduce the emotional pressure of the driver, the original non-critical information in the head-up display is hidden, and only the key information directly related to the current driving task is retained. In addition, in order to ensure that the driver can clearly and quickly obtain the required information and slow down the further fluctuation of emotion, the brightness and contrast of the head-up display are also adjusted to adapt to the driver's emotion. Exemplarily, when the driver is in the negative emotion of emotional excitement, the brightness and contrast are reduced to avoid excessive stimulation. In this way of adjusting the content of the head-up display according to the driver's emotion, the driver can easily obtain important information and reduce unnecessary interference and stimulation, thereby improving the driving safety. It should be noted that the output of the multi-modal emotion recognition model can include only two categories of emotions (i.e. positive emotion and negative emotion), or can include multiple categories of emotions, and various categories of emotions are classified as positive emotion or negative emotion according to the preset classification standard. Exemplarily, happy, relaxed, etc. are classified as positive emotion, and anxiety, tension, fatigue, etc. are classified as negative emotion.

[0061] In an embodiment of the present application, after the content of the head-up display on the vehicle is dynamically adjusted according to the preset adjustment strategy based on the emotional category, the method further comprises: monitoring in real time whether the emotional category changes, and after the emotional category changes from a negative emotion to a positive emotion, gradually presenting each non-critical information to the interface of the head-up display according to a preset priority order. Considering that the driver's emotion during driving may change, the emotional category of the driver is monitored in real time, and when the emotional category changes from a negative emotion to a positive emotion, each information in the HUD is gradually restored according to the preset priority order. For example, information with a higher priority (such as fuel consumption, temperature in the vehicle, etc.) is first presented in the HUD, and then information with a lower priority (such as entertainment information, etc.) is gradually presented. In this way, the driver's attention is not disturbed by a large amount of information suddenly appearing, thereby improving driving safety. Further, the method of the present application dynamically adjusts the presented content according to the emotional category based on a preset presentation strategy, so as to adjust the presentation mode of the information. For example, when the driver is in a negative emotion (such as a state of tension), the HUD presents information in a 2d plane, for example, a 2d guide icon in navigation information. When the driver recovers to a positive emotion (such as a state of calmness), the information is presented in a 3d form according to user settings and triggering conditions, for example, a 2d guide icon in navigation information is dynamically presented in the form of a 3d guide arrow. In addition, the content of the adjustment strategy can be changed in combination with the personal information and driving habits of the driver to ensure personalized adaptation. In addition, after identifying the current emotional category of the driver, the vehicle environment can be linked to dynamically change the lighting, music, etc. in the vehicle according to the emotion, so as to provide a better driving experience.

[0062] In summary, the present application has the following advantages: considering that when the driver is in emotional fluctuations, such as anger, anxiety or fatigue, he may have problems such as inattention, decreased judgment and prolonged reaction time, which increase the risk of traffic accidents. The present application identifies the emotional state of the driver and takes timely measures to adjust, thereby reducing the driving risk caused by emotional problems. Since different drivers may have different emotional reactions and needs when facing the same situation, the traditional HUD system fails to consider the emotional differences and personalized needs of the driver. The present application provides personalized display content and interactive mode by intelligently identifying the driver's emotions, improving the driving experience. Considering that there is a lack of real-time monitoring of the driver's emotions and rapid response in the prior art, the present application realizes rapid response and adjustment by real-time monitoring of the driver's emotional changes, improving the timeliness of the system. Since the traditional driving assistance system usually only warns the driver when he makes obvious mistakes, it ignores the potential impact of emotions on driving. The driving assistance system using the present method can actively identify emotions and take measures such as changing the HUD display content or the in-vehicle environment to prevent potential safety problems in advance. Further, considering that the environment in the cockpit (such as lighting, music, atmosphere) is usually static and does not adjust according to the driver's emotional changes. The present application adjusts the HUD display and other in-vehicle environmental factors to create a driving environment that matches the driver's emotions. In view of the problem that the amount, type and presentation of HUD display information may not be suitable for the current emotional state when the driver's emotions fluctuate, the present application adjusts the HUD display content to ensure that the information delivery matches the driver's emotional state, thereby improving the effectiveness and safety of the information. Therefore, the present application uses an efficient algorithm to ensure the real-time performance of emotion recognition and adjustment in order to quickly respond to the driver's emotional changes. In addition, the present application integrates the driving assistance system with the present method with other vehicle systems (such as navigation, entertainment, safety systems, etc.) to ensure seamless information flow and system compatibility. And through continuous learning of the driver's feedback and behavior, the emotional recognition and adjustment strategy is continuously optimized to improve the accuracy and adaptability of the final emotional recognition.

[0063] See Figure 2The head-up display adjustment system 100 based on emotion recognition comprises a data acquisition module 110, an expression recognition module 120, a physiological feature recognition module 130, a multi-modal fusion module 140 and an adjustment presentation module 150. The data acquisition module 110 is configured to acquire a sequence of facial video frames of a driver and a corresponding sequence of physiological signals. The expression recognition module 120 is configured to input the sequence of facial video frames into an expression recognition model to extract a sequence of expression features of the sequence of facial video frames. The physiological feature recognition module 130 is configured to extract a sequence of physiological features from the sequence of physiological signals according to a physiological feature extraction model. The multi-modal fusion module 140 is configured to input the sequence of expression features and the sequence of physiological features into a multi-modal emotion recognition model to generate an emotion category. The adjustment presentation module 150 is configured to dynamically adjust the content of the head-up display of a vehicle according to a preset adjustment strategy based on the emotion category.

[0064] The specific limitations of the head-up display adjustment system based on emotion recognition can refer to the limitations of the head-up display adjustment method based on emotion recognition described above, which will not be repeated here. Each module in the head-up display adjustment system based on emotion recognition can be realized by software, hardware and their combinations in whole or in part. Each module can be embedded in or independent of the processor in the computer device in hardware format, or stored in the memory of the computer device in software format, so that the processor can call the corresponding operations of each module.

[0065] It should be noted that, in order to highlight the innovative part of the present application, modules not closely related to solving the technical problems proposed in the present application are not introduced in this embodiment, but this does not mean that there are no other modules in this embodiment.

[0066] Please refer to Figure 3 The electronic device 1 can comprise a memory 12, a processor 13 and a bus, and can further comprise a computer program stored in the memory 12 and executable on the processor 13, such as a head-up display adjustment program based on emotion recognition.

[0067] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 12 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Furthermore, the memory 12 can also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 12 can not only be used to store application software and various types of data installed on the electronic device 1, such as codes for adjusting the head-up display based on emotion recognition, but can also be used to temporarily store data that has been output or is to be output.

[0068] In some embodiments, the processor 13 may be composed of an integrated circuit, such as a single packaged integrated circuit or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 1. It utilizes various interfaces and circuits to connect the various components of the entire electronic device 1. It executes or runs programs or modules stored in the memory 12 (such as a head-up display adjustment program based on emotion recognition) and calls data stored in the memory 12 to perform various functions of the electronic device 1 and process data.

[0069] The processor 13 executes the operating system and various installed applications of the electronic device 1. The processor 13 executes the applications to implement the steps in the above-mentioned head-up display adjustment method based on emotion recognition.

[0070] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into a data acquisition module 110, an expression recognition module 120, a physiological feature recognition module 130, a multimodal fusion module 140, and an adjustment and presentation module 150.

[0071] The integrated unit in the form of a software function module can be stored in a computer-readable storage medium, which can be non-volatile or volatile. The software function module is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute part of the functions of the head-up display adjustment method based on emotion recognition described in various embodiments of the present application.

[0072] In summary, the head-up display adjustment method, system, device and medium based on emotion recognition disclosed in the present application can intelligently identify the emotional adjustment of the driver to change the HUD display, so that the content of the HUD display can dynamically adapt to the current emotional state of the driver, and the personalization and real-time of the driving experience are improved. In addition, the current emotional state of the driver and other environmental factors in the vehicle are linked, and the lighting, music, etc. in the vehicle are dynamically changed according to the emotional state, which improves the matching degree of the environment in the cockpit and the driver's emotion, reduces dangerous events caused by the driver's emotional fluctuations, and provides a safer, more comfortable and intelligent driving environment for the driver. By monitoring and adjusting the driver's emotions in real time, driving errors caused by emotional problems are reduced, and driving safety is improved. According to the driver's emotions, a personalized driving environment is provided to enhance driving comfort and satisfaction. Using the vehicle assisted driving system described in the method, the system continuously learns and optimizes the ability to provide a more intelligent driving experience for the driver. Therefore, the present application effectively overcomes the various shortcomings in the prior art and has high industrial utilization value.

[0073] The above embodiments only exemplarily illustrate the principles and effects of the present application, and are not used to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by those skilled in the art without departing from the spirit and technical idea disclosed in the present application should be covered by the claims of the present application.

Claims

1. A head-up display adjustment method based on emotion recognition, characterized in that: The method comprises: Obtaining a driver's facial video frame sequence and a corresponding physiological signal sequence; Inputting the facial video frame sequence into an expression recognition model to extract an expression feature sequence of the facial video frame sequence; extracting a physiological feature sequence from the physiological signal sequence according to a physiological feature extraction model; Inputting the facial expression feature sequence and the physiological feature sequence into a multimodal emotion recognition model to generate an emotion category; Based on the emotion category, dynamically adjust the content of the head-up display on the vehicle according to a preset adjustment strategy; The adjusting of the content of the head-up display on the vehicle based on the emotion classification and according to a preset adjustment strategy includes: If the emotion category is positive, keeping the content of the head-up display unchanged; If the emotion category is a negative emotion, non-critical information preset in the head-up display is hidden, and the brightness and contrast of the head-up display are adjusted.

2. The head-up display adjustment method based on emotion recognition according to claim 1, characterized in that: The step of inputting the facial video frame sequence into an expression recognition model and extracting an expression feature sequence from the facial video frame sequence comprises: Inputting the facial video frames into the face detection module of the expression recognition model, detecting and cropping corresponding facial images from the facial video frame sequence based on a facial detection algorithm, and generating a facial image sequence; Inputting the facial image sequence into the key point detection module of the expression recognition model, locating the facial key points corresponding to each facial image based on a facial key point detection method, and obtaining a facial key point sequence corresponding to the facial image sequence; The facial key point sequence and the facial image sequence are input into the detection module of the expression recognition model, the corresponding facial images are aligned based on each facial key point, and the corresponding expression features are extracted from each aligned facial image to generate an expression feature sequence.

3. The head-up display adjustment method based on emotion recognition according to claim 2, characterized in that: The detection module is a convolutional neural network, which inputs the facial key point sequence and the facial image sequence into the detection module of the expression recognition model, aligns the corresponding facial images based on each facial key point, and extracts corresponding expression features from each aligned facial image to generate an expression feature sequence, including: For each face image: Performing alignment processing on the facial images based on corresponding facial key points; Extracting local features and corresponding global features of the facial image in layers, and fusing the local features and global features extracted in the last layer to obtain expression features of the facial image; Arrange the expression features of each facial image in sequence to generate an expression feature sequence.

4. The head-up display adjustment method based on emotion recognition according to claim 1, characterized in that: The step of extracting a physiological feature sequence from the physiological signal sequence based on a physiological feature extraction model includes: Performing fast Fourier transform on each cardiac cycle signal of the physiological signal sequence to generate frequency domain features of each cardiac cycle signal; Calculating the mean of each skin conductance of the physiological signal sequence to obtain a skin conductance feature; combining the frequency domain features and the average conductance features into a comprehensive feature sequence; The comprehensive feature sequence is input into the physiological feature extraction model, and the comprehensive feature sequence is classified based on the pre-learned weights and corresponding biases to determine the physiological feature sequence corresponding to the comprehensive feature sequence; wherein the physiological feature extraction model is a support vector machine.

5. The head-up display adjustment method based on emotion recognition according to claim 1, characterized in that: The step of inputting the facial expression feature sequence and the physiological feature sequence into a multimodal emotion recognition model to generate emotion categories includes: Inputting the expression feature sequence into the expression feature module of the multimodal emotion recognition model, and generating deep expression features based on the spatiotemporal correlation of each expression feature; wherein the expression feature module is a three-dimensional convolutional neural network; Inputting the physiological feature sequence into the physiological feature module of the multimodal emotion recognition model, and generating deep physiological features based on the time dependency of each physiological feature; wherein the physiological feature module is an LSTM; Inputting the deep expression feature and the deep physiological feature into the feature fusion module of the multimodal emotion recognition model for fusion to generate a fusion feature; The fused features are input into the output module of the multimodal emotion recognition model, the fused features are mapped to preset emotion categories, and the final emotion category is determined based on the confidence of each mapped emotion category.

6. The head-up display adjustment method based on emotion recognition according to claim 1, characterized in that: After dynamically adjusting the content of the head-up display on the vehicle based on the emotion category and according to a preset adjustment strategy, the method also includes: real-time monitoring of whether the emotion category changes, and after the emotion category changes from negative emotion to positive emotion, gradually presenting various non-critical information to the head-up display interface according to a preset priority order.

7. A head-up display adjustment system based on emotion recognition, characterized in that: The system comprises: A data acquisition module, used to acquire a driver's facial video frame sequence and a corresponding physiological signal sequence; An expression recognition module, configured to input the facial video frame sequence into an expression recognition model and extract an expression feature sequence from the facial video frame sequence; A physiological feature recognition module, configured to extract a physiological feature sequence from the physiological signal sequence based on a physiological feature extraction model; A multimodal fusion module, configured to input the facial expression feature sequence and the physiological feature sequence into a multimodal emotion recognition model to generate emotion categories; An adjustment and presentation module, configured to dynamically adjust the content of the head-up display on the vehicle based on the emotion category and a preset adjustment strategy; The adjusting of the content of the head-up display on the vehicle based on the emotion classification and according to a preset adjustment strategy includes: If the emotion category is positive, keeping the content of the head-up display unchanged; If the emotion category is a negative emotion, non-critical information preset in the head-up display is hidden, and the brightness and contrast of the head-up display are adjusted.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the vehicle-mounted terminal to implement the head-up display adjustment method based on emotion recognition as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the head-up display adjustment method based on emotion recognition according to any one of claims 1 to 6.

Images