Head-up display adjusting method, system and device based on emotion recognition and medium

Abstract

The invention relates to a head-up display adjusting method, system and device based on emotion recognition and a medium. The method comprises the following steps: acquiring a face video frame sequence of a driver and a physiological signal sequence corresponding to the face video frame sequence; inputting the face video frame sequence into an expression recognition model, and extracting an expression feature sequence of the face video frame sequence; extracting a physiological feature sequence from the physiological signal sequence according to a physiological feature extraction model; inputting the expression feature sequence and the physiological feature sequence into a multi-modal emotion recognition model to generate an emotion category; and dynamically adjusting the head-up display content on the vehicle according to a preset adjustment strategy based on the emotion category. The head-up display content is dynamically adjusted according to the emotion of the driver, and the driving safety is improved.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a head-up display method, system, vehicle-mounted terminal and medium based on emotion recognition. Background Art

[0002] With the development of science and technology, head-up display (HUD) technology has gradually been widely used in vehicles. Head-up display technology projects important driving information such as speed and navigation onto the windshield in front of the driver, allowing the driver to view key information without turning or lowering his head, thereby improving driving safety. In the actual driving process, the driver needs to adjust the driving strategy according to the ever-changing road conditions and environmental conditions. However, traditional head-up display systems can usually only provide preset fixed information displays, resulting in the driver being unable to obtain the most critical information at critical moments, thus affecting driving safety. Therefore, it is necessary to provide a head-up display adjustment method, system, device and medium based on emotion recognition. Summary of the invention

[0003] In view of the shortcomings of the prior art described above, an object of the present invention is to provide a head-up display adjustment method, system, device and medium based on emotion recognition, which improves the problem that the prior art cannot adjust the head-up display content in real time based on the driver's emotions.

[0004] To achieve the above-mentioned purpose and other related purposes, the present invention provides a head-up display adjustment method based on emotion recognition, comprising: obtaining a driver's facial video frame sequence and a corresponding physiological signal sequence; inputting the facial video frame sequence into an expression recognition model to extract an expression feature sequence of the facial video frame sequence; extracting a physiological feature sequence from the physiological signal sequence based on a physiological feature extraction model; inputting the expression feature sequence and the physiological feature sequence into a multimodal emotion recognition model to generate emotion categories; based on the emotion categories, dynamically adjusting the content of the head-up display on the vehicle according to a preset adjustment strategy.

[0005] In one embodiment of the present invention, the step of inputting the facial video frame sequence into an expression recognition model and extracting an expression feature sequence of the facial video frame sequence comprises: inputting the facial video frames into a face detection module of the expression recognition model, detecting and cropping corresponding facial images from the facial video frame sequence based on a face detection algorithm, and generating a facial image sequence; inputting the facial image sequence into a key point detection module of the expression recognition model, locating the facial key points corresponding to each facial image based on a facial key point detection method, and obtaining a facial key point sequence corresponding to the facial image sequence; inputting the facial key point sequence and the facial image sequence into the detection module of the expression recognition model, aligning the corresponding facial images based on the facial key points, and extracting corresponding expression features from the aligned facial images to generate an expression feature sequence.

[0006] In one embodiment of the present invention, the detection module is a convolutional neural network, and the facial key point sequence and the facial image sequence are input into the detection module of the expression recognition model, and the corresponding facial images are aligned based on each facial key point, and the corresponding expression features are extracted from each aligned facial image to generate an expression feature sequence, including: for each facial image: based on the corresponding facial key point, the facial image is aligned; the local features and the corresponding global features of the facial image are extracted in layers, and the local features and the global features extracted in the last layer are fused to obtain the expression features of the facial image; the expression features of each facial image are arranged in sequence to generate an expression feature sequence.

[0007] In one embodiment of the present invention, the physiological feature sequence is extracted from the physiological signal sequence based on the physiological feature extraction model, including: performing fast Fourier transform on each cardiac cycle signal of the physiological signal sequence to generate frequency domain features of each cardiac cycle signal; calculating the mean of each skin conductance of the physiological signal sequence to obtain skin conductance features; combining each of the frequency domain features and the average conductance features into a comprehensive feature sequence; inputting the comprehensive feature sequence into the physiological feature extraction model, classifying the comprehensive feature sequence based on pre-learned weights and corresponding biases, and determining the physiological feature sequence corresponding to the comprehensive feature sequence; wherein the physiological feature extraction model is a support vector machine.

[0008] In one embodiment of the present invention, the expression feature sequence and the physiological feature sequence are input into a multimodal emotion recognition model to generate an emotion category, including: inputting the expression feature sequence into an expression feature module of the multimodal emotion recognition model, and generating a deep expression feature based on the spatiotemporal correlation of each expression feature; wherein the expression feature module is a three-dimensional convolutional neural network; inputting the physiological feature sequence into a physiological feature module of the multimodal emotion recognition model, and generating a deep physiological feature based on the time dependency of each physiological feature; wherein the physiological feature module is an LSTM; inputting the deep expression feature and the deep physiological feature into a feature fusion module of the multimodal emotion recognition model for fusion to generate a fusion feature; inputting the fusion feature into an output module of the multimodal emotion recognition model, mapping the fusion feature to each preset emotion category, and determining the final emotion category based on the confidence of each mapped emotion category.

[0009] In one embodiment of the present invention, based on the emotion classification, the content of the head-up display on the vehicle is adjusted according to a preset adjustment strategy, including: if the emotion category is a positive emotion, the content of the head-up display remains unchanged; if the emotion category is a negative emotion, the preset non-critical information in the head-up display is hidden, and the brightness and contrast of the head-up display are adjusted.

[0010] In one embodiment of the present invention, after dynamically adjusting the content of the head-up display on the vehicle based on the emotion category according to a preset adjustment strategy, it also includes: real-time monitoring of whether the emotion category changes, and after the emotion category changes from negative emotion to positive emotion, gradually presenting each non-critical information to the head-up display interface according to a preset priority order.

[0011] In one embodiment of the present invention, a head-up display adjustment system based on emotion recognition is also provided, and the system includes: a data acquisition module, which is used to acquire a facial video frame sequence of a driver and a corresponding physiological signal sequence; an expression recognition module, which is used to input the facial video frame sequence into an expression recognition model to extract an expression feature sequence of the facial video frame sequence; a physiological feature recognition module, which is used to extract a physiological feature sequence from the physiological signal sequence based on a physiological feature extraction model; a multimodal fusion module, which is used to input the expression feature sequence and the physiological feature sequence into a multimodal emotion recognition model to generate an emotion category; and an adjustment presentation module, which is used to dynamically adjust the content of the head-up display on the vehicle based on the emotion category and a preset adjustment strategy.

[0012] In one embodiment of the present invention, an electronic device is also provided, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements any of the above-mentioned head-up display adjustment methods based on emotion recognition.

[0013] In one embodiment of the present invention, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes any of the above-mentioned head-up display adjustment methods based on emotion recognition.

[0014] As described above, the head-up display adjustment method, system, device and medium based on emotion recognition of the present invention have the following beneficial effects: the immune video frame sequence and physiological signal sequence of the driver are obtained at the same time, and the expression features and physiological features are extracted respectively, and these two features are input into the multimodal emotion recognition model, and this multi-dimensional information is combined so that the driver's emotional state can be more accurately identified, and the content presented by the head-up display can be dynamically adjusted according to the emotional state. By identifying different emotional states, the content of the head-up display can be quickly adjusted when the driver's emotional category changes, which improves the matching degree between the cockpit environment and the driver's emotions, thereby providing the driver with a safer, more comfortable and intelligent driving environment and improving driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic flow chart of a head-up display adjustment method based on emotion recognition provided by an embodiment of the present invention;

[0016] Figure 2 Shown is a structural block diagram of a head-up display adjustment system based on emotion recognition provided by an embodiment of the present invention;

[0017] Figure 3 Shown is a structural schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0019] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0020] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0021] The inventors found that during driving, drivers need to adjust their driving strategies according to different road conditions and environmental conditions. Traditional head-up display systems can usually only provide fixed information display and cannot be optimized according to changes in actual driving scenarios. This may cause the driver to be unable to obtain the most critical information at critical moments, thus affecting driving safety.

[0022] In view of the above situation, the present invention provides a head-up display adjustment method based on emotion recognition, which simultaneously obtains the driver's immune video frame sequence and physiological signal sequence, and extracts expression features and physiological features respectively, and inputs these two features into a multimodal emotion recognition model, combining this multi-dimensional information, so as to more accurately identify the driver's emotional state, and dynamically adjust the content presented by the head-up display according to the emotional state. By identifying different emotional states, the content of the head-up display can be quickly adjusted when the driver's emotional category changes, improving the matching degree between the cockpit environment and the driver's emotions, thereby providing the driver with a safer, more comfortable and intelligent driving environment, and improving driving safety.

[0023] See also Figure 1 ,The head-up display adjustment method based on emotion recognition includes the following steps:

[0024] S1. Obtain a driver's facial video frame sequence and a corresponding physiological signal sequence.

[0025] During driving, the driver's physiological signals can be monitored in real time by sensors installed on the steering wheel, wherein the physiological signals refer to the driver's physiological activity indicators, which can reflect the driver's emotional state. The physiological signals include but are not limited to heart rate, skin conductivity, etc. Correspondingly, the sensors include but are not limited to heart rate sensors, skin conductivity sensors, etc., as long as the driver's physiological signals can be obtained. The physiological signals obtained within the preset first sampling time are arranged in sequence to form a physiological signal sequence, wherein each physiological signal represents the physiological state at a specific time point. It should be noted that each physiological signal is timestamped, so as to achieve time alignment with the facial video frame, so as to subsequently realize multimodal emotional observation and recognition of the driver at the same time point. A high-resolution camera with an integrated infrared sensor installed inside the vehicle is used to capture video stream data that can represent the changes in the driver's facial state. The video stream collected by the camera is sampled according to a preset frame rate, and a facial video frame sequence is formed within the preset first sampling time, wherein each facial video frame represents the driver's facial state at a specific time point.

[0026] S2. Inputting the facial video frame sequence into an expression recognition model to extract an expression feature sequence of the facial video frame sequence.

[0027] The facial video frame sequence is input into the trained expression recognition model. The model analyzes the facial features in each frame and identifies the features related to the expression in the frame. After all the video frames are processed, the expression features of all frames are summarized to generate an expression feature sequence, which represents the changes in the driver's expression during the first sampling time.

[0028] In one embodiment of the present invention, the step of inputting the facial video frame sequence into an expression recognition model and extracting an expression feature sequence of the facial video frame sequence includes:

[0029] Inputting the facial video frames into the face detection module of the expression recognition model, detecting and cropping corresponding facial images from the facial video frame sequence based on a facial detection algorithm, and generating a facial image sequence;

[0030] Inputting the facial image sequence into the key point detection module of the expression recognition model, locating the facial key points corresponding to each facial image based on the facial key point detection method, and obtaining the facial key point sequence corresponding to the facial image sequence;

[0031] The facial key point sequence and the facial image sequence are input into the detection module of the expression recognition model, the corresponding facial images are aligned based on each facial key point, and the corresponding expression features are extracted from each aligned facial image to generate an expression feature sequence.

[0032] The expression recognition model includes three cascade networks: a face detection module, a key point detection module, and a detection module. The output of the previous module is used as the input of the next module. The final detection module will generate an expression feature sequence, which is used to combine with the physiological feature sequence to realize the multimodal emotion analysis and recognition of the driver. Specifically, the facial video frame is input into the face detection module of the trained expression recognition model. Based on the face detection algorithm, the facial area in the video frame is located, and a face detection frame is generated to frame the driver's face, and it is cropped out from the entire video frame to remove the background and noise effects, thereby ensuring that subsequent processing focuses only on facial features. Each frame of the cropped facial area is arranged in the order of the video frames to form a facial image sequence, wherein the type of face detection module includes but is not limited to Haar cascade classifier, SSD, YOLO series network, etc., as long as the driver's facial area can be effectively cropped, and those skilled in the art can make adaptive choices based on actual needs, and are not limited here. The generated facial image sequence is input into the key point detection module of the expression recognition model. Based on the facial key point detection algorithm, for each input facial image: by analyzing the regional features of the facial image, a series of facial key points in the image are located, and these key points are arranged in sequence to form a facial key point combination of the image. The facial key point combinations of each facial area are arranged according to the order of the facial images to form a facial key point sequence. Among them, the types of key point detection modules include but are not limited to OpenPose, Dlib facial key point detection, etc. The facial key point sequence and the facial image sequence are input into the detection module of the expression recognition model, and the facial images are aligned according to the key points to improve the angle and position differences of the facial postures, and the expression features are extracted from each aligned facial image. The expression features of all facial images are arranged in the order of the facial images to form an expression feature sequence, wherein the types of detection modules include but are not limited to convolutional neural networks, ResNet, MTCNN, etc. In the present invention, the facial images are aligned by facial key points, which can improve the image error caused by the change of head position or angle, and ensure that the facial features are consistent in each frame. This makes the extracted expression features more accurate, effectively improving the accuracy of expression recognition, and can identify the changes in facial expressions of the driver in different postures, improving the robustness of the model.

[0033] In one embodiment of the present invention, the detection module is a convolutional neural network, and the facial key point sequence and the facial image sequence are input into the detection module of the expression recognition model, the corresponding facial images are aligned based on each facial key point, and the corresponding expression features are extracted from each aligned facial image to generate an expression feature sequence, including:

[0034] For each face image:

[0035] Based on the corresponding facial key points, the facial images are aligned;

[0036] Extracting local features and corresponding global features of the facial image in layers, and fusing the local features and global features extracted in the last layer to obtain expression features of the facial image;

[0037] Arrange the expression features of each facial image in order to generate an expression feature sequence.

[0038] For each facial image, the facial features in the facial image are rotated, scaled, translated, etc. to align with the corresponding facial key points, thereby obtaining a standardized facial image and realizing image preprocessing. Each standardized facial image is input into a convolutional neural network, and local features (such as edges, texture features, etc.) in the image are extracted through the first several layers of the convolutional neural network to characterize the slight differences in expression in the image (such as the corners of the mouth rising, the eyes opening and closing, etc.). Through the latter layers of the convolutional neural network, the local features extracted in the front are gradually integrated to increase the receptive field of the convolution, thereby extracting the global features in the image to characterize the overall expression features in the image (such as smiling, angry, etc.). The global features and local features are input into the fully connected layer of the convolutional neural network, and the two features are integrated to obtain the expression features of the standardized facial image. The expression features of all standardized facial images are arranged in the order of the corresponding standardized facial images to form an expression feature sequence.

[0039] S3. Extracting a physiological feature sequence from the physiological signal sequence according to a physiological feature extraction model.

[0040] Through the physiological feature extraction model, key physiological features, such as heart rate variability or skin conductance changes, are extracted from the input physiological signal sequence (such as heart rate, skin conductivity, etc.), generating a physiological feature sequence arranged in chronological order. This sequence is used to characterize the changes in the driver's physiological or microphysiological state, so as to subsequently analyze the driver's emotional state. Among them, the types of physiological feature extraction models include but are not limited to support vector machines, random forests, deep learning models, etc.

[0041] In one embodiment of the present invention, extracting a physiological feature sequence from the physiological signal sequence based on a physiological feature extraction model includes:

[0042] Performing fast Fourier transform on each cardiac cycle signal of the physiological signal sequence to generate frequency domain features of each cardiac cycle signal;

[0043] Calculating the mean of each skin conductance of the physiological signal sequence to obtain a skin conductance feature;

[0044] Combining each of the frequency domain features and the average conductance features into a comprehensive feature sequence;

[0045] The comprehensive feature sequence is input into the physiological feature extraction model, the comprehensive feature sequence is classified based on the pre-learned weights and corresponding biases, and the physiological feature sequence corresponding to the comprehensive feature sequence is determined; wherein the physiological feature extraction model is a support vector machine.

[0046] The physiological feature sequence in this embodiment is composed of multiple physiological feature groups in sequence, and each physiological feature group includes a cardiac cycle signal and skin conductance at the same sampling time. For each cardiac cycle signal in the physiological signal sequence: the cardiac cycle signal is subjected to a fast Fourier transform, thereby converting it into a frequency domain signal, and generating the frequency domain feature of the cardiac cycle signal. Similarly, since skin conductance is related to emotion, for all skin conductances in the physiological signal sequence: the mean of all skin conductances is calculated to obtain the skin conductance feature, which is used to characterize the emotional state of the driver during this first sampling time. It can be understood that the physiological signal can include both the cardiac cycle signal and the skin conductance, or only one of them, or other signals that can characterize the physiological state of the driver. Those skilled in the art can make adaptive choices based on actual detection needs, and are not limited here. The skin conductance feature and all the frequency domain features are combined to form a comprehensive feature sequence, which integrates the dynamic heart rate feature and the static skin conductance feature. Through multi-dimensional physiological feature information, the driver's physiological state can be more accurately identified. The comprehensive feature sequence is input into the trained support vector machine. The weights and bias parameters learned during training are used to calculate each comprehensive feature input, and each comprehensive feature is mapped to a pre-set classification result. All classification results are summarized to form a physiological feature sequence in the order of each comprehensive feature. Each physiological feature represents a classification result corresponding to the corresponding comprehensive feature.

[0047] S4. Inputting the facial expression feature sequence and the physiological feature sequence into a multimodal emotion recognition model to generate an emotion category.

[0048] The expression feature sequence is combined with the physiological feature sequence and input into the multimodal emotion recognition model. The model comprehensively analyzes the driver's emotional state by integrating facial expression features and physiological signal features (such as heart rate, skin conductance, etc.), and generates corresponding emotion categories (such as positive emotions, negative emotions, etc.) to reflect the driver's current emotional state. Among them, the types of multimodal emotion recognition models include but are not limited to three-dimensional convolutional neural networks and LSTM hybrid models, multimodal support vector machines, two-stream convolutional neural networks, etc.

[0049] In one embodiment of the present invention, the step of inputting the expression feature sequence and the physiological feature sequence into a multimodal emotion recognition model to generate an emotion category includes:

[0050] Inputting the expression feature sequence into the expression feature module of the multimodal emotion recognition model, generating deep expression features according to the spatiotemporal correlation of each expression feature; wherein the expression feature module is a three-dimensional convolutional neural network;

[0051] Inputting the physiological feature sequence into the physiological feature module of the multimodal emotion recognition model, and generating deep physiological features according to the time dependency of each physiological feature; wherein the physiological feature module is LSTM;

[0052] Inputting the deep expression feature and the deep physiological feature into the feature fusion module of the multimodal emotion recognition model for fusion to generate a fusion feature;

[0053] The fused features are input into the output module of the multimodal emotion recognition model, the fused features are mapped to various preset emotion categories, and the final emotion category is determined based on the confidence of each mapped emotion category.

[0054] In this embodiment, the multimodal emotion recognition model is a three-dimensional convolutional neural network and a LSTM hybrid model. The three-dimensional convolutional neural network captures the spatial information and dynamic changes in time in the facial video frame, extracts deep expression features, and extracts deep physiological features according to the changing trend of physiological features at different time steps through LSTM. Specifically, the expression feature sequence is input into the three-dimensional convolutional neural network of the multimodal emotion recognition model, and the spatial features of the facial geometric features corresponding to each expression feature are extracted by convolution processing on each expression feature sequence in the spatial dimension. This spatial feature can characterize the subtle changes in the driver's facial expression (such as eyebrows rising, corners of the mouth drooping, etc.). In addition, by convolution processing on each expression feature in the expression feature sequence in the time dimension, based on the changes in the previous and next expression features, the dynamic changes of the expression in the entire expression feature sequence are captured, and the time features of the entire expression feature sequence are obtained. The above-mentioned spatial features and time features are fused to obtain deep expression features. This feature not only contains information about the static geometric structure of the face, but also reflects the dynamic changes of the expression over time, making emotion recognition more accurate. For physiological features: Input the physiological feature sequence into the trained LSTM, and gradually capture the long-term and short-term dependencies of the physiological features at each time step through the input gate, output gate, and forget gate mechanisms. Through these gating mechanisms, the LSTM can integrate the physiological features of the current time step with the historical features of the historical time step, and finally generate deep physiological features. Input the deep expression features and deep physiological features into the feature fusion module, fuse the two features, and generate a fusion feature containing multi-dimensional information. Input the fused features into the output module, map the fused features to the preset emotion categories (such as positive emotions, negative emotions, etc.), and select the emotion category with the highest confidence as the final emotion category to characterize the driver's current emotional state.

[0055] S5. Based on the emotion category and in accordance with a preset adjustment strategy, dynamically adjust the content of the head-up display on the vehicle.

[0056] According to different emotion categories, the corresponding display adjustment in the vehicle head-up display is triggered according to the preset adjustment strategy. Exemplarily, according to the emotion regulation strategy, the information most relevant to the driver's current emotional state is selected for display. For example, if the driver is in a nervous state, the HUD will reduce the display of non-driving critical information, such as incoming call prompts, surrounding landmark recommendations, etc., and focus on displaying safety-related information, such as vehicle speed, driving risk warnings, road target markings, etc. When the driver regains a calm state, the full amount of functional information will be displayed according to the user settings and trigger conditions.

[0057] In one embodiment of the present invention, adjusting the content of the head-up display on the vehicle based on the emotion classification and according to a preset adjustment strategy includes:

[0058] If the emotion category is positive emotion, keeping the content of the head-up display unchanged;

[0059] If the emotion category is a negative emotion, non-critical information preset in the head-up display is hidden, and the brightness and contrast of the head-up display are adjusted.

[0060] After obtaining the current emotion category of the driver through the multimodal emotion recognition model, the corresponding preset adjustment strategy is triggered according to the emotion classification result, and the corresponding head-up display adjustment strategy is selected for different emotion categories. Specifically, if the emotion category is positive emotion, it means that the driver is currently emotionally stable and can drive normally, so the original display content on the head-up display can be kept unchanged, and the original brightness and contrast will not be adjusted. If the emotion category is negative emotion, in order to reduce the emotional pressure of the driver, the original non-critical information in the head-up display is hidden, and only the critical information directly related to the current driving task is retained. In addition, in order to ensure that the driver can obtain the required information clearly and quickly and slow down further emotional fluctuations, the brightness and contrast of the head-up display will also be adjusted to adapt to the driver's emotions. For example, when the driver is in a negative emotion such as emotional excitement, the brightness and contrast are reduced to avoid overstimulation. By adjusting the content of the head-up display according to the driver's emotions, it is ensured that the driver can easily obtain important information, reduce unnecessary interference stimulation, and improve driving safety. It should be noted that the output of the multimodal emotion recognition model may include only two categories of emotions (i.e., positive emotions and negative emotions) or multiple categories of emotions, and the various categories of emotions are classified as positive emotions or negative emotions according to a preset classification standard. For example, happiness, relaxation, etc. are classified as positive emotions, and anxiety, tension, fatigue, etc. are classified as negative emotions.

[0061] In one embodiment of the present invention, after dynamically adjusting the content of the head-up display on the vehicle based on the emotion category and according to the preset adjustment strategy, it also includes: monitoring in real time whether the emotion category changes, and after the emotion category changes from negative emotion to positive emotion, gradually presenting various non-critical information to the head-up display interface according to the preset priority order. Considering that the driver's emotions may change during driving, the driver's emotion category will be monitored in real time to see if it changes, and when it changes from negative emotion to positive emotion, the various information in the HUD will be gradually restored according to the preset priority order. Exemplarily, the information with higher priority (such as fuel consumption, temperature in the car, etc.) is first presented in the HUD, and then the information with lower priority (such as entertainment information, etc.) is gradually displayed. By this way of gradually displaying according to priority, the driver's attention is prevented from being disturbed by a large amount of information that suddenly appears, so as to improve driving safety. Further, the method of the present invention will also dynamically adjust the presented content according to the emotion category and the preset presentation strategy to adjust the presentation mode of the information. Exemplarily, when the driver is in a negative emotion (such as a nervous state), the HUD will display information in a 2D plane presentation mode, such as a 2D guide icon in the navigation information. When the driver returns to a positive state (such as a calm state), the information will be presented in a 3D stereoscopic manner according to the user settings and trigger conditions. For example, the 2D guide icon in the navigation information will be dynamically presented in the form of a 3D guide arrow. In addition, the content of the adjustment strategy can be changed in combination with the driver's personal information and driving habits to ensure personalized adaptation. In addition, after identifying the driver's current emotional category, it can also be linked with the in-car environment to dynamically change the in-car lighting, music, etc. according to the emotion to provide a better driving experience.

[0062] In summary, the present invention has the following advantages: Considering that when the driver is emotionally fluctuating, such as angry, anxious or tired, he may have problems such as inattention, decreased judgment and prolonged reaction time, which all increase the risk of traffic accidents. The present invention reduces the driving risk caused by emotional problems by identifying the driver's emotional state and taking timely measures to adjust. Because different drivers may have different emotional reactions and needs when facing the same situation. The traditional HUD system fails to take into account the emotional differences and personalized needs of drivers. The present invention improves the driving experience by intelligently identifying the driver's emotions, providing personalized display content and interaction methods. Considering the lack of HUD presentation systems that monitor the driver's emotions in real time and respond quickly in the prior art, the present invention achieves rapid response and adjustment by real-time monitoring of the driver's emotional changes, thereby improving the timeliness of the system. Because traditional driving assistance systems usually issue warnings only when the driver makes obvious mistakes, they ignore the potential impact of emotions on driving. The driving assistance system using this method can actively identify emotions and take measures, such as changing the HUD display content or the in-vehicle environment, to prevent potential safety problems in advance. Furthermore, considering that the environment in the cockpit (such as lighting, music, atmosphere) is often static and will not be adjusted according to the driver's emotional changes. The present invention creates a driving environment that matches the driver's emotions by adjusting the HUD display and other in-vehicle environmental factors. In view of the problem that the amount, type and presentation of information displayed by the HUD may not be suitable for the current emotional state when the driver's emotions fluctuate, the present invention adjusts the HUD display content to ensure that the information transmission matches the driver's emotional state, thereby improving the effectiveness and safety of the information. Therefore, the present invention adopts an efficient algorithm to ensure the real-time nature of emotion recognition and regulation in order to quickly respond to the driver's emotional changes. In addition, the present invention also integrates the driving assistance system equipped with the method of the present invention with other vehicle-mounted systems (such as navigation, entertainment, safety systems, etc.) to ensure seamless information flow and system compatibility. And by continuously learning the driver's feedback and behavior, the emotion recognition and regulation strategies are continuously optimized to improve the accuracy and adaptability of the final emotion recognition.

[0063] See also Figure 2The head-up display adjustment system 100 based on emotion recognition includes: a data acquisition module 110, an expression recognition module 120, a physiological feature recognition module 130, a multimodal fusion module 140 and an adjustment presentation module 150. The data acquisition module 110 is used to obtain the driver's facial video frame sequence and the corresponding physiological signal sequence. The expression recognition module 120 is used to input the facial video frame sequence into the expression recognition model to extract the expression feature sequence of the facial video frame sequence. The physiological feature recognition module 130 is used to extract the physiological feature sequence from the physiological signal sequence based on the physiological feature extraction model. The multimodal fusion module 140 is used to input the expression feature sequence and the physiological feature sequence into the multimodal emotion recognition model to generate an emotion category. The adjustment presentation module 150 is used to dynamically adjust the content of the head-up display on the vehicle based on the emotion category and according to a preset adjustment strategy.

[0064] For the specific limitations of the head-up display adjustment system based on emotion recognition, please refer to the limitations of the head-up display adjustment method based on emotion recognition above, which will not be repeated here. Each module in the above-mentioned head-up display adjustment system based on emotion recognition can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware format, or can be stored in the memory of the computer device in software format, so that the processor can call the operations corresponding to the above modules.

[0065] It should be noted that, in order to highlight the innovative part of the present invention, the present embodiment does not introduce modules that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other modules in the present embodiment.

[0066] See also Figure 3 The electronic device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a head-up display adjustment program based on emotion recognition.

[0067] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 12 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 12 can also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 1, such as the code for adjusting the head-up display based on emotion recognition, but also can be used to temporarily store data that has been output or is to be output.

[0068] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 1, and uses various interfaces and lines to connect the various components of the entire electronic device 1, and executes or executes programs or modules stored in the memory 12 (such as a head-up display adjustment program based on emotion recognition, etc.), and calls the data stored in the memory 12 to execute various functions of the electronic device 1 and process data.

[0069] The processor 13 executes the operating system and various installed applications of the electronic device 1. The processor 13 executes the applications to implement the steps in the above-mentioned head-up display adjustment method based on emotion recognition.

[0070] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into a data acquisition module 110, an expression recognition module 120, a physiological feature recognition module 130, a multimodal fusion module 140, and an adjustment presentation module 150.

[0071] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium, and the computer-readable storage medium can be non-volatile or volatile. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to perform part of the functions of the head-up display adjustment method based on emotion recognition described in various embodiments of the present application.

[0072] In summary, the present invention discloses a head-up display adjustment method, system, device and medium based on emotion recognition, which adjusts the HUD display changes by intelligently identifying the driver's emotions, so that the content displayed by the HUD can dynamically adapt to the driver's current emotional state, thereby improving the personalization and real-time nature of the driving experience. In addition, the driver's current emotional state is linked to other environmental factors in the car, and the interior lighting, music, etc. are dynamically changed according to the emotional state, thereby improving the matching degree between the cockpit environment and the driver's emotions, reducing dangerous events caused by the driver's emotional fluctuations, and providing the driver with a safer, more comfortable and intelligent driving environment. By real-time monitoring and adjusting the driver's emotions, driving errors caused by emotional problems are reduced, and driving safety is improved. A personalized driving environment is provided according to the driver's emotions to enhance driving comfort and satisfaction. The vehicle assisted driving system described in the method provides a more intelligent driving experience for the driver through continuous intelligent learning and optimization capabilities. Therefore, the present invention effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0073] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A head-up display adjustment method based on emotion recognition, characterized in that: The method comprises: Acquire a driver's facial video frame sequence and a corresponding physiological signal sequence; Inputting the facial video frame sequence into an expression recognition model to extract an expression feature sequence of the facial video frame sequence; Extracting a physiological feature sequence from the physiological signal sequence according to a physiological feature extraction model; Inputting the expression feature sequence and the physiological feature sequence into a multimodal emotion recognition model to generate an emotion category; Based on the emotion category, the content of the head-up display on the vehicle is dynamically adjusted according to a preset adjustment strategy.

2. The head-up display adjustment method based on emotion recognition according to claim 1, characterized in that: The step of inputting the facial video frame sequence into an expression recognition model and extracting an expression feature sequence of the facial video frame sequence comprises: Inputting the facial video frames into the face detection module of the expression recognition model, detecting and cropping corresponding facial images from the facial video frame sequence based on a facial detection algorithm, and generating a facial image sequence; Inputting the facial image sequence into the key point detection module of the expression recognition model, locating the facial key points corresponding to each facial image based on the facial key point detection method, and obtaining the facial key point sequence corresponding to the facial image sequence; The facial key point sequence and the facial image sequence are input into the detection module of the expression recognition model, the corresponding facial images are aligned based on each facial key point, and the corresponding expression features are extracted from each aligned facial image to generate an expression feature sequence.

3. The head-up display adjustment method based on emotion recognition according to claim 2, characterized in that: The detection module is a convolutional neural network, and the facial key point sequence and the facial image sequence are input into the detection module of the expression recognition model, and the corresponding facial images are aligned based on each facial key point, and the corresponding expression features are extracted from each aligned facial image to generate an expression feature sequence, including: For each face image: Based on the corresponding facial key points, the facial images are aligned; Extracting local features and corresponding global features of the facial image in layers, and fusing the local features and global features extracted in the last layer to obtain expression features of the facial image; Arrange the expression features of each facial image in order to generate an expression feature sequence.

4. The head-up display adjustment method based on emotion recognition according to claim 1, characterized in that: The step of extracting a physiological feature sequence from the physiological signal sequence based on a physiological feature extraction model includes: Performing fast Fourier transform on each cardiac cycle signal of the physiological signal sequence to generate frequency domain features of each cardiac cycle signal; Calculating the mean of each skin conductance of the physiological signal sequence to obtain a skin conductance feature; Combining each of the frequency domain features and the average conductance features into a comprehensive feature sequence; The comprehensive feature sequence is input into the physiological feature extraction model, the comprehensive feature sequence is classified based on the pre-learned weights and corresponding biases, and the physiological feature sequence corresponding to the comprehensive feature sequence is determined; wherein the physiological feature extraction model is a support vector machine.

5. The head-up display adjustment method based on emotion recognition according to claim 1, characterized in that: The step of inputting the expression feature sequence and the physiological feature sequence into a multimodal emotion recognition model to generate an emotion category includes: Inputting the expression feature sequence into the expression feature module of the multimodal emotion recognition model, generating deep expression features according to the spatiotemporal correlation of each expression feature; wherein the expression feature module is a three-dimensional convolutional neural network; Inputting the physiological feature sequence into the physiological feature module of the multimodal emotion recognition model, and generating deep physiological features according to the time dependency of each physiological feature; wherein the physiological feature module is LSTM; Inputting the deep expression feature and the deep physiological feature into the feature fusion module of the multimodal emotion recognition model for fusion to generate a fusion feature; The fused features are input into the output module of the multimodal emotion recognition model, the fused features are mapped to various preset emotion categories, and the final emotion category is determined based on the confidence of each mapped emotion category.

6. The head-up display adjustment method based on emotion recognition according to claim 1, characterized in that: The adjusting the content of the head-up display on the vehicle based on the emotion classification and according to a preset adjustment strategy includes: If the emotion category is positive emotion, keeping the content of the head-up display unchanged; If the emotion category is a negative emotion, non-critical information preset in the head-up display is hidden, and the brightness and contrast of the head-up display are adjusted.

7. The head-up display adjustment method based on emotion recognition according to claim 6, characterized in that: After dynamically adjusting the content of the head-up display on the vehicle based on the emotion category and according to a preset adjustment strategy, it also includes: real-time monitoring of whether the emotion category changes, and after the emotion category changes from negative emotion to positive emotion, gradually presenting each non-critical information to the head-up display interface according to a preset priority order.

8. A head-up display adjustment system based on emotion recognition, characterized in that: The system comprises: A data acquisition module, used to acquire a driver's facial video frame sequence and a corresponding physiological signal sequence; An expression recognition module, used for inputting the facial video frame sequence into an expression recognition model to extract an expression feature sequence of the facial video frame sequence; A physiological feature recognition module, used for extracting a physiological feature sequence from the physiological signal sequence according to a physiological feature extraction model; A multimodal fusion module, used for inputting the expression feature sequence and the physiological feature sequence into a multimodal emotion recognition model to generate an emotion category; The adjustment presentation module is used to dynamically adjust the content of the head-up display on the vehicle based on the emotion category and according to a preset adjustment strategy.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the vehicle-mounted terminal implements the head-up display adjustment method based on emotion recognition as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the head-up display adjustment method based on emotion recognition as described in any one of claims 1 to 7.

Images