In-vehicle music and light adaptive adjustment method based on emotion recognition

Abstract

The invention provides an in-vehicle music and light adaptive adjustment method based on emotion recognition, and the method comprises the steps: collecting multi-modal sensor data reflecting the physiological and behavior states of a driver in real time through a plurality of vehicle-mounted integrated sensors in a vehicle, and combining the data type of the multi-modal sensor data, processing the multi-modal sensor data, extracting target features of the multi-modal sensor data, and inputting the target features of the multi-modal sensor data into a preset multi-modal fusion deep learning model for emotion recognition to obtain a current emotion state of the driver, and searching and matching a corresponding target music and light adjusting strategy in a preset emotion environment adjusting strategy library based on the current emotion state of the driver, and dynamically adjusting the music and light in the vehicle based on the target music and light adjusting strategy.

Description

Technical Field

[0001] This application relates to the field of vehicle intelligent control technology, and more specifically, to a method for adaptive adjustment of in-vehicle music and lighting based on emotion recognition. Background Technology

[0002] With the development of intelligent cockpit and human-machine interaction technologies, intelligent control of the in-vehicle environment has become an important direction for improving driving safety and comfort, which involves the adjustment of music and lighting.

[0003] Currently, in terms of music adjustment, existing technologies mostly use preset driving modes, and music playback switches according to the driving mode. Some high-end models support voice control of music playback, which is achieved by the driver actively issuing commands. In terms of lighting adjustment, the color and brightness of the ambient lighting in the car are manually adjusted by the user. Some vehicles will link with the driving mode, which is mostly triggered by fixed scenarios (such as door opening / locking / music rhythm linkage).

[0004] However, most existing methods for adjusting music and lights are based on fixed modes or manual operation, making it difficult to dynamically respond to the driver's emotional state in real time. This results in generally poor accuracy and stability of the adjustment. Furthermore, when the driver actively issues commands to control music playback, there is some operational interference, which reduces driving safety. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide an adaptive adjustment method for in-vehicle music and lighting based on emotion recognition. By collecting multimodal sensor data such as the driver's facial expressions, voice, and physiological signals for emotion recognition, it can dynamically identify and respond to the driver's emotional state in real time, and adjust the in-vehicle music and lighting environment based on the recognition results and a preset adjustment strategy. This enhances the accuracy and stability of music and lighting adjustment, and also enhances driving safety.

[0006] In a first aspect, embodiments of this application provide a method for adaptive adjustment of in-vehicle music and lighting based on emotion recognition, the method comprising: Multimodal sensor data reflecting the driver's physiological and behavioral state is collected in real time by a variety of in-vehicle integrated sensors; wherein, the multimodal sensor data includes at least facial expression data, voice data and physiological signal data; Based on the data type of the multimodal sensor data, the multimodal sensor data is processed, and the target features of the multimodal sensor data are extracted. The target features of the multimodal sensor data are input into a preset multimodal fusion deep learning model for emotion recognition to obtain the driver's current emotional state. Based on the driver's current emotional state, the system searches for and matches the corresponding target music and lighting adjustment strategies in the preset emotional environment adjustment strategy library, and dynamically adjusts the in-vehicle music and lighting based on the target music and lighting adjustment strategies.

[0007] In one possible implementation, the method further includes: Based on a pre-set built-in driver preference database, the system records the music and lighting adjustment strategies preferred by different drivers when using the device for the first time. The corresponding music and lighting adjustment strategies and the multimodal fusion deep learning model are dynamically updated based on feedback data generated by different drivers during the driving process.

[0008] In one possible implementation, the real-time acquisition of multimodal sensor data reflecting the driver's physiological and behavioral states via in-vehicle integrated sensors includes: The driver's facial expression data is collected in real time by a high-resolution infrared camera mounted above the steering wheel. The driver's voice data is collected by a high-fidelity microphone array inside the vehicle; wherein the voice data is used to analyze the driver's acoustic characteristics, including tone, speech rate, and volume. The driver's physiological signal data is collected by a target biosensor; wherein the target biosensor is integrated into the steering wheel grip inside the vehicle.

[0009] In one possible implementation, processing the multimodal sensor data in conjunction with its data type to extract target features includes: For the facial expression data, the faces in the facial expression data are detected and aligned, and the facial key points in the facial expression data are located to extract multi-dimensional facial features from the facial expression data. For the speech data, multi-dimensional acoustic features of the speech data are extracted based on a preset audio processing library; For the physiological signal data, extract the physiological signal features of the physiological signal data.

[0010] In one possible implementation, the physiological signal data includes at least heart rate variability data, skin conductance data, and respiratory rate; the step of extracting physiological signal features from the physiological signal data includes: For the heart rate variability data, the photoplethysmography (PPG) wave signal corresponding to the heart rate variability data is bandpass filtered, and the first physiological signal feature in the bandpass-filtered PPG wave signal is extracted through a preset peak detection model. For the aforementioned electrodermal response data, the photoplethysmography (PPG) signal corresponding to the electrodermal response data is low-pass filtered, and the second physiological signal feature in the low-pass filtered PPG signal is extracted. For the respiratory rate, the third physiological signal feature of the respiratory rate is extracted by modulating the respiratory rate envelope, or the third physiological signal feature of the respiratory rate is extracted based on a preset target sensor.

[0011] In one possible implementation, the step of inputting the target features of the multimodal sensor data into a preset multimodal fusion deep learning model for emotion recognition to obtain the driver's current emotional state includes: The facial features are input into a preset multimodal fusion deep learning model, and high-level semantic features of the facial features are extracted based on the convolutional neural network in the multimodal fusion deep learning model. The acoustic features and the physiological signal features are concatenated into a corresponding one-dimensional vector, and the one-dimensional vector is input into the long short-term memory network in the multimodal fusion deep learning model to extract the temporal dynamic change features of the one-dimensional vector. The high-level semantic features and the temporal dynamic change features are fused late before the fully connected layer of the multimodal fusion deep learning model to output the emotion probability of their respective emotion categories. The emotion probabilities are weighted and averaged, and the emotion probability of the driver corresponding to the emotion category is output through the Softmax layer of the multimodal fusion deep learning model. The emotion category with the highest emotion probability is selected as the driver's current emotion state.

[0012] In one possible implementation, the dynamic adjustment of in-vehicle music and lighting based on the target music and lighting adjustment strategy includes: Based on the target music and lighting adjustment strategy, the central processing unit in the vehicle generates corresponding target control commands; wherein, the target control commands include music control commands and lighting control commands; The music control command is sent to the in-vehicle music player via the API interface of the in-vehicle infotainment system, so that the music player adjusts the music based on the music control command. The light control command is sent to the ambient lighting controller inside the vehicle, so that the ambient lighting controller adjusts the light based on the light control command.

[0013] Secondly, embodiments of this application also provide an in-vehicle music and lighting adaptive adjustment device based on emotion recognition, the device comprising: The data acquisition module is used to collect multimodal sensor data reflecting the driver's physiological and behavioral state in real time through various in-vehicle integrated sensors; wherein, the multimodal sensor data includes at least facial expression data, voice data, and physiological signal data; The processing module is used to process the multimodal sensor data in combination with the data type of the multimodal sensor data, and extract the target features of the multimodal sensor data; The recognition module is used to input the target features of the multimodal sensor data into a preset multimodal fusion deep learning model for emotion recognition, so as to obtain the driver's current emotional state. The adjustment module is used to search and match the corresponding target music and lighting adjustment strategy in the preset emotional environment adjustment strategy library based on the driver's current emotional state, and dynamically adjust the in-vehicle music and lighting based on the target music and lighting adjustment strategy.

[0014] In one possible implementation, the device further includes: The input module is used to input the music and lighting adjustment strategies preferred by different drivers when using the device for the first time, based on a preset built-in driver preference database. The update module is used to dynamically update the corresponding music and lighting adjustment strategies and the multimodal fusion deep learning model based on feedback data generated by different drivers during the driving process.

[0015] In one possible implementation, the acquisition module is specifically used for: The driver's facial expression data is collected in real time by a high-resolution infrared camera mounted above the steering wheel. The driver's voice data is collected by a high-fidelity microphone array inside the vehicle; wherein the voice data is used to analyze the driver's acoustic characteristics, including tone, speech rate, and volume. The driver's physiological signal data is collected by a target biosensor; wherein the target biosensor is integrated into the steering wheel grip inside the vehicle.

[0016] In one possible implementation, the processing module is specifically used for: For the facial expression data, the faces in the facial expression data are detected and aligned, and the facial key points in the facial expression data are located to extract multi-dimensional facial features from the facial expression data. For the speech data, multi-dimensional acoustic features of the speech data are extracted based on a preset audio processing library; For the physiological signal data, extract the physiological signal features of the physiological signal data.

[0017] In one possible implementation, the physiological signal data includes at least heart rate variability data, skin conductance data, and respiratory rate; the processing module is specifically used for: For the heart rate variability data, the photoplethysmography (PPG) wave signal corresponding to the heart rate variability data is bandpass filtered, and the first physiological signal feature in the bandpass-filtered PPG wave signal is extracted through a preset peak detection model. For the aforementioned electrodermal response data, the photoplethysmography (PPG) signal corresponding to the electrodermal response data is low-pass filtered, and the second physiological signal feature in the low-pass filtered PPG signal is extracted. For the respiratory rate, the third physiological signal feature of the respiratory rate is extracted by modulating the respiratory rate envelope, or the third physiological signal feature of the respiratory rate is extracted based on a preset target sensor.

[0018] In one possible implementation, the identification module is specifically used for: The facial features are input into a preset multimodal fusion deep learning model, and high-level semantic features of the facial features are extracted based on the convolutional neural network in the multimodal fusion deep learning model. The acoustic features and the physiological signal features are concatenated into a corresponding one-dimensional vector, and the one-dimensional vector is input into the long short-term memory network in the multimodal fusion deep learning model to extract the temporal dynamic change features of the one-dimensional vector. The high-level semantic features and the temporal dynamic change features are fused late before the fully connected layer of the multimodal fusion deep learning model to output the emotion probability of their respective emotion categories. The emotion probabilities are weighted and averaged, and the emotion probability of the driver corresponding to the emotion category is output through the Softmax layer of the multimodal fusion deep learning model. The emotion category with the highest emotion probability is selected as the driver's current emotion state.

[0019] In one possible implementation, the adjustment module is specifically used for: Based on the target music and lighting adjustment strategy, the central processing unit in the vehicle generates corresponding target control commands; wherein, the target control commands include music control commands and lighting control commands; The music control command is sent to the in-vehicle music player via the API interface of the in-vehicle infotainment system, so that the music player adjusts the music based on the music control command. The light control command is sent to the ambient lighting controller inside the vehicle, so that the ambient lighting controller adjusts the light based on the light control command.

[0020] Thirdly, embodiments of this application provide an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the emotion-recognition-based adaptive adjustment method for in-vehicle music and lighting as described in any of the first aspects.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the emotion-recognition-based adaptive adjustment method for in-vehicle music and lighting as described in any one of the first aspects.

[0022] This application provides a method for adaptive adjustment of in-vehicle music and lighting based on emotion recognition. It utilizes multiple integrated in-vehicle sensors to collect multimodal sensor data reflecting the driver's physiological and behavioral states in real time. The method processes the multimodal sensor data, extracting target features, and inputs these features into a preset multimodal fusion deep learning model for emotion recognition. This yields the driver's current emotional state. Based on this state, the method searches and matches corresponding target music and lighting adjustment strategies from a preset emotional environment adjustment strategy library, dynamically adjusting the in-vehicle music and lighting accordingly. This application, by collecting multimodal sensor data such as the driver's facial expressions, voice, and physiological signals for emotion recognition, can dynamically identify and respond to the driver's emotional state in real time. Based on the recognition results, it matches preset adjustment strategies to adjust the in-vehicle music and lighting environment, enhancing the accuracy and stability of music and lighting adjustment and improving driving safety.

[0023] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of an in-vehicle music and lighting adaptive adjustment method based on emotion recognition, provided in an embodiment of this application. Figure 2 This is a flowchart illustrating a music and lighting adaptive adjustment method based on emotion recognition. Figure 3 This is a schematic diagram of the overall structure of a music and lighting adaptive adjustment system based on emotion recognition. Figure 4 This is a schematic diagram of the structure of an in-vehicle music and lighting adaptive adjustment device based on emotion recognition, according to an embodiment of this application. Figure 5 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0027] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0028] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0029] Considering the development of intelligent cockpit and human-machine interaction technology, intelligent control of the in-vehicle environment has become an important direction for improving driving safety and comfort, which involves the adjustment of music and lighting.

[0030] Currently, in terms of music adjustment, existing technologies mostly use preset driving modes, and music playback switches according to the driving mode. Some high-end models support voice control of music playback, which is achieved by the driver actively issuing commands. In terms of lighting adjustment, the color and brightness of the ambient lighting in the car are manually adjusted by the user. Some vehicles will link with the driving mode, which is mostly triggered by fixed scenarios (such as door opening / locking / music rhythm linkage).

[0031] However, most existing methods for adjusting music and lights are based on fixed modes or manual operation, making it difficult to dynamically respond to the driver's emotional state in real time. This results in generally poor accuracy and stability of the adjustment. Furthermore, when the driver actively issues commands to control music playback, there is some operational interference, which reduces driving safety.

[0032] To address this issue, this application provides an adaptive adjustment method for in-vehicle music and lighting based on emotion recognition. By collecting multimodal sensor data such as the driver's facial expressions, voice, and physiological signals for emotion recognition, it can dynamically identify and respond to the driver's emotional state in real time. Based on the recognition results, it matches a preset adjustment strategy to adjust the music and lighting environment in the vehicle, thereby enhancing the accuracy and stability of music and lighting adjustment and improving driving safety.

[0033] Figure 1 This is a flowchart of an in-vehicle music and lighting adaptive adjustment method based on emotion recognition, provided in an embodiment of this application. Figure 1 As shown in the embodiment of this application, the in-vehicle music and lighting adaptive adjustment method based on emotion recognition may specifically include: S101: Multimodal sensor data reflecting the driver's physiological and behavioral state are collected in real time through a variety of in-vehicle integrated sensors.

[0034] S102. Based on the data type of the multimodal sensor data, process the multimodal sensor data and extract the target features from the multimodal sensor data.

[0035] S103. Input the target features of the multimodal sensor data into the preset multimodal fusion deep learning model for emotion recognition to obtain the driver's current emotional state.

[0036] S104. Based on the driver's current emotional state, search and match the corresponding target music and lighting adjustment strategy in the preset emotional environment adjustment strategy library, and dynamically adjust the in-vehicle music and lighting based on the target music and lighting adjustment strategy.

[0037] In the aforementioned adaptive adjustment method for in-vehicle music and lighting based on emotion recognition, emotion recognition is achieved by collecting multimodal sensor data such as the driver's facial expressions, voice, and physiological signals. This allows for real-time dynamic identification and response to the driver's emotional state, and the adjustment of the in-vehicle music and lighting environment is adjusted based on the recognition results and a preset adjustment strategy. This enhances the accuracy and stability of music and lighting adjustment, and also improves driving safety.

[0038] The exemplary steps described above in the embodiments of this application are illustrated below with specific examples: The S101 uses multiple in-vehicle integrated sensors to collect multimodal sensor data in real time, reflecting the driver's physiological and behavioral state.

[0039] It should be noted that a unified timestamp is used to mark the multimodal sensor data in order to ensure that the multimodal sensor data is aligned on the time axis.

[0040] In this embodiment, the multimodal sensor data includes at least facial expression data, voice data, and physiological signal data. Facial expression data refers to the collected facial video data of the driver, used to capture facial muscle movements, such as frowning and changes in the corners of the mouth. Multimodal sensor data of the driver is collected in real time from various integrated vehicle sensors for subsequent processing. For example, such as... Figure 2 As shown.

[0041] In some implementations, the driver's facial expression data is collected in real time by a high-resolution infrared camera; the driver's voice data is collected by a high-fidelity microphone array inside the vehicle; and the driver's physiological signal data is collected by a biosensor.

[0042] The target infrared camera is mounted above the steering wheel and can be a high-resolution infrared camera, such as the OmniVision OV9734; voice data is used to analyze the driver's acoustic characteristics, including tone, speech rate, and volume; and the target biosensor is integrated into the steering wheel grip inside the vehicle. For example, Figure 3 As shown, Figure 3 The camera module in the image represents the target infrared camera, the microphone module represents the target high-fidelity microphone array, and the physiological signal sensor represents the target biosensor.

[0043] The target biosensors include a target photoplethysmography (PPG) sensor and a target electrode sensor; the physiological signal data include at least heart rate variability (HRV) data, skin conductance response (GSR) data, and respiratory rate.

[0044] Optionally, the driver's heart rate variability data is acquired via a target photoplethysmography (PPG) sensor; the driver's skin conductance data is acquired via a target electrode sensor; and the driver's respiratory rate is acquired via the PPG signal from the target PPG sensor. Therefore, this application employs multimodal data fusion, including facial expressions, voice, heart rate, and skin conductance, so that even if some signals fail, emotions can still be determined through other signals, significantly improving the accuracy and stability of recognition and solving the problem of single-modal susceptibility to interference.

[0045] S102, combining the data types of multimodal sensor data, processes the multimodal sensor data and extracts the target features of the multimodal sensor data.

[0046] In this embodiment, the data types of multimodal sensor data are facial expression data, voice data, and physiological signal data. The target features are the multimodal features extracted from the multimodal sensor data. The multimodal sensor data of the driver collected in step S102 is preprocessed according to the data types of the multimodal sensor data to extract the target features for subsequent processing. For example, such as... Figure 2 As shown.

[0047] In some implementations, for facial expression data, faces in the facial expression data are detected and aligned, and facial key points in the facial expression data are located to extract multi-dimensional facial features from the facial expression data; for speech data, multi-dimensional acoustic features of the speech data are extracted based on a preset audio processing library; and for physiological signal data, physiological signal features of the physiological signal data are extracted.

[0048] Among them, facial features include at least facial action unit (AU) intensity, eye closure frequency (PERCLOS), and head posture angle features; acoustic features include at least Mel frequency cepstral coefficients (MFCCs), fundamental frequency (F0), energy, speech rate, and pause duration.

[0049] Specifically, for example, for facial expression data, the OpenCV library can be used to perform face detection and alignment, and MediaPipe can be used to locate facial key points (e.g., 68-point model) to extract facial action unit (AU) intensity, eye closure frequency (PERCLOS), head pose angle features, etc. For speech data, audio processing libraries such as Librosa can be used to extract acoustic features such as Mel frequency cepstral coefficients (MFCCs), fundamental frequency (F0), energy, speech rate, and pause duration.

[0050] Optionally, when extracting physiological signal features from physiological signal data, for heart rate variability data, the photoplethysmography (PPG) wave signal corresponding to the heart rate variability data is bandpass filtered, and the first physiological signal feature in the bandpass-filtered PPG wave signal is extracted using a preset peak detection model; for skin conductance response data, the PPG wave signal corresponding to the skin conductance response data is lowpass filtered, and the second physiological signal feature in the low-pass-filtered PPG wave signal is extracted; for respiratory rate, the respiratory rate is modulated with an envelope to extract the third physiological signal feature of the respiratory rate, or the third physiological signal feature of the respiratory rate is extracted based on a preset target sensor.

[0051] The first physiological signal feature includes at least heart rate (HR) and HRV time-domain / frequency-domain indicators; the second physiological signal feature includes at least the amplitude and frequency of skin conductance level (SCL) and skin conductance response (SCR); and the third physiological signal feature includes at least respiratory rate.

[0052] Specifically, for example, after bandpass filtering (e.g., frequency 0.5-5Hz) of heart rate variability data, the first physiological signal features, such as heart rate (HR) and HRV time-domain / frequency-domain indices (e.g., SDNN, RMSSD, LF / HF ratio), are extracted using a pre-defined peak detection model. For skin conductance response data, after low-pass filtering (e.g., frequency less than 5Hz), the second physiological signal features, such as the amplitude and frequency of skin conductance level (SCL) and skin conductance response (SCR), are extracted. For respiratory rate, the respiratory rate is extracted using the modulation envelope of PPG or a dedicated sensor (e.g., a respiratory sensor). Thus, by uniformly timestamping and synchronizing multimodal sensor data, and performing preprocessing such as denoising, normalization, and feature extraction, the alignment of multi-source data in both time and space is ensured.

[0053] S103, input the target features of the multimodal sensor data into the preset multimodal fusion deep learning model for emotion recognition, and obtain the driver's current emotional state.

[0054] In this embodiment, the driver's emotional state (emotion category) can be calm, joyful, angry, sad, anxious, tired, etc. The multimodal fusion deep learning model is a pre-constructed multimodal fusion deep learning model. The target features of the multimodal sensor data extracted in step S102 are input into the multimodal fusion deep learning model for emotion recognition to obtain the driver's current emotional state (i.e., the current emotion category). For example, as Figure 2 As shown.

[0055] In some implementations, facial features are input into a pre-defined multimodal fusion deep learning model. High-level semantic features of the facial features are extracted using a convolutional neural network within the model. Acoustic features and physiological signal features are concatenated into corresponding one-dimensional vectors, which are then input into a Long Short-Term Memory (LSTM) network within the model to extract temporal dynamic change features. The high-level semantic features and temporal dynamic change features are then late-stage fused before the fully connected layer of the model, outputting the corresponding emotion probabilities for each emotion category. A weighted average of these emotion probabilities is then applied, and the driver's corresponding emotion probability is output through the Softmax layer of the model. The emotion category with the highest probability is selected as the driver's current emotional state. Here, the multimodal fusion deep learning model represents a two-stream neural network (visual stream CNN + physiological-speech stream LSTM). The two streams refer to the visual stream and the physiological-speech stream. The visual stream corresponds to the facial features of the facial expression data, while the physiological-speech stream corresponds to the acoustic features of the speech data and the physiological signal features of the physiological signal data.

[0056] Specifically, this application employs a constructed multimodal fusion deep learning model for emotion recognition and classification: for example, facial features are input into the multimodal fusion deep learning model, and convolutional neural networks (e.g., CNN, ResNet-18) are used to extract corresponding high-level semantic features of the face; acoustic features and the aforementioned physiological signal features are concatenated into a one-dimensional vector, which is then input into a long short-term memory network to capture temporal dynamic changes; the two sets of features, namely the high-level semantic features of the face and the temporal dynamic changes, are fused late before the fully connected layer, that is, the two networks output the probabilities of various emotional states respectively, and perform a weighted average, and finally output the probability distribution of the driver's emotional state through the Softmax layer.

[0057] Optionally, the driver's current emotional state can be determined by identifying the emotional state with the highest probability based on the probability distribution of the driver's emotional state.

[0058] S104: Based on the driver's current emotional state, search and match the corresponding target music and lighting adjustment strategy in the preset emotional environment adjustment strategy library, and dynamically adjust the in-vehicle music and lighting based on the target music and lighting adjustment strategy.

[0059] It should be noted that the music and lighting adjustment strategy is constructed through the following steps: For the target emotional state, the corresponding music parameters are determined; for the lighting strategy, the corresponding lighting parameters are determined. The music parameters include at least music type, BPM range, volume, and characteristics; the lighting parameters include at least color temperature, brightness, and dynamic effects.

[0060] For example, the following is a music and lighting adjustment strategy based on typical emotional states, without distinguishing the level of emotional intensity.

[0061] 1) Emotion: Calm Music strategy: Type: Soft, soothing music, such as classical music and new age music.

[0062] BPM range: 60-80 BPM.

[0063] Volume: medium to low volume (30%-50%).

[0064] Features: Natural sound effects or minimalist music.

[0065] Lighting strategy: Color temperature: Warm tone (2700K-3000K).

[0066] Brightness: Moderate (40%-60%).

[0067] Dynamic effects: slow gradient or static.

[0068] In this embodiment, the emotional environment adjustment strategy library stores music and lighting adjustment strategies corresponding to different emotional states and intensity levels. The library is designed to dynamically adjust in-vehicle music and lighting based on the driver's emotional state to optimize driving experience and safety. Based on the driver's current emotional state identified in step S103, the library searches for and matches the corresponding target music and lighting adjustment strategy, and dynamically adjusts the in-vehicle music and lighting based on the target strategy. For example, the identified emotional state (such as anger, anxiety, fatigue, etc.) can be mapped one-to-one or many-to-one with the music type, rhythm (BPM), volume, light color temperature, brightness, and dynamic mode (such as breathing, flashing) corresponding to the preset music and lighting adjustment strategy, achieving precise emotionally driven music and lighting environment adjustment.

[0069] Optionally, when dynamically adjusting in-vehicle music and lighting based on a target music and lighting adjustment strategy, the central processing unit (ECU) generates corresponding target control commands based on the target music and lighting adjustment strategy; music control commands are sent to the in-vehicle music player via the API interface of the in-vehicle infotainment system, so that the music player adjusts the music according to the music control commands; lighting control commands are sent to the in-vehicle ambient lighting controller, so that the ambient lighting controller adjusts the lighting according to the lighting control commands. For example, lighting control commands can be sent to the in-vehicle ambient lighting controller via a LIN bus. For example, such as... Figure 2 and Figure 3 As shown.

[0070] Specifically, based on the matching music and lighting adjustment strategies, the central processing unit (ECU) generates specific control commands: music control commands, sent via the API interface of the in-vehicle infotainment system, to the music player with commands such as "play a specified playlist / track," "adjust volume to X%," and "switch sound effect mode"; lighting control commands, sent via the LIN bus to the ambient lighting controller with commands such as "set color (RGB value)," "set brightness (PWM duty cycle)," and "set dynamic mode (breathing / blinking frequency)." Continuing on, upon receiving the commands, the corresponding in-vehicle infotainment system automatically switches to the target music and adjusts the volume; upon receiving the commands, the ambient lighting controller changes the color, brightness, and dynamic effects of the lights in real time, achieving adaptive adjustment of the in-vehicle environment.

[0071] In addition, when adjusting the ambient lighting, the brightness of the light can be precisely controlled by PWM signals according to the music control commands, and RGB values ​​and dynamic modes can be sent through the LIN bus to achieve smooth and gradual changes in light color, brightness and breathing / flickering frequency, thereby improving sensory comfort.

[0072] Therefore, this application can identify the driver's emotions in real time and automatically match the corresponding music and lighting adjustment strategies, realizing the transformation from "passive control" to "proactive care", and making the adjustment more timely and humane.

[0073] The in-vehicle music and lighting adaptive adjustment method based on emotion recognition provided in this application uses multiple in-vehicle integrated sensors to collect multimodal sensor data reflecting the driver's physiological and behavioral states in real time. Combining the data types of the multimodal sensor data, the method processes the data and extracts target features. These target features are then input into a preset multimodal fusion deep learning model for emotion recognition, obtaining the driver's current emotional state. Based on the driver's current emotional state, the method searches and matches corresponding target music and lighting adjustment strategies from a preset emotional environment adjustment strategy library, and dynamically adjusts the in-vehicle music and lighting based on these strategies. This in-vehicle music and lighting adaptive adjustment method based on emotion recognition, by collecting multimodal sensor data such as the driver's facial expressions, voice, and physiological signals for emotion recognition, can dynamically identify and respond to the driver's emotional state in real time. Based on the recognition results, it matches preset adjustment strategies to adjust the in-vehicle music and lighting environment, enhancing the accuracy and stability of music and lighting adjustment, and also improving driving safety.

[0074] Furthermore, based on a pre-set, built-in driver preference database, the preferred music and lighting adjustment strategies of different drivers upon first use are recorded; the corresponding music and lighting adjustment strategies and the multimodal fusion deep learning model are dynamically updated based on feedback data generated by different drivers during driving. The feedback data includes at least rating data and post-adjustment emotional changes.

[0075] Specifically, the system's built-in driver preference database (which can be a lightweight embedded database system, such as SQLite) stores personalized emotion-music-lighting matching rules (i.e., music and lighting adjustment strategies) for different drivers. Drivers can input their preferences through the settings interface upon first use. During operation, the system continuously learns and updates based on historical data. Collected feedback data (such as whether the volume was manually adjusted) can update the music and lighting adjustment strategies and the multimodal fusion deep learning model, enabling online learning or offline model optimization. Furthermore, reinforcement learning algorithms can be used to adjust parameters in the emotion environment adjustment strategy library based on feedback rewards, achieving personalized and adaptive optimization. For example, if a driver frequently switches to rock music when fatigued, the system will automatically adjust its recommendation strategy to provide a personalized experience for each driver.

[0076] Therefore, this application incorporates a driver preference database and continuously learns from user feedback, enabling the collection of user feedback and personalized, adaptive optimization of music and lighting adjustment strategies. Furthermore, the driver preference database is stored in non-volatile memory (SQLite database), supporting offline operation while also being encrypted and uploaded to the cloud, balancing real-time performance, privacy, and scalability, and achieving cross-vehicle, cross-device, and local / cloud collaborative synchronization of preference data.

[0077] Furthermore, in response to the recognition that the driver's current emotional state is a high-risk emotion, the system will proactively adjust the music to a preset music strategy and adjust the lights to a preset lighting strategy.

[0078] Specifically, this application sets up a safety-first adjustment strategy for high-risk emotions (e.g., anger, fatigue). For example, when the driver's current emotional state is detected as "anger", the music tempo and volume are automatically reduced and the lights are adjusted to cool-toned static light to avoid further stimulation; when the driver's current emotional state is detected as "fatigue", refreshing music is played and the lights simulate the dynamic changes of morning light to enhance alertness.

[0079] Therefore, this application helps drivers calm down or regain alertness when high-risk emotions are identified, reducing driving risks caused by emotional instability and contributing to improved driving safety.

[0080] Figure 4This is a schematic diagram of the structure of an in-vehicle music and lighting adaptive adjustment device based on emotion recognition, according to an embodiment of this application; as shown below. Figure 4 As shown in the figure, the in-vehicle music and lighting adaptive adjustment device 400 based on emotion recognition in this application embodiment may specifically include: The acquisition module 401 is used to acquire multimodal sensor data reflecting the driver's physiological and behavioral state in real time through a variety of in-vehicle integrated sensors; wherein, the multimodal sensor data includes at least facial expression data, voice data and physiological signal data.

[0081] The processing module 402 is used to process the multimodal sensor data in combination with the data type of the multimodal sensor data, and extract the target features of the multimodal sensor data.

[0082] The recognition module 403 is used to input the target features of the multimodal sensor data into a preset multimodal fusion deep learning model for emotion recognition, so as to obtain the driver's current emotional state.

[0083] The adjustment module 404 is used to search and match the corresponding target music and lighting adjustment strategies in the preset emotional environment adjustment strategy library based on the driver's current emotional state, and dynamically adjust the in-vehicle music and lighting based on the target music and lighting adjustment strategies.

[0084] In one possible implementation, the apparatus further includes: The input module is used to input the music and lighting adjustment strategies preferred by different drivers when using the device for the first time, based on a preset built-in driver preference database. The update module is used to dynamically update the corresponding music and lighting adjustment strategies and multimodal fusion deep learning models based on feedback data generated by different drivers during driving.

[0085] In one possible implementation, the acquisition module is specifically used for: The driver's facial expression data is collected in real time by a high-resolution infrared camera mounted above the steering wheel. The driver's voice data is collected by a high-fidelity microphone array inside the vehicle; the voice data is used to analyze the driver's acoustic characteristics, including tone, speech rate, and volume. The driver's physiological signal data is collected through a target biosensor; the target biosensor is integrated into the steering wheel grip inside the vehicle.

[0086] In one possible implementation, the processing module is specifically used for: For facial expression data, the system detects and aligns faces within the data, locates key facial points, and extracts multi-dimensional facial features. For speech data, multi-dimensional acoustic features are extracted based on a pre-defined audio processing library; For physiological signal data, extract physiological signal features from the physiological signal data.

[0087] In one possible implementation, the physiological signal data includes at least heart rate variability data, skin conductance data, and respiratory rate; the processing module is specifically used for: For heart rate variability data, the photoplethysmography (PPG) wave signal corresponding to the heart rate variability data is bandpass filtered, and the first physiological signal feature in the bandpass-filtered PPG wave signal is extracted through a preset peak detection model. For the electrodermal response data, the photoplethysmography (PPG) signal corresponding to the electrodermal response data is low-pass filtered, and the second physiological signal feature in the low-pass filtered PPG signal is extracted. For respiratory rate, the third physiological signal feature of respiratory rate is extracted by modulating the respiratory rate envelope, or the third physiological signal feature of respiratory rate is extracted based on a preset target sensor.

[0088] In one possible implementation, the identification module is specifically used for: Facial features are input into a pre-defined multimodal fusion deep learning model, and high-level semantic features of the facial features are extracted based on the convolutional neural network in the multimodal fusion deep learning model. Acoustic features and physiological signal features are concatenated into corresponding one-dimensional vectors, and the one-dimensional vectors are input into the long short-term memory network in the multimodal fusion deep learning model to extract the temporal dynamic change features of the one-dimensional vectors. High-level semantic features and temporally dynamic features are fused late in front of the fully connected layer of the multimodal fusion deep learning model to output the emotion probability of their respective emotion categories. The emotional probabilities are weighted and averaged, and the emotional probabilities of the driver's corresponding emotional category are output through the Softmax layer of the multimodal fusion deep learning model. The emotional category with the highest emotional probability is selected as the driver's current emotional state.

[0089] In one possible implementation, the adjustment module is specifically used for: Based on the target music and lighting adjustment strategy, the central processing unit in the vehicle generates corresponding target control commands; among which, the target control commands include music control commands and lighting control commands. The system sends music control commands to the in-vehicle music player via the API interface of the in-vehicle infotainment system, so that the music player can adjust the music based on the music control commands. Send a lighting control command to the ambient lighting controller inside the vehicle, so that the ambient lighting controller can adjust the lighting based on the lighting control command.

[0090] The in-vehicle music and lighting adaptive adjustment device based on emotion recognition provided in this application collects multimodal sensor data reflecting the driver's physiological and behavioral states in real time through various in-vehicle integrated sensors. Combining the data types, the device processes the multimodal sensor data, extracts target features, and inputs these features into a preset multimodal fusion deep learning model for emotion recognition. This yields the driver's current emotional state. Based on this state, the device searches and matches corresponding target music and lighting adjustment strategies from a preset emotional environment adjustment strategy library, and dynamically adjusts the in-vehicle music and lighting accordingly. This emotion recognition-based in-vehicle music and lighting adaptive adjustment device, by collecting multimodal sensor data such as the driver's facial expressions, voice, and physiological signals for emotion recognition, can dynamically identify and respond to the driver's emotional state in real time. Based on the recognition results, it matches preset adjustment strategies to adjust the in-vehicle music and lighting environment, enhancing the accuracy and stability of music and lighting adjustment and improving driving safety.

[0091] like Figure 5 As shown in the embodiment of this application, an electronic device 500 includes a processor 501, a memory 502, and a bus. The memory 502 stores machine-readable instructions executable by the processor 501. When the electronic device is running, the processor 501 communicates with the memory 502 via the bus. The processor 501 executes the machine-readable instructions to perform the steps of the above-described adaptive adjustment method for in-vehicle music and lighting based on emotion recognition.

[0092] Specifically, the memory 502 and processor 501 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 501 runs the computer program stored in the memory 502, it can execute the above-mentioned adaptive adjustment method for in-vehicle music and lighting based on emotion recognition.

[0093] Corresponding to the above-described adaptive adjustment method for in-vehicle music and lighting based on emotion recognition, this application also provides a computer-readable storage medium storing a computer program. When the computer program is run by a processor, it executes the steps of the above-described adaptive adjustment method for in-vehicle music and lighting based on emotion recognition.

[0094] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0095] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0096] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0097] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the deployment methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0098] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for adaptive adjustment of in-vehicle music and lighting based on emotion recognition, characterized in that, The method includes: Multimodal sensor data reflecting the driver's physiological and behavioral state is collected in real time by a variety of in-vehicle integrated sensors; wherein, the multimodal sensor data includes at least facial expression data, voice data and physiological signal data; Based on the data type of the multimodal sensor data, the multimodal sensor data is processed, and the target features of the multimodal sensor data are extracted. The target features of the multimodal sensor data are input into a preset multimodal fusion deep learning model for emotion recognition to obtain the driver's current emotional state. Based on the driver's current emotional state, the system searches for and matches the corresponding target music and lighting adjustment strategies in the preset emotional environment adjustment strategy library, and dynamically adjusts the in-vehicle music and lighting based on the target music and lighting adjustment strategies.

2. The method according to claim 1, characterized in that, The method further includes: Based on a pre-set built-in driver preference database, the system records the music and lighting adjustment strategies preferred by different drivers when using the device for the first time. The corresponding music and lighting adjustment strategies and the multimodal fusion deep learning model are dynamically updated based on feedback data generated by different drivers during the driving process.

3. The method according to claim 1, characterized in that, The method involves real-time acquisition of multimodal sensor data reflecting the driver's physiological and behavioral states through in-vehicle integrated sensors, including: The driver's facial expression data is collected in real time by a high-resolution infrared camera mounted above the steering wheel. The driver's voice data is collected by a high-fidelity microphone array inside the vehicle; wherein the voice data is used to analyze the driver's acoustic characteristics, including tone, speech rate, and volume. The driver's physiological signal data is collected by a target biosensor; wherein the target biosensor is integrated into the steering wheel grip inside the vehicle.

4. The method according to claim 3, characterized in that, The step of processing the multimodal sensor data in conjunction with its data type to extract target features includes: For the facial expression data, the faces in the facial expression data are detected and aligned, and the facial key points in the facial expression data are located to extract multi-dimensional facial features from the facial expression data. For the speech data, multi-dimensional acoustic features of the speech data are extracted based on a preset audio processing library; For the physiological signal data, extract the physiological signal features of the physiological signal data.

5. The method according to claim 4, characterized in that, The physiological signal data includes at least heart rate variability data, skin conductance data, and respiratory rate; the extraction of physiological signal features from the physiological signal data includes: For the heart rate variability data, the photoplethysmography (PPG) wave signal corresponding to the heart rate variability data is bandpass filtered, and the first physiological signal feature in the bandpass-filtered PPG wave signal is extracted through a preset peak detection model. For the aforementioned electrodermal response data, the photoplethysmography (PPG) signal corresponding to the electrodermal response data is low-pass filtered, and the second physiological signal feature in the low-pass filtered PPG signal is extracted. For the respiratory rate, the third physiological signal feature of the respiratory rate is extracted by modulating the respiratory rate envelope, or the third physiological signal feature of the respiratory rate is extracted based on a preset target sensor.

6. The method according to claim 5, characterized in that, The step of inputting the target features of the multimodal sensor data into a preset multimodal fusion deep learning model for emotion recognition to obtain the driver's current emotional state includes: The facial features are input into a preset multimodal fusion deep learning model, and high-level semantic features of the facial features are extracted based on the convolutional neural network in the multimodal fusion deep learning model. The acoustic features and the physiological signal features are concatenated into a corresponding one-dimensional vector, and the one-dimensional vector is input into the long short-term memory network in the multimodal fusion deep learning model to extract the temporal dynamic change features of the one-dimensional vector. The high-level semantic features and the temporal dynamic change features are fused late before the fully connected layer of the multimodal fusion deep learning model to output the emotion probability of their respective emotion categories. The emotion probabilities are weighted and averaged, and the emotion probability of the driver corresponding to the emotion category is output through the Softmax layer of the multimodal fusion deep learning model. The emotion category with the highest emotion probability is selected as the driver's current emotion state.

7. The method according to claim 1, characterized in that, The dynamic adjustment of in-vehicle music and lighting based on the target music and lighting adjustment strategy includes: Based on the target music and lighting adjustment strategy, the central processing unit in the vehicle generates corresponding target control commands; wherein, the target control commands include music control commands and lighting control commands; The music control command is sent to the in-vehicle music player via the API interface of the in-vehicle infotainment system, so that the music player adjusts the music based on the music control command. The light control command is sent to the ambient lighting controller inside the vehicle, so that the ambient lighting controller adjusts the light based on the light control command.

8. A vehicle in-car music and lighting adaptive adjustment device based on emotion recognition, characterized in that, The device includes: The data acquisition module is used to collect multimodal sensor data reflecting the driver's physiological and behavioral state in real time through various in-vehicle integrated sensors; wherein, the multimodal sensor data includes at least facial expression data, voice data, and physiological signal data; The processing module is used to process the multimodal sensor data in combination with the data type of the multimodal sensor data, and extract the target features of the multimodal sensor data; The recognition module is used to input the target features of the multimodal sensor data into a preset multimodal fusion deep learning model for emotion recognition, so as to obtain the driver's current emotional state. The adjustment module is used to search and match the corresponding target music and lighting adjustment strategy in the preset emotional environment adjustment strategy library based on the driver's current emotional state, and dynamically adjust the in-vehicle music and lighting based on the target music and lighting adjustment strategy.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the emotion-recognition-based adaptive adjustment method for in-vehicle music and lighting as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the emotion-recognition-based adaptive adjustment method for in-vehicle music and lighting as described in any one of claims 1 to 7.

Images