Driver emotion recognition and adjustment method and system based on multi-modal signals and storage medium

Abstract

The invention relates to the technical field of intelligent traffic, in particular to a multi-modal signal-based driver emotion recognition and adjustment method and system and a storage medium, and multi-modal signal-based driver emotion recognition comprises signal acquisition, face detection, feature extraction, multi-modal fusion and dimension reduction and emotion recognition. Driver emotion adjustment based on emotion recognition, including emotion evaluation, adjustment scheme design and dynamic adjustment; the method has the advantages that key features related to negative emotions are extracted by collecting multi-mode signals such as voice intonation, facial expression and skin electrical conduction, emotional state recognition is carried out on fused features by utilizing a YOLOv8 algorithm and a deep learning model, a personalized emotional regulation scheme is automatically generated by a system according to the recognized emotional state, and the emotion regulation efficiency is improved. And according to the intensity and characteristics of the negative emotion, the system dynamically selects the most adaptive strategy from a single adjustment means to multi-factor cooperative adjustment, so that the emotion of the driver can be effectively adjusted in real time.

Description

Technical Field

[0001] The present invention relates to the field of smart transportation, and specifically to a driver emotion recognition and regulation method, system and storage medium based on multimodal signals. Background Art

[0002] With the rapid development of intelligent and connected technologies, intelligent connected cars have gradually become intelligent mobile terminals that integrate diverse functions and services. In this context, intelligent cockpits with emotional interactive functions have become an important direction for the future development of the automotive industry.

[0003] Although the current smart cockpit has achieved remarkable results in improving the driving experience, it still has shortcomings in effectively dealing with the driver's emotional out-of-control and ensuring driving safety and comfort. Drivers' negative emotions are one of the main causes of traffic accidents. At present, cars lack effective recognition and regulation functions for drivers' negative emotions. Emotion recognition technology usually relies on a single signal source, such as voice or facial expressions, which makes it difficult to fully and accurately reflect the driver's true emotional state. In addition, emotion regulation methods lack pertinence and personalization, and the effect of single-factor regulation is limited, making it difficult to respond in real time and effectively improve the driver's emotional state.

[0004] Therefore, in order to solve the above technical problems, it is urgent to propose a solution that can integrate multimodal signals and realize accurate emotion recognition and personalized regulation. Summary of the invention

[0005] The purpose of the present invention is to provide a method, system and storage medium for driver emotion recognition and regulation based on multimodal signals to solve the problems raised in the above background technology. To achieve the above purpose, the present invention provides the following technical solution: a method for driver emotion recognition and regulation based on multimodal signals, the method comprising the following steps:

[0006] Driver emotion recognition based on multimodal signals, including signal acquisition, face detection, feature extraction, multimodal fusion and dimensionality reduction, and emotion recognition;

[0007] Driver emotion regulation based on emotion recognition, including emotion assessment, regulation plan design and dynamic adjustment.

[0008] Preferably, the driver emotion recognition based on multimodal signals specifically includes:

[0009] Signal acquisition: Real-time acquisition of multimodal signals from the driver during driving, including facial expressions, voice intonation, and physiological signals such as skin conduction signals;

[0010] Face detection: Use the YOLOv8 algorithm to perform real-time face detection on the collected images to ensure accurate acquisition of the driver's facial expressions;

[0011] Feature extraction: Extract pitch, speech rate, and energy change features from speech intonation; identify key point movement features from facial expressions; analyze conductivity fluctuations from skin conduction signals to form a multi-dimensional emotion feature set;

[0012] Multimodal fusion and dimensionality reduction: The features from different signal sources are fused in a multimodal manner, and the feature space redundancy is reduced through dimensionality reduction technology to generate low-dimensional, high-discrimination emotion feature vectors;

[0013] Emotion recognition: Based on the fused multimodal emotion features, combined with the deep learning model of recurrent neural network (RNN) and convolutional neural network (CNN), the temporal and spatial characteristics of emotion signals are analyzed to identify and infer the driver's current emotional state.

[0014] Preferably, the driver emotion regulation based on emotion recognition specifically includes:

[0015] Emotional assessment: Based on the identified emotional state, evaluate the current emotional intensity and valence;

[0016] Adjustment scheme design: According to the results of emotion assessment, the adjustment scheme is dynamically designed. If the emotion fluctuation is small, single-factor adjustment is selected; if the emotion fluctuation is large, multi-factor coordinated adjustment is initiated, combining olfactory, auditory and visual adjustment methods, by releasing specific smells, playing emotion-adapted music and adjusting the color of the interior light strip;

[0017] Dynamic adjustment: During the adjustment process, the driver's emotional state is continuously monitored, and the adjustment plan is dynamically adjusted based on real-time data to ensure the driver's emotional stability.

[0018] A driver emotion recognition and regulation system based on multimodal signals, applied to a driver emotion recognition and regulation method based on multimodal signals, the system comprising:

[0019] The multimodal information collection module includes an in-vehicle collection component and a physiological signal collection device. The in-vehicle collection component consists of an infrared camera installed on the rearview mirror and the inside of the A-pillar of the smart vehicle, a visible light camera, and an audio collector installed at the speaker at the driving position, which is used to collect the driver's facial expressions, eye movements, and voice intonation; the physiological signal collection device is an Empatica E4 wristband, which is worn on the driver's wrist and is used to collect heart rate, skin electrode activity, blood oxygen saturation, and skin temperature physiological data in real time;

[0020] The driver negative emotion recognition module includes a multimodal data processing module and an emotion recognition module. The multimodal data processing module processes the camera data through the YOLOv8 algorithm, extracts key features from voice intonation, facial expressions and skin conduction signals, and performs multimodal fusion and dimensionality reduction processing. The emotion recognition module recognizes and infers the driver's current emotional state based on the fused multimodal emotion features and the deep learning model of RNN and CNN.

[0021] The negative emotion regulation module consists of an emotion regulation scheme generation module and a multi-sensory emotion regulation module.

[0022] Preferably, the negative emotion regulation module specifically includes:

[0023] The emotion regulation scheme generation module receives the analysis results of the emotion recognition module in real time, evaluates the driver's emotion intensity and valence, and dynamically designs a regulation scheme that matches the driver's current emotional state, including selecting single-factor regulation when the emotion fluctuation is small and initiating multi-factor coordinated regulation when the emotion fluctuation is large;

[0024] The multi-sensory emotion regulation module includes an olfactory regulation sub-module, an auditory regulation sub-module and a visual regulation sub-module. It releases specific scents through an intelligent fragrance device, plays matching music or white noise through the on-board audio system, and adjusts the color of the LED light strip or the background color of the display screen in the car. It uses multi-sensory means to comprehensively regulate the driver's emotional state and dynamically adjusts the regulation plan based on the emotional state monitored in real time.

[0025] Preferably, the infrared camera includes an infrared camera body, which is suspended below a suspension plate, with two elastic hoops fixed to the bottom end of the suspension plate, the two elastic hoops wrapping the infrared camera body, a connecting plate fixed to the bottom end of the elastic hoops, a card slot is provided on the surface of the connecting plate, connecting blocks are plugged into the two card slots, a positioning socket is provided on the surface of the connecting block, a positioning block is plugged into the positioning socket, the positioning block is plugged into the surface of the connecting plate, a traction handle is fixed to the top end of the suspension plate, the traction handle is rotatably connected to the bottom surface of the suspension frame, and a clamping screw is screwed on the side plate of the suspension frame.

[0026] Preferably, the suspension plate is in a semi-circular ring plate structure. Edge grooves are provided on both sides of the suspension plate. The edge grooves are arc-shaped grooves, and a first rubber clamping strip is fixed inside the edge grooves. The first rubber clamping strip is in an arc-shaped sheet, and the inner ring diameter of the first rubber clamping strip is smaller than the inner ring diameter of the suspension plate. The elastic hoop plate is in an arc-shaped sheet, and the two elastic hoop plates are symmetrically distributed with respect to the suspension plate. An embedding groove is provided on the inner ring surface of the elastic hoop plate. The embedding groove is an arc-shaped groove, and a second rubber clamping strip is fixed inside the embedding groove. The second rubber clamping strip is in an arc-shaped sheet, and the inner ring diameter of the second rubber clamping strip is smaller than the inner ring diameter of the elastic hoop plate. The clamping groove is a "convex"-shaped groove, and the depth of the clamping groove is smaller than the plate width of the connecting plate. The connecting plug is in a "worker"-shaped plate structure, and the plate length of the connecting plug is equal to the depth of the clamping groove. The two parallel side plates of the connecting plug are respectively inserted into the two clamping grooves on the surfaces of the two connecting plates. Two picking grooves are provided at one end of the connecting plug extending out of the clamping groove. A positioning socket is provided on the surface of the connecting plate. The positioning socket is a "convex"

[0027] -shaped opening. The positioning socket and the clamping groove are symmetrically distributed with respect to the connecting plate. The positioning plug penetrates through the notch at one end of the positioning socket. Two blocking strips are fixed at one end of the positioning plug. The blocking strips are slidably connected in the notch at the other end of the positioning socket. A elastic band is fixed at one end of the positioning plug. The two ends of the elastic band are respectively fixed in the notch at the other end of the positioning socket, and a pull ring is fixed between the two blocking strips.

[0028] Preferably, the suspension bracket is in a "C"-shaped plate structure. Installation grooves are provided on the two parallel side plates of the suspension bracket. A circular groove is provided on the surface of the installation groove. The circular groove is a "convex"

[0029] -shaped circular opening. A wire sleeve is fixed in the notch at one end of the circular groove. A clamping screw rod is penetrated and screwed inside the wire sleeve. A plurality of拨片 are fixed at one end of the clamping screw rod. The拨片 are in a "return"

[0030] -shaped sheet, and a rubber band is fixed between the top surface and the bottom surface of the installation groove.

[0031] A computer-readable storage medium stores a computer program thereon, and when the program is executed by a processor, it implements a method for driver emotion recognition and regulation based on multi-modal signals.

[0032] Compared with the prior art, the beneficial effects of the present invention are:

[0033] The driver emotion recognition and regulation method, system and storage medium based on multimodal signals proposed in the present invention extract key features related to negative emotions by collecting multimodal signals such as voice intonation, facial expression and skin conduction, and use the YOLOv8 algorithm and deep learning model to identify the emotional state of the fused features. According to the identified emotional state, the system automatically generates a personalized emotion regulation plan, and according to the intensity and characteristics of the negative emotions, the system dynamically selects the most adaptable strategy from a single regulation method to multi-factor coordinated regulation, so as to achieve real-time and effective regulation of the driver's emotions, which can significantly improve driving safety and comfort and reduce the occurrence of traffic accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a diagram of the method architecture of the present invention;

[0035] Figure 2 is a flow chart of the method of the present invention;

[0036] Figure 3 is a system block diagram of the present invention;

[0037] Figure 4 This is a schematic diagram of the connection structure between the infrared camera body and the suspension bracket of the present invention;

[0038] Figure 5 for Figure 4 A schematic diagram of the structure enlargement in the middle;

[0039] Figure 6 for Figure 4 Front view of the middle structure;

[0040] Figure 7 for Figure 6 Structural cross-section view at AA in the middle;

[0041] Figure 8 for Figure 7 A magnified schematic diagram of the structure at B in the middle;

[0042] Fig. 9 for Figure 4 Side view of the middle structure;

[0043] Fig.10 for Fig. 9 Structural cross-section view at the middle BB;

[0044] Fig.11 It is a schematic diagram of the connection structure of the suspension plate and the elastic hoop plate of the present invention;

[0045] Fig.12 for Fig.11 Front view of the middle structure;

[0046] Fig.13 for Fig.12 Structural cross-section view at CC;

[0047] Fig.14 This is a schematic diagram of the connecting plug structure of the present invention;

[0048] Fig.15 This is a schematic diagram of the positioning plug structure of the present invention.

[0049] In the figure: infrared camera body 1, hanging plate 2, elastic hoop plate 3, connecting plate 4, side groove 5, rubber clamp strip 1 6, embedded groove 7, rubber clamp strip 2 8, card slot 9, connecting plug 10, installation port 11, positioning socket 12, positioning plug 13, baffle 14, elastic band 15, plug 16, rubber clamp 17, pick-up groove 18, pull ring 19, traction handle 20, limit groove 21, rubber sleeve 22, suspension frame 23, installation groove 24, round groove 25, silk sleeve 26, clamping screw 27, paddle 28, rubber belt 29. DETAILED DESCRIPTION

[0050] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] For example, see Figure 1-Figure 2 The present invention provides a technical solution: a driver emotion recognition and regulation method based on multimodal signals, the method comprising the following steps:

[0052] After the vehicle is started, the driver's facial expressions and eye movements are collected through infrared cameras and visible light cameras, the audio collector collects the driver's voice intonation, and the smart bracelet collects skin conduction signals. After the driver's face is detected using the YOLOv8 algorithm, the facial, voice and physiological signal data are extracted, and the fusion processing and dimensionality reduction techniques are used to comprehensively analyze the voice pitch, speech speed, key point movement of facial expressions, and conductivity fluctuations in skin conduction signals to generate low-dimensional, high-discrimination emotional feature vectors. The recurrent neural network (RNN) and convolutional neural network (CNN) models are used to analyze the fusion features, identify and infer the driver's emotional state, and evaluate the driver's emotional intensity and valence, dynamically design adjustment plans, select the most adaptable strategy from a single adjustment method to multi-factor coordinated adjustment, and dynamically adjust the adjustment intensity according to real-time data to ensure that the driver's emotions are always in a stable state.

[0053] Specifically, the intensity of emotion (I) is used to quantify the activity of emotion, with a score from 1 to 10, 10 indicating extremely strong emotion. The valence of emotion (V) is used to quantify the positive or negative nature of emotion, with a score from -5 to +5, negative values ​​indicating negative emotion and positive values ​​indicating positive emotion. The formula for selecting the adjustment plan is as follows:

[0054]

[0055] In this example, when the driver showed angry emotional characteristics, the YOLOv8 algorithm combined with the RNN-CNN model processed multimodal data, and identified and evaluated the emotional intensity (I) as 9 and the emotional valence (V) as -4, indicating that the driver was in a highly negative emotional state. According to the evaluation results, the multi-factor synergistic regulation strategy was initiated to release essential oil gases such as lavender or citrus, which have been proven to effectively relieve anger and anxiety. At the same time, the LED light strip in the car was adjusted to a soothing blue or green to simulate the seaside or forest environment, and the audio played low-frequency and deep music melodies that can produce a calming effect. If the emotional intensity and valence do not improve after the initial adjustment, the degree of adjustment will be further enhanced, the volume of the music will be increased, and the concentration of the essential oil smell will be strengthened. In extreme cases, if all measures fail to stabilize the emotions to a safe level, the emergency response program will be automatically initiated, and an emergency signal will be sent to the preset emergency contact through the in-vehicle communication function, and the driver's real-time physiological and emotional state will be provided to ensure rapid access to external assistance. This dynamic and hierarchical response mechanism can not only evaluate and adjust the driver's emotional state in real time, but also take effective measures to protect the driver and the public when emotional fluctuations may affect driving safety.

[0056] Embodiment 2, refer to the attached Figure 3 ,On the basis of the first embodiment, a driver emotion recognition and regulation system based on multimodal signals is proposed, which is mainly composed of a multimodal information acquisition module, a driver negative emotion recognition module and a negative emotion regulation module.

[0057] The multimodal information acquisition module is a driver information acquisition device based on image, sound and physiological data recognition, including an in-vehicle acquisition component and a physiological signal acquisition device. The in-vehicle acquisition component consists of an infrared camera, a visible light camera and an audio collector. The infrared camera and the visible light camera are used to record the driver's facial expressions and eye movements. Through multimodal image capture technology under different light sources, the driver's visual information is fully collected. They are installed on the rearview mirror and the inside of the A-pillar of the smart vehicle respectively; the audio collector is responsible for capturing the driver's voice tone and is installed at the speaker at the driving position. The multimodal information acquisition module further integrates Empatica's E4 wristband, which is worn on the driver's wrist and can collect physiological data including heart rate, skin electrode activity, blood oxygen saturation and skin temperature in real time to assist in judging the driver's emotional state and stress level. The in-vehicle acquisition component transmits high-speed data with the emotion recognition module through cables, while the E4 wristband maintains a stable connection with the emotion recognition module through a wireless network.

[0058] The driver negative emotion recognition module mainly includes a multimodal data processing module and an emotion recognition module. The multimodal data processing module first processes the data collected by the camera through the YOLOv8 algorithm to detect the driver's face in real time. After the driver's face is recognized, the module extracts key features from the driver's voice intonation, facial expression and skin conduction signal, and processes the data through multimodal fusion technology and dimensionality reduction technology to generate a low-dimensional, high-discrimination emotion feature vector. The emotion recognition module is based on the fused multimodal emotion features, combined with the deep learning model of recurrent neural network (RNN) and convolutional neural network (CNN) for analysis and processing, fully capturing the temporal and spatial characteristics of the emotion signal, and identifying and inferring the driver's current emotional state.

[0059] The negative emotion regulation module consists of an emotion regulation scheme generation module and a multi-sensory emotion regulation module, forming an emotion regulation system based on intelligent decision-making. The emotion regulation scheme generation module receives the analysis results of the emotion recognition module in real time, evaluates the current driver's emotion intensity and valence, identifies the driver's emotion fluctuation pattern, determines the intensity level and specific category of the emotion, and dynamically designs a regulation scheme that matches the driver's current emotional state. When the emotion fluctuation is small, the system gives priority to single-factor regulation and calls a certain regulation submodule for emotional intervention; when the emotion fluctuation is large and the driver is in a highly aroused state, the system automatically generates a multi-factor collaborative regulation scheme, combining multiple regulation methods to achieve a more comprehensive emotion regulation effect.

[0060] The multi-sensory emotion regulation module comprehensively regulates the driver's emotional state through multi-sensory means such as smell, hearing and vision. The olfactory regulation submodule accurately releases specific odor molecules through the intelligent fragrance device to regulate the driver's emotional response; the auditory regulation submodule plays music or white noise that matches the current emotional state through the car audio system to affect the driver's emotional changes; the visual regulation submodule adjusts the color of the LED light strip in the car or the background color of the display screen to change the driver's visual perception to achieve emotional regulation. During the entire regulation process, the negative emotion regulation module dynamically adjusts the emotion regulation plan according to the emotional state monitored in real time, and uses multi-dimensional sensory regulation methods to intervene and regulate the driver's emotions in real time, ensuring that the driver's emotions can be quickly and effectively relieved and optimized, thereby ensuring driving safety and driving comfort.

[0061] The working principle of the whole system is as follows: after the system detects that the vehicle is started, the multimodal information acquisition module automatically starts, and the driver's facial expressions, eye movements and voice data are collected in real time through infrared cameras, visible light cameras and audio collectors. At the same time, the E4 wristband continuously monitors physiological signals such as heart rate and skin electrical activity. All data are transmitted to the driver's negative emotion recognition module through cables or wireless networks. The multimodal data processing module first detects the driver's face through the YOLOv8 algorithm, extracts features from the data after recognizing the face, and generates efficient emotion feature vectors through fusion processing and dimensionality reduction technology. The emotion recognition module uses recurrent neural networks (RNN) and convolutional neural networks (CNN) to analyze fusion features and identify the driver's emotional state. When the system detects negative emotions, the emotion regulation scheme generation module dynamically designs regulation schemes according to the intensity and category of emotions. Under mild emotional fluctuations, the system gives priority to single-factor regulation methods, such as olfactory or auditory intervention; under high arousal, the system automatically triggers multi-factor coordinated regulation and conducts comprehensive intervention through smell, music and lighting. During the adjustment process, the system continuously collects the driver's real-time status and dynamically adjusts the adjustment intensity according to emotional changes to ensure the driver's emotional stability, thereby improving driving safety and comfort.

[0062] Embodiment 3, on the basis of embodiment 2, proposes a mechanism for installing an infrared camera on a rearview mirror, the infrared camera comprises an infrared camera body 1, the infrared camera body 1 is suspended below a suspension plate 2, two elastic hoop plates 3 are fixed to the bottom end of the suspension plate 2, the two elastic hoop plates 3 wrap the infrared camera body 1, a connecting plate 4 is fixed to the bottom end of the elastic hoop plates 3, a card slot 9 is provided on the surface of the connecting plate 4, a connecting block 10 is inserted inside the two card slots 9, a positioning socket 12 is provided on the surface of the connecting block 10, a positioning socket 13 is inserted inside the positioning socket 12, the positioning block 13 is inserted on the surface of the connecting plate 4, a traction handle 20 is fixed to the top end of the suspension plate 2, the traction handle 20 is rotatably connected to the bottom surface of a suspension frame 23, and a clamping screw 27 is screwed on the side plate of the suspension frame 23. The reason why the infrared camera body 1 is installed below the suspension bracket 23 is to fix the suspension bracket 23 behind the rearview mirror, so that the infrared camera body 1 can be adjusted at multiple angles through the traction handle 20, that is, it is convenient to adjust the monitoring direction of the infrared camera body 1.

[0063] The suspension plate 2 is a semicircular ring-shaped plate structure, and side grooves 5 are provided on both sides of the suspension plate 2. The side grooves 5 are arc-shaped grooves, and a rubber clamp 6 is fixed inside the side grooves 5. The rubber clamp 6 is an arc-shaped sheet, and the inner ring diameter of the rubber clamp 6 is smaller than the inner ring diameter of the suspension plate 2. The elastic hoop plate 3 is an arc-shaped sheet, and the two elastic hoop plates 3 are symmetrically distributed about the suspension plate 2. The inner ring surface of the elastic hoop plate 3 is provided with an embedded groove 7, and the embedded groove 7 is an arc-shaped groove. A rubber clamp 8 is fixed inside the embedded groove 7. The rubber clamp 8 is an arc-shaped sheet, and the inner ring diameter of the rubber clamp 8 is smaller than the inner ring diameter of the elastic hoop plate 3. The card groove 9 is a "convex"-shaped groove, and the depth of the card groove 9 is less than the plate width of the connecting plate 4. The connecting plug 10 is an "I"-shaped plate-shaped structure, and the plate of the connecting plug 10 The length is equal to the depth of the slot 9, and the two parallel side plates of the connecting block 10 are respectively inserted into the two slots 9 on the surfaces of the two connecting plates 4. The end of the connecting block 10 extending out of the slot 9 is provided with two picking grooves 18, and the surface of the connecting plate 4 is provided with a positioning socket 12, and the positioning socket 12 is a "convex" shape. The positioning socket 12 and the slot 9 are symmetrically distributed about the connecting plate 4. The positioning block 13 passes through the notch at one end of the positioning socket 12, and two baffles 14 are fixed at one end of the positioning block 13. The baffle 14 is slidably connected to the notch at the other end of the positioning socket 12, and an elastic band 15 is fixed to one end of the positioning block 13. The two ends of the elastic band 15 are respectively fixed in the notch at the other end of the positioning socket 12, and a pull ring 19 is fixed between the two baffles 14. When the infrared camera body 1 is installed under the hanging plate 2, the two elastic hoop plates 3 wrap the infrared camera body 1 from the bottom of the infrared camera body 1 to prevent the infrared camera body 1 from falling off, and the connecting block 10 is inserted into the two card slots 9, so that the ends of the two elastic hoop plates 3 are close together, so that the two elastic hoop plates 3 cooperate with the hanging plate 2 to tightly clamp the infrared camera body 1, and the top surface of the connecting block 10 and the bottom surface of the connecting block 10 are integrally formed with a plug 16, and the plug 16 is a "concave" shaped plate, and a rubber clip 17 is fixed inside the plug 16, and the rubber clip 17 The thickness is greater than the groove depth of the plug piece 16, and the rubber clip 17 is clamped between the plug piece 16 and the connecting plate 4 to produce elastic deformation; and the positioning block 13 is inserted into the positioning socket 12 to prevent the connecting block 10 from slipping out of the card slot 9; when the infrared camera body 1 needs to be disassembled, two fingers are hooked on the two pull rings 19 respectively, and the corresponding positioning block 13 is pulled to slide along the installation port 11. At this time, the positioning block 13 pulls the elastic band 15 to produce deformation. When the positioning block 13 is detached from the positioning socket 12, two fingers are hooked in the two extraction grooves 18, and the card slot 9 is pulled out from the connecting block 10.

[0064] The top end of the traction handle 20 is of a spherical structure. The top end of the traction handle 20 is rotatably connected in the limit groove 21. The limit groove 21 is a spherical groove and is opened on the bottom surface of the suspension bracket 23. A rubber sleeve 22 is sleeved on the top end of the traction handle 20. The rubber sleeve 22 is clamped between the traction handle 20 and the limit groove 21 to generate elastic deformation. The suspension bracket 23 is of a "C"-shaped plate-like structure. Installation grooves 24 are opened on both side plates of the suspension bracket 23 that are parallelly distributed. A circular groove 25 is opened on the surface of the installation groove 24. The circular groove 25 is a "convex"-shaped circular opening. A wire sleeve 26 is fixed in one end groove of the circular groove 25. A clamping screw 27 penetrates through and is screwed inside the wire sleeve 26. A plurality of dial plates 28 are fixed at one end of the clamping screw 27. The dial plates 28 are "hui"-shaped sheets. A rubber band 29 is fixed between the top surface and the bottom surface of the installation groove 24. The suspension bracket 23 is buckled under the rearview mirror, and then the clamping screw 27 is rotated and pushed towards the inner side of the suspension bracket 23. When the clamping screw 27 moves, the rubber band 29 is tensioned and deformed. The rubber band 29 is clamped between the clamping screw 27 and the rearview mirror. And by拨动 the infrared camera body 1, the traction handle 20 can be driven to swing under the suspension bracket 23. After the thrust stops, since the rubber sleeve 22 is tightly clamped between the traction handle 20 and the limit groove 21, the traction handle 20 is prevented from swinging randomly, thus braking the orientation of the infrared camera body 1.

[0065] Embodiment 4. On the basis of Embodiment 1, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method steps proposed in Embodiment 1 are implemented.

[0066] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principle and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents. It should be noted that there is an unclear part in the original text "拨动 the infrared camera body 1", which may need to be further clarified in the original context for more accurate translation.

Claims

1. A driver emotion recognition and regulation method based on multimodal signals, characterized in that: The method comprises the following steps: Driver emotion recognition based on multimodal signals, including signal acquisition, face detection, feature extraction, multimodal fusion and dimensionality reduction, and emotion recognition; Driver emotion regulation based on emotion recognition, including emotion assessment, regulation plan design and dynamic adjustment.

2. The driver emotion recognition and adjustment method based on multimodal signals according to claim 1 is characterized in that: Driver emotion recognition based on multimodal signals specifically includes: Signal acquisition: real-time acquisition of multimodal signals of the driver during driving, including facial expressions, voice intonation, and physiological signals such as skin conduction signals; Face detection: Use the YOLOv8 algorithm to perform real-time face detection on the collected images to ensure accurate acquisition of the driver's facial expressions; Feature extraction: Extract pitch, speech rate, and energy change features from speech intonation; identify key point movement features from facial expressions; analyze conductivity fluctuations from skin conduction signals to form a multi-dimensional emotion feature set; Multimodal fusion and dimensionality reduction: The features from different signal sources are fused in a multimodal manner, and the feature space redundancy is reduced through dimensionality reduction technology to generate low-dimensional, high-discrimination emotion feature vectors; Emotion recognition: Based on the fused multimodal emotion features, combined with the deep learning model of recurrent neural network (RNN) and convolutional neural network (CNN), the temporal and spatial characteristics of emotion signals are analyzed to identify and infer the driver's current emotional state.

3. The driver emotion recognition and adjustment method based on multimodal signals according to claim 1 is characterized in that: Driver emotion regulation based on emotion recognition specifically includes: Emotional assessment: Based on the identified emotional state, evaluate the current emotional intensity and valence; Adjustment scheme design: According to the results of emotion assessment, the adjustment scheme is dynamically designed. If the emotion fluctuation is small, single-factor adjustment is selected; if the emotion fluctuation is large, multi-factor coordinated adjustment is initiated, combining olfactory, auditory and visual adjustment methods, by releasing specific smells, playing emotion-adapted music and adjusting the color of the interior light strip; Dynamic adjustment: During the adjustment process, the driver's emotional state is continuously monitored, and the adjustment plan is dynamically adjusted based on real-time data to ensure the driver's emotional stability.

4. A driver emotion recognition and adjustment system based on multimodal signals, applied to the driver emotion recognition and adjustment method based on multimodal signals according to any one of claims 1 to 3, characterized in that: The system comprises: The multimodal information collection module includes an in-vehicle collection component and a physiological signal collection device. The in-vehicle collection component consists of an infrared camera installed on the rearview mirror and the inside of the A-pillar of the smart vehicle, a visible light camera, and an audio collector installed at the speaker at the driving position, which is used to collect the driver's facial expressions, eye movements, and voice intonation; the physiological signal collection device is an Empatica E4 wristband, which is worn on the driver's wrist and is used to collect heart rate, skin electrode activity, blood oxygen saturation, and skin temperature physiological data in real time; The driver negative emotion recognition module includes a multimodal data processing module and an emotion recognition module. The multimodal data processing module processes the camera data through the YOLOv8 algorithm, extracts key features from voice intonation, facial expressions and skin conduction signals, and performs multimodal fusion and dimensionality reduction processing. The emotion recognition module recognizes and infers the driver's current emotional state based on the fused multimodal emotion features and the deep learning model of RNN and CNN. The negative emotion regulation module consists of an emotion regulation scheme generation module and a multi-sensory emotion regulation module.

5. The driver emotion recognition and regulation system based on multimodal signals according to claim 4 is characterized in that: The negative emotion regulation module specifically includes: The emotion regulation scheme generation module receives the analysis results of the emotion recognition module in real time, evaluates the driver's emotion intensity and valence, and dynamically designs a regulation scheme that matches the driver's current emotional state, including selecting single-factor regulation when the emotion fluctuation is small and initiating multi-factor coordinated regulation when the emotion fluctuation is large; The multi-sensory emotion regulation module includes an olfactory regulation sub-module, an auditory regulation sub-module and a visual regulation sub-module. It releases specific scents through an intelligent fragrance device, plays matching music or white noise through the on-board audio system, and adjusts the color of the LED light strip or the background color of the display screen in the car. It uses multi-sensory means to comprehensively regulate the driver's emotional state and dynamically adjusts the regulation plan based on the emotional state monitored in real time.

6. The driver emotion recognition and regulation system based on multimodal signals according to claim 4, characterized in that: The infrared camera comprises an infrared camera body (1), the infrared camera body (1) is suspended below a suspension plate (2), two elastic hoop plates (3) are fixed at the bottom end of the suspension plate (2), the two elastic hoop plates (3) wrap the infrared camera body (1), a connecting plate (4) is fixed at the bottom end of the elastic hoop plate (3), a card slot (9) is provided on the surface of the connecting plate (4), a connecting plug block (10) is inserted inside the two card slots (9), a positioning socket (12) is provided on the surface of the connecting plug block (10), a positioning socket (13) is inserted inside the positioning socket (12), and the positioning plug block (13) is inserted on the surface of the connecting plate (4), a traction handle (20) is fixed at the top end of the suspension plate (2), the traction handle (20) is rotatably connected to the bottom surface of the suspension frame (23), and a clamping screw (27) is screwed on the side plate of the suspension frame (23).

7. The driver emotion recognition and regulation system based on multimodal signals according to claim 6 is characterized in that: The suspension plate (2) is in a semi-circular ring plate structure. Side grooves (5) are formed on both sides of the suspension plate (2). The side grooves (5) are arc-shaped grooves. A first rubber clamping strip (6) is fixed inside the side grooves (5). The first rubber clamping strip (6) is in an arc-shaped sheet. The inner ring diameter of the first rubber clamping strip (6) is smaller than the inner ring diameter of the suspension plate (2). The elastic hoop plate (3) is in an arc-shaped sheet. The two elastic hoop plates (3) are symmetrically distributed with respect to the suspension plate (2). An embedding groove (7) is formed on the inner ring surface of the elastic hoop plate (3). The embedding groove (7) is an arc-shaped groove. A second rubber clamping strip (8) is fixed inside the embedding groove (7). The second rubber clamping strip (8) is in an arc-shaped sheet. The inner ring diameter of the second rubber clamping strip (8) is smaller than the inner ring diameter of the elastic hoop plate (3). The clamping groove (9) is in a "convex"-shaped groove. The depth of the clamping groove (9) is smaller than the plate width of the connecting plate (4). The connecting plug (10) is in an "I"-shaped plate structure. The plate length of the connecting plug (10) is equal to the depth of the clamping groove (9). The two side plates of the connecting plug (10) distributed in parallel are respectively inserted into the two clamping grooves (9) on the surface of the two connecting plates (4). Two extraction grooves (18) are formed at one end of the connecting plug (10) extending out of the clamping groove (9). A positioning socket (12) is formed on the surface of the connecting plate (4). The positioning socket (12) is in a "convex"-shaped opening. The positioning socket (12) and the clamping groove (9) are symmetrically distributed with respect to the connecting plate (4). The positioning block (13) penetrates through the notch at one end of the positioning socket (12). Two blocking strips (14) are fixed at one end of the positioning block (13). The blocking strips (14) are slidably connected in the notch at the other end of the positioning socket (12). A elastic band (15) is fixed at one end of the positioning block (13). The two ends of the elastic band (15) are respectively fixed in the notch at the other end of the positioning socket (12). And a pull ring (19) is fixed between the two blocking strips (14).

8. The driver emotion recognition and regulation system based on multimodal signals according to claim 6, characterized in that: The top end of the traction handle (20) is in a spherical structure. The top end of the traction handle (20) is rotatably connected in the limit groove (21). The limit groove (21) is a spherical groove. The limit groove (21) is formed on the bottom surface of the suspension bracket (23). A rubber sleeve (22) is sleeved on the top end of the traction handle (20). The rubber sleeve (22) is clamped between the traction handle (20) and the limit groove (21) to generate elastic deformation.

9. The driver emotion recognition and regulation system based on multimodal signals according to claim 8, characterized in that: The suspension bracket (23) is in a "C"-shaped plate structure. Mounting grooves (24) are formed on the two side plates of the suspension bracket (23) distributed in parallel. A circular groove (25) is formed on the surface of the mounting groove (24). The circular groove (25) is a "convex"-shaped circular opening. A wire sleeve (26) is fixed in the notch at one end of the circular groove (25). A clamping screw (27) penetrates through and is screwed inside the wire sleeve (26). A plurality of dial pieces (28) are fixed at one end of the clamping screw (27). The dial pieces (28) are in a "return"-shaped sheet. A rubber band (29) is fixed between the top surface and the bottom surface of the mounting groove (24).

10. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for driver emotion recognition and regulation based on multimodal signals as described in any one of claims 1 to 3 is implemented.

Images