A music intervention method and system for driver mood regulation

Abstract

The application provides a music intervention method and system for driver emotion regulation, comprising: collecting physiological index data and audio-video index data of a driver; performing feature extraction and feature fusion on the physiological index data and the audio-video index data to obtain multi-dimensional feature data, inputting the multi-dimensional feature data into an emotion recognition model to obtain an emotion type and an emotion recognition value; inputting the emotion type and the emotion recognition value into a synchronous emotion music model for audio synthesis, and outputting a music intervention segment. The application analyzes various physiological indexes and audio-video indexes of a driver during driving, recognizes the emotional state of the driver, generates synchronous music with the driver's emotion by using an artificial intelligence algorithm, guides and regulates the negative emotional state of the driver, so as to achieve the effects of relieving stress and improving positive emotions.

Description

Technical Field

[0001] This invention relates to the field of automotive intelligent technology, and in particular to a music intervention method and system for driver emotion regulation. Background Technology

[0002] During driving, drivers are easily affected by various external and internal factors, resulting in various pressures and emotions. Especially during long-distance driving, they often unconsciously fall into a state of tension and stress, which affects their physical and mental health. Being in this state for a long time can lead to physiological imbalance, various diseases, negative emotions, and even traffic accidents.

[0003] To address the aforementioned factors influencing drivers' negative emotions, various intervention methods are needed to alleviate these emotions. Music is a common approach, as it acts as a reinforcing factor, stimulating, inducing, promoting, and improving adaptive behaviors while reducing or eliminating maladaptive behaviors, thus influencing a person's physiological state, behavior, and emotions. Existing solutions include: some systems use physiological information sensors on the driver's hand to identify and determine their current emotional state; when in a negative emotional state, the in-vehicle music playback unit plays preset tracks from a personalized music library to encourage a positive emotional state; others use facial image recognition to identify the driver's emotional category, and a cloud server pushes music corresponding to that category to the in-vehicle terminal, thus helping the driver regulate their emotions; still others determine the driver's current emotional type based on facial expression information and push at least one targeted care action, such as voice interaction chat, based on a care model that matches the current emotional type and the driver's preferences.

[0004] Analysis reveals that existing music intervention methods primarily focus on alleviating single emotions. While these methods may have a positive impact on driver emotions to some extent, they lack a precise understanding of the driver's emotional state and personalized responses. Low matching degree between music and emotion: Because the synchronicity between music and emotion is not considered, the music played may not match the driver's emotional state, resulting in an unsatisfactory music experience. Poor emotional relief effect: Focusing mainly on alleviating single emotions without synchronizing music with emotions through synchronization principles may lead to poor emotional relief, or even negative effects due to the mismatch between music playback and emotional state. Most current solutions focus on playing pre-recorded music, meaning a lack of flexibility and real-time adaptability in music selection and playback. Lack of real-time adaptability: Because pre-recorded music is played, it cannot be adjusted in real time according to the driver's emotional state. When the driver's mood changes, suitable music cannot be automatically selected and played to respond to this change. Slow music library updates: Currently, recorded music may not be updated and replaced in real time. This means that if the driver becomes fatigued or bored with the existing music library, new music resources will not be available to meet their needs. Lack of personalization: Playing pre-recorded music often fails to meet the individual needs of each driver. Each driver's emotional state and preferences may differ, and pre-recorded music may not satisfy these personalized needs. Lack of interactivity: Current patents do not allow drivers to participate in the music selection and generation process. This reduces the interaction between the driver and the system, potentially preventing drivers from having a better music experience. High technical barriers: Playing pre-recorded music requires a large amount of music resource storage space, as well as a complex music playback system. This increases the hardware cost of the entire system and the complexity of subsequent maintenance. Summary of the Invention

[0005] This invention provides a music intervention method and system for driver emotion regulation, which addresses the shortcomings of existing technologies that use music to assist in driver emotion regulation, which are too simplistic, lack flexibility, and cannot adequately adapt to complex and ever-changing driver emotion scenarios.

[0006] In a first aspect, the present invention provides a music intervention method for driver emotion regulation, comprising:

[0007] Collect physiological and audio-visual data of the driver;

[0008] Feature extraction and feature fusion are performed on the physiological indicator data and the audio and video indicator data to obtain multidimensional feature data. The multidimensional feature data is then input into the emotion recognition model to obtain the emotion type and emotion recognition value.

[0009] The emotion type and the emotion recognition value are input into a synchronous emotion music model for audio synthesis, and a music intervention segment is output.

[0010] According to the present invention, a music intervention method for driver emotion regulation is provided, which collects physiological index data and audio-visual index data of the driver, including:

[0011] The driver's heart rate variability, skin temperature, and skin resistance data are collected using wearable sensors.

[0012] The dashcam captures images of the driver's facial expressions and voice data.

[0013] According to the present invention, a music intervention method for driver emotion regulation involves feature extraction and feature fusion of the physiological indicator data and the audio-visual indicator data to obtain multidimensional feature data, including:

[0014] Deep learning technology is used to extract key feature points from facial expression images;

[0015] Speech recognition technology is used to extract pitch data, speech rate data, and volume data from speech data;

[0016] Frequency domain analysis and time domain analysis were performed on the heart rate variability data to obtain the heart rate frequency characteristics and heart rate fluctuation amplitude of the heart rate variability data;

[0017] The skin temperature data, skin resistance data, key feature points of the facial expression image, pitch data, speech rate data, and volume data of the speech data, as well as the heart rate frequency features and heart rate fluctuation amplitude of the heart rate variability data, are weighted and normalized respectively. The weighted fusion is then used to output the multidimensional feature data.

[0018] According to the present invention, a music intervention method for driver emotion regulation involves inputting the multidimensional feature data into an emotion recognition model to obtain emotion type and emotion recognition value, including:

[0019] The initial structure of the emotion recognition model consists of a convolutional neural network, a recurrent neural network, and an attention mechanism. The initial structure of the emotion recognition model is trained using labeled emotion data samples to obtain the emotion recognition model.

[0020] The multidimensional feature data is input into the emotion recognition model, and the emotion type and the emotion recognition value are output. The emotion type includes positive emotion type and negative emotion type, and the emotion recognition value includes positive emotion recognition value and negative emotion recognition value.

[0021] According to the present invention, a music intervention method for driver emotion regulation, before inputting the emotion type and the emotion recognition value into a synchronous emotion music model for audio synthesis and outputting a music intervention segment, further includes:

[0022] Identify different types of musical moods, rhythms, melodies, and harmonies;

[0023] A music library is constructed from the different types of musical moods, rhythms, melodies, and harmonies.

[0024] According to the present invention, a music intervention method for driver emotion regulation involves inputting the emotion type and the emotion recognition value into a synchronous emotion music model for audio synthesis, and outputting a music intervention segment, including:

[0025] If the emotion type is determined to be a negative emotion type, and the emotion recognition value is greater than the negative emotion threshold, the driver's emotion recognition and perception will continue.

[0026] If the emotion type is determined to be a negative emotion type, and the emotion recognition value is less than or equal to the negative emotion threshold, the driver's emotion is intervened and regulated, and the music intervention segment is generated and played.

[0027] Once the emotion type is positive and the emotion recognition value is greater than the positive emotion threshold, the music intervention segment is stopped.

[0028] According to the present invention, a music intervention method for driver emotion regulation is provided, which involves regulating the driver's emotions by generating and playing the music intervention segment, including:

[0029] Obtain the driver's current emotion recognition value, generate an intervention music emotion value based on the current emotion recognition value, and make the absolute difference between the intervention music emotion value and the current emotion recognition value less than a preset constraint value;

[0030] The driver's emotion type and the intervention music emotion value are encoded by data encoding. The emotion type is converted into a label vector and the intervention music emotion value is converted into a continuous value to obtain the emotion code.

[0031] Music fragments are retrieved from the music library based on the stated emotion type and the stated intervention music emotion value;

[0032] The emotion code and the music fragment are input into the synchronous emotion music model, and an audio waveform with a preset duration is output.

[0033] After the preset duration of playback, the emotion type and the emotion recognition value are repeatedly identified, so that the absolute difference between the intervention music emotion value and the emotion recognition value is less than the preset constraint value.

[0034] According to the present invention, a music intervention method for driver emotion regulation is provided, wherein the audio waveform includes multiple audio tracks, and the multiple audio tracks correspond to different instruments or different sound elements.

[0035] Secondly, the present invention also provides a music intervention system for driver emotion regulation, comprising:

[0036] The emotion perception module is used to collect the driver's physiological and audio-visual data.

[0037] An emotion recognition module is used to extract and fuse features from the physiological indicator data and the audio-visual indicator data to obtain multi-dimensional feature data, and input the multi-dimensional feature data into the emotion recognition model to obtain the emotion type and emotion recognition value.

[0038] The music generation module is used to input the emotion type and the emotion recognition value into the synchronous emotion music model for audio synthesis and output music intervention segments.

[0039] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the music intervention method for driver emotion regulation as described above.

[0040] The music intervention method and system for driver emotion regulation provided by this invention analyzes various physiological and audio-visual indicators of the driver during the driving process, identifies the driver's emotional state, and uses artificial intelligence algorithms to generate music synchronized with the driver's emotions, thereby guiding and regulating the driver's negative emotional state, and thus achieving the effect of relieving stress and improving positive emotions. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is one of the flowcharts of the music intervention method for driver emotion regulation provided by the present invention;

[0043] Figure 2This is the second flowchart of the music intervention method for driver emotion regulation provided by the present invention;

[0044] Figure 3 This is a flowchart of the synchronous emotional music model provided by the present invention;

[0045] Figure 4 This invention provides a synchronous music intervention emotion value curve.

[0046] Figure 5 This is a schematic diagram of the structure of the music intervention system for driver emotion regulation provided by the present invention;

[0047] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0049] Figure 1 This is one of the flowcharts illustrating the music intervention method for driver emotion regulation provided in this embodiment of the invention, such as... Figure 1 As shown, it includes:

[0050] Step 100: Collect the driver's physiological and audio-visual data;

[0051] Step 200: Perform feature extraction and feature fusion on the physiological indicator data and the audio-visual indicator data to obtain multi-dimensional feature data, and input the multi-dimensional feature data into the emotion recognition model to obtain the emotion type and emotion recognition value;

[0052] Step 300: Input the emotion type and the emotion recognition value into the synchronous emotion music model for audio synthesis, and output a music intervention segment.

[0053] Specifically, in this embodiment of the invention, emotion perception is first performed by collecting physiological data of the driver through wearable sensors and facial image data and voice audio data of the driver from a dashcam in the driver's cab. Then, emotion recognition is performed by extracting and fusing features from the collected driver emotion-related data, and inputting the multi-dimensional feature data into an emotion recognition model to identify the driver's emotion type and value. Finally, music generation is performed by inputting the output driver emotion type and value into a synchronous emotion music model for audio synthesis and post-processing. The entire process is as follows: Figure 2 As shown.

[0054] This invention analyzes various physiological and audio-visual indicators of the driver during the driving process to identify the driver's emotional state. It then uses artificial intelligence algorithms to generate music synchronized with the driver's emotions, guiding and regulating the driver's negative emotional state, thereby relieving stress and enhancing positive emotions.

[0055] Based on the above embodiments, physiological and audio-visual data of the driver are collected, including:

[0056] The driver's heart rate variability, skin temperature, and skin resistance data are collected using wearable sensors.

[0057] The dashcam captures images of the driver's facial expressions and voice data.

[0058] Specifically, the embodiments of the present invention collect indicators from two aspects:

[0059] First, physiological indicators are collected. Wearable sensors worn on the driver's body can effectively collect the driver's physiological indicators, including but not limited to heart rate, skin temperature, and skin resistance. These physiological indicators play an important role in the identification and judgment of emotional state.

[0060] Heart rate variability (HRV): Heart rate variability (HRV) refers to the variation in the difference between successive heartbeat cycles. It reflects the activity state of the cardiac autonomic nervous system and is of great significance for judging emotional state.

[0061] Skin temperature: Skin temperature regulation is part of the body's overall temperature regulation. Skin temperature may rise or fall during periods of emotional stress or anxiety.

[0062] Skin resistance: Skin resistance, also known as body resistance, is related to the amount of sweat secreted by the body's sweat glands. When the body is in a state of tension or anxiety, sweat gland secretion increases, leading to a decrease in skin resistance.

[0063] Secondly, audio and video data collection is crucial. A dashcam in the driver's cab can capture images of the driver's facial expressions and voice data. Analysis of facial expressions reveals the driver's emotional state. For example, a happy driver might smile, while an angry driver might tense up or furrow their brow. These changes in facial expressions can serve as one basis for judging the driver's emotional state. Besides facial images, voice data is also an important factor in identifying the driver's emotional state. The frequency and pitch of a person's voice can change depending on their emotional state. For example, when a driver is afraid or nervous, their voice might become higher or harsher; while when they are calm or relaxed, their voice might become deeper or softer. These vocal characteristics can also be used as a basis for judging the driver's emotional state.

[0064] Based on the above embodiments, feature extraction and feature fusion are performed on the physiological indicator data and the audio / video indicator data to obtain multidimensional feature data, including:

[0065] Deep learning technology is used to extract key feature points from facial expression images;

[0066] Speech recognition technology is used to extract pitch data, speech rate data, and volume data from speech data;

[0067] Frequency domain analysis and time domain analysis were performed on the heart rate variability data to obtain the heart rate frequency characteristics and heart rate fluctuation amplitude of the heart rate variability data;

[0068] The skin temperature data, skin resistance data, key feature points of the facial expression image, pitch data, speech rate data, and volume data of the speech data, as well as the heart rate frequency features and heart rate fluctuation amplitude of the heart rate variability data, are weighted and normalized respectively. The weighted fusion is then used to output the multidimensional feature data.

[0069] Specifically, inputting the multidimensional feature data into the emotion recognition model to obtain emotion type and emotion recognition value includes:

[0070] The initial structure of the emotion recognition model consists of a convolutional neural network, a recurrent neural network, and an attention mechanism. The initial structure of the emotion recognition model is trained using labeled emotion data samples to obtain the emotion recognition model.

[0071] The multidimensional feature data is input into the emotion recognition model, and the emotion type and the emotion recognition value are output. The emotion type includes positive emotion type and negative emotion type, and the emotion recognition value includes positive emotion recognition value and negative emotion recognition value.

[0072] Specifically, in this embodiment of the invention, after collecting various indicators, the emotion recognition step is initiated.

[0073] The first step is feature extraction and fusion, where features are extracted from the collected multidimensional data. For facial images, deep learning techniques, such as commonly used convolutional neural networks, can be used to extract key feature points of facial expressions. For speech data, speech recognition technology can be used to extract features such as pitch, speech rate, and volume. Heart rate variability data, after frequency and time domain analysis, is transformed into the frequency characteristics and fluctuation amplitude of heart rate, as well as skin temperature and skin resistance data, which can be directly used as part of physiological characteristics.

[0074] After feature extraction, feature fusion is performed to combine features from different sources. This process involves feature weighting and normalization to ensure that the impact of different features on the final emotion recognition result is balanced.

[0075] The fused multidimensional feature data is then input into an emotion recognition model, which is a complex deep neural network that combines advanced techniques such as convolutional neural networks, recurrent neural networks, and attention mechanisms. This model has been pre-trained on a large amount of labeled emotion data. After passing through the emotion recognition model, the multidimensional feature data is transformed into different emotion types and emotion values.

[0076] This invention defines 11 common emotion types: happiness, sadness, anger, fear, surprise, disgust, indifference, calmness, excitement, relaxation, positivity, and negativity. Except for the five positive emotions (happiness, calmness, excitement, relaxation, and positivity), the other six are all negative emotions.

[0077] The system also defines the categories of emotion values ​​and their corresponding ranges. The emotion values ​​for each emotion type are normalized. For positive emotions, the range is (0,1), while for negative emotions, the range is (-1,0).

[0078] Based on the above embodiments, the feature is that, before inputting the emotion type and the emotion recognition value into the synchronous emotion music model for audio synthesis and outputting the music intervention segment, the method further includes:

[0079] Identify different types of musical moods, rhythms, melodies, and harmonies;

[0080] A music library is constructed from the different types of musical moods, rhythms, melodies, and harmonies.

[0081] Specifically, before generating music, this embodiment of the invention constructs a corresponding music resource library. When selecting music resources, purely instrumental music is preferred because it can directly touch the depths of human emotions through the combination of melody, harmony, and rhythm. Compared to music with lyrics, purely instrumental music has wider applicability because it is not limited by language and can transcend cultural and contextual boundaries, touching upon common human emotions and spiritual needs. In purely instrumental music, melody is often the main thread of emotional expression. Different melodies can convey various emotions such as joy, sadness, and tranquility. Fast, lively melodies often stimulate vitality and a sense of pleasure, while slow, gentle melodies are more likely to evoke inner peace and relaxation. Harmony creates rich layers in music; through harmonic variations, music can express more complex and profound emotions. Rhythm infuses music with vitality; varying tempos can affect heartbeat and breathing, guiding people into different emotional states. Choosing instrumental music also avoids potential interference from language. Sometimes, specific lyrics may trigger negative emotions or unpleasant memories within an individual. In such cases, choosing instrumental music can avoid this interference and ensure a therapeutic effect. Instrumental music is more like an abstract language, allowing each listener to interpret it according to their own emotional state, unconstrained by external words.

[0082] Table 1 shows the detailed correspondence between music genres, rhythms, melodies, and harmonies in the music resource library.

[0083] Table 1

[0084]

[0085]

[0086] Based on the above embodiments, the emotion type and the emotion recognition value are input into a synchronous emotion music model for audio synthesis, and a music intervention segment is output, including:

[0087] If the emotion type is determined to be a negative emotion type, and the emotion recognition value is greater than the negative emotion threshold, the driver's emotion recognition and perception will continue.

[0088] If the emotion type is determined to be a negative emotion type, and the emotion recognition value is less than or equal to the negative emotion threshold, the driver's emotion is intervened and regulated, and the music intervention segment is generated and played.

[0089] Once the emotion type is positive and the emotion recognition value is greater than the positive emotion threshold, the music intervention segment is stopped.

[0090] The process includes emotional intervention and regulation for the driver, generating and playing the aforementioned music intervention segment, including:

[0091] Obtain the driver's current emotion recognition value, generate an intervention music emotion value based on the current emotion recognition value, and make the absolute difference between the intervention music emotion value and the current emotion recognition value less than a preset constraint value;

[0092] The driver's emotion type and the intervention music emotion value are encoded by data encoding. The emotion type is converted into a label vector and the intervention music emotion value is converted into a continuous value to obtain the emotion code.

[0093] Music fragments are retrieved from the music library based on the stated emotion type and the stated intervention music emotion value;

[0094] The emotion code and the music fragment are input into the synchronous emotion music model, and an audio waveform with a preset duration is output.

[0095] After the preset duration of playback, the emotion type and the emotion recognition value are repeatedly identified, so that the absolute difference between the intervention music emotion value and the emotion recognition value is less than the preset constraint value.

[0096] The audio waveform contains multiple audio tracks, each corresponding to a different instrument or sound element.

[0097] Specifically, such as Figure 3 As shown, the steps for music generation using a synchronous emotional music model in this embodiment of the invention include:

[0098] (1) Based on the driver's emotion type (ET) and emotion value (EV) generated by the emotion recognition model dr ), determine whether it is a negative emotion, if it is a negative emotion, and when EV dr EVT neg Meanwhile, it continues to sense and recognize the driver's emotions. Among them, EVT... neg EVT represents the threshold for negative emotions. neg The typical value is -0.5;

[0099] (2) When EV dr ≦EVT neg At this point, it indicates a need to intervene and regulate the driver's negative emotions, and the generation of synchronized emotional music begins, such as... Figure 4 As shown, this is represented as time 0 on the time axis;

[0100] (3) Based on the driver's current emotional state (EV) dr This generates an emotional value (EV) for the music being intervened in. mi ), EV mi The initial value must be lower than EV dr，And it satisfies the constraint |EV mi -EV dr |≦Δev, where Δev is typically 0.2;

[0101] (4) Driver's emotional type (ET) and the emotional value (EV) of the intervention music mi The data is encoded by converting emotion types into corresponding label vectors and emotion values ​​into continuous numerical values. Specifically, for emotion types, they can be converted into corresponding label vectors. For example, 11 emotion types (such as happiness, sadness, anger, etc.) can be represented as vectors consisting of 11 binary bits, where the i-th bit being 1 indicates emotion type i, and 0 indicates a different emotion. Thus, each emotion type can be represented by an 11-bit binary vector. For emotion values, they can be converted into continuous numerical values. For example, emotion values ​​0-1 can be converted into integers from 0-100, where 0 represents no emotion and 100 represents the strongest emotion. Thus, each emotion value can be represented by an integer.

[0102] (5) Based on the driver's emotional type (ET) and the emotional value (EV) of the intervention music. mi It retrieves matching music clips from a large library of music materials that have been categorized and labeled with emotion types and emotion values;

[0103] (6) Input the emotion code generated in (4) and the music fragment retrieved in (5) into the WaveNet model as input parameters. After prior training, the WaveNet model can learn the complex patterns and emotional expressions of music. The model will generate an audio waveform of duration Δt, which contains the emotional elements desired by the user;

[0104] (7) After the music plays for a duration of Δt, repeat step (1). During this process, the constraint condition |EV must be maintained. mi -EV dr ∣≦Δev;

[0105] (8) When the driver's emotion type (ET) generated by the emotion recognition model is a positive emotion, and the emotion value EV dr EVT pos At this point, it indicates that the driver's emotions have been regulated to a normal level, such as... Figure 4 The t shown n At this moment, the synchronized emotional music stops playing. (EVT) pos This represents the positive emotion threshold, which is typically 0.6.

[0106] It's important to note that the audio waveforms generated by the WaveNet model may contain multiple audio tracks representing different instruments or sound elements. To better align with musical characteristics, parameters such as rhythm, pitch, and volume can be adjusted to make the audio more harmonious and pleasing to the ear. In post-processing, these audio tracks need to be combined into a single, complete musical segment. Rhythm, pitch, and other elements are adjusted based on emotional information, and the generated audio undergoes smoothing and mixing operations to improve sound quality and coherence.

[0107] Understandably, compared to traditional music therapy that uses music libraries to regulate or recommend preferred music to improve negative emotions, this invention leverages the correlation between musical rhythm, melody, harmony, and other elements and a person's physiological and psychological states to evoke emotional resonance. Through this resonance between music and emotion, we can guide and regulate the driver's negative emotional state, thereby relieving stress and enhancing positive emotions. Furthermore, it employs artificial intelligence algorithms to dynamically generate music synchronized with the driver's emotions. This algorithm can dynamically generate music that matches the driver's current emotional type and value, and the generated audio quality is high, accurately expressing emotions.

[0108] The music intervention system for driver emotion regulation provided by the present invention is described below. The music intervention system for driver emotion regulation described below can be referred to in correspondence with the music intervention method for driver emotion regulation described above.

[0109] Figure 5 This is a schematic diagram of the structure of a music intervention system for driver emotion regulation provided in an embodiment of the present invention, as shown below. Figure 5 As shown, it includes: an emotion perception module 51, an emotion recognition module 52, and a music generation module 53, wherein:

[0110] The emotion perception module 51 is used to collect the driver's physiological index data and audio-visual index data; the emotion recognition module 52 is used to extract and fuse features from the physiological index data and the audio-visual index data to obtain multi-dimensional feature data, and input the multi-dimensional feature data into the emotion recognition model to obtain the emotion type and emotion recognition value; the music generation module 53 is used to input the emotion type and the emotion recognition value into the synchronous emotion music model for audio synthesis and output music intervention segments.

[0111] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a music intervention method for driver emotion regulation. This method includes: collecting physiological index data and audio-visual index data of the driver; performing feature extraction and feature fusion on the physiological index data and the audio-visual index data to obtain multi-dimensional feature data; inputting the multi-dimensional feature data into an emotion recognition model to obtain emotion type and emotion recognition value; inputting the emotion type and the emotion recognition value into a synchronous emotion music model for audio synthesis, and outputting a music intervention segment.

[0112] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A music intervention method for driver emotion regulation, characterized in that, include: Collect physiological and audio-visual data of the driver; Feature extraction and feature fusion are performed on the physiological indicator data and the audio and video indicator data to obtain multidimensional feature data. The multidimensional feature data is then input into the emotion recognition model to obtain the emotion type and emotion recognition value. The emotion type and the emotion recognition value are input into a synchronous emotion music model for audio synthesis, and a music intervention segment is output. The emotion type and the emotion recognition value are input into a synchronous emotion music model for audio synthesis, and the output music intervention segment includes: If the emotion type is determined to be a negative emotion type, and the emotion recognition value is greater than the negative emotion threshold, the driver's emotion recognition and perception will continue. If the emotion type is determined to be a negative emotion type, and the emotion recognition value is less than or equal to the negative emotion threshold, the driver's emotion is intervened and regulated, and the music intervention segment is generated and played. Once the emotion type is positive and the emotion recognition value is greater than the positive emotion threshold, stop playing the music intervention segment; Emotional intervention and regulation for the driver, including generating and playing the aforementioned music intervention segment, including: Obtain the driver's current emotion recognition value, generate an intervention music emotion value based on the current emotion recognition value, and make the absolute difference between the intervention music emotion value and the current emotion recognition value less than a preset constraint value; The driver's emotion type and the intervention music emotion value are encoded by data encoding. The emotion type is converted into a label vector and the intervention music emotion value is converted into a continuous value to obtain the emotion code. Music fragments are retrieved from the music library based on the stated emotion type and the stated intervention music emotion value; The emotion code and the music fragment are input into the synchronous emotion music model, and an audio waveform with a preset duration is output. After the preset duration of playback, the emotion type and the emotion recognition value are repeatedly identified, so that the absolute difference between the intervention music emotion value and the emotion recognition value is less than the preset constraint value. The audio waveform contains multiple audio tracks, which correspond to different musical instruments or different sound elements.

2. The music intervention method for driver emotion regulation according to claim 1, characterized in that, Collect the driver's physiological and audio-visual data, including: The driver's heart rate variability, skin temperature, and skin resistance data are collected using wearable sensors. The dashcam captures images of the driver's facial expressions and voice data.

3. The music intervention method for driver emotion regulation according to claim 1, characterized in that, Feature extraction and feature fusion are performed on the physiological indicator data and the audio / video indicator data to obtain multidimensional feature data, including: Deep learning technology is used to extract key feature points from facial expression images; Speech recognition technology is used to extract pitch data, speech rate data, and volume data from speech data; Frequency domain analysis and time domain analysis were performed on the heart rate variability data to obtain the heart rate frequency characteristics and heart rate fluctuation amplitude of the heart rate variability data; The skin temperature data, skin resistance data, key feature points of the facial expression image, pitch data, speech rate data, and volume data of the speech data, as well as the heart rate frequency features and heart rate fluctuation amplitude of the heart rate variability data, are weighted and normalized respectively. The weighted fusion is then used to output the multidimensional feature data.

4. The music intervention method for driver emotion regulation according to claim 3, characterized in that, The multidimensional feature data is input into the emotion recognition model to obtain the emotion type and emotion recognition value, including: The initial structure of the emotion recognition model consists of a convolutional neural network, a recurrent neural network, and an attention mechanism. The initial structure of the emotion recognition model is trained using labeled emotion data samples to obtain the emotion recognition model. The multidimensional feature data is input into the emotion recognition model, and the emotion type and the emotion recognition value are output. The emotion type includes positive emotion type and negative emotion type, and the emotion recognition value includes positive emotion recognition value and negative emotion recognition value.

5. The music intervention method for driver emotion regulation according to claim 1, characterized in that, Before inputting the emotion type and the emotion recognition value into the synchronous emotion music model for audio synthesis and outputting the music intervention segment, the process also includes: Identify different types of musical moods, rhythms, melodies, and harmonies; A music library is constructed from the different types of musical moods, rhythms, melodies, and harmonies.

6. A music intervention system for driver emotion regulation, based on the music intervention method for driver emotion regulation according to any one of claims 1 to 5, characterized in that, include: The emotion perception module is used to collect the driver's physiological and audio-visual data. An emotion recognition module is used to extract and fuse features from the physiological indicator data and the audio-visual indicator data to obtain multi-dimensional feature data, and input the multi-dimensional feature data into the emotion recognition model to obtain the emotion type and emotion recognition value. The music generation module is used to input the emotion type and the emotion recognition value into the synchronous emotion music model for audio synthesis and output music intervention segments.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the music intervention method for driver emotion regulation as described in any one of claims 1 to 5.

Images