Multi-mode vehicle-mounted music interaction system and method based on driving situation

Abstract

The invention discloses a multi-mode vehicle-mounted music interaction system and method based on a driving situation, and relates to the technical field of intelligent networked automobile man-machine interaction, and the method comprises the steps that a data collection and preprocessing module receives an original data stream and generates standardized multi-source data; the multi-modal dynamic arbitration engine module uses a sliding window synchronization mechanism to align data to generate a situation snapshot, and determines an interaction mode priority and a dynamic weight coefficient through a dynamic conflict decision matrix; the situation awareness recommendation system module constructs a three-dimensional situation feature model and generates a situation vector, calculates a recommendation score in combination with a music feature vector library and generates a playlist; and the instruction execution and feedback module executes music playing and light adjustment. According to the method, the problem of data asynchronization is solved through a sliding window mechanism, the interaction strategy is dynamically adjusted through the decision matrix to guarantee safety, emotion guidance is achieved through matching of the situation vector and the music feature vector, and the cabin interaction safety and the user experience are improved.

Description

Technical Field

[0001] This application relates to the field of human-machine interaction technology for intelligent connected vehicles, and in particular to a multimodal in-vehicle music interaction system and method based on driving scenarios. Background Technology

[0002] With the development of automotive electronics technology, in-vehicle infotainment systems have become a core component of the cabin experience. Drivers routinely select music tracks and adjust volume via touchscreen displays while driving. This visually-driven interaction method forces drivers to briefly take their eyes off the road, resulting in unmonitored driving and increasing safety hazards. Although voice interaction technology has been introduced into the cabin, wind noise at high speeds and interference from conversations inside the vehicle reduce the recognition rate of voice commands, compromising the stability of the interaction.

[0003] While existing in-vehicle systems access multi-source data, including vehicle bus, microphones, and biosensors, the data generation frequency and transmission latency of different sensors inherently differ. Traditional systems lack time-series alignment processing for multi-source heterogeneous data, making it difficult for the system to build accurate real-time driving scenario models and to dynamically arbitrate and prioritize multiple interaction methods such as touch, voice, and gestures under complex operating conditions.

[0004] Furthermore, existing in-car music playback functions primarily rely on mechanical recommendations based on users' historical preferences, neglecting changes in the real-time driving environment and the driver's physiological and psychological state. In situations where traffic congestion triggers anxiety or long-distance driving induces fatigue, music content pushed by the system that is mismatched with the current situation can exacerbate the driver's negative emotions. Simultaneously, external environmental noise fluctuates with vehicle speed, and fixed audio output parameters cannot compensate for hearing loss in real time. Moreover, the lack of a mechanism to adjust the music tempo according to the user's physiological characteristics makes it difficult to effectively regulate and guide the driver's emotions. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal in-vehicle music interaction system and method based on driving scenarios, which at least solves the technical problems of existing in-vehicle systems where the lack of time alignment in multi-source data processing leads to decision-making bias, single interaction methods pose safety hazards in complex environments, and music recommendation logic is detached from real-time driving scenarios and cannot regulate the emotional state of drivers and passengers.

[0006] This invention provides the following solution:

[0007] The first aspect of this invention provides a multimodal in-vehicle music interaction system based on driving scenarios. The system is configured to integrate vehicle hardware resources and software algorithms to address the problems of asynchronous multi-source data, conflicting interaction methods, and low contextual adaptability of music recommendations in the in-vehicle environment. The system mainly includes a data acquisition and preprocessing module, a multimodal dynamic arbitration engine module, a context-aware recommendation system module, and an instruction execution and feedback module.

[0008] The data acquisition and preprocessing module, serving as the system's sensing entry point, is responsible for receiving the raw data stream from the data acquisition unit. This raw data stream encompasses vehicle state data describing the vehicle's physical motion, occupant state data describing the user's physiological and behavioral characteristics, and environmental state data describing external and internal environmental conditions. To eliminate dimensional differences and noise interference between data from different sensors, the module performs data denoising and normalization processing, outputting standardized multi-source data to provide a unified data foundation for subsequent decision-making and recommendations.

[0009] The multimodal dynamic arbitration engine module is the core of the system's decision-making, primarily addressing the temporal asynchrony of multi-source data and security conflicts in interaction methods. To address data asynchrony caused by inconsistent sampling frequencies of different sensors, the module employs a timestamp-based sliding window synchronization mechanism. This mechanism defines a fixed-length sliding window, captures and aggregates multi-source data within the same time interval, generating a snapshot of the current situation reflecting the complete state at that moment, ensuring subsequent decisions are based on time-aligned data. Based on this snapshot, the module uses a dynamic conflict decision matrix to adjudicate current interaction requests. This matrix contains multi-level logical branches, dynamically determining the priority of interaction methods and dynamic weight coefficients for subsequent recommendation calculations based on changes in vehicle speed, noise levels, and passenger biometrics. This mechanism ensures optimized reliability and user experience in human-computer interaction while prioritizing driving safety.

[0010] The context-aware recommendation system module is responsible for achieving accurate music content matching. Based on standardized multi-source data, the module constructs a three-dimensional context feature model from three dimensions: driving state, biometrics, and environmental data, generating a context vector quantifying the current scene. Simultaneously, the module combines song feature vectors stored in a music feature vector library with user historical preference data for recommendation calculation. The dimensions of the song feature vectors correspond to the context vectors, respectively representing the applicable driving intensity, user physiological activation level, and environmental noise level. The module calculates the similarity between the context vector and the song feature vector, and combines user preferences and dynamic weight coefficients to derive a recommendation score for each song. This calculation logic can flexibly switch recommendation strategies between safety-first, emotion-first, or user-preference-first based on the weights output by the arbitration engine.

[0011] The instruction execution and feedback module is responsible for translating decision results into physical actions. This module controls the operation mode of the human-computer interaction unit according to the determined interaction method priority and executes playback operations based on the generated music playlist. To further enhance the auditory experience and emotional guidance effect, the module also features sound field compensation and dynamic BPM adjustment functions. It dynamically adjusts audio output parameters based on the ambient noise level and fine-tunes the music tempo based on the user's physiological characteristics, achieving various coordinations between music rhythm and the user's physiological rhythm.

[0012] A second aspect of this invention provides a multimodal in-vehicle music interaction method based on driving scenarios. This method achieves intelligent in-vehicle music interaction through a series of ordered data processing and decision-making steps.

[0013] The method first performs data acquisition and preprocessing steps to obtain raw data streams containing vehicle, personnel and environmental information, and then cleans and standardizes them to obtain standardized multi-source data.

[0014] Subsequently, the method performs a multimodal arbitration step. A sliding window synchronization mechanism is used to align asynchronous, standardized multi-source data along the timeline and generate a contextual snapshot. This contextual snapshot is then input into a preset dynamic conflict decision matrix for matching, outputting the priority of interaction methods in the current scenario and dynamic weight coefficients for music recommendation. This step establishes the system's interaction strategy and recommendation focus at the current moment.

[0015] Next, the method performs context awareness and recommendation calculation steps. A context vector is constructed based on standardized multi-source data, incorporating driving intensity, user physiological activation level, and cabin noise level. The similarity between the context vector and song feature vectors in a music feature vector library is calculated, and combined with user historical preference data and dynamic weighting coefficients, a recommendation score is calculated for each song. The logic for calculating the recommendation score can respond to different driving scenarios: in general scenarios, it balances contextual adaptation and user preferences; in specific scenarios, it focuses on achieving emotion regulation through matching and guiding music energy with the user's physiological state.

[0016] Finally, the method executes the instruction execution steps. Based on the generated music playlist and interaction mode priority, the vehicle hardware is driven to perform music playback, lighting adjustment, and interaction mode switching, and the final output effect is optimized using sound field compensation and dynamic beat adjustment technology.

[0017] In summary, this invention solves the problem of asynchronous data through a sliding window mechanism, addresses the issues of interaction security and conflict through a dynamic conflict decision matrix, and resolves the problem of context adaptation through three-dimensional feature modeling and vector matching, thereby constructing a safe, reliable, and emotion-regulating intelligent in-vehicle music interaction solution.

[0018] The above solution achieves the following beneficial technical effects:

[0019] This application utilizes a multimodal dynamic arbitration engine module and a timestamp-based sliding window synchronization mechanism to align asynchronous multi-source data and generate a contextual snapshot. It also employs a dynamic conflict decision matrix to dynamically adjust the priority of interaction methods based on vehicle speed and noise levels. This approach resolves the temporal asynchrony issue of multi-source sensor data, automatically shielding high-risk touch operations and switching to voice and gesture interaction in high-speed driving and high-noise environments. This improves the efficiency and reliability of human-machine interaction while ensuring driving safety.

[0020] This application constructs a three-dimensional contextual feature model through a context-aware recommendation system module, incorporating driving intensity features, user physiological activation features, and cabin noise levels. It then combines dynamic weighting coefficients with a music feature vector library for similarity matching. This setup allows the system to flexibly switch recommendation strategies between safety priority, emotion regulation priority, and user preference priority based on the urgency of the current driving situation. Consequently, it alleviates driving fatigue and calms agitated emotions by playing music that matches the energy level of the current situation.

[0021] This application utilizes sound field compensation and dynamic BPM adjustment functions through a command execution and feedback module. It dynamically adjusts the speaker equalizer parameters based on environmental data and fine-tunes the music tempo based on the user's physiological activation characteristics. This approach compensates for hearing loss caused by external environmental noise and, through the subconscious synergy between music rhythm and the user's circadian rhythm, enhances the auditory experience within the cabin and strengthens the positive guidance effect on the emotional state of passengers. Attached Figure Description

[0022] Figure 1 This is a flowchart of a multimodal dynamic arbitration engine provided in one embodiment of the present invention;

[0023] Figure 2 This is a flowchart of a three-dimensional feature modeling process provided in one embodiment of the present invention;

[0024] Figure 3 This is a flowchart of three-dimensional context feature modeling and recommendation provided in a specific embodiment of the present invention. Detailed Implementation

[0025] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] See attached document Figure 1 This invention provides a multimodal in-vehicle music interaction system based on driving scenarios. The system includes a central processing unit, a memory, a data acquisition unit, a human-computer interaction unit, and a network communication unit.

[0027] The central processing unit (CPU) is the control core of the system and is implemented using an in-vehicle system-on-a-chip (SoC). The CPU is responsible for executing instructions stored in memory, processing data from the data acquisition unit, and generating control commands based on the processing results to drive the human-machine interface unit.

[0028] The memory is connected to the central processing unit and is used to store the operating system, applications, and temporary data and configuration parameters generated during system operation. The memory includes random access memory (RAM) and non-volatile memory (ROM or flash memory). A user history database is also built into the memory to record the user's historical playback behavior for different songs (including single-track loop counts, skip records, favorites, and cumulative playback counts) to generate user preference data.

[0029] The data acquisition unit is responsible for acquiring raw data from the vehicle, occupants, and environment. The data acquisition unit includes a CAN bus interface, camera, microphone array, biosensors, and gesture recognition unit.

[0030] The CAN bus interface connects to the vehicle's Controller Area Network (CAN) bus to acquire vehicle status data in real time. This vehicle status data includes vehicle speed data, acceleration data, steering wheel angle data, accelerator pedal depth data, brake pedal depth data, and turn signal status data.

[0031] The camera is installed inside the vehicle cabin, facing the driver and passengers. It is used to capture images of the drivers' and passengers' facial micro-expressions and body postures.

[0032] The microphone array is positioned on the top of the vehicle cabin or the front console to collect audio signals from inside the cabin, including voice commands from the driver and passengers and ambient noise within the cabin.

[0033] Biosensors are integrated into the steering wheel, seat, or seatbelt to acquire the driver's physiological data non-contactly, including heart rate and respiratory rate data.

[0034] The gesture recognition unit includes an infrared sensor or a time-of-flight (ToF) sensor to capture the hand gesture data of the driver and passengers.

[0035] The human-computer interaction unit (HCI) is responsible for enabling information exchange between the system and the user. The HCI includes a touchscreen display, speakers, and an ambient lighting controller.

[0036] Touchscreen displays are used to visually present information to users and receive touch input commands from users.

[0037] The speaker is used to play music, system prompts, and voice feedback.

[0038] The ambient lighting controller is used to control the color, brightness, and flashing mode of the ambient lighting in the cabin according to instructions issued by the central processing unit.

[0039] The network communication unit is used to establish a wireless connection between the system and the external network, and to obtain weather information and real-time traffic information from the network server.

[0040] In this embodiment, the memory, data acquisition unit, human-computer interaction unit, and network communication unit all establish data communication connections with the central processing unit through the system bus to realize data exchange and instruction transmission.

[0041] See attached document Figure 1 The system of this invention functionally comprises multiple functional modules embedded in memory and executed by a central processing unit. These functional modules include: a data acquisition and preprocessing module, a multimodal dynamic arbitration engine module, a context-aware recommendation system module, and an instruction execution and feedback module.

[0042] The data acquisition and preprocessing module receives the raw data stream from the data acquisition unit. The raw data stream includes vehicle status data, occupant status data, and environmental status data. The module performs preprocessing operations on the raw data stream, including data denoising and data normalization, ultimately generating standardized multi-source data. This standardized multi-source data is then output to the multimodal dynamic arbitration engine module and the context-aware recommendation system module.

[0043] The multimodal dynamic arbitration engine module receives standardized multi-source data from the data acquisition and preprocessing module, as well as user interaction commands from the human-computer interaction unit. Based on preset conflict decision rules, the module adjudicates concurrent user interaction commands, determining the priority of the interaction method in the current scenario. The module also performs dynamic weight allocation based on the standardized multi-source data and outputs the interaction method priority and dynamic weight coefficients to the context-aware recommendation system module and the command execution and feedback module.

[0044] The context-aware recommendation system module receives standardized multi-source data from the data acquisition and preprocessing module, as well as dynamic weight coefficients from the multimodal dynamic arbitration engine module. Based on the standardized multi-source data, the module constructs a contextual feature model from three dimensions and combines it with the song feature vectors of each song in the music library stored in memory, along with the user's historical preference data, to calculate a recommendation score using a pre-defined scoring model. The module then generates a sorted music playlist based on the recommendation score and outputs the playlist to the instruction execution and feedback module.

[0045] The instruction execution and feedback module receives adjudicated user interaction instructions from the multimodal dynamic arbitration engine module and music playlists from the context-aware recommendation system module. Based on the instructions and playlists, the module generates hardware control instructions. These instructions are used to drive the speakers in the human-machine interface unit to play music, display information on the touchscreen, and control the ambient lighting controller to adjust the cabin lighting effects.

[0046] One function of the data acquisition and preprocessing module is to collect driving status data. The module establishes communication with the vehicle's controller area network bus through the system's CAN bus interface.

[0047] The data acquisition and preprocessing module listens for and receives data frames transmitted on the controller area network bus via the CAN bus interface. The module parses the received data frames according to the preset message identifier (MessageID) to extract the vehicle's dynamic operating parameters.

[0048] The extracted vehicle dynamic operating parameters constitute driving status data. Driving status data includes: vehicle speed data, longitudinal and lateral acceleration data, steering wheel angle data, accelerator pedal depth data, brake pedal depth data, and turn signal status data.

[0049] The data acquisition and preprocessing module periodically performs the above data extraction operation at a preset sampling frequency, thereby generating a series of driving status data streams with timestamps, and providing the data streams to subsequent functional modules for processing.

[0050] Another function of the data acquisition and preprocessing module is to collect driver and passenger status data. Driver and passenger status data consists of two parts: physiological indicator data and behavioral characteristic data.

[0051] The module collects physiological data of drivers and passengers through biosensors integrated into the system. These biosensors are located in the steering wheel, seat, and seat belt, and use photoplethysmography (PPG) technology to acquire heart rate and respiratory rate data.

[0052] The module captures facial images of drivers and passengers using cameras within the system. The internal computer vision processing unit analyzes these images, first locating key facial features, and then, based on the position and changes of these features, outputting micro-expression classification results, gaze direction, and the number of yawns per unit time. The micro-expression classification results include categories such as fatigue, joy, and irritability.

[0053] The module also collects the voice signals of the driver and passengers through the system's microphone array. The audio signal processing unit inside the module analyzes the voice signals and extracts acoustic features. The acoustic features include the fundamental frequency (pitch) and speech rate of the voice signal.

[0054] The module also collects the gesture data of drivers and passengers through the gesture recognition unit in the system.

[0055] Physiological indicators, micro-expression classification results, gaze direction, number of yawns per unit time, acoustic features, and gesture data collectively constitute the driver and passenger status data. The data acquisition and preprocessing module adds a timestamp to each set of data in the driver and passenger status data stream and transmits the timestamped data stream to subsequent functional modules.

[0056] Another function of the data acquisition and preprocessing module is to collect environmental status data. This environmental status data consists of cabin acoustic data and external environmental information.

[0057] The module collects audio signals from within the cockpit using a microphone array within the system. The audio signal processing unit inside the module analyzes the audio signals and calculates the current sound pressure level (SPL) as a quantitative indicator of the noise level within the cockpit.

[0058] The module acquires external environmental information through the system's network communication unit. It then sends a request containing the vehicle's current geographical coordinates to a pre-set weather information server via this network communication unit. The server responds to the request and returns a data packet containing the current weather conditions, categorized as sunny, rainy, or snowy.

[0059] The module also uses a network communication unit to combine the vehicle's geographic location coordinates to obtain the current road type information from map data services. The road type information includes classifications such as highways, urban roads, and rural roads.

[0060] The data acquisition and preprocessing module adds timestamps to the collected cabin noise level, weather conditions and road type information to form an environmental status data stream, and then transmits the data stream to subsequent functional modules.

[0061] Another function of the data acquisition and preprocessing module is to perform preprocessing operations on the acquired driving status data, passenger status data, and environmental status data.

[0062] Preprocessing operations include data denoising and data normalization. For continuous numerical signals, including heart rate and respiratory rate data, the module applies a digital low-pass filter to remove high-frequency noise components from the data.

[0063] For numerical data with different physical dimensions, including vehicle speed and heart rate data, the module employs normalization processing to map the values ​​to a unified numerical range of [0, 1]. This operation eliminates the differences in numerical ranges between different data dimensions.

[0064] Normalization is performed according to a minimum-maximum scaling formula.

[0065] ;

[0066] in: This represents the normalized value; This represents the current original value; This represents the preset minimum value for the data type. This indicates the preset maximum value for the data type.

[0067] For categorical data, which includes weather conditions and road type information, the module uses the One-Hot Encoding method to convert the categorical data into a multi-dimensional binary vector.

[0068] The data acquisition and preprocessing module combines the preprocessed and normalized data to form standardized multi-source data, and outputs the standardized multi-source data to the multimodal dynamic arbitration engine module and the context-aware recommendation system module.

[0069] See attached document Figure 2 One function of the multimodal dynamic arbitration engine module is to realize the synchronous processing of asynchronous data from multiple sources.

[0070] The multimodal dynamic arbitration engine module receives multiple timestamped data streams from the data acquisition and preprocessing module. The data streams include driving status data streams, passenger status data streams, and environmental status data streams, and the generation frequency and transmission latency of the data streams are different.

[0071] To align asynchronous data streams, the multimodal dynamic arbitration engine module incorporates a sliding window synchronization mechanism. This mechanism defines a sliding window of fixed duration, preferably set to 400 milliseconds (ms).

[0072] The module establishes an independent data buffer for each input data stream. When a new data point arrives, the data point, along with its timestamp, is stored in the corresponding buffer.

[0073] The sliding window synchronization mechanism advances with a preset sliding step size. At each time step, the mechanism defines a time interval based on the current timestamp. All data points whose timestamps fall within the time interval are extracted from their respective buffers.

[0074] All extracted data points are combined into a set, which includes multi-source data inputs such as gesture data, heart rate data, and voice signal data. For data types with multiple measurements within a time interval, such as heart rate data, the mechanism applies a preset aggregation function, such as a mean function, to calculate a single representative value.

[0075] All the data after aggregation together constitute a synchronized context snapshot. A context snapshot is a data structure that contains a unified state description of all data sources within a time interval.

[0076] The multimodal dynamic arbitration engine module then performs dynamic weight allocation on the data in the context snapshot, generating dynamic weight coefficients for subsequent recommendations.

[0077] The module then inputs the scenario snapshot and weight allocation results into a dynamic conflict decision matrix. The dynamic conflict decision matrix is ​​a set of pre-defined conditional action rules.

[0078] The module takes a context snapshot as input and performs a match within a dynamic conflict decision matrix. Upon successful matching, the module outputs a final interaction decision. This decision includes the current interaction rules and a set of dynamic weighting coefficients for music recommendation.

[0079] The dynamic conflict decision matrix also has a self-learning function. The module continuously records the user's interaction history data (e.g., whether the user performed a "skip" or "adjust volume" operation after recommending a specific song). Based on the historical data, the module dynamically adjusts the weight parameters of each decision rule in the matrix, thereby evolving from initial general rules to personalized intelligent decisions that adapt to the user's individual habits.

[0080] The rule-based decision-making logic of the dynamic conflict decision matrix is ​​divided into three main branches:

[0081] The first branch is the vehicle speed adjudication branch. The rules include a safety priority adjudication layer. When the vehicle speed data in the scenario snapshot exceeds a preset safe speed threshold, the module matches the corresponding safety priority rule and generates an interaction rule. The interaction rule is defined as prioritizing voice interaction, disabling touch functionality on all areas of the touchscreen except for navigation and emergency call functions, and generating a command to broadcast a safety prompt voice through the speaker.

[0082] The second branch is the reliability assessment branch. When the cabin noise level in the scenario snapshot exceeds a preset noise threshold (which will lead to a decrease in voice recognition rate), the module matches the corresponding reliability assessment rule. The rule is defined as: increasing the priority weight of gesture interaction and touch interaction, while decreasing the priority of voice interaction, and generating prompts to guide the user to use gesture operation or close the window.

[0083] The third branch is the multi-passenger biosignal adjudication branch. The module first detects the number of passengers in the vehicle.

[0084] If there is only one passenger (driver) in the vehicle, the judgment is made directly based on the driver's biosignals.

[0085] If there are multiple passengers in the vehicle, the module comprehensively considers the biometrics of all individuals. If the biometric data of both the driver and passengers remain within a stable range (i.e., the rate of change is below a preset threshold) within a preset time window, the module calculates the average of the biometric data of all individuals as a unified user status feature. If any user's biometrics undergo a significant change within a short period (i.e., the change gradient exceeds a preset threshold, such as a sudden increase in the heart rate of the front passenger), the module assigns increased weight to the user with the most significant change in biometric signal to prioritize responding to their potential emergency or emotional needs.

[0086] The dynamic conflict decision matrix also includes a biosignal-based adjudication layer. When the heart rate value in the contextual snapshot consistently exceeds a preset heart rate threshold, and the micro-expression classification result indicates irritability, the module matches the corresponding emotion regulation rule. The rule is defined as: the dynamic weight coefficients used for music recommendation... Set a first preset value (e.g., a high value) to the dynamic weighting coefficient. Set a second preset value (e.g., a low value) to ensure that the system prioritizes responding to current safety and mood regulation needs, rather than user preferences.

[0087] See attached document Figure 3 , Figure 3 This is a flowchart of a three-dimensional context feature modeling and recommendation process according to an embodiment of the present invention. One function of the context-aware recommendation system module is to construct a three-dimensional context feature model, quantizing and fusing arbitrated multi-source input data into a unified context vector.

[0088] The context-aware recommendation system module receives standardized multi-source data from the data acquisition and preprocessing module. The module employs a three-dimensional feature modeling approach, constructing features from three dimensions: for the driving state dimension, it collects acceleration and steering wheel angle data; for the biometric dimension, it collects heart rate and micro-expression data; and for the environmental data dimension, it collects noise and weather data.

[0089] The module first calculates a driving intensity characteristic value. This driving intensity characteristic value is calculated using a weighted summation formula.

[0090] ;

[0091] in: This represents the characteristic value of driving intensity; Preset weighting coefficients representing acceleration; A preset weighting coefficient representing the steering wheel angular velocity; This represents the normalized value of the vehicle's longitudinal acceleration obtained from driving status data; This represents the normalized value of the steering wheel angular velocity obtained from driving status data.

[0092] The module then calculates a user's physiological activation characteristic value. This characteristic value is derived using a comprehensive scoring formula.

[0093] ;

[0094] in: This represents the user's physiological activation characteristic value; The preset weighting coefficients representing heart rate; Preset weighting coefficients representing micro-expressions; Preset weighting coefficients representing the frequency of yawning; This represents the normalized value of the current heart rate obtained from the driver and passenger status data; This represents a normalized value of a preset baseline heart rate; This represents the pleasure score obtained from the micro-expression classification results; This represents the normalized value of the number of yawns per unit time obtained from driver and passenger status data.

[0095] Finally, the module combines the calculated driving intensity feature value, user physiological activation feature value, and environmental state feature extracted from standardized multi-source data into a multi-dimensional context vector. .

[0096] ;

[0097] in: This represents the final generated context vector; This represents the characteristic value of driving intensity; This represents the user's physiological activation characteristics (e.g., reflecting the user's excitement or fatigue level); This represents a normalized value of the cabin noise level obtained from environmental condition data (reflecting the level of environmental noise).

[0098] The context-aware recommendation system module will generate context vectors The output is then fed into the subsequent scoring model for processing.

[0099] Another function of the context-aware recommendation system module is to construct a corresponding song feature vector for each song in the music library stored in memory during the system initialization phase or when the music library is updated.

[0100] Unlike existing technologies, the song feature vector of this invention not only includes audio physical features but also the contextual label applicable to the song. The context-aware recommendation system module generates a contextual vector for each song using a pre-set audio analysis algorithm. Song feature vectors with one-to-one dimension .

[0101] ;

[0102] in: Indicates the first The feature vector of the song; This indicates the recommended driving intensity level for the song (for example, fast-paced rock music has a higher value, suitable for aggressive driving or to stay alert). This indicates the physiological activation level of the user matched with the song (e.g., high-energy music matches high excitement). This indicates the ambient noise level to which the song is suitable. These values ​​are all mapped to the range [0, 1].

[0103] The context-aware recommendation system module performs the above labeling process for all songs in the music library and stores the generated set of feature vectors for all songs in memory.

[0104] See attached document Figure 3 Another function of the context-aware recommendation system module is to calculate music recommendation scores based on a scoring model. The module receives context vectors generated by itself. It reads music feature vector library and user history preference data from memory, and receives dynamic weight coefficients from the multimodal dynamic arbitration engine module.

[0105] The context-aware recommendation system module utilizes the received context vectors Each song feature vector in the music feature vector library User preference for the song and dynamic weighting coefficients and Calculate a recommendation score for each song in the music library. .

[0106] Recommended score The calculation uses the following linear weighted formula:

[0107] ;

[0108] in: Indicates the first The final recommended score for the song; This represents the dynamic weight coefficients related to context fit, output by the multimodal dynamic arbitration engine module. This represents the dynamic weight coefficients related to user preferences, output by the multimodal dynamic arbitration engine module. The cosine similarity between the context vector and the song feature vector is represented. Indicates user U's relationship with the first... The preference value for a song is calculated based on the user's historical behavior (such as number of plays and collection records) and ranges from [0, 1].

[0109] Cosine similarity The calculation formula is as follows:

[0110] ;

[0111] in: Representation of context vector In the The values ​​of each dimension; Indicates the first The feature vector of the song In the The values ​​of each dimension; Indicates the number of dimensions of a vector.

[0112] The context-aware recommendation system module repeatedly performs the above scoring calculation process for all songs in the music library.

[0113] After calculating the scores for all songs, the module assigns all songs their recommended scores. Sort the music in descending order to generate a sorted playlist.

[0114] Finally, the system executes the output of two branches based on the model results:

[0115] The first branch is the music playback branch. The context-aware recommendation system module outputs the sorted music playlist to the instruction execution and feedback module, matching music with different rhythms to adapt to different scenarios.

[0116] The second branch is the ambient lighting adjustment branch. The instruction execution and feedback module generates control instructions based on the rhythm of the music and the recommended results to drive the ambient lighting controller to adjust the ambient lighting.

[0117] The command execution and feedback module, when executing music playback commands, also includes sound field compensation and dynamic BPM adjustment functions. The sound field compensation function monitors the cabin noise level in real time through a microphone array and dynamically adjusts the speaker equalizer parameters and output gain using a sound field compensation algorithm to compensate for the impact of ambient noise on music clarity. The dynamic BPM adjustment function fine-tunes the music playback speed (BPM) based on the user's physiological activation characteristics to guide the user's heart rate or respiratory rate towards a stable state, thereby reducing user agitation.

[0118] This invention provides a method for intelligent intervention in highway fatigue driving scenarios, which may include the following specific steps:

[0119] The data acquisition and preprocessing module first obtains real-time multi-source data through the system interface. In this embodiment, the module detects the vehicle's speed as 115 km / h (highway) via the CAN bus interface, and confirms that the vehicle is maintaining a constant speed. Simultaneously, the module obtains the road type information as "highway" through the network communication unit.

[0120] At the same time, the data acquisition and preprocessing module captures facial images of drivers and passengers through cameras. Analysis shows that the number of yawns per unit time exceeds the preset fatigue threshold, and the gaze direction frequently shifts downward.

[0121] The context-aware recommendation system module calculates context feature values ​​based on standardized multi-source data and constructs context vectors. In this scenario, the calculated Example values ​​are [0.8, 0.2, 0.9]. 0.8 represents high driving intensity (high speed), 0.2 represents low user excitement (fatigue), and 0.9 represents high environmental noise (high wind noise).

[0122] The multimodal dynamic arbitration engine module identified that the scenario involved driving safety (fatigue driving) and triggered a safety priority rule. The module then applied scenario-adaptive weights. Set it to a high value of 0.8 to weight user preferences. Set it to a low value of 0.2 to force the system to recommend energizing music instead of the soothing music that users usually prefer.

[0123] The context-aware recommendation system module calculates the recommendation scores for two candidate songs:

[0124] Song 1 (rock music), its feature vector ;

[0125] Song 2 (instrumental), its feature vector .

[0126] Assuming the user's preference for song 1 Preference for song 2 .

[0127] Calculate the matching degree using the cosine similarity formula:

[0128] The calculated value is approximately 0.86 (high match).

[0129] The calculated value is approximately 0.26 (low match).

[0130] Calculate the final score using the scoring formula:

[0131] ;

[0132] .

[0133] The system prioritizes recommending rock music based on scores. This is intended to refresh the mind and combat fatigue safely. The instruction execution and feedback module then plays the rock music and controls the ambient lighting to adjust to the corresponding cool-toned, high-brightness mode.

[0134] This invention also provides a method for calming and regulating the irritability of drivers and passengers in urban traffic congestion, which may include the following specific steps:

[0135] The data acquisition and preprocessing module monitors vehicle dynamics data in real time through the CAN bus interface and detects that the vehicle is in a congested state with frequent starts and stops (low speed, frequent braking).

[0136] Meanwhile, the data acquisition and preprocessing module detected that the driver and passengers' micro-expressions were "irritable" and their heart rate was consistently higher than the baseline value.

[0137] Context-aware recommendation system module constructs context vectors At this point, the vector reflects low driving speed but high user physiological activation (irritability, high arousal).

[0138] The multimodal dynamic arbitration engine module identifies abnormal user emotions and triggers emotion regulation rules. The module dynamically adjusts weight coefficients to ensure that the recommendation results guide the user's emotions back to a stable state.

[0139] When calculating recommendation scores, the context-aware recommendation system module searches for feature vectors. With the current context vector The system selects the music with the highest matching degree. Based on the "homogeneity principle," the system first matches music that is similar in energy to the user's current high arousal emotion (irritability) but has a positive guiding effect (because according to the cosine similarity algorithm, the high arousal context vector will naturally match the high-energy song feature vector). Then, the instruction execution and feedback module gradually reduces the music tempo through the dynamic BPM adjustment function, guiding the user's emotion to transition to a soothing state.

[0140] The instruction execution and feedback module plays the generated playlist through the speaker and adjusts the cabin lighting in sync with the music rhythm via the ambient lighting controller to perform the final soothing action.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal in-vehicle music interaction system based on driving scenarios, characterized in that, The system includes: The data acquisition and preprocessing module is configured to receive a raw data stream from the data acquisition unit, which includes vehicle status data, driver and passenger status data, and environmental status data, and to perform data denoising and data normalization processing on the raw data stream to generate standardized multi-source data. The multimodal dynamic arbitration engine module is configured to receive the standardized multi-source data, apply a timestamp-based sliding window synchronization mechanism to align the asynchronous standardized multi-source data and generate a contextual snapshot, and then determine the priority of the interaction method and the dynamic weight coefficient for music recommendation based on the contextual snapshot through a dynamic conflict decision matrix. The context-aware recommendation system module is configured to receive the standardized multi-source data and the dynamic weight coefficients, construct a three-dimensional context feature model based on the standardized multi-source data and generate a context vector, and calculate the recommendation score of each song in the music feature vector library by combining the song feature vectors stored in the music feature vector library and the user's historical preference data to generate a sorted music playlist. The instruction execution and feedback module is configured to control the interaction mode of the human-machine interaction unit according to the priority of the interaction method, and to control the human-machine interaction unit to perform music playback and cabin lighting adjustment according to the music playlist.

2. The multimodal in-vehicle music interaction system based on driving scenarios according to claim 1, characterized in that, The data acquisition unit includes: A CAN bus interface connected to the vehicle controller local area network bus is used to collect the vehicle status data, which includes vehicle speed data, acceleration data, and steering wheel angle data. A camera facing the driver and passenger seats is used to capture facial micro-expression images and body posture images from the driver and passenger status data; A microphone array positioned in the cockpit is used to collect the cabin noise level from the environmental status data and the voice signal from the occupant status data. Biosensors integrated into the cockpit contact components are used to collect heart rate and respiratory rate data from the occupant status data; The gesture recognition unit is used to collect gesture data from the driver and passenger status data.

3. The multimodal in-vehicle music interaction system based on driving scenarios according to claim 1, characterized in that, The sliding window synchronization mechanism in the multimodal dynamic arbitration engine module is configured as follows: Define a sliding window of fixed time length and establish an independent data buffer for each input raw data stream; Advance by a preset sliding step size, and define a time interval based on the current timestamp at each time step; Extract all data points falling within the time interval, apply an aggregation function to data types with multiple measurements to calculate a single representative value, and combine the aggregated multi-source data to form the context snapshot.

4. The multimodal in-vehicle music interaction system based on driving scenarios according to claim 1, characterized in that, The dynamic conflict decision matrix in the multimodal dynamic arbitration engine module contains the following adjudication logic branches: Vehicle speed adjudication branch: When the vehicle speed data in the scenario snapshot exceeds the preset safe vehicle speed threshold, an interaction rule is generated that prioritizes voice interaction and disables touch function. Reliability assessment branch: When the cabin noise level in the scenario snapshot exceeds the preset noise threshold, increase the priority weight of gesture interaction and touch interaction, and decrease the priority weight of voice interaction. Multi-passenger biosignal adjudication branch: When there are multiple passengers in the vehicle, calculate the gradient of changes in each passenger's biosignature data, and assign the interaction weight to the passenger with the largest gradient of changes in biosignature data.

5. A multimodal in-vehicle music interaction system based on driving scenarios according to claim 4, characterized in that, The dynamic conflict decision matrix also includes a self-learning function: The self-learning function is configured to continuously record the user's interaction history data and feedback operations on the recommendation results, and dynamically adjust the weight parameters of each adjudication rule in the dynamic conflict decision matrix based on the interaction history data.

6. A multimodal in-vehicle music interaction system based on driving scenarios according to claim 1, characterized in that, The three-dimensional context feature model and context vector constructed by the context-aware recommendation system module are generated in the following way: Based on the acceleration data and steering wheel angle data in the vehicle state data, the driving intensity characteristic value is calculated by weighted summation. Based on the heart rate data, the pleasure score corresponding to the facial micro-expression images, and the yawning frequency value in the driver and passenger status data, the user's physiological activation characteristic value is calculated through a comprehensive score. The driving intensity feature value, the user physiological activation feature value, and the cabin noise level value obtained from the environmental state data are combined to form the context vector.

7. A multimodal in-vehicle music interaction system based on driving scenarios according to claim 6, characterized in that, The method for constructing the song feature vector in the context-aware recommendation system module is as follows: The tempo, energy value, and valence characteristics of each song are extracted using audio analysis algorithms. Generate the song feature vector corresponding to the context vector dimension. The song feature vector includes the driving intensity feature value applicable to the song recommendation, the user physiological activation feature value matching the song, and the cabin noise level value applicable to the song.

8. A multimodal in-vehicle music interaction system based on driving scenarios according to claim 7, characterized in that, The specific logic for calculating the recommendation score in the context-aware recommendation system module is as follows: Calculate the cosine similarity between the context vector and the song feature vector; Obtain the user's preference value for the song, which is calculated based on the user's historical preference data; The recommendation score is obtained by linearly weighting and summing the cosine similarity and the preference value using the dynamic weight coefficients. When the dynamic weight coefficient output by the multimodal dynamic arbitration engine module indicates that the need to prioritize responding to emotion regulation needs, the recommended score is configured to increase the ranking weight of songs that are similar in energy to the current user's physiological activation characteristic value.

9. A multimodal in-vehicle music interaction system based on driving scenarios according to claim 1, characterized in that, The instruction execution and feedback module also includes sound field compensation and dynamic BPM adjustment functions: The sound field compensation function is configured to dynamically adjust the equalizer parameters and output gain of the speaker based on the current environmental state data. The dynamic BPM adjustment function is configured to adjust the tempo parameters of music playback based on the user's physiological activation characteristic value output by the context-aware recommendation system module.

10. A multimodal in-vehicle music interaction method based on driving context, characterized in that, The method, applied to a multimodal in-vehicle music interaction system based on driving scenarios as described in any one of claims 1-9, comprises the following steps: Vehicle status data, driver and passenger status data, and environmental status data are collected as raw data streams. Data denoising and normalization processing are performed on the raw data streams to obtain standardized multi-source data. The standardized multi-source data is aligned and a contextual snapshot is generated by applying a timestamp-based sliding window synchronization mechanism. The scenario snapshot is matched with a preset dynamic conflict decision matrix to output the priority of the interaction method and the dynamic weight coefficient for music recommendation. Based on the standardized multi-source data, a context vector is constructed that includes driving intensity feature values, user physiological activation feature values, and cabin noise level values. Calculate the cosine similarity between the context vector and the song feature vector of each song in the music feature vector library, and combine the preference value corresponding to the user's historical preference data and the dynamic weight coefficient to calculate the recommendation score of each song. A music playlist is generated based on the recommended scores; The interaction mode is controlled according to the priority of the interaction method, music is played according to the music playlist, and playback parameters are adjusted using sound field compensation and dynamic BPM adjustment functions, while the cabin lighting is adjusted synchronously.

Images